Records

How record operations work at the worker level — generating, validating, deduplicating, and inserting records with citation metadata.

Overview

The ai_generate_records task type generates new records using AI based on a prompt and the dataset's field definitions, then inserts them directly into dataset_records. It supports chunked generation, deduplication, web search grounding, and per-field citation metadata.

This task follows the same worker lifecycle as other tasks (poll → claim → process → complete), but unlike ai_response, it reads and writes the datasets and dataset_records tables directly.

Database Tables

Workers handling record tasks need access to these tables in addition to tasks and task_events:

dataset_records Table

Each row is a single record in a dataset. Workers insert new rows here during ai_generate_records and update existing rows during ai_generate_fields backfill.

_meta Column

Every AI-generated record must have a _meta column containing per-field citations and generation provenance. This is a separate JSONB column on dataset_records, not inside the data column, to keep user data clean.

{
  "citations": {
    "Company": [{ "url": "https://example.com/page", "title": "Source Page", "snippet": "Acme Corp is..." }],
    "Price": [{ "url": "https://pricing.example.com", "title": "Pricing", "snippet": "$29/mo" }]
  },
  "generatedBy": "ai",
  "taskId": "task_xyz789",
  "model": "gpt-5-nano",
  "generatedAt": "2026-02-22T10:30:00.000Z"
}

ai_generate_records

This task type generates dataset records using AI and inserts them directly into dataset_records. The worker reads the dataset's field definitions, generates records in chunks, deduplicates against existing data, and writes each record with citation metadata in the _meta column.

Input Schema

The task's input column contains the validated generation request. The API server pre-populates fields, selectedFields, fieldSources, and existingUniqueValues from the dataset before creating the task.

{
  "prompt": "Generate 50 SaaS companies with pricing data",
  "count": 50,
  "model": "gpt-5-nano",
  "temperature": 0.7,
  "streaming": true,
  "max_output_tokens": 10000,
  "web_search": {
    "enabled": true,
    "sources": 5,
    "include_urls": ["https://example.com"],
    "exclude_urls": [],
    "x_search": false,
    "citations": true
  },
  "chunk_size": 50,
  "datasetId": "ds_abc123",
  "fields": [
    { "id": "fld_a1b2c3d4e5f6", "name": "Company", "type": "text", "required": true, "unique": true, "ai": { "search": { "sources": 10, "citations": true } } },
    { "id": "fld_b2c3d4e5f6g7", "name": "Price", "type": "number", "ai": { "search": {} } },
    { "id": "fld_c3d4e5f6g7h8", "name": "Website", "type": "url", "ai": { "search": {} } },
    { "id": "fld_d4e5f6g7h8i9", "name": "ID", "type": "uuid", "autoGenerate": "uuid" },
    { "id": "fld_e5f6g7h8i9j0", "name": "Category", "type": "text", "enum": ["B2B", "B2C", "Enterprise"], "ai": { "model": "claude-sonnet-4-20250514" } },
    { "id": "fld_f6g7h8i9j0k1", "name": "Notes", "type": "text", "ai": false },
    { "id": "fld_g7h8i9j0k1l2", "name": "Revenue Score", "type": "formula", "formula": "Price * 12" }
  ],
  "selectedFields": ["Company", "Price", "Website", "Category"],
  "fieldSources": {
    "Company": "web_search",
    "Price": "web_search",
    "Website": "web_search",
    "Category": "web_search"
  },
  "fieldWebSearchConfigs": {
    "Company": { "sources": 10, "citations": true }
  },
  "fieldModels": {
    "Category": "claude-sonnet-4-20250514"
  },
  "fieldPrompts": {
    "Company": "Research the company and provide their official description"
  },
  "fieldTemperatures": {
    "Company": 0.5
  },
  "fieldStreaming": {
    "Company": true
  },
  "fieldMaxOutputTokens": {
    "Company": 2000
  },
  "existingUniqueValues": {
    "Company": ["Acme Corp", "Globex"]
  }
}

Non-Selected Field Handling

Fields in fields but NOT in selectedFields are not sent to the AI. Instead, the worker assigns values using this priority:

1. If field.type === "formula" → skip entirely (do not include in data object)
2. If field.autoGenerate === "uuid" → generate a UUID v4 string
3. If field.autoGenerate === "createdAt" or "updatedAt" → current ISO 8601 datetime
4. If field.default is defined → use that value
5. Otherwise → set to null

Output Schema

Written to task.output when the task completes successfully:

{
  "inserted": 47,
  "skipped": 3,
  "skippedReasons": [
    { "data": { "Company": "Acme Corp", "Price": 29, "Category": "B2B" }, "reason": "Duplicate value 'Acme Corp' for unique field 'Company'" },
    { "data": { "Company": "Globex", "Price": 49, "Category": "Enterprise" }, "reason": "Duplicate value 'Globex' for unique field 'Company'" }
  ],
  "totalChunks": 5,
  "tokenUsage": {
    "input_tokens": 12500,
    "output_tokens": 8400,
    "total_tokens": 20900
  }
}

Chunked Generation

Large generation requests are split into chunks to stay within LLM token limits and provide incremental progress:

• Break count into chunks of chunk_size (e.g., 100 records with chunk_size 50 = 2 chunks)
• For each chunk: generate records via AI, validate against field schema, insert into dataset_records
• Update task progress after each chunk: Math.round((chunkIndex + 1) / totalChunks * 100)
• Accumulate token usage across all chunks
• If a chunk fails, previously inserted records are kept and the error is reported

Deduplication

For fields marked as unique in the field definitions, the worker applies a three-layer deduplication strategy:

1. Pre-prompt: Pass existingUniqueValues to the AI prompt so it avoids generating known duplicates. Also include values generated in previous chunks.
2. Post-validate within batch: After generation, check for duplicates within the current chunk.
3. Post-validate against DB: Check generated unique values against existing database records.

Skipped records and reasons are tracked in output.skipped and output.skippedReasons.

Anti-Hallucination

When web search is enabled, the worker follows a search-then-extract pipeline to prevent hallucinated data:

• Search-then-extract: When web_search.enabled is true, search for relevant sources first, then pass retrieved content as grounding context for the AI model.
• Citation-required mode: When web_search.citations is true, instruct the model to cite sources for each field value. Citations are stored in _meta.citations.
• Null over guessing: The model should output null for fields where no reliable source is found, rather than hallucinating values.
• Never fabricate: URLs, statistics, and factual claims must come from actual sources.

Shared Requirements

• Token usage: Accumulated across all AI calls, written to task.output.tokenUsage
• Progress: Updated via task.progress (0-100) after each chunk/batch
• Cancellation: Task status checked every few chunks — processing stops if cancelled
• Error handling: On failure, status set to failed, error message recorded, completed_at set
• Structured output: All AI calls use JSON schema for parseable responses

Provider Routing

The API server resolves the provider via task.metadata.resolvedProvider. For backward compatibility, the worker also detects the provider from the model prefix:

All providers are accessed via the OpenAI-compatible SDK with different base URLs and API keys. Provider mapping from API server names: openai→openai, anthropic→anthropic, google→gemini, openrouter→xai.