Records

How record operations work at the worker level — generating, validating, deduplicating, and inserting records with citation metadata.

Overview

The ai_generate_records task type generates new records using AI based on a prompt and the dataset's field definitions, then inserts them directly into dataset_records. It supports chunked generation, deduplication, web search grounding, and per-field citation metadata.

This task follows the same worker lifecycle as other tasks (poll → claim → process → complete), but unlike ai_response, it reads and writes the datasets and dataset_records tables directly.

Database Tables

Workers handling record tasks need access to these tables in addition to tasks and task_events:

dataset_records Table

Each row is a single record in a dataset. Workers insert new rows here during ai_generate_records and update existing rows during ai_generate_fields backfill.

_meta Column

Every AI-generated record must have a _meta column containing per-field citations and generation provenance. This is a separate JSONB column on dataset_records, not inside the data column, to keep user data clean.

{
  "citations": {
    "Company": [{ "url": "https://example.com/page", "title": "Source Page", "snippet": "Acme Corp is..." }],
    "Price": [{ "url": "https://pricing.example.com", "title": "Pricing", "snippet": "$29/mo" }]
  },
  "generatedBy": "ai",
  "taskId": "task_xyz789",
  "model": "gpt-5-nano",
  "generatedAt": "2026-02-22T10:30:00.000Z"
}

ai_generate_records

This task type generates dataset records using AI and inserts them directly into dataset_records. The worker reads the dataset's field definitions, generates records in chunks, deduplicates against existing data, and writes each record with citation metadata in the _meta column.

Input Schema

The task's input column contains the validated generation request. The API server pre-populates fields, selectedFields, fieldSources, and existingUniqueValues from the dataset before creating the task.

{
  "prompt": "Generate 50 SaaS companies with pricing data",
  "count": 50,
  "model": "gpt-5-nano",
  "temperature": 0.7,
  "web_search": {
    "enabled": true,
    "sources": 5,
    "include_urls": ["https://example.com"],
    "exclude_urls": [],
    "x_search": false,
    "citations": true
  },
  "chunk_size": 10,
  "datasetId": "ds_abc123",
  "fields": [
    { "name": "Company", "type": "text", "required": true, "unique": true, "aiGenerate": true, "aiSource": "web_search" },
    { "name": "Price", "type": "number", "aiGenerate": true, "aiSource": "web_search" },
    { "name": "Website", "type": "url", "aiGenerate": true, "aiSource": "web_search" },
    { "name": "ID", "type": "uuid", "autoGenerate": "uuid" },
    { "name": "Category", "type": "text", "enum": ["B2B", "B2C", "Enterprise"], "aiGenerate": true },
    { "name": "Notes", "type": "text", "aiGenerate": false },
    { "name": "Revenue Score", "type": "formula", "formula": "Price * 12" }
  ],
  "selectedFields": ["Company", "Price", "Website", "Category"],
  "source_override": "web_search",
  "fieldSources": {
    "Company": "web_search",
    "Price": "web_search",
    "Website": "web_search",
    "Category": "web_search"
  },
  "existingUniqueValues": {
    "Company": ["Acme Corp", "Globex"]
  }
}

Non-Selected Field Handling

Fields in fields but NOT in selectedFields are not sent to the AI. Instead, the worker assigns values using this priority:

1. If field.type === "formula" → skip entirely (do not include in data object)
2. If field.autoGenerate === "uuid" → generate a UUID v4 string
3. If field.autoGenerate === "createdAt" or "updatedAt" → current ISO 8601 datetime
4. If field.default is defined → use that value
5. Otherwise → set to null

Output Schema

Written to task.output when the task completes successfully:

{
  "inserted": 47,
  "skipped": 3,
  "skippedReasons": [
    "Duplicate value 'Acme Corp' for unique field 'Company'",
    "Duplicate value 'Globex' for unique field 'Company'",
    "Duplicate within batch: 'NewCo' for field 'Company'"
  ],
  "totalChunks": 5,
  "tokenUsage": {
    "input_tokens": 12500,
    "output_tokens": 8400,
    "total_tokens": 20900
  }
}

Chunked Generation

Large generation requests are split into chunks to stay within LLM token limits and provide incremental progress:

• Break count into chunks of chunk_size (e.g., 50 records with chunk_size 10 = 5 chunks)
• For each chunk: generate records via AI, validate against field schema, insert into dataset_records
• Update task progress after each chunk: Math.round((chunkIndex + 1) / totalChunks * 100)
• Accumulate token usage across all chunks
• If a chunk fails, previously inserted records are kept and the error is reported

Deduplication

For fields marked as unique in the field definitions, the worker applies a three-layer deduplication strategy:

1. Pre-prompt: Pass existingUniqueValues to the AI prompt so it avoids generating known duplicates. Also include values generated in previous chunks.
2. Post-validate within batch: After generation, check for duplicates within the current chunk.
3. Post-validate against DB: Check generated unique values against existing database records.

Skipped records and reasons are tracked in output.skipped and output.skippedReasons.

Anti-Hallucination

When web search is enabled, the worker follows a search-then-extract pipeline to prevent hallucinated data:

• Search-then-extract: When web_search.enabled is true, search for relevant sources first, then pass retrieved content as grounding context for the AI model.
• Citation-required mode: When web_search.citations is true, instruct the model to cite sources for each field value. Citations are stored in _meta.citations.
• Null over guessing: The model should output null for fields where no reliable source is found, rather than hallucinating values.
• Never fabricate: URLs, statistics, and factual claims must come from actual sources.

Shared Requirements

• Token usage: Accumulated across all AI calls, written to task.output.tokenUsage
• Progress: Updated via task.progress (0-100) after each chunk/batch
• Cancellation: Task status checked every few chunks — processing stops if cancelled
• Error handling: On failure, status set to failed, error message recorded, completed_at set
• Structured output: All AI calls use JSON schema for parseable responses

Provider Routing

The model string in the task input determines which AI provider handles the request:

All providers are accessed via the OpenAI-compatible SDK with different base URLs and API keys. The worker resolves the correct client at runtime based on the model prefix.