Fields

How field generation works at the worker level — generating new field definitions, updating dataset schemas, and backfilling existing records.

Overview

The ai_generate_fields task type generates new field definitions using AI and updates the dataset schema. Optionally, if backfill is true and existing records are present, the worker generates values for the new fields across all existing records.

This task follows the same worker lifecycle as other tasks (poll → claim → process → complete), but unlike ai_response, it reads and writes the datasets and dataset_records tables directly.

Database Tables

dataset_records Table

During backfill, the worker updates existing records to add values for the new fields.

Task Configuration

Input Schema

The task's input column contains the validated generation request. AI configuration is wrapped in an ai object. The API server pre-populates existingFields and recordCount from the dataset before creating the task.

{
  "count": 5,
  "backfill": true,
  "ai": {
    "prompt": "Add industry classification and founding year fields",
    "model": "gpt-5-nano",
    "temperature": 0.5,
    "streaming": false,
    "max_output_tokens": 5000
  },
  "datasetId": "ds_abc123",
  "existingFields": [
    { "id": "fld_a1b2c3d4e5f6", "name": "Company", "type": "text" },
    { "id": "fld_g7h8i9j0k1l2", "name": "Price", "type": "number" }
  ],
  "recordCount": 150
}

Task Metadata

The API server writes a metadata object on the task row when creating it. The input.model field may contain "auto" (the default) — workers should use the resolved values from metadata instead.

{
  "resolvedModel": "gpt-4o",
  "resolvedProvider": "openai"
}

Output Schema

Written to task.output when the task completes successfully:

{
  "fieldsAdded": ["Industry", "Founded"],
  "recordsBackfilled": 150,
  "tokenUsage": {
    "input_tokens": 9800,
    "output_tokens": 6200,
    "total_tokens": 16000
  }
}

Task Events

Field generation is not a streaming task, so workers are not required to write task_events rows. However, workers may optionally write events for observability. Recommended event types:

Progress should always be tracked via task.progress (0-100) regardless of whether events are written.

Field Definition Shape

Each generated field must follow this structure. The worker is responsible for generating valid field IDs and ensuring names don't collide with existing fields.

[
  {
    "id": "fld_Kx9mR2vL4wQn",
    "name": "Industry",
    "type": "text",
    "description": "The company's primary industry sector",
    "enum": ["SaaS", "Fintech", "Healthcare", "E-commerce", "AI/ML", "Other"],
    "required": true,
    "ai": { "search": { "citations": true } }
  },
  {
    "id": "fld_pQ3rS7tU8vWx",
    "name": "Founded Year",
    "type": "integer",
    "description": "Year the company was founded",
    "exampleValue": 2015,
    "ai": { "search": { "citations": true } }
  },
  {
    "id": "fld_aB4cD9eF0gHi",
    "name": "Sentiment Score",
    "type": "number",
    "description": "Computed sentiment of the company's market position (-1 to 1)",
    "ai": { "prompt": "Rate the company's market sentiment from -1 (negative) to 1 (positive)" }
  },
  {
    "id": "fld_jK5lM1nO2pQr",
    "name": "Personal Notes",
    "type": "text",
    "description": "User's personal notes about the company"
  }
]

ID Generation

Workers must generate field IDs themselves — the API server does not assign them. Use "fld_" + 12 random characters from a-zA-Z0-9. Example: "fld_Kx9mR2vL4wQn". Each ID must be unique within the dataset's field array.

Name Collision Handling

If the AI generates a field name that already exists in existingFields, the worker should either retry generation with an instruction to avoid those names, or append a suffix (e.g., "Industry_2"). Do not silently overwrite existing field definitions.

AI Property on Generated Fields

When generating new field definitions, workers must decide whether and how to set the ai property. This property controls how the field behaves during future AI record generation. The decision should be based on the nature of the field:

ai.search Sub-Object

The search property within ai configures web search grounding for the field. Its presence signals that the field's data should come from web sources rather than model knowledge.

During backfill, the worker reads each new field's ai.search config to decide whether to use web search for that field's values. Fields are grouped by search config and web search is run once per group.

Processing Steps

Complete step-by-step flow for processing an ai_generate_fields task:

1. Fetch the dataset by input.datasetId. If locked = true, fail the task with error "Dataset is locked". If deleted_at is not null, fail with "Dataset has been deleted".
2. Read resolved model from task.metadata.resolvedModel and task.metadata.resolvedProvider. Never use input.ai.model directly (may be "auto").
3. Generate field definitions via AI using structured output (JSON array). Pass the user prompt (input.ai.prompt), input.count (number of fields to generate), existing field names (for context and collision avoidance), and the dataset title.
4. Generate field IDs for each new field: "fld_" + 12 random alphanumeric chars.
5. Validate names — ensure no generated field name collides with any name in input.existingFields. If collision detected, skip the colliding field.
6. Update the dataset — append new field definitions to the existing fields JSONB array. Write the fields_generated event.
7. If backfill — when input.backfill = true and input.recordCount > 0, batch through existing records and generate values for the new fields (see Backfill Pattern below).
8. Write output to task.output and set status = 'completed', completed_at = NOW().

SQL: Update Dataset Fields

Append new field definitions to the existing JSONB array:

UPDATE datasets
SET    fields = fields || $newFields::jsonb,
       updated_at = NOW(),
       updated_by = $userId
WHERE  id = $datasetId;

-- $newFields is a JSON array of the new FieldDefinition objects
-- $userId is task.created_by

SQL: Backfill a Record

Merge new field values and metadata into an existing record:

UPDATE dataset_records
SET    data = data || $newFieldValues::jsonb,
       _meta = COALESCE(_meta, '{}'::jsonb) || $newMeta::jsonb,
       updated_at = NOW(),
       updated_by = $userId
WHERE  id = $recordId;

-- $newFieldValues is { "Industry": "SaaS", "Founded": 2015 }
-- $newMeta merges into existing _meta, preserving prior citations
-- $userId is task.created_by

Error Handling

• If the dataset is locked or deleted, set task.status = 'failed', task.error = reason, and task.completed_at = NOW().
• If a backfill batch fails, keep previously updated records and continue to the next batch. Report the failure in a backfill_progress event.
• Check task.status for cancellation every few batches. If cancelled, write partial output and set status to 'cancelled'.

Backfill Pattern

When backfill is true and recordCount > 0, the worker populates new fields across all existing records:

1. First, generate and validate the new field definitions. Update the dataset's fields column using append SQL (fields || $newFields::jsonb).
2. Group new fields by their ai.search config: fields with a search object are grouped for web search, fields with ai: {} or ai: { "prompt": "..." } are synthetic, fields without ai use default synthetic generation.
3. Fetch existing records in batches (25 at a time) to manage token limits.
4. For each batch: run web search for search-enabled field groups, read existing record data, generate new field values via AI (separating synthetic vs. web-search-grounded fields in the prompt), update each record using merge SQL.
5. Each backfilled record's _meta always includes generatedBy, taskId, model, and generatedAt. Existing per-field citations are preserved (merged, not overwritten).
6. Update task progress after each batch.
7. Token usage is accumulated across field generation and all backfill batches.
8. If backfill is false, omitted, or recordCount is 0, only the field definitions are added — existing records are not modified.

Record _meta

When backfilling records, workers update each record's _meta column with generation metadata:

{
  "generatedBy": "ai",
  "taskId": "task_abc123",
  "model": "gpt-4o",
  "generatedAt": "2025-01-15T10:30:00Z",
  "citations": {
    "Industry": [
      { "url": "https://example.com/article", "title": "Industry Report", "snippet": "..." }
    ],
    "Founded": [
      { "url": "https://example.com/company", "title": "Company Profile" }
    ]
  }
}

If the record already has a _meta value (e.g., from prior AI generation), merge the new metadata — do not overwrite existing citations for other fields.

Provider Routing

The API server resolves the provider via task.metadata.resolvedProvider. For backward compatibility, the worker also detects the provider from the model prefix:

Provider mapping from API server names: openai→openai, anthropic→anthropic, google→gemini, openrouter→xai.

Shared Requirements

• Token usage: Accumulated across field generation and all backfill batches, written to task.output.tokenUsage
• Progress: Updated via task.progress (0-100) after each backfill batch
• Cancellation: Task status checked between batches — processing stops if cancelled
• Error handling: On failure, status set to failed, error message recorded, completed_at set
• Structured output: All AI calls use JSON schema for parseable responses