Documents → Warehouse table

Goal: every document (PDF / text / JSON-as-text) uploaded to a bucket gets entities + PII extracted into a structured warehouse table. Query the table directly, build dashboards, run SQL — no separate ETL.

Template: entity_pii_extraction — K3’s production template for structured entity + PII detection from text/PDF documents. Warehouse-compatible (emits rows).

Shape:


  Document upload  ──►  bucket  ──►  rule matches  ──►  entity_pii_extraction
                                                              │
                                                              ▼
                                                       Warehouse table
                                                       (entities + PII rows)
                                                              │
                                                              ▼
                                                           SQL / queries

Prerequisites

dodil CLI installed and dodil auth login done — CLI Basics

A bucket — we’ll use kb-prod:


dodil data bucket create kb-prod -d "Document intake"

You do not create the table first. K3 persists the table entity and the pipeline binding; the Delta table itself materializes lazily in the plane on first ingest via Scriptum’s k3_table_ensure_schema. K3 deliberately does not derive columns from the template’s outputs.

1. Find a warehouse-compatible template

Only templates the Tables facet accepts can back a table pipeline. Ask for that list directly rather than guessing:


dodil data table templates
# classification                 Automatically classify documents by type, topic, language, and sensitivity.
# document_triage                Rapid document triage for intake pipelines — …
# entity_pii_extraction          Extract structured entities and detect PII from any document.
# … 17 in total

This is ListTemplates with facet=PIPELINE_FACET_TABLE — the one filter that genuinely narrows server-side. (dodil data template list --label warehouse_compatible=true looks like it should do the same, but the label filter is ignored and returns all 34.)

entity_pii_extraction extracts structured entities (people, organizations, locations, dates, identifiers) and detects PII (emails, phones, SSNs, credit cards, …). Inspect its contract:


dodil data template get entity_pii_extraction -o json | jq '{id, category, modalities, labels}'
dodil data template get entity_pii_extraction -o json | jq '.contract'

Live labels include category: "analysis, action", type: entity_extraction, modality: "text, pdf", warehouse_compatible: "true", pipeline: ingestion, unit: object, status: ready, and vertical: "compliance, healthcare, finance, legal, government". Its top-level category field is core.

A table template must declare @accepts_extension. CreateTablePipeline refuses one that doesn’t, because the derived rule globs come from it and a table pipeline with no accepted extensions can never fire.

2. Create the table pipeline

One call writes the store_entities table row and the pipelines row bound to it. The name you pass is the table name:


dodil data table pipeline create entities \
  --bucket kb-prod \
  --template entity_pii_extraction \
  --description "Entities + PII extracted from intake documents" \
  -o json

That is POST /kb-prod/pipelines/table → CreateTablePipeline. Capture the pipeline id:


export PIPELINE_ID="<pipelineId from the response>"

Flag	Short	Description
`--template`	`-t`	Required. Template id from `dodil data table templates`.
`--folder-prefix`	—	Only files under this prefix trigger the pipeline. Writes a 0-byte S3 folder marker so the prefix renders in the object explorer before the first upload. Must not contain `..` or `**`.
`--description`	`-d`	Description

Verify:


dodil data table pipeline list --bucket kb-prod
# pipe_a1b2...    entities    entity_pii_extraction

storeEntityKind is gone. The flat store_entity_* fields were reserved on Pipeline and folded into destination. Read .destination.facet — it reports PIPELINE_FACET_TABLE. Note the enum name says TABLE while the stored kind string is still "warehouse", which is what you’ll see in IngestJob.pipelineKind. Same thing, two vocabularies, one historical column that was never migrated.

.destination.table.columns is empty on purpose for every pipeline-mode table — the live schema is a plane fact. DESCRIBE it through the tables-gateway (dodil data table describe entities -b kb-prod).

3. Look up the bucket’s internal source


export SOURCE_ID=$(dodil data source list -b kb-prod -o json \
  | jq -r '.sources[] | select(.name == "internal") | .sourceId')
echo "internal source = $SOURCE_ID"

4. Create the rule — mandatory

CreateTablePipeline does not auto-create an ingest rule — its proto comment says so outright: “callers own rule scope, same as the vector facet.” Without this step the table exists as a binding and never receives a row.

Cover the document types you want extracted. entity_pii_extraction accepts text + PDF — adjust the patterns to your shape:


# All PDFs under intake/ and all .txt / .json under intake/
dodil data ingest add intake-entities \
  --bucket kb-prod \
  --source "$SOURCE_ID" \
  --collection "$PIPELINE_ID" \
  --include "intake/**/*.pdf" \
  --include "intake/**/*.txt" \
  --include "intake/**/*.json"

Capture the rule ID for later filtering:


export RULE_ID="<rule_id from the add response>"

To restrict by MIME (e.g. only specific content types despite the path), use ingest update — the CLI now covers all rule mutations:


dodil data ingest update "$RULE_ID" -b kb-prod \
  --include-mime application/pdf \
  --include-mime text/plain \
  --include-mime application/json \
  --max-size 10485760

5. Upload test documents


# A PDF
curl -sSL https://example.com/sample-contract.pdf -o contract.pdf
dodil data object create ./contract.pdf -b kb-prod -k intake/contracts/acme-2026.pdf
 
# A JSON ticket (treated as text)
cat > ticket.json <<'EOF'
{
  "id": "TKT-1042",
  "customer": { "name": "Jane Doe", "email": "[email protected]" },
  "subject": "Login issues",
  "body": "Hi, I can't sign in to my account. My phone is +1-555-0142."
}
EOF
dodil data object create ./ticket.json -b kb-prod -k intake/tickets/TKT-1042.json

6. Watch the ingest jobs


# Jobs for this pipeline
dodil data ingest jobs --bucket kb-prod --pipeline "$PIPELINE_ID" -o json \
  | jq '.jobs[] | {object: .object.key, status, rowsWritten, batchesReceived, updatedAt}'
 
# Or block on one until it terminates
dodil data ingest watch <job-id> -b kb-prod

When jobs complete, rowsWritten is the count of structured rows written to the warehouse table:


[
  {
    "object": "intake/contracts/acme-2026.pdf",
    "status": "INGEST_STATUS_COMPLETED",
    "rowsWritten": 23,
    "batchesReceived": 3
  },
  {
    "object": "intake/tickets/TKT-1042.json",
    "status": "INGEST_STATUS_COMPLETED",
    "rowsWritten": 4,
    "batchesReceived": 1
  }
]

For warehouse pipelines the meaningful counters are rowsWritten + batchesReceived — chunksCreated and embeddingsCreated stay at zero (vector-specific).

7. Query the warehouse

The extracted rows are in entities now, and it is an ordinary Delta table on the tables plane — so query it with SQL, through whichever client you already use. Confirm the real schema first; K3 records no columns for a pipeline-generated table, so DESCRIBE is the source of truth:


dodil data table describe entities -b kb-prod

The column names below (source_key, entity_type, entity_value, is_pii, …) are illustrative of what entity_pii_extraction emits — check yours against that output before wiring anything to them.

psql

The bucket speaks the Postgres wire. dodil data connect kb-prod prints a ready-made connection string — note sslmode=require: the pg wire terminates TLS. Don’t downgrade it to prefer, which silently falls back to plaintext, and your password on this wire is your service-account secret.


-- All PII detected in one ticket
SELECT entity_type, entity_value
FROM entities
WHERE source_key = 'intake/tickets/TKT-1042.json'
  AND is_pii = true;
 
-- Top organizations across the extracted contracts
SELECT entity_value, COUNT(*) AS n
FROM entities
WHERE entity_type = 'ORGANIZATION'
  AND source_key LIKE 'intake/contracts/%'
GROUP BY 1
ORDER BY 2 DESC
LIMIT 20;
 
-- Which uploads produced nothing? (join intake coverage against extraction)
SELECT source_key, COUNT(*) AS entities
FROM entities
GROUP BY 1
HAVING COUNT(*) = 0;

Python (psycopg)


import psycopg
 
# dodil data connect kb-prod  → prints the DSN (contains a live token; keep it out of logs)
with psycopg.connect(DSN) as conn, conn.cursor() as cur:
    cur.execute(
        """
        SELECT entity_type, entity_value, confidence
        FROM entities
        WHERE source_key = %s AND is_pii = true
        ORDER BY confidence DESC
        """,
        ("intake/tickets/TKT-1042.json",),
    )
    for entity_type, value, confidence in cur:
        print(f"{entity_type:16} {value:40} {confidence:.2f}")

The Postgres wire is the only path offering prepared statements, COPY FROM STDIN, cursors and savepoints — worth knowing when the extraction volume grows.

dodil data

dodil data table query takes the SQL as a positional argument and scopes with -b only — there is no --table flag:


dodil data table query -b kb-prod \
  "SELECT entity_type, COUNT(*) AS n
     FROM entities
    WHERE is_pii = true
    GROUP BY 1 ORDER BY 2 DESC"

Reads are read-your-writes by default; the --freshness flag is deprecated and ignored.

Full SQL surface, dialect and type vocabulary: Tables.

Common gotchas

Symptom	Cause	Fix
Uploads land, no job ever spawns	No ingest rule — `CreateTablePipeline` doesn’t make one	`dodil data ingest list -b kb-prod -p "$PIPELINE_ID"`. Empty? Go back to step 4.
`FAILED_PRECONDITION` at create time	The template doesn’t declare `@accepts_extension`, so no rule globs can be derived	Pick one from `dodil data table templates` — that list is already filtered to `facet=TABLE`
`dodil data table describe` says the table doesn’t exist	Nothing has ingested yet — the Delta table materializes lazily on first ingest	Upload a matching object and wait for one job to reach `COMPLETED`
`.destination.table.columns` is empty	Expected. K3 never records columns for a pipeline-generated table	`DESCRIBE` through the tables-gateway; the plane owns the live schema
Job `COMPLETED` but `rowsWritten == 0`	Document had no extractable entities (e.g. blank scan)	Spot-check with a richer doc; if widespread, inspect the contract via `template get`
`FAILED` on PDFs but text files work	OCR / parsing failure on the PDF	Read `errorDetails`, then `dodil scriptum thread steps <threadId>` for the failing step. Consider chaining `ocr_extraction`.
`pipelineKind` shows `"free"` instead of `"warehouse"`	The pipeline has no destination bound	Confirm you used `table pipeline create`, not bare `pipeline create` (which never creates a destination)
Rows appear duplicated after re-upload	K3 re-runs the pipeline on every successful PUT, overwrites included	Dedup is the table’s job — declare merge keys / a primary key on the table
Failed-job rate spikes after a template change	Stricter template inputs	Retry one job in place first (`POST /:bucket/ingest/jobs/{job_id}/retry`) to confirm the fix before mass replay

Variations

Same recipe shape, different warehouse-compatible template:

Goal	Template	What rows look like
Document type / topic / language tags	`classification`	One row per document with classification labels
Summaries + keywords	`summarization`	Summary text + extracted keywords per document
Sentiment + intent	`sentiment_intent_analysis`	Sentiment scores, intent, topics, urgency, toxicity
Triage (classify + extract)	`document_triage`	Combined output for intake routing
OCR text-only	`ocr_extraction`	Raw text rows per page / region
Translation	`translation`	Translated text rows
Audio → transcript	`audio_transcription`	Speaker-diarized transcript segments
Object detection	`object_detection`	Bounding boxes + class labels per detection
Image understanding	`image_understanding`	Combined OCR + detection + LLM reasoning
Code symbols + deps	`code_intelligence`	Symbol-level rows + dependency edges
Product catalog enrichment	`product_catalog_enrichment`	Enriched attribute rows per SKU
Customer reviews	`review_analysis`	Sentiment + toxicity + keyword rows per review
Video surveillance	`video_surveillance`	Tracked objects + activity classifications

All follow this exact pattern — change step 2’s --template. Every one of them appears in dodil data table templates, which is the authoritative list of what the Tables facet accepts (17 templates).

The invoice-intake recipe installs the invoice_parsing variant with its rule already bound: dodil data recipe install invoice-intake -b finance --folder invoices/.