Pipeline-bound Table

Goal: drop unstructured documents into a bucket → K3 extracts structured rows into a Tables table automatically. No manual INSERTs, no schema declaration — the Scriptum template owns both. You write only the SELECT at the end.

Template used: entity_pii_extraction — extracts named entities + PII from text/PDF documents. The pattern works identically for any warehouse-compatible template.

Shape:


   CreateTablePipeline          you create an ingest rule
   (table row + pipeline row)   scoped to the paths you want
                    │                     │
                    └──────────┬──────────┘
                               ▼
                     document upload  ──►  rule matches  ──►  ingest job
                                                                  │
                                                     (Scriptum template runs)
                                                                  │
                                                                  ▼
                                                          Tables table
                                                   (schema lazy-materialized
                                                    on first ingest)
                                                                  │
                                                                  ▼
                                                             SQL queries

Prerequisites

A bucket — kb-prod:


dodil data bucket create kb-prod -d "Document intake"

Tables are implicit per bucket — nothing to enable.
Pipelines is wired — confirm dodil data template list returns results.

1. Browse warehouse-compatible templates

dodil data table templates lists only the templates with the table facet (filtered server-side, PIPELINE_FACET_TABLE). The catalog is org-scoped — no -b flag needed:


# All table-compatible templates
dodil data table templates -o json
 
# Narrow by free text or by label — both server-side
dodil data table templates --search pii -o json
dodil data table templates --label modality=pdf -o json

Pick entity_pii_extraction. Modalities: text, pdf. Accepted extensions: pdf, txt, docx.

dodil data table templates takes only --search and --label. The broader catalog across all pillars — with --category filtering — is dodil data template list.

2. Inspect the chosen template’s contract


dodil data template get entity_pii_extraction -o json

You’ll see labels like:


{
  "category": "analysis",
  "type": "entity_extraction",
  "pipeline": "ingestion",
  "modality": "text, pdf",
  "warehouse_compatible": "true",
  "vertical": "compliance, healthcare, finance, legal, government"
}

The template must declare @accepts_extension — its accepted-extensions list is what you derive rule globs from, and CreateTablePipeline refuses a template without one (“a table pipeline whose template accepts no file extensions can never fire”).

Output columns are deliberately absent. K3 does not derive the table’s schema from the template’s contract; Scriptum materializes it on first ingest via k3_table_ensure_schema.

3. Create the pipeline-bound table

dodil data


# The command name IS the table name.
dodil data table pipeline create entities \
  --bucket kb-prod \
  --template entity_pii_extraction \
  --description "Auto-extracted entities + PII from intake/" \
  --folder-prefix intake

K3 writes two rows, in one motion (the destination is rolled back if the pipeline row fails):

Created	What it is	Where it lives
`entities` table destination	The warehouse entity. No schema yet — Delta materializes lazily on first ingest. `status` is `active` immediately: it describes the binding, not whether the table exists in the plane.	Pipelines service (`store_entities`, kind `warehouse`)
A pipeline	Bound to `entity_pii_extraction` with the `entities` destination attached	Pipelines service (`Pipeline`)

It does not create an ingest rule. Rule scope is the caller’s job — the same as the vector facet. An earlier version auto-created a root-scope rule, which duplicated rules whenever a recipe layered a folder-scoped one on top. Step 4 creates it.

--folder-prefix is not rule scoping. It writes a 0-byte S3 marker at intake/ so the folder renders in the object explorer (and in the AWS console, mc, Cyberduck) before anything is uploaded. The marker key ends in / with no extension, so it can never match a rule glob and never triggers an ingest. It must be a literal path segment — .. and ** are rejected.

Confirm the pipeline exists:


dodil data table pipeline list --bucket kb-prod -o json

This is ListPipelines with facet=table — the same thing as GET /:bucket/pipelines?facet=table.

4. Create the ingest rule

Bind uploads to the pipeline. Derive the globs from the template’s accepted extensions:


PIPELINE_ID=$(dodil data table pipeline list -b kb-prod -o json \
  | jq -r '.pipelines[] | select(.name == "entities") | .pipelineId')
 
dodil data ingest add entities-intake -b kb-prod \
  -c "$PIPELINE_ID" \
  -i 'intake/**/*.pdf' -i 'intake/**/*.txt' -i 'intake/**/*.docx'

-c/--collection carries the pipeline id — the flag name predates the pillar split. -i/--include is repeatable.

This is a regular IngestRule; manage it with dodil data ingest. To pause ingestion without deleting the table:


dodil data ingest list -b kb-prod -p "$PIPELINE_ID" -o json    # find the rule id
dodil data ingest update "$RULE_ID" -b kb-prod --enabled=false
dodil data ingest update "$RULE_ID" -b kb-prod --enabled=true  # resume

**/*.pdf matches recursively; *.pdf matches only the bucket root. If documents upload and no job spawns, this is almost always why.

5. Upload test documents


dodil data object create ./contract.pdf -b kb-prod -k intake/contracts/acme-2026.pdf
dodil data object create ./ticket.txt   -b kb-prod -k intake/tickets/TKT-1042.txt
 
# Or a whole tree at once — keys are paths relative to the directory
dodil data object create ./docs -b kb-prod -k intake/ --recursive

Each upload matching the rule spawns an ingest job → runs entity_pii_extraction → writes rows into entities. Objects also arrive over the S3 wire (aws s3 cp, boto3, rclone) and trigger the same rule.

6. Watch the ingest jobs


dodil data ingest jobs -b kb-prod -p "$PIPELINE_ID" -o json
 
# Follow one to completion
dodil data ingest watch "$JOB_ID" -b kb-prod

Status progression for a successful run:


PENDING ─► PROCESSING ─► COMPLETED
                │
                ▼  transient error
            RETRYING ─► PROCESSING ─► COMPLETED   (automatic retry)
                │
                ▼  permanent error (max attempts)
              FAILED

For replay / retry semantics, see Pipelines → Replay & Retry.

7. See the materialized schema

After the first successful ingest, the schema is known:

psql


DESCRIBE entities;
-- or the psql shorthand
\d entities

Exact columns vary by template — every Scriptum template defines its own output schema, and K3 learns it only when rows arrive. Expect string columns for the extracted values, a source-key column tying each row back to its object, and typed columns for scores and offsets.

8. Query the auto-extracted rows

From here it is an ordinary table. Nothing about it being pipeline-written changes how you read it:


-- All PII detected in one document
SELECT entity_type, entity_value
FROM entities
WHERE source_key = 'intake/tickets/TKT-1042.txt'
  AND is_pii = true;
 
-- Top organizations across all ingested contracts
SELECT entity_value, COUNT(*) AS n
FROM entities
WHERE entity_type = 'ORGANIZATION'
  AND source_key LIKE 'intake/contracts/%'
GROUP BY 1
ORDER BY 2 DESC
LIMIT 20;

psql


psql "host=pg.uk-lon-1.dodil.io port=5432 dbname=kb-prod \
      user=dk_XXXX password=$DODIL_SECRET sslmode=require" \
  -c "SELECT entity_type, COUNT(*) FROM entities GROUP BY 1 ORDER BY 2 DESC"

Python (psycopg)


rows = conn.execute(
    "SELECT entity_value, COUNT(*) AS n FROM entities"
    " WHERE entity_type = %s AND source_key LIKE %s"
    " GROUP BY 1 ORDER BY 2 DESC LIMIT 20",
    ("ORGANIZATION", "intake/contracts/%"),
).fetchall()

dodil data


dodil data table query \
  "SELECT entity_type, COUNT(*) AS n FROM entities GROUP BY 1 ORDER BY 2 DESC" \
  --bucket kb-prod

Reads are read-your-writes, so extracted rows are queryable as soon as the job commits them — there is no freshness knob and nothing to wait for. Column names and enum-ish values (PERSON, ORGANIZATION, EMAIL, …) are template-defined; check DESCRIBE and a SELECT DISTINCT rather than assuming.

Common gotchas

Symptom	Cause	Fix
Documents upload but no ingest job spawns	You never created an ingest rule — `CreateTablePipeline` does not create one	Do step 4
Rule exists but still no jobs	Globs don’t match your paths	`*/.pdf` is recursive, `*.pdf` is not; check `includePatterns` against the actual key
`template '…' has no @accepts_extension annotation`	The template can’t derive rule globs, so it could never fire	Pick a template with the annotation, or add it and re-publish via Scriptum
”a table destination named ’…’ already exists in bucket ’…’”	Name collision — the pipeline name is the table name	Pick another name, or delete the existing pipeline
`DESCRIBE entities` says the table doesn’t exist	Delta materializes lazily; no document has completed ingest yet	Upload a matching document and wait for a `COMPLETED` job
`describe` shows columns but the table is empty	The template extracted nothing (blank document, unsupported encoding)	Spot-check with a richer document; inspect the template contract for required inputs
Some jobs `FAILED`, others `COMPLETED`	The template can’t process specific documents (corrupt PDF, wrong encoding, unsupported MIME)	Read the job error; for permanent failures exclude or pre-process the file. See Replay & Retry.
Wrong facet — vector rows, not table rows	You picked a template with the vector facet	`dodil data table templates` only lists table-facet templates; use it to choose
Re-uploading a document duplicates rows	The pipeline re-runs on every successful PUT, including overwrites	Dedup downstream, or choose a template whose output carries a stable key

Variations — other table-facet templates

Same recipe shape, different template. Swap --template and the schema materializes from the chosen one on first ingest:

Goal	Template
Document type / topic / language tags	`classification`
Summary + keywords	`summarization`
Sentiment + intent + toxicity	`sentiment_intent_analysis`
Triage (classify + extract)	`document_triage`
OCR-only (no analysis)	`ocr_extraction`
Translate documents	`translation`
Audio → transcripts	`audio_transcription`
Object detection in images	`object_detection`
Full image analysis	`image_understanding`
Code symbols + dependencies	`code_intelligence`
Product attribute enrichment	`product_catalog_enrichment`
Customer review analytics	`review_analysis`
Video surveillance	`video_surveillance`

Run dodil data table templates for the live list — the catalog is the source of truth, not this table.

Pipeline-bound Table

Prerequisites

1. Browse warehouse-compatible templates

2. Inspect the chosen template’s contract

3. Create the pipeline-bound table

dodil data

HTTP

4. Create the ingest rule

5. Upload test documents

6. Watch the ingest jobs

7. See the materialized schema

psql

dodil data

8. Query the auto-extracted rows

psql

Python (psycopg)

dodil data

Common gotchas

Variations — other table-facet templates

See also