dodil data pipeline

Manage pipelines. Full CRUD — create, list, get, update, delete. Mirrors Pipelines — API Reference.

A pipeline is a Scriptum template + options + optional destination. See Core Concepts → Pipeline for the type signature.

pipeline create never creates a destination. It wraps CreatePipeline, the non-creating creator: it makes a free pipeline, or binds to a destination that already exists. To create a destination and its pipeline together, use a facet creator instead — dodil data vector collection add, dodil data table pipeline create, or dodil data object destination create. Use pipeline create when you want a second pipeline over a destination you already have, or a free pipeline with no destination at all.

Creating a pipeline does not create an ingest rule. Whichever creator you use, nothing runs until you bind a rule with dodil data ingest add.

dodil data pipeline create

dodil data pipeline create [name] -b BUCKET \
  (--scriptum TEMPLATE | --spawn-from-template TEMPLATE) \
  [--destination STORE_ENTITY_ID] [--options key=value]

Exactly one of --scriptum or --spawn-from-template is required — the CLI errors if you pass neither or both. The two are historical aliases that now mean the same thing (“this is the template id to dispatch”), since the worker builds per-call env at dispatch time.

Example:

# Second pipeline over a vector collection that already exists
dodil data pipeline create embed-contracts \
  -b kb-prod \
  --scriptum text_embedding_index \
  --destination ent_a1b2... \
  -o json
 
# A free pipeline — runs the script, persists nothing durable
dodil data pipeline create summarize-only \
  -b kb-prod \
  --spawn-from-template summarization \
  --options output_dest=return \
  -o json

Capture pipelineId from the JSON response for the next step (binding it to a rule).

--options-json does not exist on create. It is an update-only flag. On create the flag is --options key=value, repeatable.

dodil data pipeline list

dodil data pipeline list -b BUCKET

Lists every pipeline in the bucket — free ones included. The human table prints ID / NAME / TEMPLATE, but note the CLI renders Pipeline.name in both the NAME and TEMPLATE columns, so they always look identical. Use -o json for the real fields.

Example:

dodil data pipeline list -b kb-prod -o json \
  | jq '.pipelines[] | {pipelineId, name, scriptumTemplate,
                        facet: .destination.facet,
                        destination: .destination.name}'

The flat storeEntityKind / storeEntityName / storeEntityId fields are gone — they were folded into the nested destination message. A free pipeline has no destination key at all.

Filters (facet, free_only, store_entity_id, pagination) exist on the API but aren’t surfaced as flags on this command. Two CLI commands do hardcode a facet: dodil data vector collection list (facet=VECTOR) and dodil data table pipeline list (facet=TABLE).

dodil data pipeline get

dodil data pipeline get [pipeline_id] -b BUCKET

Fetches one pipeline by ID.

Example:

dodil data pipeline get pipe_a1b2... -b kb-prod -o json

dodil data pipeline update

dodil data pipeline update [pipeline_id] -b BUCKET [flags]

Patches mutable fields. At least one update flag is required (the CLI errors out otherwise — refuses no-op updates).

Examples:

# Rename
dodil data pipeline update pipe_a1b2... -b kb-prod --name embed-contracts-v2
 
# Re-bind to a different vector collection
dodil data pipeline update pipe_a1b2... -b kb-prod --store-entity-id ent_new...
 
# Tighten chunking parameters
dodil data pipeline update pipe_a1b2... -b kb-prod \
  --options-json '{"chunk_size":"1500","chunk_overlap":"200"}'
 
# Multiple at once
dodil data pipeline update pipe_a1b2... -b kb-prod \
  --name embed-contracts-v2 \
  --options-json '{"chunk_size":"1500"}'

--options-json is a full replace of the options map, not a merge. Pass the complete set of options you want to keep.

dodil data pipeline delete

dodil data pipeline delete [pipeline_id] -b BUCKET

Deletes the pipeline. Does not cascade — any rule whose pipeline_id was this pipeline will keep existing with an empty binding and will fail at ingest time. Either delete dependent rules first or re-bind them with dodil data ingest update --pipeline against another pipeline.

It also deletes only the wiring K3 owns. Dropping the Delta table, vector collection or S3 prefix the destination backs is plane state, and tenants do that themselves through the tables-gateway.

Example:

dodil data pipeline delete pipe_a1b2... -b kb-prod

To tear down a whole recipe-installed chain (rules → pipelines → destinations) in one idempotent call, use BatchDeleteArtifacts — POST /:bucket/pipelines/_batch-delete. It sorts refs into dependency order so FKs unwind cleanly, and treats a missing ref as already-deleted. Not in the CLI today.

See also

• Pipelines — API Reference — full surface incl. the facet creators and BatchDeleteArtifacts
• dodil data template — browse what Scriptum templates exist
• dodil data ingest — bind a pipeline to a source via a rule
• Core Concepts → Pipeline — type signature