Skip to Content
We are live but in Staging 🎉
PipelinesAPI ReferencePipelines

Pipelines — API Reference

Package: dodil.data.pipeline.v1 · Service: PipelineService

A pipeline is a Scriptum template + options + optional destination — the recipe that rules trigger against an object. Pipelines are bucket-scoped and first-class rows; rules reference them by pipeline_id. See Core Concepts → Pipeline.

This service is the single surface for “what runs, and where its output lands”. It absorbed the three per-pillar route trees that used to say the same thing three ways — /:bucket/vector/…, /:bucket/tables/pipelines and /:bucket/objects/destinations are all gone, replaced by the facet creators below.

RPCHTTP
CreateVectorPipelinePOST /:bucket/pipelines/vector
CreateTablePipelinePOST /:bucket/pipelines/table
CreateObjectPipelinePOST /:bucket/pipelines/object
CreatePipelinePOST /:bucket/pipelines
GetPipelineGET /:bucket/pipelines/:pipeline_id
ListPipelinesGET /:bucket/pipelines
UpdatePipelinePATCH /:bucket/pipelines/:pipeline_id
DeletePipelineDELETE /:bucket/pipelines/:pipeline_id
BatchDeleteArtifactsPOST /:bucket/pipelines/_batch-delete

Creating a pipeline does not start ingestion. None of the facet creators write an ingest rule — binding one with CreateRule is a separate, mandatory step. A collection or table created without a rule is real, listable, and will never ingest anything.

gRPC setup — grpcurl, endpoints, reflection, and field-name casing — is covered once in Conventions → Using gRPC.

Facets

PipelineFacet is the wire form of the destination’s kind. Note the one name that does not match its stored column: TABLE maps to the stored string "warehouse".

FacetCreatorDestination configNotes
PIPELINE_FACET_VECTORCreateVectorPipelineVectorConfigtemplate_id required
PIPELINE_FACET_TABLECreateTablePipelineTableConfigtemplate_id required, and the template must declare @accepts_extension
PIPELINE_FACET_OBJECTCreateObjectPipelineObjectConfigtemplate_id and config.target_prefix required
PIPELINE_FACET_GRAPH——Representable and listable, but nothing creates one. There is no CreateGraphPipeline and no GraphConfig; graph writes land through the Cypher surface on the tables plane.

CreateVectorPipeline

Registers a vector collection and the index pipeline bound to it, in one motion. Every schema-shaping fact — embed_model, dimensions, distance_metric, sparse_mode, embedding_type — is resolved from the template’s ScriptContract. Callers cannot override them; template_inputs only carries the contract’s declared runtime inputs.

The physical collection materializes lazily in the plane on first ingest.

Request

dodil data vector collection add contracts \
  -b kb-prod \
  -t text_embedding_index \
  -d "Q2 contract corpus"

Response

The index Pipeline row, with its destination attached.

Two observed behaviours worth knowing before you read the response.

  1. Pipeline.name comes back as the template id, not the name you sent. The server writes name: req.template_id on the pipeline row (services/pipeline/create_vector.rs:254); your name lands on destination.name. So a collection created as contracts returns a pipeline named text_embedding_index. Read destination.name for the user-facing label. This is a logged source bug, not intended behaviour — do not build display logic on Pipeline.name.
  2. Two pipeline rows are created, not one. After the index pipeline, the server derives a search-side counterpart by the *_index → *_search convention (text_embedding_index → text_embedding_search) and binds it via store_entities.search_pipeline_id. Only the index row is returned, but ListPipelines will show both, so one vector collection yields two rows in the listing. The search row is best-effort: if it fails the collection is still indexable, just not searchable through the offload.

CreateTablePipeline

Persists the table entity and the pipeline binding — which template feeds which table name. The Delta table itself materializes lazily in the plane on first ingest via Scriptum’s k3_table_ensure_schema; K3 deliberately does not derive columns from the template’s outputs, which is why TableConfig.columns comes back empty for every pipeline-mode table. The live schema is a plane fact — DESCRIBE it through the tables-gateway.

The template must declare @accepts_extension. A table pipeline with no accepted extensions can never fire.

Request

dodil data table pipeline create triage_results \
  -b kb-prod \
  -t document_triage \
  --folder-prefix intake/ \
  -d "Triaged intake documents"

folder_prefix only writes a 0-byte S3 folder marker so the prefix renders in the object explorer before the first upload. It is not a filter — it does not scope what the pipeline ingests. Scoping is the ingest rule’s job.

CreateObjectPipeline

Pipeline output written back to CephS3 at a deterministic key derived from the source key plus config.target_prefix. Both the destination row and the pipeline row are written together — the destination is rolled back if the pipeline row fails.

Request

dodil data object destination create summaries \
  -b kb-prod \
  -t summarization \
  --target-prefix summaries/ \
  --excluded-prefix summaries/ \
  -d "Markdown summaries"

Cycle prevention lives in excluded_prefix. Ingest rules skip source objects under any destination’s excluded_prefix; without it an object pipeline re-ingests its own output forever. Empty falls back to _pipeline-out/. If the destination writes into a folder it also reads from, set excluded_prefix equal to target_prefix to self-exclude.

CreatePipeline

The non-creating creator: it makes a pipeline that is either free (no destination) or bound to a destination that already exists. It cannot create a destination — that is what keeps it from overlapping the facet creators, and why it needs no per-facet validation.

It is not redundant. A destination may back several pipelines — two templates writing into the same object prefix, an index/search pair over one collection — and this is the only RPC that adds the second one.

Both script_source variants now mean the same thing. spawn_from_template is a historical alias for scriptum_template: both are “the template id to dispatch”, because the worker builds per-call env at dispatch time and there is nothing left to bake in at create.

Request

dodil data pipeline create embed-contracts \
  -b kb-prod \
  --scriptum text_embedding_index \
  --destination ent_a1b2... \
  --options chunk_size=1000 --options chunk_overlap=150

--destination is the store_entity_id of an existing destination. Omit it for a free pipeline.

Response

A Pipeline row. The server validates that the store_entity exists and belongs to the same org and bucket.

GetPipeline

Request

dodil data pipeline get pipe_a1b2... -b kb-prod

Response

{
  "pipelineId": "pipe_a1b2...",
  "bucket": "kb-prod",
  "name": "text_embedding_index",
  "scriptumTemplate": "text_embedding_index",
  "options": { "chunk_size": "1000", "chunk_overlap": "150" },
  "createdAt": "1716843600000",
  "updatedAt": "1716843600000",
  "destination": {
    "storeEntityId": "ent_a1b2...",
    "facet": "PIPELINE_FACET_VECTOR",
    "name": "contracts",
    "description": "Q2 contract corpus",
    "status": "DESTINATION_STATUS_ACTIVE",
    "engineId": "eng_a1b2...",
    "vector": {
      "dimensions": 1024,
      "distanceMetric": "DISTANCE_METRIC_COSINE",
      "chunkSize": 1000,
      "chunkOverlap": 150,
      "sparseMode": "SPARSE_MODE_BM25",
      "embeddingType": "EMBEDDING_TYPE_FLOAT",
      "modality": "text",
      "embeddingSource": "EMBEDDING_SOURCE_PIPELINE",
      "templateId": "text_embedding_index",
      "embedModel": "bge-m3",
      "physicalName": "kb-prod__contracts"
    }
  }
}

store_entity_id, store_entity_kind and store_entity_name are gone from Pipeline — fields 6/7/8 are reserved. One destination is now described by one destination message, whose facet is the store_entities.kind column and whose config oneof is the config jsonb, typed per facet. destination absent means a free pipeline. The typed configs are in Core Concepts → Pipeline.

ListPipelines

Replaces the retired vector/ListCollections, object/ListObjectDestinations and tables/ListTables. Filter by facet for the per-pillar view, by store_entity_id for one destination, or set free_only for pipelines with no destination at all.

facet and free_only are mutually exclusive — a free pipeline has no facet to match, so the server rejects the combination rather than answering an empty list to a contradictory question. An unrecognised ?facet= value is likewise a 400, not a silently-widened filter: ?facet=tables and ?facet=warehouse both error with unknown facet '…' — expected one of vector, table, object, graph.

Request

dodil data pipeline list -b kb-prod

The CLI sends no filters — it lists every pipeline in the bucket. For the per-facet views use the pillar commands, which call the same RPC with a facet set:

dodil data vector collection list -b kb-prod
dodil data table pipeline list -b kb-prod
dodil data object destination list -b kb-prod

store_entity_kind no longer exists. Field 2 is reserved and the stringly-typed ?store_entity_kind=vector query is not read. Use ?facet=vector. Note also that the facet value is table, not warehouse — warehouse is the stored column value and is explicitly rejected as a query value.

The HTTP handler does not accept pagination. GET /:bucket/pipelines reads only facet, store_entity_id and free_only, and sends pagination: None. ?page_size= and ?page_token= are ignored — the response carries the whole bucket. Pagination is reachable over gRPC only.

Response

{
  "pipelines": [
    {
      "pipelineId": "pipe_a1b2...",
      "bucket": "kb-prod",
      "name": "text_embedding_index",
      "scriptumTemplate": "text_embedding_index",
      "options": {},
      "createdAt": "1716843600000",
      "updatedAt": "1716843600000",
      "destination": {
        "storeEntityId": "ent_a1b2...",
        "facet": "PIPELINE_FACET_VECTOR",
        "name": "contracts",
        "status": "DESTINATION_STATUS_ACTIVE"
      }
    },
    {
      "pipelineId": "pipe_c3d4...",
      "bucket": "kb-prod",
      "name": "text_embedding_search",
      "scriptumTemplate": "text_embedding_search",
      "options": {},
      "createdAt": "1716843600000",
      "updatedAt": "1716843600000",
      "destination": {
        "storeEntityId": "ent_a1b2...",
        "facet": "PIPELINE_FACET_VECTOR",
        "name": "contracts",
        "status": "DESTINATION_STATUS_ACTIVE"
      }
    }
  ],
  "pagination": { "nextPageToken": "", "totalCount": "2" }
}

Both rows above are one collection. CreateVectorPipeline writes an index pipeline and a search-side counterpart against the same destination.storeEntityId. Deduplicate on destination.storeEntityId if you are rendering collections rather than pipelines.

UpdatePipeline

Patch-style. optional fields use proto3 presence; the options map uses the StringMap wrapper from dodil.data.common.v1 so you can distinguish “leave alone” (absent) from “clear” (present + empty) from “replace” (present + non-empty).

This also replaces the retired object/UpdateObjectDestination — renaming a destination or editing its facet config happens here, on the pipeline that owns it, via the destination field.

Request

dodil data pipeline update pipe_a1b2... \
  -b kb-prod \
  --name embed-contracts-v2 \
  --options-json '{"chunk_size":"1500","chunk_overlap":"200"}'

Re-bind to a different destination:

dodil data pipeline update pipe_a1b2... -b kb-prod --store-entity-id ent_new...

The CLI cannot edit the destination config in place — use HTTP or gRPC for that.

The facet of an existing destination is immutable. destination is rejected when the pipeline has no destination, and when the supplied config variant does not match the destination’s facet. Per-facet validation applies exactly as on create — an ObjectConfig with a blank target_prefix is refused here too, so update cannot walk a destination into a state create would have rejected.

Response

A Pipeline row.

DeletePipeline

Request

dodil data pipeline delete pipe_a1b2... -b kb-prod

Response

Empty (DeletePipelineResponse {}).

This deletes the pipeline row, not the data. The destination’s rows survive — the Delta table, the vector collection and the S3 prefix are plane state, and K3 never reaches into the data plane to drop them on your behalf. Drop your own data through the tables-gateway (DROP TABLE and friends).

Deleting a pipeline doesn’t cascade to rules either — any rule whose pipeline_id was this pipeline keeps existing with its binding empty, because the server can no longer resolve the deleted pipeline. New ingest jobs for those rules will fail. Either delete dependent rules first or re-bind them with UpdateRule.

BatchDeleteArtifacts

Multi-artifact idempotent delete for the recipe-uninstall flow. A recipe creates a chain of rule → pipeline → destination; this is the inverse. The handler sorts refs by kind into dependency order (rules → pipelines → destinations) so foreign keys unwind cleanly without the caller knowing about ordering.

Idempotent: NotFound on any ref is treated as already-deleted. Per-ref results carry success / skip / error, so a partial failure does not block the batch.

Request

curl -sS -X POST "https://api.data.dodil.io/kb-prod/pipelines/_batch-delete" \
  -H "Authorization: Bearer $DODIL_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "refs": [
      { "kind": "ARTIFACT_KIND_RULE",        "id": "rule_a1b2..." },
      { "kind": "ARTIFACT_KIND_PIPELINE",    "id": "pipe_a1b2..." },
      { "kind": "ARTIFACT_KIND_DESTINATION", "id": "ent_a1b2..."  }
    ]
  }'

Response

{
  "results": [
    { "ref": { "kind": "ARTIFACT_KIND_RULE", "id": "rule_a1b2..." }, "deleted": true,  "error": "" },
    { "ref": { "kind": "ARTIFACT_KIND_PIPELINE", "id": "pipe_a1b2..." }, "deleted": false, "error": "" }
  ]
}

Read each result as a three-way outcome:

deletederrorMeaning
trueemptyThe wiring row was removed.
falseemptyThe ref was already gone — idempotent success.
falsenon-emptyThe delete failed.

Wiring only. This removes the rows K3 owns; it never drops the data a destination backs. There is no wipe_data flag — it was removed because it could not do what it claimed: the vector branch reported success while deleting only the Postgres row, since the drop it delegated to was decommissioned. Dropping a Delta table, a vector collection or an S3 prefix is plane state, and you do it yourself through the tables-gateway.

See also

CredentialsVector Collections