Pipelines — API Reference

The Pipelines domain spans three gRPC services across three proto packages:

PipelineService is the centre of the platform. The pillar split collapsed three route trees — /:bucket/vector/…, /:bucket/tables/pipelines and /:bucket/objects/destinations — into pipeline facets, so creating a vector collection, a pipeline-written table, or an object destination are all one shape now. Pipelines is the canonical definition; the Vector and SQL sections point here.

For the typed domain model (Source, Credential, Pipeline, Template, IngestRule, IngestJob) see Core Concepts. For headers and error envelope see Conventions.

Creating a pipeline does not start ingestion. No facet creator writes an ingest rule — binding one is a separate, mandatory step. This is the single most common way to end up with a collection or table that looks correct and never receives an object. See Rules.

Sections

• Sources — CreateSource · GetSource · ListSources · UpdateSource · DeleteSource
• Credentials — StoreCredential · ListCredentials · DeleteCredential · OAuth flow (GetOAuthUrl · ExchangeOAuthCode · RefreshOAuthToken)
• Pipelines — CreateVectorPipeline · CreateTablePipeline · CreateObjectPipeline · CreatePipeline · GetPipeline · ListPipelines · UpdatePipeline · DeletePipeline · BatchDeleteArtifacts
• Vector Collections — the vector facet in depth: CreateVectorPipeline, the collection CRUD it shares with the other facets, and where BYO-embedding collections went
• Templates — ListTemplates · GetTemplate
• Rules — CreateRule · GetRule · ListRules · UpdateRule · DeleteRule
• Sync — TriggerDiscovery · TriggerIngestion · GetSyncStatus
• Jobs — TriggerIngest · GetIngestStatus · ListIngestJobs · RetryIngestJob

Retired surfaces

Documented here because they were previously part of this reference and calls against them now fail.

Wire conventions

HTTP query parameters are not uniform — each handler declares its own struct, and they disagree. Check the endpoint’s own page rather than assuming:

Where a listing says “no pagination”, the HTTP handler passes pagination: None and returns every matching row; ?page_size= and ?page_token= are silently ignored. Pagination on those endpoints is reachable over gRPC only.

Endpoint roots

Auth: bearer JWT in Authorization. See Conventions.

A note on routing authority

Rules carry both pipeline_id (the authoritative pointer) and binding (server-derived display info — pipeline name, kind, destination). Always route on pipeline_id. The binding is for rendering only and is empty when a rule outlives its pipeline.

Two display fields are similarly untrustworthy as identity:

• Pipeline.name is written as the template id, not the name you supplied, on the vector facet — your name lands on destination.name. Render from destination.name. This is a logged source bug.
• ListPipelines returns two rows per vector collection — an index pipeline and its derived *_search counterpart, both pointing at the same destination.storeEntityId. Deduplicate on that id when listing collections rather than pipelines.

Other protocols

Pipelines are configured here (control plane), but their outputs are ordinary engine data, reachable over every wire: query pipeline-written tables with psql on pg.uk-lon-1.dodil.io:5432, search pipeline-built collections with Qdrant/Pinecone SDKs, read source objects over S3. See Connect & wire adapters.

See also

• Core Concepts — every type signature, end-to-end
• Quickstart — wire all six entities in 5 minutes
• CLI Guide — every dodil data command in this domain
• Conventions — auth headers + error envelope