Pipelines — CLI Guide

The dodil data CLI covers six Pipeline-domain command groups:

dodil data source — sources on a bucket (create, list, get)
dodil data credential — store credentials + run the OAuth flow (current CLI surface: store, oauth-url, oauth-exchange)
dodil data pipeline — full pipeline CRUD (create, list, get, update, delete)
dodil data vector collection — the vector facet creator (add, list, get, delete) — auto-embed-on-ingest collections
dodil data object destination — the object facet creator (create, list, get, update, delete) — derived files written back to the bucket
dodil data template — browse the Scriptum template catalog (list, get, plus vector templates)
dodil data ingest — rule lifecycle + sync triggers + jobs (rules: add, get, list, update, delete; jobs: jobs, watch; triggers: trigger, trigger-discovery)
dodil data recipe — one-command provisioning of complete ingest stacks (list, show, install)

For install, auth, output format, and global flags see CLI Basics. For the underlying API contracts see Pipelines — API Reference.

Quick setup


dodil auth login

Opens a browser to log in; for non-interactive / CI use, set DODIL_SERVICE_ACCOUNT_ID + DODIL_SERVICE_ACCOUNT_SECRET (or pass --service-account-id/--service-account-secret). After login, the CLI resolves endpoint, token, and org from ~/.config/dodil/config.yaml. Most pipeline commands take --bucket / -b to scope.

CLI surface vs API surface

The CLI is leaner than the underlying API on purpose — it covers the most-common operations. Anything missing has a direct HTTP equivalent in the API Reference.

Domain	API coverage	CLI coverage	Use API directly for
Source	full CRUD (`Create`, `Get`, `List`, `Update`, `Delete`)	`create`, `list`, `get`	`update`, `delete` — `PATCH` / `DELETE /:bucket/sources/:source_id`
Credential	`Store`, `List`, `Delete`, `RefreshOAuthToken` + the OAuth flow. `GetCredential` and `ValidateCredential` are retired	`store`, `oauth-url`, `oauth-exchange`	`list`, `delete`, `refresh` — `/admin/credentials`
Pipeline	3 facet creators + `CreatePipeline` + full CRUD + `BatchDeleteArtifacts`	`pipeline` (bind-only CRUD); facet creators live under `vector collection add`, `table pipeline create`, `object destination create`	`BatchDeleteArtifacts` (`POST /:bucket/pipelines/_batch-delete`)
Template	`ListTemplates` + `GetTemplate`	`list` (`--category`, `--search`, `--label`), `get`	the `facet` filter — the CLI hardcodes it in `table templates` / `vector templates` only
Rule	full CRUD	full CRUD — `add` / `get` / `list` (with `-p` pipeline filter) / `update` / `delete`	— ✅ complete
Sync / Jobs	`TriggerIngest`, `TriggerDiscovery`, `TriggerIngestion`, `GetSyncStatus`, `ListIngestJobs`, `GetIngestStatus`, `RetryIngestJob`	`trigger`, `trigger-discovery`, `jobs` (`-r`, `-p`, `--status`, `--page-size`/`--page-token`/`--all`), `watch`	`TriggerIngest` (single object + per-event options), `GetSyncStatus`, `RetryIngestJob` — use `curl`

--label on template list is accepted and then ignored. Verified live: dodil data template list --label anything=at-all returns the whole catalog. --category and --search do filter server-side. Filter labels with jq until this is fixed.

Quickstart shape

A typical pipeline-domain CLI session looks like this:


# 1. Browse what's available
dodil data template list -o json
 
# 2. Create the destination + its pipeline in one call
#    (vector collection → CreateVectorPipeline; table → `table pipeline create`)
PIPELINE_ID=$(dodil data vector collection add docs-index \
  -b kb-prod --template text_embedding_index -o json | jq -r '.pipelineId')
 
# 3. Bind an ingest rule — MANDATORY. Nothing runs without it.
SOURCE_ID=$(dodil data source list -b kb-prod -o json \
  | jq -r '.sources[] | select(.name == "internal") | .sourceId')
dodil data ingest add pdf-rule \
  -b kb-prod \
  --source "$SOURCE_ID" \
  --collection "$PIPELINE_ID" \
  --include "**/*.pdf"
 
# 4. Upload an object — auto-triggers the pipeline
dodil data object create ./contract.pdf -b kb-prod -k contracts/contract.pdf
 
# 5. Watch the job to a terminal state
dodil data ingest jobs -b kb-prod -p "$PIPELINE_ID"
dodil data ingest watch <job-id> -b kb-prod

Steps 2 and 3 together are exactly what dodil data recipe install document-rag -b kb-prod does in one command — see dodil data recipe.

See Quickstart for the full end-to-end walkthrough.

Pipelines — CLI Guide

Quick setup

CLI surface vs API surface

Quickstart shape

See also