`dodil data ingest`

Rules, triggers, and jobs. Aliased as dodil data rule. The single group covers three concerns:

Rule lifecycle — full CRUD: add, get, list, update, delete
Sync triggers — trigger (dispatch matched objects), trigger-discovery (scan source for new objects)
Jobs — jobs (list, filter, paginate), watch (block on one job until it terminates)

--bucket / -b is persistent on the group.

The rule is what makes a pipeline run. Creating a pipeline — through any facet creator or pipeline create — does not create a rule. dodil data ingest add is the mandatory second step. The only shortcut is dodil data recipe install, which issues both calls for you.

Rules

`dodil data ingest add`


dodil data ingest add [name] \
  -b BUCKET \
  --source SOURCE_ID \
  --collection PIPELINE_ID \
  [--include PATTERN] [--include PATTERN ...]

Creates an ingest rule. The --collection flag maps to the API’s pipeline_id — historical naming the CLI hasn’t caught up on.

Flag	Short	Type	Description
`--bucket`	`-b`	string	Required (persistent).
`--source`	`-s`	string	Required. Source ID the rule fires against.
`--collection`	`-c`	string	Required. Pipeline ID (despite the name) to bind to.
`--include`	`-i`	string list	Include glob patterns (case-insensitive, supports `` / `` / `?` — see Glob syntax). Default: `/` (match everything). Repeatable.

Examples:


# Index all PDFs under contracts/ via your embedding pipeline
dodil data ingest add pdf-rule \
  -b kb-prod \
  --source "$SOURCE_ID" \
  --collection "$PIPELINE_ID" \
  --include "contracts/**/*.pdf"
 
# Match anything (default --include)
dodil data ingest add all-rule \
  -b kb-prod \
  --source "$SOURCE_ID" \
  --collection "$PIPELINE_ID"
 
# Multiple include patterns
dodil data ingest add docs-rule \
  -b kb-prod \
  --source "$SOURCE_ID" \
  --collection "$PIPELINE_ID" \
  --include "docs/**/*.pdf" \
  --include "docs/**/*.md"

Exclude patterns, MIME filters, size thresholds, and priority aren’t on CLI flags today. Set them via UpdateRule over the API after add, or create the rule via the API directly with CreateRule.

`dodil data ingest get`


dodil data ingest get [rule_id] -b BUCKET

Example:


dodil data ingest get rule_a1b2... -b kb-prod -o json

`dodil data ingest list`


dodil data ingest list -b BUCKET [-s SOURCE_ID] [-p PIPELINE_ID]

Flag	Short	Description
`--source`	`-s`	Filter by source ID
`--pipeline`	`-p`	Filter by pipeline ID

Examples:


# All rules in the bucket
dodil data ingest list -b kb-prod -o json
 
# Filter by source
dodil data ingest list -b kb-prod -s "$SOURCE_ID" -o json
 
# All rules bound to a specific pipeline (useful when you have multiple sources feeding one pipeline)
dodil data ingest list -b kb-prod -p "$PIPELINE_ID" -o json

`dodil data ingest update`


dodil data ingest update [rule_id] -b BUCKET [flags]

Patches mutable fields on a rule. Pass only the flags you want to change.

Flag	Type	Description
`--name`	string	Rename the rule
`--description`	string	Replace description
`--include`	string list	Replace include glob patterns (empty = clear)
`--exclude`	string list	Replace exclude glob patterns (empty = clear)
`--include-mime`	string list	Replace include MIME types (empty = clear)
`--exclude-mime`	string list	Replace exclude MIME types (empty = clear)
`--min-size`	int64	Minimum object size in bytes
`--max-size`	int64	Maximum object size in bytes (`0` = no limit)
`--enabled`	bool	Enable (`true`) or disable (`false`) — pauses the rule without deleting it
`--priority`	int32	Higher value = matched first
`--pipeline`	string	Re-bind the rule to a different pipeline ID

Examples:


# Pause a rule without deleting it
dodil data ingest update rule_a1b2... -b kb-prod --enabled=false
 
# Re-enable
dodil data ingest update rule_a1b2... -b kb-prod --enabled=true
 
# Tighten the include set
dodil data ingest update rule_a1b2... -b kb-prod --include "contracts/2026/*.pdf"
 
# Add MIME filtering on top
dodil data ingest update rule_a1b2... -b kb-prod \
  --include-mime application/pdf
 
# Re-bind to a new pipeline (e.g. after spawning a v2 pipeline)
dodil data ingest update rule_a1b2... -b kb-prod --pipeline pipe_new...
 
# Combine — rename + change priority + re-enable
dodil data ingest update rule_a1b2... -b kb-prod \
  --name "contracts-rule-v2" \
  --priority 200 \
  --enabled=true

List-typed flags REPLACE the existing values. --include with an empty list clears the include set; with one or more patterns it replaces the whole set. Same for --exclude, --include-mime, --exclude-mime. To accumulate, list-get the existing values first, then pass the combined list.

`dodil data ingest delete`


dodil data ingest delete [rule_id] -b BUCKET

Example:


dodil data ingest delete rule_a1b2... -b kb-prod

Existing in-flight jobs continue; new ones stop. To pause without deleting, use update --enabled=false above.

Sync triggers

`dodil data ingest trigger-discovery`

Scans a source for new / changed objects.


dodil data ingest trigger-discovery -b BUCKET -s SOURCE_ID [--rule RULE_ID] [--full-sync]

Flag	Short	Type	Description
`--source`	`-s`	string	Required. Source to scan.
`--rule`	`-r`	string	Scope discovery to one rule’s matchers
`--full-sync`	—	bool	Ignore the source’s checkpoint and rescan from scratch

Examples:


# Delta scan
dodil data ingest trigger-discovery -b kb-prod -s "$SOURCE_ID"
 
# Full rescan (after fixing pipeline options / rules)
dodil data ingest trigger-discovery -b kb-prod -s "$SOURCE_ID" --full-sync
 
# Narrow to one rule's matchers
dodil data ingest trigger-discovery -b kb-prod -s "$SOURCE_ID" --rule "$RULE_ID"

`dodil data ingest trigger`

Dispatches already-discovered objects to ingest — replay / backfill.


dodil data ingest trigger -b BUCKET -s SOURCE_ID \
  [--rule RULE_ID] [--source-object-id OBJ_ID] [--retry-failed]

Flag	Short	Type	Description
`--source`	`-s`	string	Required. Source to ingest from.
`--rule`	`-r`	string	Restrict to one rule’s pending objects
`--source-object-id`	—	string	Restrict to a single source-object row
`--retry-failed`	—	bool	Include objects in `FAILED` / `PARTIAL` status

Examples:


# Replay all failed/partial objects
dodil data ingest trigger -b kb-prod -s "$SOURCE_ID" --retry-failed
 
# Replay a single object
dodil data ingest trigger -b kb-prod -s "$SOURCE_ID" --source-object-id obj_a1b2...

Two job-level operations have no CLI command yet. TriggerIngest (POST /:bucket/ingest) is the one-shot single-object ingest that skips the source queue and is the only path accepting a per-event options overlay. RetryIngestJob (POST /:bucket/ingest/jobs/{job_id}/retry) re-runs one existing FAILED/PARTIAL job in place, keeping the same job_id. Both are on the API — see Replay & Retry.

Jobs

`dodil data ingest jobs`


dodil data ingest jobs -b BUCKET [-r RULE_ID] [-p PIPELINE_ID] \
  [--status STATUS] [--page-size N] [--page-token TOKEN] [--all]

Lists ingest jobs in the bucket. All filtering is server-side, including status.

Flag	Short	Description
`--rule`	`-r`	Filter by rule ID
`--pipeline`	`-p`	Filter by pipeline ID
`--status`	—	Server-side status filter: `pending`, `processing`, `completed`, `failed`, `partial`, `retrying`. Also accepts the full enum name (`INGEST_STATUS_FAILED`) or its numeric value.
`--page-size`	—	Max jobs per page (`0` = server default, ~50)
`--page-token`	—	Page token from a previous response’s `nextPageToken`
`--all`	—	Follow pagination to the end and return every matching job

Examples:


# All jobs in the bucket
dodil data ingest jobs -b kb-prod -o json
 
# Failed jobs only — server-side, no jq needed
dodil data ingest jobs -b kb-prod --status failed -o json
 
# Every failed job for one pipeline, following pagination to the end
dodil data ingest jobs -b kb-prod -p "$PIPELINE_ID" --status failed --all -o json
 
# Only jobs from one rule
dodil data ingest jobs -b kb-prod -r "$RULE_ID" -o json

An unrecognised --status fails fast with the valid list rather than silently returning everything. The human table prints JOB ID / STATUS / PIPELINE / OBJECT / THREAD with the INGEST_STATUS_ prefix trimmed, plus a Showing N of M job(s) footer that names the next page token when there is one.

Status enum, counters, and Scriptum thread linkage are documented at Core Concepts → IngestJob.

`dodil data ingest watch`


dodil data ingest watch <job-id> -b BUCKET [--wait 5m] [--interval 3s]

Polls one job until it reaches a terminal state — COMPLETED, FAILED or PARTIAL — printing a line on every observed change and then a result summary. This is the command to reach for after an upload; jobs is for surveying, watch is for waiting.

Flag	Type	Default	Description
`--wait`	duration	`5m`	How long to keep watching before giving up
`--interval`	duration	`3s`	Poll interval


dodil data ingest watch job_a1b2... -b kb-prod
# [3s]  PENDING (batches=0, embeddings=0)
# [9s]  PROCESSING (batches=2, embeddings=18)
# [21s] COMPLETED (batches=5, embeddings=47)
#
# Job job_a1b2... COMPLETED
#   object:   papers/attention.pdf
#   pipeline: text_embedding_index (vector)
#   chunks=47 embeddings=47 rows=0 vector=success
#   thread:   thread_a1b2... ('dodil scriptum thread steps thread_a1b2...' for per-step detail)

Notes on behaviour worth knowing:

With -o json the progress lines are suppressed and only the final IngestJob is printed — safe to pipe.
Transient control-plane blips are ridden out; watch only gives up after three consecutive failed polls.
Timing out is not a failure of the job. The job keeps running server-side; the error message says so and names the Scriptum thread so you can run dodil scriptum thread steps <id> for per-step detail.
Every ingest job executes as a Scriptum thread — the thread id in the summary is the handle for debugging a failure.

dodil data ingest

Rules

dodil data ingest add

dodil data ingest get

dodil data ingest list

dodil data ingest update

dodil data ingest delete

Sync triggers

dodil data ingest trigger-discovery

dodil data ingest trigger

Jobs

dodil data ingest jobs

dodil data ingest watch

See also

`dodil data ingest`

`dodil data ingest add`

`dodil data ingest get`

`dodil data ingest list`

`dodil data ingest update`

`dodil data ingest delete`

`dodil data ingest trigger-discovery`

`dodil data ingest trigger`

`dodil data ingest jobs`

`dodil data ingest watch`