Operations

Day-2 operations and incident triage. The shortest path from “something feels off” to “I know which surface to inspect.”

1. Service health + readiness

curl -sS "https://api.data.dodil.io/health"
curl -sS "https://api.data.dodil.io/healthz"
curl -sS "https://api.data.dodil.io/ready"
curl -sS "https://api.data.dodil.io/readyz"
curl -sS "https://api.data.dodil.io/metrics"

All return 200 + a short body on a healthy K3. /metrics is Prometheus-shaped — scrape it for SLO dashboards. These five paths are the only ones that bypass auth — see Auth & Access → Unauthenticated endpoints.

2. Bucket + capacity baseline

There is no engine to check — tables, vector, and graph are implicit per bucket (the engine plane was retired). dodil data engine get errors; use the reservation and the table listing instead.

# Storage — list your org's buckets
dodil data bucket list
 
# Capacity — what hot budget this bucket holds (never errors, even with no reservation)
dodil data reservation get -b "$BUCKET" -o json | jq '{capacity_gb, state, floors}'
# state: RESERVATION_STATE_ACTIVE (held) or RESERVATION_STATE_PENDING_CAPACITY (backordered)
 
# Tables — what exists on the tables-gateway (this is SQL `SHOW TABLES`, not a control-plane RPC)
dodil data table list -b "$BUCKET"
 
# Vector — a collection is a pipeline with a vector facet; list what exists
dodil data vector collection list --bucket "$BUCKET"

There is no explicit vector setup step — a vector collection is materialized the first time you run dodil data vector collection add <name> -b $BUCKET --template <id>. If the collection list is empty, no vector workload has been configured on the bucket yet. See Reservation & Hot Cache for what the reservation shape means.

3. Pipelines health

# All rules in the bucket. NOTE: `ingest list` ignores `-o json` and always
# prints a table — columns: ID, NAME, COLLECTION, TYPE, ENABLED.
dodil data ingest list --bucket "$BUCKET"
 
# All jobs in the bucket — focus on FAILED + RETRYING.
# `ingest jobs -o json` returns {"pagination":{}} (no `jobs` key) on an empty
# bucket, so guard every filter with `.jobs // []`.
dodil data ingest jobs --bucket "$BUCKET" -o json \
  | jq '[(.jobs // [])[] | select(.status | IN("INGEST_STATUS_FAILED", "INGEST_STATUS_RETRYING"))] |
        group_by(.status) |
        map({status: .[0].status, count: length})'
 
# Stuck-pending count (ingest backlog)
dodil data ingest jobs --bucket "$BUCKET" -o json \
  | jq '[(.jobs // [])[] | select(.status == "INGEST_STATUS_PENDING")] | length'

Things to look for:

• Repeated FAILED with the same errorDetails across many objects → a template / pipeline misconfiguration. Replay after fixing — see Pipelines → Replay & Retry.
• Stuck PENDING for > 1 minute → check whether the worker is healthy via /metrics.
• Persistent RETRYING with attempt N/M reaching N=M and flipping to FAILED → transient failure exhausted retries; likely a real problem (upstream model timeout, missing credential, etc.).
• A rule whose COLLECTION names a pipeline that no longer exists → orphaned rule (the pipeline was deleted). Re-bind: dodil data ingest update <rule_id> -b $BUCKET --collection <new_pipeline_id> (the CLI’s --collection maps to the API’s pipeline_id).

4. Search smoke check

dodil data vsearch runs pure KNN against a table’s vector column. It requires -t/--table (a collection is a table with a vector column; --collection is an alias) and --column; there is no -c shorthand. Give it a query with either --text (embedded client-side via ignite models) or --vector.

# Simplest: let vsearch embed the query text for you
dodil data vsearch -b "$BUCKET" -t "$TABLE" --column "$VECTOR_COLUMN" \
  --text "health probe query" --metric cosine --top-k 5
 
# Or pass a pre-computed vector
VEC=$(dodil ignite models embed "health probe query" -o json | jq -r '.embedding|join(",")')
dodil data vsearch -b "$BUCKET" -t "$TABLE" --column "$VECTOR_COLUMN" --vector "$VEC" --top-k 5

Scores are distances (cosine/euclidean/dot) — closest match first, so they sort ascending, not descending.

If you get zero results:

1. Confirm the table + vector column exist: dodil data table describe "$TABLE" -b $BUCKET — look for a vector<N> column.
2. Confirm ingest completed for some objects: dodil data ingest jobs -b $BUCKET -o json | jq '[(.jobs // [])[] | select(.status == "INGEST_STATUS_COMPLETED")] | length'
3. For multi-collection retrieval, dodil data search "<text>" -b $BUCKET fuses results across collections with reciprocal-rank fusion. Note there is no dense+BM25 hybrid on the plane — see Feature Status.

5. Tables drain + maintenance

For tables under active write load. Note dodil data table describe returns the table’s schema only (column, type, nullable, pk, default) — it runs DESCRIBE on the tables-gateway and does not expose WAL / drain counters. The way to check drain health is to run compact and read how much it drained:

# Drain the write-ahead log into the base table. JSON keys:
#   wal_entries_processed, wal_unique_keys, drained_high_ulid
dodil data table compact "$TABLE" --bucket "$BUCKET" -o json | jq '{wal_entries_processed, wal_unique_keys}'
# wal_entries_processed == 0 → nothing left to drain (the compactor is keeping up).
# A large, non-decreasing count across repeated runs → the compactor is behind.

Then optionally bin-pack small files. optimize returns {table_name, version, optimize_type, files_added, files_removed} (no byte counts):

dodil data table optimize "$TABLE" --bucket "$BUCKET" -o json | jq '{files_added, files_removed, version}'
 
# Reclaim stale files past the 168h retention floor (there is no bypass flag).
# vacuum returns {table_name, version, dry_run, files_deleted}. Preview with --dry-run first.
dodil data table vacuum "$TABLE" --bucket "$BUCKET" --dry-run -o json | jq '{dry_run, files_deleted}'

For full per-RPC maintenance details, see Tables → Maintenance and the Tables Maintenance CLI guide.

6. Common failure patterns

What the auth-layer status codes (401 / 402 / 403) mean is defined in Auth & Access; the table below maps operational symptoms to concrete fixes.

7. Where to dig deeper

See also

• Conventions — auth headers + error envelope
• Feature Status — what’s live vs roadmap
• Auth and Access — auth modes detail