Replay & retry

Goal: recover from ingest failures and re-run objects after pipeline changes — without dropping the rest of your ingest stream.

Why this matters: ingest pipelines run async over big object volumes. Some jobs fail transiently (network blip, model timeout, transient backend slowness). Some fail permanently (corrupt PDF, malformed schema). K3 retries the first kind automatically; the second kind needs your action. This recipe walks the full diagnose-and-recover loop.

Shape:

   list FAILED jobs   ──►   read .error / .errorDetails / .threadId
                                       │
                                       ▼
                              decide: bulk or surgical?
                                       │
         ┌──────────────────┬──────────┴───────────┬──────────────────┐
         ▼                  ▼                      ▼                  ▼
  all failures on    re-run THIS job         one object as a    full re-scan of
  one source/rule    in place, same id       brand-new job      source (full_sync)
  TriggerIngestion   RetryIngestJob          TriggerIngest      then re-dispatch
  (retry_failed)     (FAILED/PARTIAL only)   (+options overlay)

The four scopes are genuinely different operations, not aliases:

Prerequisites

• A bucket with rules + pipelines already wired (see PDF → Vector or Documents → Warehouse)
• Some ingest history — at least a handful of jobs across COMPLETED / FAILED / PARTIAL (you can force a failure by uploading a corrupt PDF or pointing a rule at an unsupported MIME)

We’ll use kb-prod as the bucket throughout.

1. List FAILED jobs in the bucket

The status filter is server-side, and the CLI exposes it. Don’t pull the whole bucket down and grep it:

# Every failed job, following pagination to the end
dodil data ingest jobs -b kb-prod --status failed --all -o json \
  | jq '.jobs[] | {jobId, object: .object.key, pipelineName, error, updatedAt}'
 
# Scope to one pipeline or one rule
dodil data ingest jobs -b kb-prod --status failed -p "$PIPELINE_ID" -o json
dodil data ingest jobs -b kb-prod --status failed -r "$RULE_ID" -o json

--status accepts the short form (failed), the full enum name (INGEST_STATUS_FAILED), or the numeric value; anything else fails fast with the valid list. Without --all you get one page (server default ~50) and a nextPageToken to continue from.

The same filter over HTTP:

curl -sS "https://api.data.dodil.io/kb-prod/ingest/jobs?statusFilter=INGEST_STATUS_FAILED&pagination.pageSize=100" \
  -H "Authorization: Bearer $DODIL_TOKEN" \
  | jq '.jobs[] | {jobId, object: .object.key, pipelineName, ruleId, error}'

Want the PARTIAL bucket (some output landed, some didn’t)?

dodil data ingest jobs -b kb-prod --status partial --all -o json \
  | jq '.jobs[] | {jobId, object: .object.key, embeddingsCreated, embeddingsWritten, vectorStatus, error}'

PARTIAL is common in vector pipelines when some chunks embedded successfully but embeddingsWritten < embeddingsCreated — the collection write phase didn’t land everything. vectorStatus narrows it further: "failed" (the persist step errored) versus "skipped" (the script produced no embeddings at all) are different bugs.

2. Read the failure detail

JOB_ID="<a failed job_id from step 1>"
 
curl -sS "https://api.data.dodil.io/kb-prod/ingest/jobs/$JOB_ID" \
  -H "Authorization: Bearer $DODIL_TOKEN" \
  | jq '{status, error, errorDetails, threadId, batchesReceived,
         chunksCreated, embeddingsCreated, embeddingsWritten, rowsWritten, objectsWritten}'

Four fields matter most for diagnosis:

Follow the thread id straight into the step that broke:

dodil scriptum thread steps "<threadId>"

dodil data ingest watch <job-id> -b kb-prod prints the same summary — including the thread id and the exact command above — whenever a job it is watching reaches a terminal state.

Common failure shapes you’ll see — first-pass triage:

3. How RETRYING works

If you see jobs in INGEST_STATUS_RETRYING, K3 has already detected a transient failure and is retrying automatically. You don’t need to do anything — wait for the terminal state.

Retries are NATS JetStream redelivery, not an application-level loop, and two budgets share one cap:

The effective maximum is the smaller of the two, except for a job waiting on an engine, which gets the full 30. Only errors the handler classifies as retryable are redelivered at all — a permanent failure goes straight to FAILED on the first attempt, without ever passing through RETRYING. Status flow:

PENDING ─► PROCESSING ─► COMPLETED
                │
                ▼  transient error
            RETRYING ─► PROCESSING ─► COMPLETED   (success path)
                │
                ▼  same / different transient error
            RETRYING ─► PROCESSING ─► COMPLETED   (eventually succeeds)
                                  ╲
                                   ╲  max attempts reached
                                    ╲►  FAILED

The error field on a RETRYING job reads attempt N/M: <last error> — useful for watching progress without polling internals. After max attempts, RETRYING → FAILED with the last error preserved.

4. Replay strategies — pick the right scope

K3 gives you four replay scopes. Pick by how much you want to re-dispatch, and by whether you want the existing job row re-run or a new one.

A. One existing job, in place — RetryIngestJob

This is the first thing to reach for after fixing a pipeline. It re-runs the job you already have: same job_id, no new row, per-attempt state reset, status flipped to RETRYING immediately so a UI polling the job sees the change at once.

JOB_ID="<a failed job_id from step 1>"
 
curl -sS -X POST "https://api.data.dodil.io/kb-prod/ingest/jobs/$JOB_ID/retry" \
  -H "Authorization: Bearer $DODIL_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{}' \
  | jq '.job | {jobId, status, error}'

Then watch it to a terminal state:

dodil data ingest watch "$JOB_ID" -b kb-prod

Three properties worth knowing before you rely on it:

• Only FAILED and PARTIAL jobs qualify. A PENDING, PROCESSING, RETRYING or COMPLETED job is refused with FAILED_PRECONDITION: only failed or partial jobs can be retried — re-kicking a running job would duplicate work, and re-running a completed one is a different use case.
• The pipeline and its options are re-resolved, not replayed. The handler deliberately does not reuse the options stored on the job row, so a fix you made to the pipeline (a corrected model, new chunk size) takes effect on the retry. This is exactly what makes “fix the pipeline, then retry” work.
• Worker idempotency does the rest. The event is re-published with the same job_id; the worker looks the row up by id and treats it as a fresh attempt.

RetryIngestJob has no CLI command today — use the route above.

B. Single object as a new job — surgical

For “run this object again as a brand-new job”, or to test a pipeline change against a known case without touching the original job row:

# Via TriggerIngest — re-runs the matching rule's pipeline
curl -sS -X POST "https://api.data.dodil.io/kb-prod/ingest" \
  -H "Authorization: Bearer $DODIL_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "object": { "bucket": "kb-prod", "key": "intake/contracts/acme-2026.pdf" }
  }'
 
# Force a specific pipeline (override rule resolution)
curl -sS -X POST "https://api.data.dodil.io/kb-prod/ingest" \
  -H "Authorization: Bearer $DODIL_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "object": { "bucket": "kb-prod", "key": "intake/contracts/acme-2026.pdf" },
    "pipelineId": "pipe_a1b2..."
  }'
 
# Per-event option overlay (one-off parameter sweep without modifying the pipeline)
curl -sS -X POST "https://api.data.dodil.io/kb-prod/ingest" \
  -H "Authorization: Bearer $DODIL_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "object": { "bucket": "kb-prod", "key": "intake/contracts/acme-2026.pdf" },
    "pipelineId": "pipe_a1b2...",
    "options": { "chunk_size": "2000" }
  }'

The CLI doesn’t expose TriggerIngest directly today — use the API. See Jobs — API Reference.

options here is the only per-event overlay in the system. RetryIngestJob re-resolves options from the pipeline; TriggerIngestion uses the pipeline’s own. If you want to sweep a parameter without editing the pipeline, this is the one path that does it.

C. All failed objects in a source — bulk replay

For “I fixed the pipeline, now re-run everything that failed”:

# Replay every FAILED / PARTIAL object on this source through their matching rules
dodil data ingest trigger --bucket kb-prod \
  --source "$SOURCE_ID" \
  --retry-failed
 
# Or scope to one rule's failures
dodil data ingest trigger --bucket kb-prod \
  --source "$SOURCE_ID" \
  --rule "$RULE_ID" \
  --retry-failed

The response includes a dispatched count — how many objects K3 queued for re-ingest:

{
  "accepted": true,
  "message": "Replay started for 42 failed objects",
  "dispatched": 42
}

D. Full re-discover (after rules / pipeline options change)

For “the rule’s match set changed (you tightened or loosened the include pattern), or the pipeline now produces different output and you want to re-run everything that would match today”:

# Discover from scratch — ignores the source's etag checkpoint
dodil data ingest trigger-discovery --bucket kb-prod \
  --source "$SOURCE_ID" \
  --full-sync
 
# Then re-dispatch any pending objects (you can combine with --retry-failed)
dodil data ingest trigger --bucket kb-prod --source "$SOURCE_ID" --retry-failed

For external (Preview) sources this is the standard backfill pattern; for the internal S3 source it’s effectively a “scan and re-fire rules on everything in the bucket.”

5. Replaying onto a different pipeline

A common shape: the template or its options were wrong, you built a corrected pipeline, and now you want the same objects processed by the new one. Re-binding the rule is what makes this replay — the rule is the only thing that decides which pipeline an object reaches.

# 1. Point the existing rule at the new pipeline
dodil data ingest update "$RULE_ID" -b kb-prod --pipeline "$NEW_PIPELINE_ID"
 
# 2. Confirm the binding resolved
dodil data ingest get "$RULE_ID" -b kb-prod
# ID: rule_a1b2...  Collection: pipe_new...  Type: vector  Enabled: true
 
# 3. Now re-dispatch. Nothing replays on its own — the rule change only
#    affects future dispatches until you trigger one.
dodil data ingest trigger -b kb-prod -s "$SOURCE_ID" --rule "$RULE_ID" --retry-failed

Three things follow from how the binding works, and each of them bites someone eventually:

• Re-binding alone replays nothing. UpdateRule rewrites pipeline_id and that is all. Objects already ingested stay ingested; you must trigger a re-dispatch (step 3 above) or a full re-discovery.
• RetryIngestJob picks up the new pipeline for free. It re-resolves the pipeline through the rule rather than reusing what the job row stored — so on a job that has a rule_id, retrying after a re-bind runs the new pipeline. On a job with no rule_id (a manual trigger), it resolves by pipeline name instead.
• The old pipeline’s output is not cleaned up. Rows already written to a table, embeddings already in a collection, blobs already at an object prefix — all stay. Re-binding changes what happens next, never what already happened. Dedup is the destination’s problem: give the table merge keys, or write to a fresh destination and cut over.

If you delete the old pipeline instead of re-binding, dependent rules survive with a pipeline_id pointing at nothing and an empty binding, and they fail at ingest time. Re-bind first, delete second.

6. Verify the replay landed

Watch the same query you used in step 1 — the failed jobs should move through PROCESSING to COMPLETED:

# Tail until nothing is in flight for this rule
while true; do
  inflight=$(for s in pending processing retrying; do
    dodil data ingest jobs -b kb-prod -r "$RULE_ID" --status "$s" --all -o json \
      | jq '.jobs | length'
  done | paste -sd+ - | bc)
  echo "in-flight: $inflight"
  [ "$inflight" = "0" ] && break
  sleep 5
done

Then summarize the new terminal-state distribution:

dodil data ingest jobs -b kb-prod -r "$RULE_ID" --all -o json \
  | jq '[.jobs[].status] | group_by(.) | map({status: .[0], count: length})'

You’re aiming for INGEST_STATUS_COMPLETED to dominate. Anything stuck in FAILED after a replay needs another round of diagnosis.

When to not retry

A few classes of failure where retry just burns more CPU:

• Object permanently corrupt / unsupported — fix or remove the source object, then re-upload. Retrying without changing the input gets the same error.
• Template mismatch — the template requires options you didn’t provide (or the wrong type). Fix pipeline.options first, then replay.
• Destination is gone — your vector collection or warehouse table was deleted. Recreate / re-bind the pipeline before replaying.
• Quota exhausted — replay will hit the same quota wall. Resolve quotas in the Storage admin first.

A useful gate: if error contains “attempt N/N” (i.e. K3 already exhausted the retry budget), assume the failure is permanent until you change something. Just re-firing the same job will land in FAILED again — and because RetryIngestJob re-resolves options from the pipeline, retrying without editing the pipeline reproduces the identical run.

Surgical re-run with a pipeline override

When you’re debugging “did my new pipeline options fix this?” without touching the production pipeline:

# Test the same object against a different (test) pipeline
curl -sS -X POST "https://api.data.dodil.io/kb-prod/ingest" \
  -H "Authorization: Bearer $DODIL_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "object": { "bucket": "kb-prod", "key": "intake/contracts/acme-2026.pdf" },
    "pipelineId": "pipe_test_a1b2..."
  }'
 
# Or test new options without changing the pipeline at all
curl -sS -X POST "https://api.data.dodil.io/kb-prod/ingest" \
  -H "Authorization: Bearer $DODIL_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "object": { "bucket": "kb-prod", "key": "intake/contracts/acme-2026.pdf" },
    "pipelineId": "pipe_a1b2...",
    "options": { "chunk_size": "2000", "chunk_overlap": "400" }
  }'

The options field on the request is a per-event overlay — it doesn’t modify the pipeline. Spawn N jobs with N different option sets to A/B test before committing changes via pipeline update.

Common gotchas

See also

• PDF → Vector — the canonical RAG ingest recipe you might be retrying from
• Documents → Warehouse — the warehouse-side recipe
• dodil data ingest — jobs --status, watch, and the rule-update flags used above
• Sync — API Reference — TriggerDiscovery, TriggerIngestion, GetSyncStatus
• Jobs — API Reference — TriggerIngest, GetIngestStatus, ListIngestJobs, RetryIngestJob
• Core Concepts → IngestJob — the full status enum + counters