Jobs — API Reference

Package: dodil.data.ingest.v1 · Service: IngestService

An ingest job is one execution of one pipeline against one object. The Jobs RPCs let you fire a one-shot ingest, look up a specific job, list jobs in a bucket with filters, or re-run a single failed one. See Core Concepts → IngestJob.

RPC	HTTP
`TriggerIngest`	`POST /:bucket/ingest`
`GetIngestStatus`	`GET /:bucket/ingest/jobs/:job_id`
`ListIngestJobs`	`GET /:bucket/ingest/jobs`
`RetryIngestJob`	`POST /:bucket/ingest/jobs/:job_id/retry`

gRPC setup — grpcurl, endpoints, reflection, and field-name casing — is covered once in Conventions → Using gRPC.

Status

IngestStatus has six meaningful values plus the zero default. These exact strings are what the API emits and accepts.

Value	Meaning
`INGEST_STATUS_UNSPECIFIED`	Zero value — never a real job state.
`INGEST_STATUS_PENDING`	Row created, not yet picked up.
`INGEST_STATUS_PROCESSING`	A worker is running the pipeline.
`INGEST_STATUS_COMPLETED`	Terminal success.
`INGEST_STATUS_FAILED`	Terminal failure.
`INGEST_STATUS_PARTIAL`	Some output landed, some did not.
`INGEST_STATUS_RETRYING`	A previous attempt failed transiently and NATS JetStream will redeliver. `error` carries `"attempt N/M: <last error>"` so progress is visible.

RETRYING is not terminal and needs no action from you — redelivery is automatic. FAILED and PARTIAL are terminal; those are the ones RetryIngestJob exists for.

Counters

IngestJob carries one counter set, and which fields are populated depends on the pipeline’s kind. A counter that does not apply stays 0 — a 0 does not mean the stage ran and produced nothing.

Field	Populated for	Meaning
`chunks_created`	vector	Chunks the splitter produced.
`embeddings_created`	vector	Embeddings computed.
`embeddings_written`	vector	Embeddings actually persisted — compare against `embeddings_created` to spot a partial write.
`rows_written`	warehouse (table)	Rows landed in the Delta table.
`objects_written`	object	Blobs the ObjectWriter wrote at the destination. Always `0` or `1` today — one blob per source object — but typed as a count so future multi-output pipelines need no schema bump.
`batches_received`	all	Streamed yields consumed from the Scriptum thread.
`output_size`	all	Bytes of Scriptum output.

vector_status is a separate free-form string on vector jobs: null, "success", "failed" or "skipped". thread_id links to the Scriptum thread; error_details carries the long-form failure text when error is only a summary.

`TriggerIngest`

One-shot ingest of a single object — skips discovery. Useful for testing pipelines, re-ingesting after a change, or piping in an object from an external system without going through the source sync layer.

By default K3 resolves which pipeline to run by matching the object against the bucket’s rules. Pass pipeline_id or rule_id to override.

Request

HTTP

Resolve pipeline from matching rules:


curl -sS -X POST "https://api.data.dodil.io/kb-prod/ingest" \
  -H "Authorization: Bearer $DODIL_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "object": { "bucket": "kb-prod", "key": "contracts/acme-2026.pdf" }
  }'

Force a specific pipeline + per-event option overlay:


curl -sS -X POST "https://api.data.dodil.io/kb-prod/ingest" \
  -H "Authorization: Bearer $DODIL_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "object": { "bucket": "kb-prod", "key": "contracts/acme-2026.pdf" },
    "pipelineId": "pipe_a1b2...",
    "options": { "chunk_size": "2000" }
  }'

Response

HTTP


{
  "job": {
    "jobId": "job_a1b2...",
    "object": { "bucket": "kb-prod", "key": "contracts/acme-2026.pdf" },
    "status": "INGEST_STATUS_PENDING",
    "pipelineId": "pipe_a1b2...",
    "pipelineName": "embed-contracts",
    "pipelineKind": "vector",
    "createdAt": "1716843600000"
  }
}

The response contains the freshly created IngestJob, typically in PENDING or already PROCESSING. Poll GetIngestStatus for terminal state, or watch it with dodil data ingest watch <job-id>.

Note that TriggerIngest is the one RPC in this service with no bucket field — it is annotated post: "/{object.bucket}/ingest", so the bucket is read from object.bucket in the body and mirrored into the path.

The options map on the request is a per-event overlay on top of the pipeline’s stored options — useful for one-off parameter sweeps without modifying the pipeline itself.

`GetIngestStatus`

Request

HTTP


curl -sS "https://api.data.dodil.io/kb-prod/ingest/jobs/job_a1b2..." \
  -H "Authorization: Bearer $DODIL_TOKEN"

Response

An IngestJob — see Counters above for which fields are populated per pipeline kind, and Core Concepts → IngestJob for the full type.

`ListIngestJobs`

Filter by status_filter, pipeline_id, or rule_id. Cursor pagination, default page size 50.

This endpoint’s query parameters are camelCase, unlike its siblings: the handler deserializes ListJobsQuery with rename_all = "camelCase", so it reads ?pipelineId=, ?ruleId=, ?pageSize= and ?pageToken=. Snake-case forms are silently ignored — a ?pipeline_id= filter returns the whole bucket rather than erroring. status_filter is the one exception: it keeps status and status_filter as explicit aliases.

Request

dodil data


dodil data ingest jobs -b kb-prod --status failed
dodil data ingest jobs -b kb-prod -p pipe_a1b2... --page-size 100
dodil data ingest jobs -b kb-prod -r rule_a1b2... --all

--status takes the short lowercase form: pending, processing, completed, failed, partial, retrying. --all follows pagination and returns every matching job.

HTTP

All failed jobs:


curl -sS "https://api.data.dodil.io/kb-prod/ingest/jobs?status=failed" \
  -H "Authorization: Bearer $DODIL_TOKEN"

The status filter accepts either the short lowercase name or the full wire enum — the handler strips an INGEST_STATUS_ prefix and lowercases before matching, so ?status=INGEST_STATUS_FAILED works too. An unrecognised value is dropped, leaving the listing unfiltered.

Jobs for one pipeline — note the camelCase keys:


curl -sS "https://api.data.dodil.io/kb-prod/ingest/jobs?pipelineId=pipe_a1b2...&pageSize=100" \
  -H "Authorization: Bearer $DODIL_TOKEN"

Jobs for one rule:


curl -sS "https://api.data.dodil.io/kb-prod/ingest/jobs?ruleId=rule_a1b2..." \
  -H "Authorization: Bearer $DODIL_TOKEN"

gRPC


rpc ListIngestJobs(ListIngestJobsRequest) returns (ListIngestJobsResponse);
 
message ListIngestJobsRequest {
  string bucket = 1;
  optional IngestStatus status_filter = 2;
  optional string pipeline_id = 3;
  optional string rule_id = 4;
  dodil.data.common.v1.PaginationRequest pagination = 5;
}

All failed jobs:


grpcurl \
  -H "Authorization: Bearer $DODIL_TOKEN" \
  -d '{
    "bucket": "kb-prod",
    "status_filter": "INGEST_STATUS_FAILED"
  }' \
  $K3_GRPC \
  dodil.data.ingest.v1.IngestService/ListIngestJobs

Jobs for one pipeline:


grpcurl \
  -H "Authorization: Bearer $DODIL_TOKEN" \
  -d '{
    "bucket": "kb-prod",
    "pipeline_id": "pipe_a1b2...",
    "pagination": { "page_size": 100 }
  }' \
  $K3_GRPC \
  dodil.data.ingest.v1.IngestService/ListIngestJobs

Jobs for one rule:


grpcurl \
  -H "Authorization: Bearer $DODIL_TOKEN" \
  -d '{
    "bucket": "kb-prod",
    "rule_id": "rule_a1b2..."
  }' \
  $K3_GRPC \
  dodil.data.ingest.v1.IngestService/ListIngestJobs

Response

HTTP


{
  "jobs": [
    {
      "jobId": "job_a1b2...",
      "object": { "bucket": "kb-prod", "key": "contracts/acme-2026.pdf" },
      "status": "INGEST_STATUS_COMPLETED",
      "pipelineId": "pipe_a1b2...",
      "pipelineName": "embed-contracts",
      "pipelineKind": "vector",
      "ruleId": "rule_a1b2...",
      "chunksCreated": 47,
      "embeddingsCreated": 47,
      "rowsWritten": 0,
      "objectsWritten": 0,
      "createdAt": "1716843600000",
      "updatedAt": "1716843680000",
      "threadId": "thread_a1b2...",
      "vectorStatus": "success",
      "outputSize": 18429,
      "batchesReceived": 5,
      "embeddingsWritten": 47
    }
  ],
  "pagination": { "nextPageToken": "", "totalCount": "1" }
}

rowsWritten and objectsWritten are 0 here because this is a vector job, not because those stages produced nothing — see Counters.

`RetryIngestJob`

Re-runs a single failed or partial job in place. It resets the per-attempt state on the existing row and re-publishes an IngestEvent with the same job_id — no new row is created and no schema bump happens, so the job’s history stays on one record.

Use this for one bad object. For bulk replay across a source, use TriggerIngestion(retry_failed = true) instead.

Request

HTTP


curl -sS -X POST "https://api.data.dodil.io/kb-prod/ingest/jobs/job_a1b2.../retry" \
  -H "Authorization: Bearer $DODIL_TOKEN"

Response

The reset IngestJob, with status back to INGEST_STATUS_RETRYING and its counters cleared.

There is no dodil data command for single-job retry. The CLI’s retry surface is dodil data ingest trigger --retry-failed, which is the bulk TriggerIngestion path, not this one. Watch a retried job with dodil data ingest watch <job-id>.

Jobs — API Reference

Status

Counters

TriggerIngest

Request

HTTP

gRPC

Response

HTTP

gRPC

GetIngestStatus

Request

HTTP

gRPC

Response

ListIngestJobs

Request

dodil data

HTTP

gRPC

Response

HTTP

gRPC

RetryIngestJob

Request

HTTP

gRPC

Response

See also

`TriggerIngest`

`GetIngestStatus`

`ListIngestJobs`

`RetryIngestJob`