Core Concepts — Pipelines

Every type signature below is verbatim from proto/proto-k3/ (k3_source.proto, k3_pipeline.proto, k3_ingest.proto). Pillar packages are dodil.data.source.v1, dodil.data.pipeline.v1, dodil.data.ingest.v1. Wire encodings: gRPC follows the proto types directly; HTTP uses pbjson (camelCase, int64 as strings, enums as wire-name strings).

The six types: Source, Credential, Pipeline, Template, Rule, IngestJob. Their lifecycle and how they compose:

   bucket ──┬── Source (where to read from)
            │     │
            │     └── Credential (how to auth)
            │
            ├── Pipeline (Scriptum template + destination)
            │     │
            │     └── Template (catalog entry)
            │
            └── Rule (filter + pipeline binding)
                  │
                  └── IngestJob (one execution per matched object)

Source

Where K3 reads objects from. Every bucket has an auto-created internal S3 source the moment you call CreateBucket — you don’t register it, name it, or authenticate it. Every direct upload to the bucket flows through your rules.

Preview — external sources (SOURCE_PROVIDER_GOOGLE_DRIVE, _SHAREPOINT, _CONFLUENCE, _GITHUB) are available in the API but are still solidifying for production use. The internal S3 path is the production-ready channel today.

{
  "sourceId": "src_a1b2…",
  "bucket": "kb-prod",
  "provider": "SOURCE_PROVIDER_GOOGLE_DRIVE",
  "name": "finance-drive",
  "description": "Q2 finance docs",
  "rootPath": "/Q2",
  "providerAccount": "drive-id-123",
  "syncIntervalSeconds": "3600",
  "enabled": true,
  "status": "SOURCE_STATUS_ACTIVE",
  "totalObjects": "1247",
  "indexedObjects": "1240",
  "totalBytes": "8923471024",
  "createdAt": "1716840000000",
  "updatedAt": "1716843600000",
  "lastSyncAt": "1716843600000",
  "lastSuccessAt": "1716843600000"
}

message Source {
  string source_id = 1;
  string bucket = 2;
  SourceProvider provider = 3;
  string name = 4;
  string description = 5;
  string root_path = 6;                       // root folder/prefix to sync
  string provider_account = 7;                // bucket name, drive ID, repo URL, etc.
  int64 sync_interval_seconds = 8;            // 0 = manual sync only
  bool enabled = 9;
  SourceStatus status = 10;
  int64 total_objects = 11;
  int64 indexed_objects = 12;
  int64 total_bytes = 13;
  int64 created_at = 14;
  int64 updated_at = 15;
  int64 last_sync_at = 16;                    // 0 = never
  int64 last_success_at = 17;                 // 0 = never
  optional string last_error = 18;
}
 
enum SourceProvider {
  SOURCE_PROVIDER_UNSPECIFIED   = 0;
  SOURCE_PROVIDER_INTERNAL_S3   = 1;
  SOURCE_PROVIDER_GOOGLE_DRIVE  = 10;
  SOURCE_PROVIDER_SHAREPOINT    = 11;
  SOURCE_PROVIDER_CONFLUENCE    = 12;
  SOURCE_PROVIDER_GITHUB        = 20;
}
 
enum SourceStatus {
  SOURCE_STATUS_UNSPECIFIED = 0;
  SOURCE_STATUS_PENDING     = 1;
  SOURCE_STATUS_SYNCING     = 2;
  SOURCE_STATUS_ACTIVE      = 3;
  SOURCE_STATUS_ERROR       = 4;
  SOURCE_STATUS_DISABLED    = 5;
}

Key facts:

• provider = SOURCE_PROVIDER_INTERNAL_S3 is the auto-created per-bucket source. It fires on every direct S3 PUT to the bucket.
• sync_interval_seconds = 0 means manual sync only — useful when you want explicit control via TriggerDiscovery.
• provider_account is provider-shaped: for Google Drive it’s a drive ID; for GitHub a repo URL; for InternalS3 the bucket name itself.
• total_objects / indexed_objects / total_bytes are aggregates the source maintains as it discovers and ingests.
• status reflects the source’s most recent sync attempt — SOURCE_STATUS_ACTIVE is the steady-state success. A freshly auto-created internal source reports SOURCE_STATUS_PENDING, because it has not run a sync yet. That is normal and does not stop direct uploads from firing rules; don’t gate your setup on seeing ACTIVE.
• The auto-created internal source is created with name: "internal", description: "Auto-created internal S3 source", enabled: true, sync_interval_seconds: 300.
• Source has full CRUD on the API — CreateSource, GetSource, ListSources, UpdateSource (PATCH /:bucket/sources/:source_id) and DeleteSource. The CLI exposes only the first three.

Credential

How K3 authenticates to an external source. Credentials are org-scoped and may optionally link to a source_id. The secret payload is a oneof keyed on credential_type.

Preview — credentials are used by external sources, which are Preview. The internal S3 source needs no credential. The shape below reflects the API as it stands today; expect refinements as external source support hardens.

{
  "info": {
    "credentialId": "cred_a1b2…",
    "sourceId": "src_a1b2…",
    "provider": "SOURCE_PROVIDER_GOOGLE_DRIVE",
    "credentialType": "CREDENTIAL_TYPE_OAUTH2",
    "displayName": "gdrive-main",
    "isPrimary": true,
    "isValid": true,
    "createdAt": "1716840000000",
    "updatedAt": "1716843600000",
    "expiresAt": "1716847200000"
  },
  "oauth2": {
    "accessToken": "ya29.…",
    "refreshToken": "1//…",
    "expiresAt": "1716847200000",
    "tokenType": "Bearer",
    "scopes": ["https://www.googleapis.com/auth/drive.readonly"]
  }
}

enum CredentialType {
  CREDENTIAL_TYPE_UNSPECIFIED      = 0;
  CREDENTIAL_TYPE_OAUTH2           = 1;
  CREDENTIAL_TYPE_API_KEY          = 2;
  CREDENTIAL_TYPE_ACCESS_KEY       = 3;       // S3-style access_key + secret_key
  CREDENTIAL_TYPE_SERVICE_ACCOUNT  = 4;
  CREDENTIAL_TYPE_PAT              = 5;
}
 
message CredentialInfo {
  string credential_id = 1;
  string source_id = 2;                       // optional link to a source
  SourceProvider provider = 3;
  CredentialType credential_type = 4;
  string display_name = 5;
  bool is_primary = 6;
  bool is_valid = 7;
  int64 created_at = 8;
  int64 updated_at = 9;
  int64 expires_at = 10;                      // 0 = never
}
 
// One of these payload messages travels inside `oneof credential_data`:
message AccessKeyCredential {
  string access_key_id = 1;
  string secret_access_key = 2;
  string region = 3;
  string endpoint = 4;
}
message OAuth2Credential {
  string access_token = 1;
  string refresh_token = 2;
  int64 expires_at = 3;
  string token_type = 4;
  repeated string scopes = 5;
}
message ApiKeyCredential         { string api_key = 1; }
message ServiceAccountCredential { string json_key = 1; }
message PatCredential            { string token = 1; string username = 2; }

Key facts:

• Secrets are stored in HashiCorp Vault behind K3. ListCredentials returns only CredentialInfo — never a payload.
• GetCredential is RETIRED, with no replacement. It returned raw access/refresh tokens, API keys, service-account JSON and PATs at viewer tier; a control plane whose job is to hold secrets in Vault must not offer an org-level RPC that reads them back in plaintext. There is no route and no RPC. To rotate an expired OAuth token, call RefreshOAuthToken (POST /admin/credentials/{credential_id}/refresh), which refreshes it in place.
• ValidateCredential is RETIRED. It validated nothing — it echoed the stored is_valid, which is only ever written true, and stamped validated_at: now(), implying a live provider check it never performed. A real probe would be a new RPC that actually calls the provider; none exists yet.
• The surviving surface is exactly four RPCs: StoreCredential, ListCredentials, DeleteCredential, RefreshOAuthToken, plus GetOAuthUrl / ExchangeOAuthCode for the authorization-code flow.
• is_primary is a per-(source_id, provider) flag — at most one primary per pairing. Used by sync to pick which credential to authenticate with.

Pipeline

The recipe: a Scriptum template name + per-pipeline options + an optional destination. First-class row in the K3 DB. Rules and any other triggers reference pipelines by pipeline_id.

{
  "pipelineId": "pipe_a1b2…",
  "bucket": "kb-prod",
  "name": "text_embedding_index",
  "scriptumTemplate": "text_embedding_index",
  "options": {
    "chunk_size": "1000",
    "chunk_overlap": "150"
  },
  "createdAt": "1716840000000",
  "updatedAt": "1716843600000",
  "recipeName": "document-rag",
  "destination": {
    "storeEntityId": "ent_a1b2…",
    "facet": "PIPELINE_FACET_VECTOR",
    "name": "docs-index",
    "description": "",
    "status": "DESTINATION_STATUS_ACTIVE",
    "engineId": "eng_a1b2…",
    "createdAt": "1716840000000",
    "updatedAt": "1716840000000",
    "vector": {
      "dimensions": 768,
      "distanceMetric": "DISTANCE_METRIC_COSINE",
      "sparseMode": "SPARSE_MODE_NONE",
      "embeddingType": "EMBEDDING_TYPE_FLOAT",
      "modality": "text",
      "embeddingSource": "EMBEDDING_SOURCE_PIPELINE",
      "templateId": "text_embedding_index",
      "embedModel": "jina-embeddings-v4",
      "physicalName": "k3_1f2e3d4c…"
    }
  }
}

message Pipeline {
  string pipeline_id = 1;
  string bucket = 2;
  string name = 3;                            // user-visible label
  string scriptum_template = 4;               // Scriptum template id (the "what")
  map<string, string> options = 5;            // pipeline-level config overlay
 
  // 6/7/8 = store_entity_id, store_entity_kind, store_entity_name.
  // Folded into `destination` below.
  reserved 6, 7, 8;
  reserved "store_entity_id", "store_entity_kind", "store_entity_name";
 
  int64 created_at = 9;
  int64 updated_at = 10;
  // Stable recipe.id (e.g. "document-rag") when installed via a recipe.
  optional string recipe_name = 11;
  // The `store_entities` row this pipeline writes into.
  // ABSENT = free pipeline: runs Scriptum, persists nothing durable.
  Destination destination = 12;
}
 
message Destination {
  string store_entity_id = 1;                 // store_entities.id
  PipelineFacet facet = 2;                    // store_entities.kind
  string name = 3;
  string description = 4;
  DestinationStatus status = 5;
  string engine_id = 6;                       // opaque; no engine RPC exists
  int64 created_at = 7;
  int64 updated_at = 8;
  oneof config {                              // the `config` jsonb, typed per facet
    VectorConfig vector = 10;
    TableConfig  table  = 11;
    ObjectConfig object = 12;
  }
}
 
enum PipelineFacet {
  PIPELINE_FACET_UNSPECIFIED = 0;   // no filter / free pipeline
  PIPELINE_FACET_VECTOR      = 1;   // store_entities.kind = "vector"
  PIPELINE_FACET_TABLE       = 2;   // store_entities.kind = "warehouse"
  PIPELINE_FACET_OBJECT      = 3;   // store_entities.kind = "object"
  PIPELINE_FACET_GRAPH       = 4;   // representable; nothing creates one yet
}
 
enum DestinationStatus {
  DESTINATION_STATUS_UNSPECIFIED = 0;
  DESTINATION_STATUS_CREATING    = 1;
  DESTINATION_STATUS_ACTIVE      = 2;
  DESTINATION_STATUS_ERROR       = 3;
}

Key facts:

• store_entity_id / store_entity_kind / store_entity_name no longer exist on Pipeline — fields 6, 7 and 8 are reserved. Everything they said now lives on the nested destination message: destination.storeEntityId, destination.facet, destination.name. Code reading the old flat fields gets nothing back.
• A pipeline with no destination is a free pipeline — it runs the Scriptum script and persists nothing durable; you observe its output through the ingest job.
• PIPELINE_FACET_TABLE maps to the stored string "warehouse". The column kept its historical value; the wire uses the current vocabulary. Expect "warehouse" wherever a kind string appears (IngestJob.pipeline_kind, PipelineBinding.pipeline_kind) and PIPELINE_FACET_TABLE wherever the enum does.
• Filter ListPipelines with facet (or free_only for the complement — the two are mutually exclusive and the server rejects them together). Route via pipeline_id.
• Pipeline.name is not the name you typed for the facet creators. CreateVectorPipeline puts your name on destination.name and sets Pipeline.name to the template id, because the row it names is the index pipeline. dodil data vector collection list therefore prints the same string under NAME and TEMPLATE.
• Two script_source modes on CreatePipeline — scriptum_template or spawn_from_template. They are now historical aliases for the same thing: both mean “this is the template id to dispatch”, since the worker builds per-call env at dispatch time.
• options is an overlay on top of the Scriptum script’s defaults. Validated server-side against the template’s ScriptContract.
• CreatePipeline never creates a destination. It is the non-creating creator: it makes a free pipeline, or binds to a destination that already exists (server validates it belongs to the same org + bucket). Creating a destination is the job of a facet creator — CreateVectorPipeline, CreateTablePipeline, CreateObjectPipeline. There is no graph creator.
• Re-bind with UpdatePipeline and a new store_entity_id (empty string = make the pipeline free). UpdatePipeline.destination edits the bound destination in place, but a destination’s facet is immutable — supplying a config variant that doesn’t match is rejected.

Creating a pipeline does NOT create an ingest rule

Worth its own heading because it is the single most common way a working-looking setup silently does nothing. Every facet creator’s proto comment says it explicitly — CreateTablePipelineRequest: “This handler does NOT auto-create an ingest rule — callers own rule scope, same as the vector facet.”

A pipeline with no Rule pointing at it never runs. Binding the rule is a separate, mandatory call. The one exception is dodil data recipe install, which is a client-side orchestrator that issues the pipeline call and the CreateRule call for you.

Template

A row in the Scriptum template catalog. Org-scoped, not bucket-scoped — served from GET /admin/templates, not a bucket path. K3 ships a production catalog of 34 templates across six categories: embedding (10), core (9), actions (7), vision (4), ecommerce (2), financials (2). See API Reference → Templates → The catalog for the full list with descriptions and modalities.

message Template {
  string id = 1;                              // Scriptum template ID
  string name = 2;
  string description = 3;
  repeated string tags = 4;
  repeated string tools_required = 5;
  map<string, string> labels = 6;             // structured key-value labels
  string category = 7;                        // "embedding" | "core" | "actions" | …
  // Typed I/O contract — proxied verbatim from Scriptum's GetTemplate.
  // UI form generators walk this; K3 reads it server-side to validate
  // caller-supplied template inputs and to resolve schema-shaping facts.
  dodil.data.common.v1.ScriptContract contract = 8;
  // Modalities the template handles, from its `modality` label.
  repeated string modalities = 9;
  // File extensions it accepts, from its contract (lowercase, no dot).
  // Drives an ingest rule's glob patterns.
  repeated string accepted_extensions = 10;
  repeated string accepted_content_types = 11;
  // Object-facet templates only: "markdown" | "text" | "json" | "file".
  string output_kind = 12;
}

Key facts:

• The real category values are actions, core, ecommerce, embedding, financials, vision. There is no extraction, analytics or structuring category — passing one to --category returns nothing.
• --category and --search filter server-side; --label does not. ListTemplatesRequest.labels is accepted on the wire and then ignored — dodil data template list --label anything=at-all returns the entire catalog unchanged. Filter labels client-side with jq until that lands.
• The facet filter on ListTemplates is the real per-pillar narrowing, and it is what dodil data table templates uses (PIPELINE_FACET_TABLE → 17 templates). PipelineFacet replaced the old Pillar enum, which said the same four things.
• accepted_extensions / accepted_content_types are the modality gate. A table pipeline’s template must declare @accepts_extension — CreateTablePipeline refuses one that doesn’t, because the derived rule globs come from it and a table pipeline with no accepted extensions can never fire.
• contract is a typed schema — ScriptContract, detailed in the next section. It tells you what inputs the template accepts and what shape its outputs take.
• For full Scriptum semantics see the Scriptum docs (separate product) — K3 surfaces only the catalog.

ScriptContract

The typed I/O contract carried on Template.contract (every pillar’s Template surfaces it). It’s K3’s anti-corruption mirror of Scriptum’s contract — re-homed into dodil.data.common.v1 so pillar protos expose a template’s schema without importing scriptum.proto. Two consumers: form generators (UI / CLI) walk inputs to render per-field controls; the server validates caller-supplied inputs against inputs at creation time (pipeline options, CreateVectorPipeline.template_inputs, table-pipeline inputs).

message ScriptContract {
  repeated TypedField inputs  = 1;              // what callers may set (validated server-side)
  repeated TypedField outputs = 2;              // shape of what the script emits
  repeated TypedField stream  = 3;              // streamed / intermediate fields
  repeated TypedField env     = 4;              // environment inputs (keys, model handles)
  repeated EnumDef    enums   = 5;              // named enums referenced by TypeRef.enum_name
  repeated string accepted_content_types = 6;   // input MIME globs (`image/*`); empty = no restriction
  repeated string accepted_extensions    = 7;   // lowercase, no leading dot; empty = unrestricted
}
 
// One field in inputs / outputs / stream / env.
message TypedField {
  string  name               = 1;
  TypeRef type               = 2;
  string  default_value_json = 3;   // JSON literal: `"hi"`, `42`, `true`, `[1,2]`; empty = no default
  bool    required           = 4;   // required if `true` OR no default_value_json
  string  description        = 5;
}
 
// A field's type — oneof over four kinds, plus constraints keyed to the kind.
message TypeRef {
  oneof kind {
    TypeRefPrimitive primitive = 1;   // text|number|integer|boolean|object|binary|date|timestamp|any
    string           enum_name = 2;   // names an entry in ScriptContract.enums
    ListType         list      = 3;   // list<T> — recursive
    ObjectShape      object    = 4;   // anonymous record of named TypedFields — recursive
  }
  NumericConstraints number_constraints = 10;   // numeric kinds
  StringConstraints  string_constraints = 11;   // text kind
  ListConstraints    list_constraints   = 12;   // list kind
}
message ListType    { TypeRef item = 1; }
message ObjectShape { repeated TypedField fields = 1; }
 
// Constraints — only the message matching the field's kind is set; bounds optional (absent = unconstrained).
message NumericConstraints { optional double minimum = 1; optional double maximum = 2; bool exclusive_min = 3; bool exclusive_max = 4; optional double multiple_of = 5; }
message StringConstraints  { optional uint32 min_length = 1; optional uint32 max_length = 2; optional string pattern = 3; }  // pattern = ECMA-262 regex
message ListConstraints    { optional uint32 min_items = 1; optional uint32 max_items = 2; bool unique_items = 3; }
 
// Enums — a field with kind=enum_name resolves to one of these; `base` sets wire serialization.
enum EnumBase   { ENUM_BASE_UNSPECIFIED = 0; ENUM_BASE_TEXT = 1; ENUM_BASE_INTEGER = 2; ENUM_BASE_NUMBER = 3; }  // string | int64 | double
message EnumDef     { string name = 1; EnumBase base = 2; repeated EnumVariant variants = 3; }
message EnumVariant { oneof value { string value_text = 1; int64 value_integer = 2; double value_number = 3; } string description = 4; }  // value matches EnumDef.base

What the proto doesn’t say outright:

• list and object nest — a TypeRef recurses to describe list<object{…}> and deeper.
• Defaults are JSON-encoded strings, not native values — decode default_value_json as a JSON literal before use.
• accepted_* is the modality gate — align an ingest rule’s MIME / extension filters with these, or the rule won’t match what the template can actually ingest.

Reading a contract from the CLI:

dodil data template get text_embedding_index -o json | jq '.contract.inputs'

Rule

The trigger: which objects in which source should fire which pipeline. Pure binding — rules don’t own destinations; they reference a pipeline that does.

{
  "ruleId": "rule_a1b2…",
  "bucket": "kb-prod",
  "sourceId": "src_a1b2…",
  "name": "pdf-contracts",
  "description": "All PDFs under contracts/",
  "includePatterns": ["contracts/**/*.pdf"],
  "excludePatterns": ["contracts/**/draft-*"],
  "includeMimeTypes": ["application/pdf"],
  "excludeMimeTypes": [],
  "minSizeBytes": "0",
  "maxSizeBytes": "0",
  "enabled": true,
  "priority": 100,
  "createdAt": "1716840000000",
  "updatedAt": "1716843600000",
  "pipelineId": "pipe_a1b2…",
  "binding": {
    "pipelineName": "embed-contracts",
    "pipelineKind": "vector",
    "destinationId": "ent_a1b2…",
    "destinationKind": "vector",
    "destinationName": "docs-index"
  }
}

message IngestRule {
  string rule_id = 1;
  string bucket = 2;
  string source_id = 3;
  string name = 4;
  string description = 5;
  // Path matching (glob patterns)
  repeated string include_patterns = 6;
  repeated string exclude_patterns = 7;
  // Content filters
  repeated string include_mime_types = 8;
  repeated string exclude_mime_types = 9;
  int64 min_size_bytes = 10;
  int64 max_size_bytes = 11;                  // 0 = no limit
  bool enabled = 12;
  int32 priority = 13;                        // higher = matched first
  int64 created_at = 14;
  int64 updated_at = 15;
  // Authoritative pipeline pointer
  string pipeline_id = 16;
  // Server-derived display info — DO NOT use for routing
  PipelineBinding binding = 17;
  // Stable recipe.id (e.g. "document-rag") when created via a recipe
  // install. Empty for hand-installed rules.
  optional string recipe_name = 18;
}
 
// Resolved view of pipeline_id → pipeline → destination. Empty when
// `pipeline_id` is unresolvable (rule outlived a deleted pipeline).
message PipelineBinding {
  string pipeline_name = 1;
  string pipeline_kind = 2;                   // "vector" | "warehouse" | "free"
  string destination_id = 3;                  // store_entity.id, empty for free
  string destination_kind = 4;                // mirrors store_entity.kind
  string destination_name = 5;                // collection or table name
}

Key facts:

• Matching is conjunctive across filter axes: an object must satisfy include_patterns AND not match exclude_patterns AND satisfy MIME include/exclude AND be within [min_size_bytes, max_size_bytes]. Each axis is OR’d within itself (any include pattern matching is enough). Excludes are checked before includes — if an exclude matches, K3 rejects the object without considering includes.
• priority is int32, higher values evaluated first. Use it to express “specific rules before general ones.”
• Multiple rules can match the same object — each match spawns its own IngestJob. Use enabled = false to pause a rule without deleting it.
• binding is display-only — server-resolved from pipeline_id. Don’t make routing decisions from it; if you delete a pipeline, dependent rules still exist with pipeline_id pointing at nothing and binding empty.

Glob pattern syntax

include_patterns and exclude_patterns use K3’s in-house glob matcher with these tokens:

Behavior:

• Case-insensitive (ASCII). **/*.pdf matches Report.PDF, IMG_1234.JPG matches **/*.jpg, README.MD matches readme.md. The matcher lowercases both pattern and key before comparing — same convention as other production glob implementations (globset, fast-glob, minimatch). Bucket keys + extensions are always ASCII so K3 uses cheap ASCII case-folding, not full Unicode.
• ** matches zero or more path segments — **/test.txt matches both test.txt (at the root) AND a/b/c/test.txt (nested). Trailing / after ** is optional.

Not supported (yet — work around with multiple patterns):

• POSIX character classes — [a-z], [!a] don’t expand
• Brace expansion — {pdf,docx} is treated literally
• Escaping — there’s no way to match a literal *, ?, or ** in the object key itself

For either-or expansion, use multiple include patterns:

dodil data ingest add multi-ext -b kb-prod \
  --source "$SOURCE_ID" --collection "$PIPELINE_ID" \
  --include "**/*.pdf" --include "**/*.docx" --include "**/*.txt"

MIME pattern syntax

include_mime_types and exclude_mime_types use a simpler matcher than globs:

Not supported: type wildcards (*/json doesn’t work), brace expansion (text/{plain,html}), or any other shape. List multiple exact / subtype-wildcard entries instead.

Evaluation order (worth knowing for debugging)

K3 evaluates a rule against an object in this exact order, short-circuiting on the first reject:

1. exclude_patterns — if any match, reject (early-exit; cheap)
2. include_patterns — if non-empty and none match, reject
3. exclude_mime_types — if any match, reject
4. include_mime_types — if non-empty and none match, reject
5. max_size_bytes (when > 0) — if object exceeds, reject
6. min_size_bytes (when > 0) — if object is smaller, reject

If all axes pass, the object matches.

IngestJob

One execution of one pipeline against one object. The unit of observability for everything that happens after a rule matches.

{
  "jobId": "job_a1b2…",
  "object": { "bucket": "kb-prod", "key": "contracts/acme-2026.pdf" },
  "status": "INGEST_STATUS_COMPLETED",
  "pipelineId": "pipe_a1b2…",
  "pipelineName": "embed-contracts",
  "pipelineKind": "vector",
  "ruleId": "rule_a1b2…",
  "chunksCreated": 47,
  "embeddingsCreated": 47,
  "rowsWritten": 0,
  "createdAt": "1716843600000",
  "updatedAt": "1716843680000",
  "threadId": "thread_a1b2…",
  "vectorStatus": "success",
  "outputSize": 18429,
  "batchesReceived": 5,
  "embeddingsWritten": 47
}

enum IngestStatus {
  INGEST_STATUS_UNSPECIFIED = 0;
  INGEST_STATUS_PENDING     = 1;
  INGEST_STATUS_PROCESSING  = 2;
  INGEST_STATUS_COMPLETED   = 3;
  INGEST_STATUS_FAILED      = 4;
  INGEST_STATUS_PARTIAL     = 5;
  // Previous attempt failed transiently and the job will be retried
  // by NATS JetStream redelivery. The job's `error` field carries
  // "attempt N/M: <last error>" so the UI can show progress.
  INGEST_STATUS_RETRYING    = 6;
}
 
message IngestJob {
  string job_id = 1;
  dodil.data.common.v1.ObjectRef object = 2;
  IngestStatus status = 3;
  // Pipeline linkage — mirrors IngestRule.binding for display.
  string pipeline_id = 4;
  string pipeline_name = 5;
  string pipeline_kind = 6;                   // "vector" | "warehouse" | "object" | "free"
  optional string rule_id = 7;                // absent for manual triggers
  optional string error = 8;
  // Run results — populated per pipeline kind
  int32 chunks_created = 9;
  int32 embeddings_created = 10;
  int32 rows_written = 11;                    // warehouse only
  int64 created_at = 12;
  int64 updated_at = 13;
  optional string thread_id = 14;             // Scriptum thread ID
  optional string vector_status = 15;         // null | "success" | "failed" | "skipped"
  optional int32 output_size = 16;            // bytes of Scriptum output
  optional string error_details = 17;
  int32 batches_received = 18;                // streamed yields from WatchThread
  int32 embeddings_written = 19;              // count actually persisted
  // Object-pipeline counter — how many S3 objects the ObjectWriter
  // wrote at the destination. v1 is always 0 or 1 (single blob per
  // source); reserved as a count for future multi-output pipelines.
  int32 objects_written = 20;
}

Key facts:

• status transitions are not always linear — transient failures retry automatically through RETRYING → PROCESSING → COMPLETED, or land in FAILED after the final attempt.
• rule_id is absent for jobs spawned via TriggerIngest (single-object manual trigger) without a rule_id override.
• pipeline_kind controls which counters populate: vector → chunks_created / embeddings_created / embeddings_written; warehouse → rows_written; object → objects_written (0 or 1 in v1); free → none.
• thread_id is the Scriptum thread ID — useful for cross-referencing with Scriptum logs / observability.
• vector_status is a fine-grained sub-status for vector pipelines specifically: did the embedding-persist phase succeed (success), fail (failed), or skip because the script produced no embeddings (skipped)?
• batches_received vs embeddings_written lets you spot partial writes: Scriptum yielded N batches but only M embeddings landed in the collection.

When ingest runs

Three triggers can cause a pipeline to run against an object — all produce the same IngestJob shape, only rule_id differs:

Reliability:

• Transient failures retry automatically via NATS JetStream redelivery (the INGEST_STATUS_RETRYING state). Two budgets share one cap: ordinary transient failures stop at the logical failure-retry budget (K3_INGEST_FAILURE_RETRY_BUDGET, default 5), while a job parked on a still-provisioning engine may use the full hard redelivery cap (K3_INGEST_MAX_DELIVER, default 30) so it outlasts a slow spin-up.
• After the effective maximum the job becomes INGEST_STATUS_FAILED with the last error in error / error_details. Only retryable errors are redelivered at all — a permanent failure goes straight to FAILED on the first attempt.
• Three replay scopes, coarsest to finest:
  • TriggerIngestion on a source (retry_failed = true) — bulk, every failed/partial object on that source.
  • RetryIngestJob (POST /:bucket/ingest/jobs/{job_id}/retry) — re-runs one existing job in place: same job_id, no new row, per-attempt state reset. Only FAILED and PARTIAL jobs qualify; anything else returns FAILED_PRECONDITION.
  • TriggerIngest on a single object — spawns a new job, and is the only path that accepts a per-event options overlay.

See Recipes → Replay & retry for the full triage loop.

See also

• Quickstart — wire all six entities together in 5 minutes
• API Reference — gRPC + HTTP for every Source / Pipeline / Ingest RPC
• CLI Guide — dodil data source · pipeline · template · ingest · credential
• Vector — destination kind for vector pipelines
• Tables — destination kind for warehouse pipelines
• Conventions — auth headers, error envelope, wire encoding rules