Pipeline Collection

Goal: stand up a searchable vector collection where every uploaded document gets chunked + embedded + indexed automatically — no manual ingestion code.

Template used: text_embedding_index — the canonical text/PDF/docx/HTML/audio/video RAG-ingest template. The same pattern works for any *_embedding_index template in the vector catalog.

Shape:

   document upload  ──►  bucket  ──►  YOUR ingest rule matches
                                                  │
                                                  ▼
                                       index pipeline runs in Scriptum
                                       (chunks + embeds via Ignite)
                                                  │
                                                  ▼
                                          vector collection
                                  (lazy-materialized on first ingest)
                                                  │
                                                  ▼
                                    POST /:bucket/search/vector
                                  (the bound *_embedding_search pipeline)

Prerequisites

• dodil CLI + dodil auth login done — CLI Basics
• A bucket — kb-prod:

  dodil data bucket create kb-prod -d "RAG corpus"

Vector is enabled by default — no engine to configure; collections provision on demand.

Check this flow end-to-end before you rely on it. K3’s half is fully cut over to the tabled plane — both the ingest and search dispatch envs carry only K3_DB_ID + the bucket SA pair + the embed shape, and a test asserts the retired MILVUS_* / VBASE_* keys are absent (crates/k3-ingest/src/dispatch_env.rs:22-37, crates/k3-scriptum/src/env.rs:211-213). What is not confirmable from dodil-k3 or dodil-tables is whether Scriptum’s vector_store_* native tools have been repointed at tabled; dodil-k3/docs/vector_search_readpath_followup.md tracks that as the outstanding item. Until it lands, ingest and search fail softly — as a job error or a per-collection failReason — not with a clear signal that the tool is pointed at a decommissioned plane.

1. Browse + pick a template

dodil data vector templates -o json | jq '.templates[] | {id, modalities, acceptedExtensions}'

For text + PDF + docx ingest, pick text_embedding_index. Inspect its contract to see what (if any) template_inputs it needs:

dodil data template get text_embedding_index -o json | jq '.contract.inputs'

text_embedding_index has no required runtime inputs — everything has a contract default. Other templates may require inputs:

2. Create the collection

For text_embedding_index (no required inputs), the CLI works directly:

export PIPELINE_ID=$(dodil data vector collection add docs -b kb-prod \
  --description "PDF / docx / HTML embeddings" \
  --template text_embedding_index \
  -o json | jq -r '.pipelineId')

For object_embedding_index — whose labels input is an array, and --set sends only strings — use the API:

curl -sS -X POST "https://api.data.dodil.io/kb-prod/pipelines/vector" \
  -H "Authorization: Bearer $DODIL_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "bucket": "kb-prod",
    "name": "products",
    "description": "Product image objects",
    "templateId": "object_embedding_index",
    "templateInputs": {
      "labels": ["bottle", "bag", "shoe", "watch", "jewelry"]
    }
  }'

What K3 created

CreateVectorPipeline writes three rows and talks to no data plane:

Inspect what you got — get takes the pipeline id:

dodil data vector collection get "$PIPELINE_ID" -b kb-prod \
  | jq '{pipelineId, scriptumTemplate, name: .destination.name, status: .destination.status, vector: .destination.vector}'

3. Create the ingest rule

CreateVectorPipeline does not derive a rule. Callers own rule scope — that is stated in the proto (k3_pipeline.proto:523-524) and there is no rule-creation call anywhere under bin/api/src/services/pipeline/. Skip this step and your uploads will sit in the bucket, unembedded.

dodil data ingest add docs-corpus -b kb-prod \
  -c "$PIPELINE_ID" \
  -i '**/*.pdf' -i '**/*.txt' -i '**/*.md'
 
RULE_ID=$(dodil data ingest list -b kb-prod -p "$PIPELINE_ID" -o json | jq -r '.rules[0].ruleId')

-c / --collection is a historical flag name — it sets CreateRuleRequest.pipeline_id. Mirror the globs on the template’s acceptedExtensions from step 1; a rule whose patterns don’t match your keys never fires.

4. Upload documents

# A PDF
curl -sSL https://arxiv.org/pdf/1706.03762.pdf -o attention.pdf
dodil data object create ./attention.pdf -b kb-prod -k papers/attention.pdf
 
# Plain text
cat > intro.txt <<'EOF'
The Transformer architecture introduced multi-head attention as a way for
the model to jointly attend to information from different representation
subspaces at different positions.
EOF
dodil data object create ./intro.txt -b kb-prod -k papers/intro.txt

Each upload matches your rule → spawns an ingest job → runs text_embedding_index → writes embeddings to the docs collection.

5. Watch the ingest jobs

dodil data ingest jobs -b kb-prod -p "$PIPELINE_ID" -o json \
  | jq '.jobs[] | {object: .object.key, status, chunksCreated, embeddingsWritten}'

Status path: PENDING → PROCESSING → COMPLETED; the full IngestStatus set also has FAILED, PARTIAL and RETRYING (the last carries attempt N/M: <last error> in error). dodil data ingest watch <job-id> -b kb-prod blocks until a job is terminal. When done:

[
  {"object": "papers/attention.pdf", "status": "INGEST_STATUS_COMPLETED", "chunksCreated": 47, "embeddingsWritten": 47},
  {"object": "papers/intro.txt",      "status": "INGEST_STATUS_COMPLETED", "chunksCreated": 1,  "embeddingsWritten": 1}
]

chunksCreated == embeddingsWritten is the happy path. If embeddingsWritten is lower, see Pipelines → Replay & Retry.

6. Search

Semantic text search over HTTP — the bound text_embedding_search pipeline embeds the query server-side and runs retrieval (see Recipes → Hybrid Search). This route is the search path for pipeline collections; the wire adapters are not, because the physical table is named k3_<uuid> and you never named its vector column.

import os, requests
 
BASE = "https://api.data.dodil.io"
HEADERS = {"Authorization": f"Bearer {os.environ['DODIL_TOKEN']}"}
 
resp = requests.post(
    f"{BASE}/kb-prod/search/vector",
    headers=HEADERS,
    json={
        "text": "what is multi-head attention",
        "collectionNames": ["docs"],
        "topK": 5,
    },
).json()
 
for w in resp.get("warnings", []):
    print("WARN:", w)
 
for r in resp["results"]:
    print(round(r["score"], 4), r["object"]["key"], (r.get("content") or "")[:200])

curl -sS -X POST "https://api.data.dodil.io/kb-prod/search/vector" \
  -H "Authorization: Bearer $DODIL_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "text": "what is multi-head attention",
    "collectionNames": ["docs"],
    "topK": 5
  }' | jq '{results: [.results[] | {score, object: .object.key}], warnings, collectionStatuses}'

Sending {"vector": [...]} here returns UNIMPLEMENTED — the pre-embedded fast path went with Milvus and the tabled KNN read-path is still pending. For KNN by vector, own the table and the column: see External Collection.

7. Inspect the resolved schema

Every schema fact came from the template’s ScriptContract at create time — inspect it before doing multi-collection search, because the compatibility group key is (embedModel, dimensions, embeddingType):

dodil data vector collection get "$PIPELINE_ID" -b kb-prod \
  | jq '{
      status: .destination.status,
      v: (.destination.vector | {dimensions, embeddingType, distanceMetric, sparseMode, embedModel, modality, physicalName})
    }'

Shape of the answer (the exact values come from the template you picked, not from K3):

{
  "status": "DESTINATION_STATUS_ACTIVE",
  "v": {
    "dimensions": 1024,
    "embeddingType": "EMBEDDING_TYPE_FLOAT",
    "distanceMetric": "DISTANCE_METRIC_COSINE",
    "sparseMode": "SPARSE_MODE_BM25",
    "embedModel": "jina-embeddings-v4",
    "modality": "text",
    "physicalName": "k3_a1b2c3d4…"
  }
}

K3’s own fallbacks, when a contract declares nothing, are cosine / float / sparse_mode = none / modality = text — embed_model and dimensions have no fallback and a template that can’t resolve them is rejected with FAILED_PRECONDITION at create time.

Common gotchas

Variations — other vector templates

Same recipe, different template:

All four are created the same way — three rows, no rule — and all four need a rule of your own.

Cleanup

# 1. Disable the rule first (stops new ingests)
dodil data ingest update "$RULE_ID" -b kb-prod --enabled=false
 
# 2. Delete the rule, then the pipeline
dodil data ingest delete "$RULE_ID" -b kb-prod
dodil data vector collection delete "$PIPELINE_ID" -b kb-prod
 
# 3. (Optional) Delete the objects and the bucket
dodil data object remove papers/attention.pdf -b kb-prod
dodil data object remove papers/intro.txt -b kb-prod
dodil data bucket delete kb-prod

dodil data vector collection delete and dodil data pipeline delete are the same RPC (DeletePipeline) — you need one, not both. For a whole recipe-installed chain, POST /:bucket/pipelines/_batch-delete unwinds rules → pipelines → destinations in dependency order, idempotently.

Deleting a pipeline removes wiring, not data. Dropping the physical vector collection on the plane is something tenants do themselves through the tables-gateway (k3_pipeline.proto:149-152).

See also

• External Collection — opposite shape: you own the embedding pipeline, push vectors directly
• Multi-collection Search — search docs + code + assets in one query
• Hybrid + Rerank — improve precision on the pipeline-mode collection from this recipe
• Pipelines → Replay & Retry — recover from failed ingests
• Templates — API Reference — vector catalog spec