PDF → Vector collection

Goal: every PDF uploaded to a bucket is automatically chunked, embedded, and indexed into a vector collection — searchable end-to-end without any glue code.

Template: text_embedding_index — K3’s production template for text/PDF/docx/HTML/audio/video embedding. Resolves a source object, chunks it, embeds the chunks, writes them to a vector collection.

Shape:

   PDF upload  ──►  bucket  ──►  rule matches  ──►  text_embedding_index
                                                          │
                                                          ▼
                                                  Vector collection
                                                  (chunks + embeddings)
                                                          │
                                                          ▼
                                                   Vector search

Prerequisites

• dodil CLI installed and dodil auth login done — CLI Basics
• A bucket (we’ll use kb-prod):

  dodil data bucket create kb-prod -d "RAG corpus"

Nothing else. You do not provision a vector engine or a collection beforehand — step 2 creates both the collection and its index pipeline, and the physical collection materializes lazily in the plane on first ingest.

The one-command version. dodil data recipe install document-rag -b kb-prod does steps 2 and 4 together, including the rule. This page is the hand-wired version so each piece is visible. See dodil data recipe.

1. Inspect the template

Look at the template’s typed contract before you spawn anything — it tells you what inputs it accepts:

dodil data template get text_embedding_index -o json | jq '{id, category, labels, modalities}'
dodil data template get text_embedding_index -o json | jq '.contract.inputs'

text_embedding_index reports category: "embedding" and modalities: ["text","pdf","docx","html","audio","video"].

You cannot override the schema-shaping facts. embed_model, dimensions, distance_metric, sparse_mode and embedding_type are resolved from the contract at create time, never from the caller — that is exactly why template_id is required. For text_embedding_index they resolve to jina-embeddings-v4 / 768 dimensions / cosine / SPARSE_MODE_NONE / float. What you can pass are the template’s declared contract inputs, via --set key=value.

2. Create the collection and its pipeline

A vector collection is a pipeline with a vector destination. One call writes both rows:

dodil data vector collection add docs-index \
  --bucket kb-prod \
  --template text_embedding_index \
  -o json

That is POST /kb-prod/pipelines/vector → CreateVectorPipeline. Capture the pipeline id — everything downstream addresses the pipeline, not the collection:

export PIPELINE_ID="<pipelineId from the response>"

Verify what you got:

dodil data vector collection get "$PIPELINE_ID" --bucket kb-prod -o json \
  | jq '{name, scriptumTemplate,
         facet: .destination.facet,
         collection: .destination.name,
         status: .destination.status,
         dims: .destination.vector.dimensions,
         model: .destination.vector.embedModel,
         physical: .destination.vector.physicalName}'

facet should read PIPELINE_FACET_VECTOR. Two things will look odd and are both correct:

• name is text_embedding_index, not docs-index. The name you passed lands on destination.name; Pipeline.name is set to the template id because the row it names is the index pipeline.
• The old flat fields are gone. storeEntityId, storeEntityKind and storeEntityName were reserved on Pipeline and folded into destination. Anything still reading .storeEntityKind gets nothing.

Per-pipeline options (chunking and friends) are tuned afterwards with pipeline update --options-json — note that flag exists only on update, never on create:

dodil data pipeline update "$PIPELINE_ID" --bucket kb-prod \
  --options-json '{"chunk_size":"1000","chunk_overlap":"150"}'

--options-json is a full replace of the options map, not a merge.

3. Look up the bucket’s internal source

Every bucket gets an auto-created internal source on CreateBucket. We need its source_id to attach a rule:

export SOURCE_ID=$(dodil data source list -b kb-prod -o json \
  | jq -r '.sources[] | select(.name == "internal") | .sourceId')
echo "internal source = $SOURCE_ID"

Its status reads SOURCE_STATUS_PENDING on a fresh bucket — normal, and no obstacle to direct uploads firing rules. See CLI Guide → source.

4. Create the rule — without this, nothing runs

CreateVectorPipeline does not create an ingest rule. The proto is explicit that callers own rule scope. A collection with no rule bound to it will sit there indefinitely while every PDF you upload sails past. This step is not optional.

The rule says “any PDF anywhere in this bucket fires the pipeline”:

dodil data ingest add pdf-rule \
  --bucket kb-prod \
  --source "$SOURCE_ID" \
  --collection "$PIPELINE_ID" \
  --include "**/*.pdf"

The CLI’s --collection flag maps to the API’s pipeline_id — historical naming.

Tighter scoping (e.g. only contracts/ PDFs) — replace the include pattern:

dodil data ingest add contracts-pdf-rule \
  --bucket kb-prod \
  --source "$SOURCE_ID" \
  --collection "$PIPELINE_ID" \
  --include "contracts/**/*.pdf"

Exclude / MIME / size filters aren’t on ingest add flags — set them with ingest update after creation:

# Capture the rule ID first (CLI emits the IngestRule on add)
RULE_ID="<rule_id from the add response>"
 
# Exclude drafts, require ≥ 1 KB
dodil data ingest update "$RULE_ID" -b kb-prod \
  --exclude "**/draft-*" \
  --min-size 1024

5. Upload a test PDF

# Any PDF — here, the "Attention Is All You Need" paper
curl -sSL https://arxiv.org/pdf/1706.03762.pdf -o attention.pdf
 
dodil data object create ./attention.pdf -b kb-prod -k papers/attention.pdf

The PUT completes immediately; the pipeline runs asynchronously.

6. Watch the ingest job

# All jobs for our rule
dodil data ingest jobs --bucket kb-prod --rule "$RULE_ID" -o json
 
# Or filter by pipeline — useful when multiple rules feed the same pipeline
dodil data ingest jobs --bucket kb-prod --pipeline "$PIPELINE_ID" -o json \
  | jq '.jobs[] | {jobId, status, chunksCreated, embeddingsCreated, embeddingsWritten}'

To block until it finishes rather than polling by hand:

dodil data ingest watch <job-id> -b kb-prod --wait 10m

watch prints a line on every observed change and a final summary with the counters and the Scriptum thread id — the handle you need if it fails.

Status progression for a healthy run:

PENDING ─► PROCESSING ─► COMPLETED
                │
                ▼  transient failure
            RETRYING ─► PROCESSING ─► COMPLETED   (K3 retries automatically)
                │
                ▼  permanent failure (max attempts reached)
              FAILED

When COMPLETED, you’ll see:

{
  "jobId": "job_a1b2…",
  "status": "INGEST_STATUS_COMPLETED",
  "chunksCreated": 47,
  "embeddingsCreated": 47,
  "embeddingsWritten": 47,
  "vectorStatus": "success",
  "batchesReceived": 5
}

chunksCreated == embeddingsCreated == embeddingsWritten is the happy path. If embeddingsWritten < embeddingsCreated, some embeddings didn’t land in the collection — inspect the Vector primitive for collection health.

Cross-check the object’s pipeline status without leaving Storage:

dodil data object show papers/attention.pdf -b kb-prod -o json \
  | jq '.pipelineStatuses[]'

One entry per rule that ran — aggregates align with the job’s counters.

7. Search the result

The embeddings are live. The native way to ask is dodil data search — K3 embeds the query server-side with the paired text_embedding_search template, fans out across the bucket’s collections, and returns ranked chunks:

dodil data search "what is multi-head attention" -b kb-prod --top-k 10 --rerank
# 10 result(s) in 1840ms (semantic)
# SCORE    OBJECT                                   CHUNK
# 0.8213   papers/attention.pdf                     #12
#          The Transformer uses multi-head attention in three different ways …

PHYS=$(dodil data vector collection get "$PIPELINE_ID" -b kb-prod -o json \
  | jq -r '.destination.vector.physicalName')
 
dodil data vsearch -b kb-prod -t "$PHYS" --column embedding \
  --text "what is multi-head attention" --top-k 10 --metric cosine -o json
# {"matches":[{"id":"…","score":0.8213}, …],"count":10}

curl -sS -X POST "https://api.data.dodil.io/kb-prod/search/vector" \
  -H "Authorization: Bearer $DODIL_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"text": "what is multi-head attention", "topK": 10, "rerank": true}'

Two things this platform does not do, despite what you may expect. There is no dense + BM25 hybrid retrieval — it is absent from the tabled plane rather than partially built, and the vector-store tool errors out if enable_bm25 or query_texts is set rather than silently degrading to dense-only. And pre-embedded queries to POST /:bucket/search/vector return UNIMPLEMENTED (that fast path retired with Milvus); send that endpoint text, and use vsearch when you have a vector in hand.

Common gotchas

Variations

Same recipe shape, different template:

All four follow this exact recipe — change step 2’s --template and step 1’s expectations of the contract. Each has its own resolved embed_model and dimensions; check the collections reference before assuming they interoperate. Collections that share a dimension but use different models never co-mingle on the search side.

The image-rag and code-rag recipes install the visual_embedding_index and code_embedding_index variants with matching globs in one command.

See also

• Replay & Retry — recover from FAILED jobs, replay after pipeline changes
• Documents → Warehouse — same flow but writing structured rows instead of embeddings
• Quickstart — the abbreviated version of this recipe
• dodil data recipe — recipe install document-rag does this whole page in one command
• Templates → The catalog — the other 33 production templates
• Vector — the destination primitive (vector collections)