Quickstart

Five minutes from here you’ll have a bucket where every uploaded PDF auto-indexes into a vector collection and is searchable.

We’ll wire up the full chain: source → pipeline → rule → upload → ingest job → search.

This is the canonical RAG-ingest flow. The same shape works for warehouse pipelines (PDFs → rows) and free pipelines (PDFs → side-effects) — see Recipes when you’re done.

Prerequisites

• dodil CLI installed and dodil auth login done — see CLI Basics.
• A bucket. We’ll use kb-quickstart. If you don’t have one:

  dodil data bucket create kb-quickstart -d "Pipelines quickstart"

That’s it — you do not need to provision a vector engine or a collection up front. Step 2 creates the collection and its index pipeline in one command, and the physical collection materializes lazily in the data plane on first ingest.

In a hurry? dodil data recipe install document-rag -b kb-quickstart does steps 2 and 4 in one command — it creates the vector pipeline and binds the ingest rule. The rest of this page is the same chain wired by hand, so you can see each piece. See dodil data recipe.

1. Discover available Scriptum templates

K3 ships a production template catalog — 34 templates across six categories (actions, core, ecommerce, embedding, financials, vision). For “PDF → vector embeddings”, the right template is text_embedding_index (its modalities are text, pdf, docx, html, audio, video). List the full catalog with:

dodil data template list -o json

Inspect the typed contract for our chosen template:

dodil data template get text_embedding_index -o json

The response includes a typed ScriptContract — input fields, output schema, required tools, and labels. K3 uses this server-side to validate the template_inputs you pass at create time, and to resolve the collection’s schema-shaping facts (embed_model, dimensions, distance_metric, sparse_mode) — you cannot override those.

2. Create the vector collection

A vector collection is a pipeline with a vector destination. One command writes both rows — the store_entities collection row and the index pipelines row bound to it:

dodil data vector collection add docs-index \
  --bucket kb-quickstart \
  --template text_embedding_index \
  -o json

This is POST /kb-quickstart/pipelines/vector → CreateVectorPipeline. Capture the pipeline id — it is what everything downstream addresses:

export PIPELINE_ID="<pipelineId from the create response>"

The name you passed lands on destination.name, not on Pipeline.name. Pipeline.name is set to the template id, because the row it names is the index pipeline. So dodil data vector collection list prints text_embedding_index under both NAME and TEMPLATE, and the docs-index you typed shows up as destination.name. That is observed behaviour, not a bug in your setup.

Pass template inputs with --set key=value (repeatable) if the template’s contract requires any.

3. Find the internal source

You don’t create it — it already exists. Every bucket gets an internal S3 source the moment CreateBucket runs. You only need to look up its source_id to wire a rule against it:

export SOURCE_ID=$(dodil data source list -b kb-quickstart -o json \
  | jq -r '.sources[] | select(.name == "internal") | .sourceId')

See CLI Guide → source for the full source list / source get surface. Its live status is SOURCE_STATUS_PENDING on a fresh bucket — that is normal and does not stop direct uploads from firing rules.

4. Create the rule — this step is mandatory

Creating a pipeline does not create an ingest rule. k3_pipeline.proto says it outright for every facet creator: “callers own rule scope”. Skip this step and your pipeline will sit there while every upload sails past it. The rule is the trigger: “for every object in the internal source whose key matches *.pdf, run this pipeline.”

dodil data ingest add pdf-rule \
  --bucket kb-quickstart \
  --source "$SOURCE_ID" \
  --collection "$PIPELINE_ID" \
  --include "**/*.pdf"

Capture the rule id from the response — you’ll filter jobs by it in step 6:

export RULE_ID="<ruleId from the add response>"

The CLI’s --collection flag maps to the API’s pipeline_id field — historical naming. It takes a pipeline id, not a collection name. See dodil data ingest for the full surface.

5. Upload a PDF

# Grab any PDF, or use one of yours
curl -sSL https://arxiv.org/pdf/1706.03762.pdf -o attention.pdf
 
dodil data object create ./attention.pdf -b kb-quickstart -k papers/attention.pdf

That’s it. K3 matches your rule against the upload, spawns an IngestJob, runs the Scriptum template, and lands embeddings in your vector collection.

6. Watch the ingest job

# All jobs in the bucket — the most recent is yours
dodil data ingest jobs --bucket kb-quickstart -o json
 
# Or filter by pipeline / by rule
dodil data ingest jobs --bucket kb-quickstart -p "$PIPELINE_ID" -o json
dodil data ingest jobs --bucket kb-quickstart -r "$RULE_ID" -o json

Better: block until the job reaches a terminal state, printing every observed change:

dodil data ingest watch <job-id> -b kb-quickstart
# [3s]  PENDING (batches=0, embeddings=0)
# [9s]  PROCESSING (batches=2, embeddings=18)
# [21s] COMPLETED (batches=5, embeddings=47)
#
# Job job_a1b2… COMPLETED
#   object:   papers/attention.pdf
#   pipeline: text_embedding_index (vector)
#   chunks=47 embeddings=47 rows=0 vector=success
#   thread:   thread_a1b2… ('dodil scriptum thread steps thread_a1b2…' for per-step detail)

watch defaults to --wait 5m --interval 3s; raise --wait for large media. Status progression for a successful run:

PENDING ─► PROCESSING ─► COMPLETED
                │
                ▼  on transient error
            RETRYING ─► PROCESSING ─► COMPLETED   (K3 retries automatically)
                │
                ▼  on permanent error
              FAILED

Once COMPLETED, the job’s chunks_created and embeddings_created counters tell you how the object decomposed. See Core Concepts → IngestJob for the full status enum.

You can also inspect the object directly — pipeline_statuses on ObjectInfo reflects per-rule indexing state:

dodil data object show papers/attention.pdf -b kb-quickstart -o json
# Look for `pipelineStatuses[]` — one entry per rule that ran

7. Search the result

The embeddings are now live. The native way to ask the question is dodil data search — K3 embeds your query server-side with the matching text_embedding_search template, fans out across the bucket’s collections, and returns ranked chunks:

dodil data search "what is multi-head attention" -b kb-quickstart --top-k 10
# 10 result(s) in 1840ms (semantic)
# SCORE    OBJECT                                   CHUNK
# 0.8213   papers/attention.pdf                     #12
#          The Transformer uses multi-head attention in three different ways …

Add --rerank for server-side reranking, or --table to restrict to one destination table.

If you already hold a query embedding and want raw low-latency KNN instead, dodil data vsearch goes straight to the tables data plane (TablesVector.QueryVectors). It addresses a table plus a column, not a collection name — the physical table is destination.vector.physicalName:

PHYS=$(dodil data vector collection get "$PIPELINE_ID" -b kb-quickstart -o json \
  | jq -r '.destination.vector.physicalName')
 
dodil data vsearch -b kb-quickstart -t "$PHYS" --column embedding \
  --text "what is multi-head attention" --top-k 10 -o json

--text embeds client-side via ignite models before the KNN; --model must match the model that embedded the table’s vectors. Pass --vector with comma-separated floats if you already have one.

There is no dense + BM25 hybrid. dodil data vsearch is pure KNN, and hybrid retrieval is absent from the tabled plane rather than partially built — the vector-store tool errors out if enable_bm25 or query_texts is set instead of silently degrading to dense-only. Pre-embedded queries against POST /:bucket/search/vector return UNIMPLEMENTED (that path retired with Milvus); send it text.

What you just built

Every subsequent PDF you PUT to kb-quickstart automatically follows the same chain. Add more rules for different extensions, MIMEs, or paths — they all dispatch in parallel, one ingest job per rule per object.

Cleanup

# Delete the rule first (stops new ingests but keeps existing jobs)
dodil data ingest delete "$RULE_ID" --bucket kb-quickstart
 
# Delete the collection — this is DeletePipeline, addressed by pipeline id
dodil data vector collection delete "$PIPELINE_ID" --bucket kb-quickstart
 
# Delete the object + the bucket
dodil data object remove papers/attention.pdf -b kb-quickstart
dodil data bucket delete kb-quickstart

Deleting the wiring does not delete the data. K3 removes the rows it owns — rule, pipeline, destination — but never reaches into the data plane to drop the physical collection. Tenants drop their own plane state through the tables-gateway. The same rule applies to BatchDeleteArtifacts, the multi-artifact teardown behind recipe uninstall.

Next steps

• Recipes — full worked flows (PDF → vector, S3 → warehouse, replay & retry)
• Core Concepts — every type signature, the event flow in detail
• API Reference — gRPC + HTTP for Source / Pipeline / Ingest services
• CLI Guide — every dodil data command in the pipeline domain