RAG Knowledge Base

Goal: stand up a working RAG (retrieval-augmented generation) corpus on K3. Upload PDFs / docs / text → K3 chunks + embeds + indexes them automatically → query it with one search call.

Primitives used: Storage (the bucket + S3 upload) → Pipelines (the auto-generated rule wired by Vector at collection-create time) → Vector (the collection + search).

Shape:


   ┌──────────┐
   │  Your    │
   │ documents│
   └────┬─────┘
        │  aws s3 cp / dodil data object create
        ▼
   ┌──────────────────────────────────────────────────┐
   │ Storage — kb-platform bucket                      │
   └──────────────────────────────────────────────────┘
        │  auto-rule fires (globs from template's acceptedExtensions)
        ▼
   ┌──────────────────────────────────────────────────┐
   │ Pipelines — text_embedding_index Scriptum         │
   │ runs per uploaded object → chunks + embeds        │
   └──────────────────────────────────────────────────┘
        │
        ▼
   ┌──────────────────────────────────────────────────┐
   │ Vector — `docs` collection                        │
   │ (pipeline-mode, dense embeddings)                 │
   └──────────────────────────────────────────────────┘
        │
        ▼
   ┌──────────────────────────────────────────────────┐
   │ App layer: search route → top-K chunks → LLM      │
   └──────────────────────────────────────────────────┘

Prerequisites

dodil CLI installed + dodil auth login — see CLI Basics
(Optional) aws-cli configured against K3 for S3-style uploads — see Storage → S3 Compatibility

1. Create the bucket


# Storage primitive — create the bucket
dodil data bucket create kb-platform -d "Production RAG knowledge base"

No engine setup step. Both the Tables and Vector engines are wired at bucket creation — vector capacity is provisioned on demand the moment you run dodil data vector collection add. There is no vector store create and nothing to poll.

Prefer one command? dodil data recipe install document-rag -b kb-platform provisions the collection + pipeline + rule of step 2 below in one shot — see Pipelines → CLI → recipe. It uses the same text_embedding_index template, names the collection document_rag_docs (resources are named <prefix>_<suffix>; override with --name-prefix), and wires the rule for **/*.pdf, **/*.docx, **/*.md, **/*.txt, **/*.html. Add --dry-run to print the plan first. The rest of this page does it piece by piece so you can see each moving part.

2. Pick a template + create the collection


# Browse vector-pillar templates (facet=vector → the `embedding/` catalog)
dodil data vector templates -o json | jq '.templates[] | {id, name, acceptedExtensions}'

Ten templates carry the vector facet, in *_index / *_search pairs: text_embedding_*, visual_embedding_*, code_embedding_*, face_embedding_* and object_embedding_*. The _index half is the ingestion pipeline you bind here; the _search half is what the search route runs at query time.

For PDF + docx + HTML + markdown + plain text, pick text_embedding_index (@accepts_extension txt, md, html, htm, pdf, docx). Inspect its contract:


dodil data template get text_embedding_index -o json | jq '.contract.inputs'

Only collection_name and artifact_id are required, and the pipeline supplies both. Everything else defaults — chunk_size 2000, chunk_overlap, dimensions 768, embed_model jina-embeddings-v4.

Create a pipeline-mode collection — K3 atomically creates the collection + a Scriptum pipeline + an auto-generated ingest rule:


dodil data vector collection add docs -b kb-platform \
  --description "Production RAG corpus" \
  --template text_embedding_index
# → Vector collection 'docs' (pipeline pl_…) created in bucket 'kb-platform'.

A vector collection is a pipeline with a vector facet, so everything downstream is keyed by the pipeline id. collection get and collection delete take that id positionally — not the collection name:


export PIPELINE_ID=$(dodil data vector collection list -b kb-platform -o json \
  | jq -r '.pipelines[] | select(.name == "docs") | .pipelineId')
 
dodil data vector collection get "$PIPELINE_ID" -b kb-platform

Confirm the auto-rule is enabled:


dodil data ingest list -b kb-platform -p "$PIPELINE_ID" -o json \
  | jq '.rules[] | {ruleId, name, includePatterns, enabled}'

Expect something like includePatterns: ["**/*.pdf", "**/*.txt", "**/*.docx", "**/*.html"] and enabled: true.

3. Upload documents — three ways

boto3


import boto3
 
s3 = boto3.client(
    "s3",
    endpoint_url="https://object.uk-lon-1.dodil.io",
    aws_access_key_id=SERVICE_ACCOUNT_ID,      # SigV4 needs a service account —
    aws_secret_access_key=SERVICE_ACCOUNT_SECRET,  # an API key cannot sign
    region_name="us-east-1",
)
 
s3.upload_file("./attention.pdf", "kb-platform", "papers/attention.pdf")

An API key (dk_…) cannot SigV4-sign — only the argon2 hash of its secret is stored. On the object door it rides as Authorization: Bearer dk_… instead. For SDK uploads, use a service account.

aws-cli


# One-time setup
aws configure --profile dodil-k3
# AWS Access Key ID: <your Dodil service-account ID>
# AWS Secret Access Key: <your Dodil service-account secret>
# Default region name: us-east-1
# Default output format: json
 
# Bulk-upload an existing folder
aws s3 sync ./my-docs/ s3://kb-platform/papers/ \
  --endpoint-url https://object.uk-lon-1.dodil.io \
  --profile dodil-k3

dodil data


curl -sSL https://arxiv.org/pdf/1706.03762.pdf -o attention.pdf
dodil data object create ./attention.pdf -b kb-platform -k papers/attention.pdf
 
# whole directory tree; -k becomes a key prefix
dodil data object create ./my-docs -b kb-platform -k papers/ --recursive

@aws-sdk/client-s3 works identically — same endpoint, same auth. See Storage → S3 Compatibility for setup snippets.

Every upload (CLI or S3 SDK) fires through the auto-generated rule → spawns an ingest job → runs text_embedding_index → writes chunks + embeddings to the docs collection.

4. Watch the ingest pipeline


dodil data ingest jobs -b kb-platform -p "$PIPELINE_ID" -o json \
  | jq '.jobs[] | {object: .object.key, status, chunksCreated, embeddingsWritten}'

Status path: PENDING → PROCESSING → COMPLETED. Happy path: chunksCreated == embeddingsWritten. If embeddingsWritten is lower, see Pipelines → Replay & Retry.

5. Search — text in, ranked chunks out

The search route embeds your query with the collection’s paired text_embedding_search template and returns ranked chunks with their text.

dodil data


dodil data search "what is multi-head attention" \
  -b kb-platform \
  --table docs \
  --top-k 5

--table (alias --collection) is repeatable and pins the search to those collections; omit it to search every eligible collection in the bucket.

curl


curl -sS -X POST "https://api.data.dodil.io/kb-platform/search/vector" \
  -H "Authorization: Bearer $DODIL_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "text": "what is multi-head attention",
    "collectionNames": ["docs"],
    "topK": 5
  }' | jq '{
       tookMs,
       results: [.results[] | {
         score,
         object: .object.key,
         content: (.content | .[0:200] + "...")
       }]
     }'

Python (requests)


import os, requests
 
r = requests.post(
    "https://api.data.dodil.io/kb-platform/search/vector",
    headers={
        "Authorization": f"Bearer {os.environ['DODIL_TOKEN']}",
        "Content-Type": "application/json",
    },
    json={
        "text": "what is multi-head attention",
        "collectionNames": ["docs"],
        "topK": 5,
    },
).json()
 
for hit in r["results"]:
    print(hit["score"], hit["object"]["key"], hit["content"][:200])

The whole JSON body is text + collectionNames + topK (plus a minScore that is parsed but never applied). The bucket comes from the path, not the body. content comes back on every hit — there is no includeContent switch.

There is no metadata pre-filter on this route. k3-api hands the search template a hardcoded empty filter, so you cannot narrow by source_key, folder, or any other field in the request. Narrow by pinning collectionNames, or filter client-side on results[].object.key. If you need real predicate-plus-KNN in one statement, drop to the data plane and use pgvector operators in SQL — WHERE … ORDER BY emb <=> … applies the predicate and the KNN together.

Hybrid is not live. Every shipping *_embedding_index template pins enable_bm25 = false — the ingest tool hard-errors if it is set true, and the plane has no BM25 implementation. Searches today are dense-only. See Vector → Search for what the RRF merge in that route actually does (it merges across collection groups, not retrieval tiers).

Pre-embedded KNN — when you already hold the vector

dodil data vsearch is data-plane KNN over a table’s vector column. It needs -b, -t/--table (alias --collection, no -c shorthand) and --column:


# --text embeds client-side for you; --vector takes a comma-separated literal
dodil data vsearch -b kb-platform --table docs --column embedding \
  --text "what is multi-head attention" --top-k 5 --metric cosine

6. Wire into your application

A typical RAG loop in Python:


import os, requests, openai
 
K3 = "https://api.data.dodil.io"
HEADERS = {
    "Authorization": f"Bearer {os.environ['DODIL_TOKEN']}",
    "Content-Type": "application/json",
}
 
def rag_query(question: str) -> str:
    # 1. Retrieve top-5 chunks from K3 (the template owns retrieval)
    resp = requests.post(
        f"{K3}/kb-platform/search/vector",
        headers=HEADERS,
        json={
            "text": question,
            "collectionNames": ["docs"],
            "topK": 5,
        },
    ).json()
 
    # 2. Build the context — `content` is on every hit, no opt-in needed
    chunks = [hit["content"] for hit in resp["results"]]
    context = "\n\n---\n\n".join(chunks)
 
    # 3. Hand to your LLM
    completion = openai.OpenAI().chat.completions.create(
        model="gpt-4o-mini",
        messages=[
            {"role": "system", "content": "Answer the question using the provided context. Cite sources by file name."},
            {"role": "user", "content": f"Context:\n{context}\n\nQuestion: {question}"},
        ],
    )
    return completion.choices[0].message.content
 
print(rag_query("Explain self-attention"))

Drop-in Node / Go / Rust equivalents — the K3 HTTP API speaks pbjson; any HTTP client works.

7. Operational maintenance

Add new documents

Same upload commands — every new object hits the auto-generated rule → ingest job → vectors land in docs. No re-configuration needed.

Backfill after rule changes

Edit the rule to broaden coverage (e.g. add .md to includePatterns), then retroactively re-ingest objects that now match:


RULE_ID=$(dodil data ingest list -b kb-platform -p "$PIPELINE_ID" -o json | jq -r '.rules[0].ruleId')
 
# Add .md to the include patterns
dodil data ingest update "$RULE_ID" -b kb-platform \
  --include "**/*.pdf" --include "**/*.docx" --include "**/*.txt" --include "**/*.md"
 
# Re-discover the source (internal-S3) and dispatch ingestion for matched objects
SOURCE_ID=$(dodil data source list -b kb-platform -o json \
  | jq -r '.sources[] | select(.name == "internal") | .sourceId')
 
dodil data ingest trigger-discovery -b kb-platform -s "$SOURCE_ID" --full-sync

For replay of failed jobs specifically, see Pipelines → Replay & Retry.

Pause ingestion temporarily


# Disable the rule — uploads still happen, just no ingestion
dodil data ingest update "$RULE_ID" -b kb-platform --enabled=false
 
# Re-enable when ready
dodil data ingest update "$RULE_ID" -b kb-platform --enabled=true

Inspect what’s indexed


# The collection's resolved config (pipeline id, not name)
dodil data vector collection get "$PIPELINE_ID" -b kb-platform -o json
 
# Per-object-key chunk status — via Storage's ObjectInfo
dodil data object show papers/attention.pdf -b kb-platform -o json \
  | jq '.pipelineStatuses[]'   # one entry per rule that ran on this object

Common gotchas

Symptom	Cause	Fix
Upload succeeds but no ingest job spawns	Object path doesn’t match auto-rule globs	List the rule’s `includePatterns`; remember `*/.pdf` is recursive, `*.pdf` is not
Jobs `COMPLETED` but `search` returns nothing	First-ingest vector index build still in progress	Wait 10–30 s after first ingest, then re-search
Search returns stale results after re-upload	Vector index updates async after re-ingest	Delete + re-upload as a new key, or wait for the re-ingest job to reach `COMPLETED`. There is no freshness selector on the search route
Different doc types yielding very different chunk counts	`text_embedding_index` chunks by token count (`chunk_size` defaults to 2000); long docs → many chunks	Set the template inputs at create time — `dodil data vector collection add … --set chunk_size=1000 --set chunk_overlap=100`
Latency spikes after corpus grows past ~100K chunks	Default KNN search params too low for large corpora	Run tuned KNN on the data plane via the Qdrant / Pinecone adapters (e.g. Qdrant `params.hnsw_ef`)

Cleanup


# Pause ingestion first
dodil data ingest update "$RULE_ID" -b kb-platform --enabled=false
 
# Delete in this order (no cascade)
dodil data ingest delete "$RULE_ID" -b kb-platform
 
# The collection IS the pipeline — either verb deletes it, by pipeline id
dodil data vector collection delete "$PIPELINE_ID" -b kb-platform
 
# (Optional) Drop the bucket + objects
dodil data bucket delete kb-platform

RAG Knowledge Base

Prerequisites

1. Create the bucket

2. Pick a template + create the collection

3. Upload documents — three ways

boto3

aws-cli

dodil data

4. Watch the ingest pipeline

5. Search — text in, ranked chunks out

dodil data

curl

Python (requests)

Pre-embedded KNN — when you already hold the vector

6. Wire into your application

7. Operational maintenance

Add new documents

Backfill after rule changes

Pause ingestion temporarily

Inspect what’s indexed

Common gotchas

Cleanup

See also