Mixed-Media Library

Goal: a single bucket holds images, videos, audio clips, and PDFs; K3 embeds them all into one Vector collection; you search by file (image-by-image, audio-by-audio) or by text (“a brown leather handbag”). Visual + cross-modal retrieval, no glue code.

Primitives used: Storage (mixed-media uploads via S3 SDK or CLI) → Pipelines (auto-rule wired by Vector’s visual_embedding_index template) → Vector (one collection, searched by uploaded file or by text).

Shape:

   images / video / audio / PDFs  ──upload──►  Storage bucket
                                                    │
                                                    ▼  auto-rule fires
                                          visual_embedding_index Scriptum
                                            (handles each modality:
                                             - image → single embed
                                             - video → sampled frame embeds
                                             - audio → windowed embeds
                                             - pdf   → page-render embeds)
                                                    │
                                                    ▼
                                          Vector — `assets` collection
                                          (one collection, multimodal)
                                                    │
                          ┌─────────────────────────┴─────────────────────────┐
                          │                                                   │
                          ▼                                                   ▼
              Search by file (multipart upload)                    Search by text (JSON)
              "find similar to this image"                         "brown handbag"

Prerequisites

• dodil CLI + dodil auth login
• A bucket — kb-platform:

  dodil data bucket create kb-platform -d "Mixed-media asset library"

• No engine setup step — vector capacity is provisioned on demand when you run dodil data vector collection add below.

1. Create the multimodal collection

visual_embedding_index handles all four modalities — image, video frames, audio, and PDF page renders — through one collection. Its declared extension set is jpg, jpeg, png, gif, webp, mp4, mov, webm, mp3, wav, flac, pdf. Schema and embed_model come from the template’s contract:

dodil data template get visual_embedding_index -o json | jq '.contract.inputs'

Defaults worth knowing: embed_model jina-embeddings-v4, dimensions 1024 (the only permitted value for this template — the text one defaults to 768), image_max_width / image_max_height 512, frame_every 10 s, key_frames_only true, and max_size_bytes 10485760 (10 MB — a tenth of the text template’s ceiling). Override any of them with --set:

dodil data vector collection add assets -b kb-platform \
  --description "Mixed-media library — images / video / audio / PDFs" \
  --template visual_embedding_index \
  --set frame_every=5
# → Vector collection 'assets' (pipeline pl_…) created in bucket 'kb-platform'.
 
export PIPELINE_ID=$(dodil data vector collection list -b kb-platform -o json \
  | jq -r '.pipelines[] | select(.name == "assets") | .pipelineId')

Inspect the resolved collection (by pipeline id — collection get does not take the name):

dodil data vector collection get "$PIPELINE_ID" -b kb-platform -o json

Visual collections are dense-only, like every other collection on the platform: no shipping template enables BM25, and the plane has no BM25 implementation.

Inspect the auto-rule’s globs — they should cover image / video / audio / PDF extensions:

dodil data ingest list -b kb-platform -p "$PIPELINE_ID" -o json \
  | jq '.rules[] | {ruleId, includePatterns, includeMimeTypes, enabled}'

2. Upload mixed media — pick your client

K3 speaks native S3 — see Storage → S3 Compatibility for setup.

# Set up profile once
aws configure --profile dodil-k3
export AWS_PROFILE=dodil-k3
export AWS_ENDPOINT_URL=https://object.uk-lon-1.dodil.io
 
# Sync a folder of mixed assets
aws s3 sync ./local-assets/ s3://kb-platform/library/ \
  --content-type "image/jpeg" \
  --exclude "*" --include "*.jpg" --include "*.jpeg"
aws s3 sync ./local-assets/ s3://kb-platform/library/ \
  --content-type "video/mp4" \
  --exclude "*" --include "*.mp4"
aws s3 sync ./local-assets/ s3://kb-platform/library/ \
  --content-type "audio/wav" \
  --exclude "*" --include "*.wav"

import boto3, mimetypes, pathlib
 
s3 = boto3.client(
    "s3",
    endpoint_url="https://object.uk-lon-1.dodil.io",
    aws_access_key_id=SERVICE_ACCOUNT_ID,          # SigV4 requires a service
    aws_secret_access_key=SERVICE_ACCOUNT_SECRET,  # account — API keys can't sign
    region_name="us-east-1",
)
 
for path in pathlib.Path("./local-assets").rglob("*"):
    if not path.is_file():
        continue
    ctype = mimetypes.guess_type(path.name)[0] or "application/octet-stream"
    s3.upload_file(
        str(path), "kb-platform", f"library/{path.name}",
        ExtraArgs={"ContentType": ctype},   # sets the MIME the auto-rule matches on
    )

# Whole tree at once; -k is the key prefix
dodil data object create ./local-assets -b kb-platform -k library/ --recursive

Setting --content-type per upload matters. K3’s auto-rule matches on MIME types; the visual pipeline also dispatches to different embedders based on content-type (image vs video vs audio). For mass-uploads where MIME is wrong, fix it before ingest fires — the search side has no content-type override for stored objects (a file query carries its own content type on the multipart part, but that only affects the query).

Per-file, one key at a time:

dodil data object create ./bag-001.jpg   -b kb-platform -k library/bags/bag-001.jpg
dodil data object create ./audio-001.mp3 -b kb-platform -k library/audio/audio-001.mp3
dodil data object create ./clip-001.mp4  -b kb-platform -k library/video/clip-001.mp4
dodil data object create ./catalog.pdf   -b kb-platform -k library/pdf/catalog.pdf

3. Watch the ingest jobs

dodil data ingest jobs -b kb-platform -p "$PIPELINE_ID" -o json \
  | jq '.jobs[] | {object: .object.key, status, chunksCreated, embeddingsWritten}'

Modality-specific behavior to expect:

If embeddings_written is consistently lower than chunks_created, see Pipelines → Replay & Retry.

4. Search

The route takes two request shapes, distinguished by content type:

There is no s3Key field. A file query means uploading the bytes on the request; K3 stages them at a temp key under .k3-tmp/search/, hands the template a 5-minute presigned URL, and reaps the object when the search returns. Max upload 50 MiB.

A. By an image — file query (multipart)

curl -sS -X POST "https://api.data.dodil.io/kb-platform/search/vector" \
  -H "Authorization: Bearer $DODIL_TOKEN" \
  -F "file=@./query-bag.jpg;type=image/jpeg" \
  -F "collection_name=assets" \
  -F "top_k=20" | jq '.results[] | {score, object: .object.key}'

import os, requests
 
with open("./query-bag.jpg", "rb") as fh:
    r = requests.post(
        "https://api.data.dodil.io/kb-platform/search/vector",
        headers={"Authorization": f"Bearer {os.environ['DODIL_TOKEN']}"},
        files={"file": ("query-bag.jpg", fh, "image/jpeg")},
        data={"collection_name": "assets", "top_k": "20"},
    ).json()
 
for hit in r["results"]:
    print(hit["score"], hit["object"]["key"])

The file’s own content type routes it to the right modality — image/*, audio/* and video/* all go to the visual embedder. Set it explicitly (;type=image/jpeg in curl, the 3-tuple in requests) rather than relying on sniffing.

B. By text

Visual embedders trained CLIP-style understand text-to-image matching directly. Send a text description; K3 embeds it on the same vector axis as your images:

dodil data search "brown leather handbag with gold hardware" \
  -b kb-platform --table assets --top-k 20

curl -sS -X POST "https://api.data.dodil.io/kb-platform/search/vector" \
  -H "Authorization: Bearer $DODIL_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "text": "brown leather handbag with gold hardware",
    "collectionNames": ["assets"],
    "topK": 20
  }' | jq '.results[] | {score, object: .object.key}'

Quality of text-to-image recall depends on the underlying visual embedder. Test both shapes against your corpus to see which works better for your queries.

C. File + text together

Send text alongside file in the same multipart request. It is carried as the rerank text, not as a second query vector:

curl -sS -X POST "https://api.data.dodil.io/kb-platform/search/vector" \
  -H "Authorization: Bearer $DODIL_TOKEN" \
  -F "file=@./query-bag.jpg;type=image/jpeg" \
  -F "text=brown leather handbag with gold hardware" \
  -F "collection_name=assets" \
  -F "top_k=50" | jq '.results[] | {score, object: .object.key}'

Honest caveat: the combined signal is inert today. The text, rerank and search_mode fields are parsed into the search request and then never read — ranking comes from the file’s embedding alone. The fields are accepted for wire compatibility, so the request above succeeds; it just behaves like §A. Send the two queries separately and merge client-side if you need both signals now.

See Vector → Multimodal Search for more combined-signal patterns.

D. Audio similarity — same shape

curl -sS -X POST "https://api.data.dodil.io/kb-platform/search/vector" \
  -H "Authorization: Bearer $DODIL_TOKEN" \
  -F "file=@./query-sound.wav;type=audio/wav" \
  -F "collection_name=assets" \
  -F "top_k=10" | jq '.results[] | {score, object: .object.key}'

E. Narrowing results

There is no metadata pre-filter on this route. k3-api sends the search template a hardcoded empty filter, so you cannot scope a query to library/bags/ in the request. Two things that do work: pin collection_name / collectionNames to separate collections per category, or filter the returned results[].object.key client-side. For genuine predicate-plus-KNN in one statement, use pgvector operators in SQL on the data plane.

5. Display results — building a product-search UI

# Sketch: build a "more like this" UI
import os, requests
 
K3 = "https://api.data.dodil.io"
AUTH = {"Authorization": f"Bearer {os.environ['DODIL_TOKEN']}"}
 
def find_similar(image_path: str, top_k: int = 20):
    """File query — the bytes ride on the request; there is no s3Key shape."""
    with open(image_path, "rb") as fh:
        resp = requests.post(
            f"{K3}/kb-platform/search/vector",
            headers=AUTH,
            files={"file": (os.path.basename(image_path), fh, "image/jpeg")},
            data={"collection_name": "assets", "top_k": str(top_k)},
        ).json()
 
    # Each result has object.key — turn into a presigned download URL for the UI
    return [
        {
            "score": hit["score"],
            "key": hit["object"]["key"],
            "url": presigned_url(hit["object"]["bucket"], hit["object"]["key"]),
        }
        for hit in resp["results"]
    ]
 
def presigned_url(bucket: str, key: str) -> str:
    # Storage's GetObjectUrl — a GET, key in the path
    r = requests.get(f"{K3}/{bucket}/objects/{key}/url", headers=AUTH).json()
    return r["url"]

The presigned URL flow is critical for UIs — Storage’s GetObjectUrl is GET /:bucket/objects/:key/url and returns a K3-signed URL good for 3600 s by default (dodil data object url <key> -b <bucket> --expires <seconds> from the CLI); you don’t need to proxy bytes through your app.

Common gotchas

Cleanup

RULE_ID=$(dodil data ingest list -b kb-platform -p "$PIPELINE_ID" -o json | jq -r '.rules[0].ruleId')
 
dodil data ingest update "$RULE_ID" -b kb-platform --enabled=false
dodil data ingest delete "$RULE_ID" -b kb-platform
dodil data vector collection delete "$PIPELINE_ID" -b kb-platform
dodil data bucket delete kb-platform

See also

• Vector → Multimodal Search — deeper on the multimodal patterns (face-search, open-vocab object detection)
• Storage → S3 Compatibility — bulk-upload via aws-cli / SDKs
• Storage → Browser Upload — presigned PUT for direct-from-browser uploads (great for user-generated content libraries)
• Vector → External Collection — when you want a different visual embedder than K3’s default
• RAG Knowledge Base — same shape but for text; pair both for a multimodal-RAG product