Multimodal Search

Goal: search a vector collection by a file — an image to find similar images, a clip to find similar clips. Upload the file with the search request; the bound *_embedding_search template embeds it server-side, then searches.

Template used: visual_embedding_index. Also in the catalog: face_embedding_index (face embeddings, arcface) and object_embedding_index (open-vocabulary object detection).

Shape:


   query file (multipart upload, ≤ 50 MB)
          │
          ▼
   POST /:bucket/search/vector  →  staged at a temp S3 key,
          │                        reaped by a drop guard on return
          ▼
   content type → modality → only matching collections take part
          │
          ▼
   *_embedding_search template mints a presigned URL, fetches,
   extracts, embeds the file server-side
          │
          ▼
   results, RRF-merged across groups (single group = raw scores)

Modality routing — the exact rule

The file part’s content type picks the modality, and a file query with no explicit collectionNames auto-filters to collections whose modality matches. From detect_modality (dodil-k3/bin/api/src/services/search/search.rs:159-170):

Content type	Modality
`image/`, `audio/`, `video/*`	`visual`
`text/x-*`, `application/javascript`	`code`
everything else	`text`

A collection’s modality comes from its template’s type label at create time, defaulting to text. Note that all three of image, audio and video collapse into the single visual bucket — there is no separate audio or video modality.

Set the part’s content type explicitly. If your client sends application/octet-stream, the query routes to text and your visual collections are silently excluded. In curl that’s ;type=image/jpeg.

Prerequisites

A visual-template-backed collection, plus an ingest rule — CreateVectorPipeline derives none:


PIPELINE_ID=$(dodil data vector collection add product-images -b kb-prod \
  --description "Product photo library" \
  --template visual_embedding_index -o json | jq -r '.pipelineId')
 
dodil data ingest add product-images-rule -b kb-prod \
  -c "$PIPELINE_ID" -i 'products/**/*.jpg' -i 'products/**/*.png'
 
dodil data object create ./img-001.jpg -b kb-prod -k products/img-001.jpg
dodil data object create ./img-002.jpg -b kb-prod -k products/img-002.jpg

Verify the jobs completed:


dodil data ingest jobs -b kb-prod -p "$PIPELINE_ID" -o json \
  | jq '.jobs[] | {object: .object.key, status, chunksCreated}'

1. Search by image upload

Python (requests)


import os, requests
 
resp = requests.post(
    "https://api.data.dodil.io/kb-prod/search/vector",
    headers={"Authorization": f"Bearer {os.environ['DODIL_TOKEN']}"},
    files={"file": ("example-bag.jpg", open("example-bag.jpg", "rb"), "image/jpeg")},
    data={"collection_name": "product-images", "top_k": "20"},
).json()
 
for r in resp["results"]:
    print(round(r["score"], 4), r["object"]["key"])

requests sets the part’s content type from the third tuple element — that string is what drives the modality routing, so don’t omit it.

Sample response (top 5):


[
  {"score": 0.0164, "object": "products/img-042.jpg"},
  {"score": 0.0161, "object": "products/img-128.jpg"},
  {"score": 0.0158, "object": "products/img-007.jpg"},
  {"score": 0.0155, "object": "products/img-091.jpg"},
  {"score": 0.0152, "object": "products/img-205.jpg"}
]

The multipart contract

Field	Required	Notes
`file`	yes	Missing → `INVALID_ARGUMENT: file field is required`
`collection_name`	no	Repeat the field to name several
`collection_names`	no	Comma-separated list; folded into the same list as `collection_name`
`top_k`	no	Default `10`; unparseable values fall back to `10`
`text`	no	Parsed into `rerank_text`, which drives no K3-side logic
`search_mode`	no	Parsed and discarded
`rerank`	no	`"true"` / `"1"`; parsed and discarded

Omit the collection fields entirely to fan out across every collection matching the file’s modality.

2. Other file types

The same route and the same fields handle any file — what actually gets embedded is the template’s business, so check the catalog before assuming a modality ships:


dodil data vector templates -o json \
  | jq '.templates[] | {id, modalities, acceptedExtensions, acceptedContentTypes}'

The shipped picker describes visual_embedding_index as images and video frames. Audio and PDF-page-render support depends on what the deployed template’s contract declares — if acceptedContentTypes doesn’t list audio/*, ingest will not embed your clips even though a query with an audio/wav part still routes to the visual modality. The routing rule and the template’s capabilities are two separate things.

3. Face search (`face_embedding_index`)

For face recognition, use the face-specific template — it embeds detected face crops (arcface-family embeddings):


FACES=$(dodil data vector collection add faces -b kb-prod \
  --template face_embedding_index -o json | jq -r '.pipelineId')
 
dodil data ingest add faces-rule -b kb-prod -c "$FACES" -i 'members/**/*.jpg'
 
dodil data object create ./member-001.jpg -b kb-prod -k members/member-001.jpg
 
curl -sS -X POST "https://api.data.dodil.io/kb-prod/search/vector" \
  -H "Authorization: Bearer $DODIL_TOKEN" \
  -F "file=@./unknown-face.jpg;type=image/jpeg" \
  -F "collection_name=faces" \
  -F "top_k=5"

Face and visual collections both carry the visual modality but use different models, so they land in different compatibility groups and are RRF-merged rather than ranked together — exactly the jina-v4-vs-arcface case the group key exists for.

4. Object detection — open-vocabulary

object_embedding_index detects objects against a caller-defined vocabulary. Its labels input is an array, and --set sends only strings, so this one needs the API:


curl -sS -X POST "https://api.data.dodil.io/kb-prod/pipelines/vector" \
  -H "Authorization: Bearer $DODIL_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "bucket": "kb-prod",
    "name": "products-by-object",
    "templateId": "object_embedding_index",
    "templateInputs": {
      "labels": ["bottle", "bag", "shoe", "watch", "jewelry", "headphones"]
    }
  }'

Then add a rule and search exactly as above. template_inputs is validated against the template’s ScriptContract at the boundary — a required field with no contract default must be present, and contract-fixed fields are rejected outright.

Common gotchas

Symptom	Cause	Fix
`INVALID_ARGUMENT: file too large: NMB max is 50MB`	Over the service’s 50 MB cap (the route’s body limit is 50 MiB too)	Downscale / trim the query file
Image query returns no results but collections have data	The part’s content type wasn’t `image/*`, so the query routed to the `text` modality	Set it explicitly (`;type=image/jpeg`, or the 3-tuple in `requests`)
`NOT_FOUND: no collections matched`	A name in `collection_name` / `collection_names` doesn’t exist	Names come from `.destination.name`
Collection missing from results and `collectionStatuses`	Not `active`, or no bound search pipeline	Read `warnings[]`
Face query returns 0 results despite a clear face	No face detected (too small, low contrast, angled)	Pre-crop to roughly face-only; try a higher-res version
Audio or PDF query returns nothing	The deployed template may not accept that content type	Check `acceptedContentTypes` from `dodil data vector templates`
Object-detection collection misses an obvious object	The object wasn’t in the `labels` vocabulary passed at create time	Re-create with the extra label — open-vocab models only detect what you name
`RESOURCE_EXHAUSTED` under load	Per-pod search semaphore full	Retry — it protects Scriptum from overload

Performance notes

Server-side embedding of the query file is the long pole for multimodal queries — much slower than text. The file is staged to a temp S3 key, handed to the template as a short-lived presigned URL, and reaped by a drop guard when the search returns; that round trip is on every request. For latency-sensitive workflows, pre-embed query files on your side and run KNN over the wire adapters — note that POST /:bucket/search/vector itself rejects pre-embedded vectors with UNIMPLEMENTED.