Vector Collections — API Reference

Package: dodil.data.pipeline.v1 · Service: PipelineService

There is no VectorService. The vector pillar was folded into the pipeline catalog: dodil.data.vector.v1 no longer exists, and the whole /:bucket/vector/… route tree is gone. A vector collection is a pipeline with a vector facet — one store_entities row (kind='vector') plus the index pipeline bound to it, created in a single call. Every read and delete goes through the generic pipeline RPCs, filtered or addressed by facet.

RPC	HTTP	Was
`CreateVectorPipeline`	`POST /:bucket/pipelines/vector`	`AddVectorPipeline`
`ListPipelines`	`GET /:bucket/pipelines?facet=vector`	`ListCollections`
`GetPipeline`	`GET /:bucket/pipelines/:pipeline_id`	`GetCollection`
`UpdatePipeline`	`PATCH /:bucket/pipelines/:pipeline_id`	(new)
`DeletePipeline`	`DELETE /:bucket/pipelines/:pipeline_id`	`DeleteCollection`

AddVectorCollection — the manual / EXTERNAL creation mode — was removed with the Milvus/VBase decommission. See BYO embeddings.

gRPC setup — grpcurl, endpoints, reflection, and field-name casing — is covered once in Conventions → Using gRPC.

The identifier story

A vector collection has two ids, and the one you address it by is the pipeline’s:

Id	Where it comes from	What it’s for
`pipelineId`	the `pipelines` row	Get / update / delete the collection. This is what `dodil data vector collection get\|delete` takes.
`destination.storeEntityId`	the `store_entities` row	The destination itself. Only `BatchDeleteArtifacts` addresses it directly.
`destination.vector.physicalName`	generated `k3_<uuid>`	The physical collection name in the data plane. Stored under the legacy jsonb key `milvus_collection`; the plane it names is `tabled`.

There is no collection_id. The collection name you pass to CreateVectorPipeline lands on destination.name, not on Pipeline.name — Pipeline.name is set to the template id, because the row it names is the index pipeline.

`CreateVectorPipeline`

Template-driven. K3 writes three rows and talks to no data plane at all: the store_entities collection row, the index pipelines row, and — best-effort, by the *_index → *_search convention — the search-side pipeline, bound via store_entities.search_pipeline_id. The physical collection materializes lazily in the plane on first ingest via the index template’s vector_store_ensure_schema.

Schema-shaping facts (embed_model, dimensions, distance_metric, sparse_mode, embedding_type) are resolved from the template’s ScriptContract — callers cannot override them. template_id is required for exactly that reason: with no template there is nothing to resolve from. A template whose contract resolves no embed_model, or a non-positive dimensions, is refused with FAILED_PRECONDITION.

What the five shipping index templates actually resolve to:

Template	`embed_model`	`dimensions`	`modality`
`text_embedding_index`	`jina-embeddings-v4`	768	`text`
`code_embedding_index`	`jina-embeddings-v4`	1024	`code`
`visual_embedding_index`	`jina-embeddings-v4`	1024	`visual`
`object_embedding_index`	`jina-embeddings-v4` (detector `mm-gdino-large`)	1024	`object`
`face_embedding_index`	`arcface-r100` (detector `scrfd-10g`)	512	`face`

modality is read from the template’s type label, not its modality label.

Distance, sparse mode and embedding type are defaults, not declarations. No shipping template declares distance_metric or sparse_mode in its contract, so every collection created today falls through to cosine and SPARSE_MODE_NONE. Only code_embedding_index and visual_embedding_index declare embedding_type; the rest fall through to float. The templates additionally pin enable_bm25 = false and note that hybrid is not available on tabled — the ingest tool hard-errors if it is set true. The other DistanceMetric / SparseMode / EmbeddingType enum values are representable on the wire but nothing produces them.

Request

HTTP

Text embedding (no required template inputs):


curl -sS -X POST "https://api.data.dodil.io/kb-prod/pipelines/vector" \
  -H "Authorization: Bearer $DODIL_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "name": "docs",
    "description": "PDF / docx / HTML embeddings",
    "templateId": "text_embedding_index"
  }'

Object detection embedding (requires labels per the template’s ScriptContract):


curl -sS -X POST "https://api.data.dodil.io/kb-prod/pipelines/vector" \
  -H "Authorization: Bearer $DODIL_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "name": "products",
    "description": "Product image object detection",
    "templateId": "object_embedding_index",
    "templateInputs": {
      "labels": ["bottle", "bag", "shoe", "watch", "jewelry"]
    }
  }'

The bucket comes from the URL path and overrides any bucket in the body.

From the CLI: dodil data vector collection add docs -b kb-prod -t text_embedding_index (--set key=value, repeatable, for templateInputs).

Response

A Pipeline — the index pipeline row, with the collection attached as its destination.

HTTP


{
  "pipelineId": "8f1c…",
  "bucket": "kb-prod",
  "name": "text_embedding_index",
  "scriptumTemplate": "text_embedding_index",
  "options": {},
  "createdAt": "1756598400000",
  "updatedAt": "1756598400000",
  "destination": {
    "storeEntityId": "a1b2…",
    "facet": "PIPELINE_FACET_VECTOR",
    "name": "docs",
    "description": "PDF / docx / HTML embeddings",
    "status": "DESTINATION_STATUS_ACTIVE",
    "engineId": "e3f4…",
    "vector": {
      "dimensions": 768,
      "distanceMetric": "DISTANCE_METRIC_COSINE",
      "chunkSize": 500,
      "chunkOverlap": 50,
      "enableBm25": false,
      "sparseMode": "SPARSE_MODE_NONE",
      "embeddingType": "EMBEDDING_TYPE_FLOAT",
      "modality": "text",
      "embeddingSource": "EMBEDDING_SOURCE_PIPELINE",
      "templateId": "text_embedding_index",
      "embedModel": "jina-embeddings-v4",
      "templateInputs": {},
      "physicalName": "k3_a1b2…"
    }
  }
}

What K3 creates on CreateVectorPipeline:

The store_entities collection row, status = active immediately (the physical collection materializes lazily on first ingest, so there is no provisioning step to await).

The index pipeline row, bound to the template. If it fails, the destination is rolled back.

Best-effort, the search pipeline row (*_index → *_search) bound via store_entities.search_pipeline_id. If this fails the collection is still indexable, but POST /:bucket/search/vector will skip it with a warning.

It does not create an ingest rule. Callers own rule scope — add one with CreateRule pointing at the returned pipelineId.

Errors

Code	When
`INVALID_ARGUMENT`	`bucket`, `name` or `template_id` blank; `template_inputs` fails contract validation
`ALREADY_EXISTS`	A vector destination of that name already exists in the bucket
`FAILED_PRECONDITION`	The template has no `ScriptContract`, or its contract resolves no `embed_model` / no positive `dimensions`
`UNAVAILABLE`	Scriptum is not reachable — the contract cannot be fetched

`ListPipelines`

The vector view is ListPipelines narrowed to the vector facet. An unrecognised ?facet= value is a 400, not “no filter” — ?facet=vectors will not silently return every pipeline in the bucket.

Request

HTTP


curl -sS "https://api.data.dodil.io/kb-prod/pipelines?facet=vector" \
  -H "Authorization: Bearer $DODIL_TOKEN"

Query params: facet (vector | table | object | graph, case-insensitive), store_entity_id, free_only.

Response

HTTP


{
  "pipelines": [
    {
      "pipelineId": "8f1c…",
      "bucket": "kb-prod",
      "name": "text_embedding_index",
      "scriptumTemplate": "text_embedding_index",
      "options": {},
      "destination": {
        "storeEntityId": "a1b2…",
        "facet": "PIPELINE_FACET_VECTOR",
        "name": "docs",
        "status": "DESTINATION_STATUS_ACTIVE",
        "vector": {
          "dimensions": 768,
          "embedModel": "jina-embeddings-v4",
          "distanceMetric": "DISTANCE_METRIC_COSINE",
          "sparseMode": "SPARSE_MODE_NONE",
          "embeddingType": "EMBEDDING_TYPE_FLOAT",
          "embeddingSource": "EMBEDDING_SOURCE_PIPELINE",
          "templateId": "text_embedding_index",
          "modality": "text",
          "physicalName": "k3_a1b2…"
        }
      }
    }
  ],
  "pagination": null
}

Two rows per collection. ListPipelines filters on the destination’s facet, not on the pipeline’s role, so a template-created collection returns both its index pipeline and its search pipeline — two entries sharing one destination.storeEntityId. Tell them apart by scriptumTemplate (…_index vs …_search), or collapse on destination.storeEntityId to count collections.

pagination is always null: pagination is not implemented at the repository layer, and the HTTP handler never sends a PaginationRequest. The full bucket list comes back in one response.

`GetPipeline`

Request

HTTP


curl -sS "https://api.data.dodil.io/kb-prod/pipelines/8f1c…" \
  -H "Authorization: Bearer $DODIL_TOKEN"

Response

A Pipeline with its destination attached — same shape as the create response.

A pipeline_id belonging to another org or another bucket reports NOT_FOUND, not PERMISSION_DENIED, so the RPC does not leak another tenant’s rows. A non-UUID pipeline_id is INVALID_ARGUMENT.

`UpdatePipeline`

Rename the collection or edit its destination config in place. The destination’s facet is immutable — supplying a table or object config against a vector destination is rejected, as is a destination block on a free pipeline.

HTTP


curl -sS -X PATCH "https://api.data.dodil.io/kb-prod/pipelines/8f1c…" \
  -H "Authorization: Bearer $DODIL_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "destination": {
      "description": "Product manuals + release notes"
    }
  }'

`DeletePipeline`

This deletes the pipeline row only. It does not delete the store_entities collection row, and it does not touch the vector data in the plane. K3 deletes the wiring it owns; dropping the data is plane state and the tenant’s own call through the tables-gateway.

To remove the destination as well, use BatchDeleteArtifacts (POST /:bucket/pipelines/_batch-delete) with an ARTIFACT_KIND_DESTINATION ref alongside the ARTIFACT_KIND_PIPELINE and ARTIFACT_KIND_RULE refs. It sorts refs into dependency order (rules → pipelines → destinations) and is idempotent: an already-deleted ref is a success-skip.

Request

HTTP


curl -sS -X DELETE "https://api.data.dodil.io/kb-prod/pipelines/8f1c…" \
  -H "Authorization: Bearer $DODIL_TOKEN"

Response

Empty (DeletePipelineResponse {}).

BYO embeddings — the `EXTERNAL` mode is retired

AddVectorCollection was the only creator of EMBEDDING_SOURCE_EXTERNAL collections, and it went with the Milvus/VBase decommission. EmbeddingSource.EMBEDDING_SOURCE_EXTERNAL still exists on the enum and existing rows still report it, but nothing produces one any more — the mode is read-only. dodil data vector collection add-manual is likewise gone.

Bring-your-own embeddings are now a tabled data-plane operation: a collection is a table with a VECTOR(dim) column.

SQL

Over the Postgres wire (pg.uk-lon-1.dodil.io:5432, sslmode=require):


CREATE TABLE ada_embeddings (
  id        VARCHAR PRIMARY KEY,
  content   VARCHAR,
  embedding VECTOR(1536)
);

Then KNN with the pgvector operators (<-> L2, <#> inner product, <=> cosine):


SELECT id, content, embedding <=> '[0.12,0.03,…]'::vector AS distance
FROM ada_embeddings
ORDER BY distance
LIMIT 10;

VECTOR(n) and vector<n> are the same type. An optional ANN index is CREATE INDEX … USING hnsw (embedding vector_cosine_ops) — the accepted operator classes are vector_l2_ops, vector_ip_ops and vector_cosine_ops, nothing else.

dodil data


dodil data table create ada_embeddings -b kb-prod \
  --columns-json '[
    {"name":"id","type":"VARCHAR"},
    {"name":"content","type":"VARCHAR"},
    {"name":"embedding","type":"VECTOR(1536)"}
  ]' \
  --merge-key id
 
dodil data vsearch -b kb-prod -t ada_embeddings \
  --column embedding --vector "0.12,0.03,…" --metric cosine

There is no --column flag on table create — columns are supplied as a JSON array via --columns-json. Short type aliases are normalized (string/text → VARCHAR, int/long → BIGINT); anything else, including VECTOR(1536), passes through uppercased.

HTTP (tables door)

Writes go to the typed vector facet on the tables door, not api.data.dodil.io:


curl -sS -X POST "https://table.uk-lon-1.dodil.io/v1/vector/ada_embeddings/upsert" \
  -H "x-db-id: $DODIL_DB_ID" \
  -H "Authorization: Bearer $DODIL_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "column": "embedding",
    "points": [{ "id": "doc-1", "values": [0.12, 0.03] }]
  }'

TablesVector (dodil.tables.v1) serves UpsertVectors, QueryVectors, FetchVectors, DeleteVectors and DescribeVectorStats at /v1/vector/:table/{upsert,query,fetch,delete,stats}. The x-db-id header carries the database; its absence is a 401. The table is a path segment — there is no /v1/databases/{db}/vector/… form.

Qdrant and Pinecone SDKs speak to the same tables, via the wire adapters — endpoints and credentials in Connect & wire adapters.

The plane’s honest vector surface. Metric on dodil.tables.v1 is exactly three values — COSINE, EUCLIDEAN, DOT_PRODUCT. There is no Hamming, Jaccard or BM25 metric, QueryVectors carries no metadata filter or namespace, and DeleteVectors has no filter arm (ids or all: true, never both). VectorMatch.score is the metric’s distance — lower is closer — not a similarity.

Vector Collections — API Reference

The identifier story

CreateVectorPipeline

Request

HTTP

gRPC

Response

HTTP

gRPC

Errors

ListPipelines

Request

HTTP

gRPC

Response

HTTP

gRPC

GetPipeline

Request

HTTP

gRPC

Response

UpdatePipeline

HTTP

gRPC

DeletePipeline

Request

HTTP

gRPC

Response

BYO embeddings — the EXTERNAL mode is retired

SQL

dodil data

HTTP (tables door)

See also

`CreateVectorPipeline`

`ListPipelines`

`GetPipeline`

`UpdatePipeline`

`DeletePipeline`

BYO embeddings — the `EXTERNAL` mode is retired