Core Concepts — Object Storage

The Object Storage domain is defined by the proto package dodil.data.storage.v1. From the proto itself:

Owns: buckets, bucket policy (S3-style ACL), CORS, and the S3-like admin surface for objects (list, info, delete, presigned URL). Does NOT own: sources or their credentials (see dodil.data.source.v1), rules / sync / ingest jobs (see dodil.data.ingest.v1), or the pillar engines (vector / warehouse). Storage stays thin: the data plane, not the intelligence plane.

Each entity below has a gRPC view (the proto message) and an HTTP view (the JSON your client sees on the wire). Use the toggle at the top of each section to flip — your choice sticks across the page and across visits.

What to expect in the HTTP view:

• Field names are camelCase (e.g. storageUsedBytes).
• Enums are wire-name strings (e.g. "BUCKET_STATUS_ACTIVE").
• int64 values are JSON strings — JavaScript can’t represent them as numbers without precision loss.
• Every field is always present — defaults are not omitted.
• Timestamps are Unix milliseconds.

Bucket

The central entity. Most K3 APIs are bucket-scoped — sources, rules, vector engines, and table engines all attach here.

{
  "name": "demo",
  "description": "Workflow bucket",
  "status": "BUCKET_STATUS_ACTIVE",
  "internalSourceId": "src_01HEFGHIJK",
  "storageUsedBytes": "1048576",
  "storageQuotaBytes": "0",
  "objectCount": "42",
  "sourceCount": "3",
  "ruleCount": "2",
  "indexCount": "1",
  "createdAt": "1700000000000",
  "updatedAt": "1700001000000",
  "accessMode": "BUCKET_ACCESS_MODE_PRIVATE",
  "drn": "drn:dodil:k3:uk-lon-1:org_01HABC:bucket/demo"
}

message Bucket {
  string name = 1;                            // unique per tenant
  string description = 2;
  BucketStatus status = 3;
  string internal_source_id = 4;              // auto-created INTERNAL_S3 source
  int64 storage_used_bytes = 5;
  int64 storage_quota_bytes = 6;              // 0 = unlimited
  int64 object_count = 7;
  int64 source_count = 8;
  int64 rule_count = 9;
  int64 index_count = 10;
  int64 created_at = 11;
  int64 updated_at = 12;
  BucketAccessMode access_mode = 13;
  // Canonical IAM resource name — `drn:dodil:k3:{region}:{org}:bucket/{name}`.
  // Server-built so clients never hand-assemble DRNs.
  string drn = 14;
}
 
enum BucketStatus {
  BUCKET_STATUS_UNSPECIFIED = 0;
  BUCKET_STATUS_CREATING = 1;
  BUCKET_STATUS_ACTIVE = 2;
  BUCKET_STATUS_DISABLED = 3;
  BUCKET_STATUS_DELETING = 4;
}
 
enum BucketAccessMode {
  BUCKET_ACCESS_MODE_UNSPECIFIED = 0;         // treated as PRIVATE
  BUCKET_ACCESS_MODE_PRIVATE = 1;
  BUCKET_ACCESS_MODE_PUBLIC = 2;
  BUCKET_ACCESS_MODE_CUSTOM = 3;
}

Notes:

• internal_source_id references an auto-created INTERNAL_S3 source managed by the Source service — a direct S3 PUT triggers the same ingest rules as an external sync.
• The *_count and storage_used_bytes fields are server-maintained aggregates.
• CreateBucket accepts access_mode, but the CLI’s dodil data bucket create does not expose it — a new bucket is PRIVATE until you dodil data bucket update <name> --access-mode public.

Access modes

access_mode on a bucket governs who can read or write its objects. The bucket owner always wins in all three modes; a denial is a flat 403 AccessDenied in every case, deliberately indistinguishable from “no such bucket”:

UNSPECIFIED is treated as PRIVATE. “Write” means PutObject / DeleteObject; GetObject, HeadObject and ListBucket are reads.

Access mode is only evaluated on api.data.dodil.io. The S3 byte plane at object.uk-lon-1.dodil.io refuses any request that carries no recognised credential with 401 AccessDenied before a bucket is ever looked at, and it exposes no /orgs/ route. Anonymous, mode-aware access is the control-plane S3 route:

https://api.data.dodil.io/orgs/<org_name>/<bucket>/<key>

The org slug and bucket name are resolved as a pair, so a mismatch returns S3 NoSuchBucket rather than leaking another tenant’s bucket. Path segments the control plane reserves for management endpoints — objects, policy, sources, rules, ingest, search, metadata, indexes, vector-store — are never resolved as object keys on this route.

See S3 Compatibility → Public buckets for end-to-end SDK examples.

Object

Type: ObjectInfo. The metadata view returned by the admin surface (ListObjects, GetObjectInfo). The bytes themselves flow through the K3 HTTP gateway, not the gRPC API.

{
  "bucket": "demo",
  "key": "docs/sample.pdf",
  "size": "1048576",
  "etag": "\"d41d8cd98f00b204e9800998ecf8427e\"",
  "contentType": "application/pdf",
  "lastModified": "1700000500000",
  "userMetadata": {
    "source": "drive"
  },
  "storageClass": "STANDARD",
  "indexStatus": "indexed",
  "pipelineStatuses": [
    {
      "ruleId": "rule_01HEFG",
      "ruleName": "docs-rule",
      "pipelineKind": "vector",
      "indexStatus": "indexed",
      "destinationId": "col_01HIJK"
    }
  ]
}

message ObjectInfo {
  string bucket = 1;
  string key = 2;
  int64 size = 3;
  string etag = 4;
  string content_type = 5;
  int64 last_modified = 6;
  map<string, string> user_metadata = 7;
  string storage_class = 8;
  // Aggregate index status across pipelines: none|pending|indexed|failed|partial
  string index_status = 9;
  // Per-pipeline status (one entry per rule that triggered on this object).
  // The detail fields are flat strings — clients read them directly without
  // pulling in the ingest service types.
  repeated ObjectPipelineStatus pipeline_statuses = 10;
}
 
message ObjectPipelineStatus {
  string rule_id = 1;
  string rule_name = 2;
  string pipeline_kind = 3;                   // "vector" | "warehouse" | "free"
  string index_status = 4;                    // "pending" | "indexed" | "failed" | "partial"
  string destination_id = 5;                  // store_entity.id (empty for free pipelines)
}

Notes:

• pipeline_statuses is the bridge between the storage plane and the ingestion plane: each entry tells you which rule fired for this object, what kind of pipeline ran, and the current index state.
• pipeline_kind is a flat string ("vector" / "warehouse" / "free") so callers can stay decoupled from the ingest service types.

Policy

Type: BucketPolicy. S3-style access control, modeled on the AWS bucket-policy contract. Bound to a bucket via SetBucketPolicy; BucketAccessMode.CUSTOM on Bucket signals a policy is in effect.

{
  "version": "2024-01-01",
  "statements": [
    {
      "sid": "PublicRead",
      "effect": "POLICY_EFFECT_ALLOW",
      "principal": { "aws": ["*"] },
      "actions": ["s3:GetObject"],
      "resources": ["public/*"]
    }
  ]
}

message BucketPolicy {
  string version = 1;                         // e.g. "2024-01-01"
  repeated PolicyStatement statements = 2;
}
 
message PolicyStatement {
  string sid = 1;
  PolicyEffect effect = 2;
  PolicyPrincipal principal = 3;
  repeated string actions = 4;                // "s3:GetObject", "s3:PutObject", ...
  repeated string resources = 5;              // object-key globs, e.g. "public/*"
}
 
message PolicyPrincipal {
  repeated string aws = 1;                    // "*" anonymous, or org IDs
}
 
enum PolicyEffect {
  POLICY_EFFECT_UNSPECIFIED = 0;
  POLICY_EFFECT_ALLOW = 1;
  POLICY_EFFECT_DENY = 2;
}

Notes:

• actions are matched exactly against five recognised names: s3:GetObject, s3:PutObject, s3:DeleteObject, s3:ListBucket, s3:HeadObject. There is no wildcard action — s3:* matches nothing, and a statement left with no recognised action is dropped from the policy entirely.
• resources are bare object-key globs (* and ?), not bucket-qualified paths and not ARNs: public/* opens public/… while private/* stays closed on the same bucket. Omitting resources defaults to *.
• principal.aws accepts "*" for anonymous access or one or more org IDs for tenant-scoped policies.
• Evaluation is the S3 algorithm: an explicit POLICY_EFFECT_DENY wins, any surviving ALLOW grants, and no matching statement is an implicit deny.
• A policy is only consulted when access_mode is CUSTOM. Setting one on a PUBLIC or PRIVATE bucket changes nothing until you flip the mode.
• The HTTP door also accepts an AWS-shaped document — Version / Statement / Sid / Effect: "Allow" / Principal: {"AWS": "*"} / Action / Resource with ARNs (arn:aws:s3:::demo/public/* is stripped to public/*), a bare string wherever a list is allowed, and a single statement object instead of an array. An ARN naming a different bucket is a 400, not a silent no-match. Condition, NotAction, NotResource and NotPrincipal are refused with a 400 naming the key — K3 does not evaluate them, and honouring the document while ignoring them would grant more access than it states.

CORS

Type: BucketCorsConfiguration. Mirrors S3 CORS one-to-one.

{
  "corsRules": [
    {
      "allowedOrigins": ["https://app.example.com"],
      "allowedMethods": ["GET", "PUT"],
      "allowedHeaders": ["*"],
      "exposeHeaders": ["ETag"],
      "maxAgeSeconds": 3600
    }
  ]
}

message BucketCorsConfiguration {
  repeated CorsRule cors_rules = 1;
}
 
message CorsRule {
  repeated string allowed_origins = 1;
  repeated string allowed_methods = 2;
  repeated string allowed_headers = 3;
  repeated string expose_headers = 4;
  int32 max_age_seconds = 5;
}

Notes:

• Rules are evaluated in order; the first rule matching origin (and, on a preflight, method and requested headers) wins and supplies every injected header.
• allowed_origins supports * and a single wildcard segment — https://*.example.com matches by prefix + suffix. Exact matches are case-insensitive.
• An empty allowed_methods matches any method, per S3’s permissive behaviour. Be explicit.
• expose_headers is what a browser is allowed to read off the response — ETag is not CORS-safelisted, so an upload flow that reads the etag must list it.
• Per-bucket CORS is enforced on api.data.dodil.io only. That door answers the OPTIONS preflight from these rules and injects Access-Control-Allow-Origin on the response. The object.uk-lon-1.dodil.io byte plane carries one fixed fleet-wide origin allow-list instead (localhost, dodil.io, dodil.cloud by default) and never consults BucketCorsConfiguration. Point browser traffic that needs your own origin at the control-plane route.
• These rules are K3’s own. The S3 ?cors subresource passes through to the storage backend and is never read by K3 — see S3 Compatibility.

PresignedURL

Produced by the GetObjectUrl RPC. The URL carries a K3 custom token signed with the server’s presign_secret (impl: services/storage/objects.rs). It is not an AWS SigV4 or SigV2 URL.

// GET /demo/objects/docs%2Fsample.pdf/url?expires_in_seconds=1800
// — key is URL-encoded in the path; there is no request body.
 
// Response
{
  "url": "https://object.uk-lon-1.dodil.io/demo/docs/sample.pdf?X-K3-Token=9f3c…&X-K3-Expires=1700003600&X-K3-Org=org_01HABC",
  "expiresAt": "1700003600000"
}

message GetObjectUrlRequest {
  string bucket = 1;
  string key = 2;
  int64 expires_in_seconds = 3;               // default 3600
}
 
message GetObjectUrlResponse {
  string url = 1;
  int64 expires_at = 2;
}

Notes:

• expires_in_seconds defaults to 3600 (1h) and is capped at 86400 (24h) by the handler — ask for more and you get 24 h. expires_at in the response is Unix milliseconds; X-K3-Expires in the URL is Unix seconds.
• The URL carries three query parameters, all required: X-K3-Token (HMAC-SHA256 hex), X-K3-Expires, X-K3-Org. The HMAC covers {org, bucket, key, expires} together, so the URL is bound to exactly one object — editing the key invalidates it.
• The host is whatever the deployment sets as its public S3 proxy URL; in production that is the object endpoint (object.uk-lon-1.dodil.io), which verifies X-K3-Token with the same shared verifier the control-plane route uses. Both doors accept it.
• GET only. It grants a download, never an upload — browser uploads need a SigV4 presign instead (see Recipes → Browser upload).
• AWS SigV4 / SigV2 are separately accepted by the S3 proxy as incoming auth schemes — those are client-signed URLs. GetObjectUrl is the server-issued, K3-token flow.

See also

• Object Storage — overview
• API Reference
• CLI Guide
• Conventions — auth, headers, error envelope
• CLI Basics — install + common flags