S3 Compatibility

K3’s object gateway speaks native S3 on the byte plane. Point any S3 SDK at it with your Dodil API key and existing code works — aws s3 sync, boto3.client('s3'), @aws-sdk/client-s3, MinIO mc, rclone — with two changes: the endpoint URL, and path-style addressing.

What this page covers: credentials, the operation matrix (and the honest unsupported list), auth modes, error codes, limits, and end-to-end setup for the four most common clients.

Credentials & endpoint

Lead credential: a Dodil API key — an opaque dk_{lookup}{secret} token (the lookup is 16 base32 characters). Issue one from the console (Bucket → API Keys or Org → API Keys) or the CLI:

dodil auth apikey issue --name s3-ci --role k3.editor    # secret shown once

With no --sa/--user, IAM automatically mints a managed service account and binds the key to it — that’s the simple path, nothing extra to create. Every API key is bound to a service account (SA) principal, and that binding matters for SigV4 below.

Two secrets, two jobs. The secret half of a dk_{lookup}{secret} token is a bearer credential only: send the whole token as Authorization: Bearer dk_…. SigV4 is symmetric HMAC, so its signing secret is different — it is the client secret of the service account the key is bound to. When you sign with dk_{lookup} as the access key ID, the gateway maps the lookup to its bound SA and verifies your signature against that SA’s client secret. Never put the dk_ secret half in an SDK’s secret_access_key field — the signature won’t verify.

If you don’t want the SigV4 dance at all, the byte plane also accepts the full dk_ token as a Bearer header (see Auth modes below) — but SigV4 is what every standard S3 SDK speaks. Basic auth is not accepted on the S3 byte plane — that form belongs to the tables-gateway wires (Postgres, Bolt, HTTP).

Housekeeping: dodil auth apikey rotate <lookup> mints a new secret while keeping the same lookup (so the access key ID is stable), and dodil auth apikey revoke <lookup> propagates within ~30 seconds.

Setup — pick your client

# One-time configuration
aws configure --profile dodil-k3
# AWS Access Key ID:     <access_key_id>
# AWS Secret Access Key: <secret_access_key>
# Default region name:   us-east-1
# Default output format: json
 
# Test — list a bucket you own (not bare `aws s3 ls`, see the note below)
aws s3 ls s3://kb-prod/ \
  --endpoint-url https://object.uk-lon-1.dodil.io \
  --profile dodil-k3

export AWS_PROFILE=dodil-k3
export AWS_ENDPOINT_URL=https://object.uk-lon-1.dodil.io   # aws-cli v2.15+
 
aws s3 ls s3://kb-prod/
aws s3 cp ./hello.txt s3://kb-prod/greetings/hello.txt

import boto3
from botocore.config import Config
 
s3 = boto3.client(
    "s3",
    endpoint_url="https://object.uk-lon-1.dodil.io",
    aws_access_key_id="<access_key_id>",
    aws_secret_access_key="<secret_access_key>",
    region_name="us-east-1",
    config=Config(s3={"addressing_style": "path"}),
)
 
# Verify
s3.list_objects_v2(Bucket="kb-prod")
 
# Upload
s3.upload_file("hello.txt", "kb-prod", "greetings/hello.txt")
 
# Presign a 1-hour download URL
url = s3.generate_presigned_url(
    "get_object",
    Params={"Bucket": "kb-prod", "Key": "greetings/hello.txt"},
    ExpiresIn=3600,
)

import {
  S3Client,
  ListObjectsV2Command,
  PutObjectCommand,
  GetObjectCommand,
} from "@aws-sdk/client-s3";
import { getSignedUrl } from "@aws-sdk/s3-request-presigner";
 
const s3 = new S3Client({
  endpoint: "https://object.uk-lon-1.dodil.io",
  region: "us-east-1",
  forcePathStyle: true,
  credentials: {
    accessKeyId: "<access_key_id>",
    secretAccessKey: "<secret_access_key>",
  },
});
 
// Verify
await s3.send(new ListObjectsV2Command({ Bucket: "kb-prod" }));
 
// Upload
await s3.send(new PutObjectCommand({
  Bucket: "kb-prod",
  Key: "greetings/hello.txt",
  Body: "hello, k3",
  ContentType: "text/plain",
}));
 
// Presign a 1-hour download URL
const url = await getSignedUrl(
  s3,
  new GetObjectCommand({ Bucket: "kb-prod", Key: "greetings/hello.txt" }),
  { expiresIn: 3600 },
);

mc alias set dodil-k3 \
  https://object.uk-lon-1.dodil.io \
  <access_key_id> \
  <secret_access_key> \
  --api S3v4 --path on
 
mc ls dodil-k3/kb-prod
mc cp ./hello.txt dodil-k3/kb-prod/greetings/hello.txt
mc share download --expire 1h dodil-k3/kb-prod/greetings/hello.txt

No checksum or payload-signing tweaks are needed. Modern SDKs default to flexible checksums and Content-Encoding: aws-chunked streaming bodies (STREAMING-UNSIGNED-PAYLOAD-TRAILER); the gateway verifies your signature over the headers you sent, decodes the chunk framing, and re-signs a plain body to the backend. Leave request_checksum_calculation / AWS_REQUEST_CHECKSUM_CALCULATION alone. The one consequence: x-amz-sdk-checksum-algorithm and x-amz-checksum-* are dropped at the edge, so end-to-end checksum validation is not enforced by the backend — the ETag is still returned.

Root ListBuckets (aws s3 ls with no bucket) is not the operation you want here. It reaches the backend outside any org-scoped bucket rewrite, so it does not return your short bucket names. Use dodil data bucket list or the ListBuckets admin RPC to enumerate buckets, and scope every S3 call to a bucket you already know.

What’s supported

The honest shape of the byte plane: it is a verifying reverse proxy, not a re-implementation of S3. It authenticates you, binds the request to your org, rewrites the bucket to its backend composite, and forwards to the CephS3 backend. It does not maintain an allow-list of S3 actions — so a verb it does not recognise is not blocked, it is simply passed through and answered by the backend, with none of K3’s control-plane semantics attached to it.

That distinction is what the three tables below encode. Supported means K3 owns the behaviour end-to-end. Passes through means the bytes reach the backend and you get a plausible S3 answer, but K3’s ACL, pipelines, quota and admin RPCs know nothing about it. Not supported means don’t build on it.

Object operations

Multipart upload

The full lifecycle is supported, and it is the path every SDK takes automatically above its multipart threshold — see Recipes → Multipart for large files.

K3 sets no part-size floor or ceiling of its own — the backend’s S3 rules apply. What does bind is the edge: every request body is buffered whole in memory before it is forwarded, and the public ingress caps a single request body at 5 GB. Leave your SDK’s default part size alone; it is already well inside that.

Bucket operations

Use the admin plane for bucket lifecycle and configuration. Creating, deleting, and configuring a bucket are control-plane operations. Doing them over the S3 wire produces backend state that K3 does not know about, and the drift is silent.

Not supported

Nothing in this list is explicitly rejected at the edge — the request reaches the backend. What “not supported” means is that K3 does not model it: the admin RPCs, ACL evaluator, quota accounting and ingest pipelines all ignore it, so anything you set here is invisible to every other surface and may be discarded by a future backend change.

Public buckets — anonymous access

Anonymous reads use a different door. object.uk-lon-1.dodil.io requires a credential on every request — an unsigned request is refused with 401 AccessDenied before any bucket is looked at. Anonymous access is served by the control-plane S3 route on api.data.dodil.io, which is the only one that understands the org-scoped /orgs/<org_name>/ path prefix and evaluates a bucket’s access mode for an anonymous caller. Use the signed object endpoint for your own traffic and the org-scoped URL for public sharing.

Three access modes gate this, and the owner always wins in all three:

A CUSTOM policy statement is matched on effect, principal, action (s3:GetObject, s3:PutObject, s3:DeleteObject, s3:ListBucket, s3:HeadObject) and a key-prefix resource — so public/* can be open while private/* stays closed on the same bucket. See Policy.

# Direct invocation
aws s3 ls \
  --endpoint-url https://api.data.dodil.io/orgs/<org_name> \
  --no-sign-request \
  s3://<bucket>/
 
aws s3 cp \
  --endpoint-url https://api.data.dodil.io/orgs/<org_name> \
  --no-sign-request \
  s3://<bucket>/path/to/file.pdf .

[profile dodil-k3-public]
endpoint_url = https://api.data.dodil.io/orgs/<org_name>
region = us-east-1
s3 =
    addressing_style = path

import boto3
from botocore import UNSIGNED
from botocore.config import Config
 
s3 = boto3.client(
    "s3",
    endpoint_url="https://api.data.dodil.io/orgs/<org_name>",
    config=Config(signature_version=UNSIGNED, s3={"addressing_style": "path"}),
)
 
# Read-only operations work; writes return 403.
s3.list_objects_v2(Bucket="<bucket>")
s3.download_file("<bucket>", "path/to/file.pdf", "./file.pdf")

# Object download
curl -O https://api.data.dodil.io/orgs/<org_name>/<bucket>/path/to/file.pdf
 
# Reference in HTML / markdown
# <img src="https://api.data.dodil.io/orgs/<org_name>/<bucket>/logo.svg" />

Anonymous writes return 403 AccessDenied — owners must use signed requests for uploads and deletes. To switch a bucket to PUBLIC: dodil data bucket update <name> --access-mode public (private | public | custom).

Per-bucket CORS is evaluated on this same control-plane route: it answers the OPTIONS preflight and injects Access-Control-Allow-Origin from the bucket’s CORS rules. The object endpoint carries a single fixed CORS allow-list for the console instead, so a browser reading a public bucket cross-origin must use the org-scoped URL.

Auth modes accepted on the byte plane

Region from the credential scope (…/<yyyymmdd>/<region>/s3/aws4_request) is whatever the client picks — the S3 wire derives the signing key from it, so any consistent region works.

For the SigV4 / SigV2 forms the access key ID is dk_{lookup} or an SA clientId, and the signing secret is the bound SA’s client secret — see Credentials & endpoint for the distinction.

The K3 token form is the one GetObjectUrl and dodil data object url mint. All three query parameters are required and the HMAC covers {org, bucket, key, expires} together, so the URL is bound to exactly one object — changing the key invalidates it. X-K3-Expires is an absolute Unix-seconds deadline. The requested TTL defaults to 3600 s and is capped at 86400 s (24 h) — ask for more and you get 24 h. GET only: it grants a download, never an upload, so browser uploads need a SigV4 presign instead.

There is no anonymous mode in this table. A request on the object endpoint that carries none of these is refused with 401 — see Public buckets.

End-to-end: aws-cli ↔ dodil data

Objects uploaded via raw S3 are first-class K3 objects — admin RPCs see them, pipelines fire on them, presigned URLs work over them. Buckets are the exception: create them on the admin plane (see Bucket operations above).

# 1. Create the bucket on the admin plane — NOT `aws s3 mb`
dodil data bucket create kb-s3-demo
 
# 2. Upload an object over raw S3
echo "hello from aws-cli" > hello.txt
aws s3 cp hello.txt s3://kb-s3-demo/greetings/hello.txt
 
# 3. Inspect via K3 admin — same object, enriched with ingest status
dodil data object show greetings/hello.txt -b kb-s3-demo -o json
 
# 4. Issue a K3-signed share URL (shorter than SigV4 presign)
dodil data object url greetings/hello.txt -b kb-s3-demo --expires 3600

The takeaway: S3 in, K3 features out. Objects are interoperable; bucket lifecycle belongs to the control plane.

Error response shape

Errors on the byte plane come back as S3 XML (compatible with every S3 SDK) with a real Code element; the admin plane uses a JSON envelope. The codes K3 itself mints:

<!-- 403 — ACL denial, quota exhaustion, or a signature that did not verify -->
<?xml version="1.0" encoding="UTF-8"?>
<Error>
  <Code>AccessDenied</Code>
  <Message>access denied</Message>
</Error>

Every denial renders the same flat AccessDenied on purpose: a prober must not be able to tell “no such bucket” from “not yours” from “policy says no”. The real reason is in the server-side logs, never in the response.

Codes that describe the object, not the request — NoSuchKey, NoSuchBucket, NoSuchUpload, EntityTooLarge — are produced by the storage backend and passed through verbatim:

<!-- 404 — bucket or key not found (from the backend) -->
<?xml version="1.0" encoding="UTF-8"?>
<Error>
  <Code>NoSuchKey</Code>
  <Message>The specified key does not exist.</Message>
</Error>

Standard SDKs surface all of these as typed exceptions (ClientError in boto3, S3ServiceException in the JS SDK). For admin RPCs (dodil data ... / gRPC), see Conventions for the JSON error envelope.

Limits

Next steps

• Recipes — direct-from-browser upload, multipart for large files, static-site hosting
• API Reference — gRPC + HTTP admin contracts
• Pipelines — auto-indexing on upload (the K3-specific feature S3 doesn’t have)
• Auth and Access — auth-mode internals