Direct-from-browser upload

Goal: let a signed-in user upload a file from their browser directly to K3 — no bytes flowing through your application server.

Why: your server stays stateless, the upload runs at the network’s full bandwidth, and you don’t pay for inbound bandwidth twice.

Shape:

[browser]  --(1) GET /api/upload-url-------------->  [your backend]
[browser]  <--(2) { url, key, expiresAt }---------  [your backend]
[browser]  --(3) PUT <url> + file body----------->  [k3 gateway]

The presigned URL is single-use, time-limited, and scoped to one bucket+key+method. Your backend never sees the file.

Presign against api.data.dodil.io, not the object endpoint. Both doors verify a SigV4-presigned PUT, but only the control-plane route evaluates per-bucket CORS. object.uk-lon-1.dodil.io carries one fixed fleet-wide origin allow-list instead (localhost, dodil.io, dodil.cloud by default) and never reads BucketCorsConfiguration — so a browser on https://app.example.com gets a CORS failure there no matter what you set. The trade is body size: the control-plane ingress caps a single request body at 1 GB, against 5 GB on the object endpoint. Above 1 GB, go multipart — each part is its own request.

1. Configure CORS on the bucket

The browser will issue a cross-origin PUT and read response headers (etag). The bucket needs a CORS rule that allows it. Do this once per bucket:

cat > cors.json <<'EOF'
{
  "corsRules": [
    {
      "allowedOrigins": ["https://app.example.com"],
      "allowedMethods": ["PUT", "GET", "HEAD"],
      "allowedHeaders": ["*"],
      "exposeHeaders": ["ETag"],
      "maxAgeSeconds": 3600
    }
  ]
}
EOF
 
dodil data bucket cors set kb-prod -f cors.json

Adjust allowedOrigins to match your frontend origin — a single wildcard segment is supported (https://*.example.com), and ["*"] should stay in local development only. Rules are evaluated in order and the first origin+method match wins, so put your narrow rules first. See CORS — API Reference for the full shape.

2. Mint a presigned PUT URL on the backend

You have two paths, both supported by K3:

For uploads you need SigV4 presign, because GetObjectUrl issues GET-only URLs. The DODIL_ACCESS_KEY_ID / DODIL_SECRET_ACCESS_KEY values come from your Dodil API key — see S3 Compatibility for what goes in each field. Implementation:

// backend route: GET /api/upload-url?filename=foo.pdf
import { S3Client, PutObjectCommand } from "@aws-sdk/client-s3";
import { getSignedUrl } from "@aws-sdk/s3-request-presigner";
import { randomUUID } from "node:crypto";
 
const s3 = new S3Client({
  endpoint: process.env.K3_ENDPOINT,           // https://api.data.dodil.io
  region: "us-east-1",
  forcePathStyle: true,
  credentials: {
    accessKeyId: process.env.DODIL_ACCESS_KEY_ID!,
    secretAccessKey: process.env.DODIL_SECRET_ACCESS_KEY!,
  },
});
 
export async function uploadUrlHandler(req, res) {
  const { filename, contentType } = req.query;
 
  // Namespace under the user, never trust the raw filename
  const key = `uploads/${req.user.id}/${randomUUID()}/${filename}`;
 
  const url = await getSignedUrl(
    s3,
    new PutObjectCommand({
      Bucket: "kb-prod",
      Key: key,
      ContentType: contentType,
    }),
    { expiresIn: 300 }   // 5 minutes — short-lived
  );
 
  res.json({ url, key, expiresAt: Date.now() + 300_000 });
}

# backend route: GET /api/upload-url?filename=foo.pdf
import os, uuid
import boto3
from botocore.config import Config
 
s3 = boto3.client(
    "s3",
    endpoint_url=os.environ["K3_ENDPOINT"],
    region_name="us-east-1",
    aws_access_key_id=os.environ["DODIL_ACCESS_KEY_ID"],
    aws_secret_access_key=os.environ["DODIL_SECRET_ACCESS_KEY"],
    config=Config(s3={"addressing_style": "path"}),
)
 
def upload_url(filename: str, content_type: str, user_id: str) -> dict:
    key = f"uploads/{user_id}/{uuid.uuid4()}/{filename}"
    url = s3.generate_presigned_url(
        "put_object",
        Params={
            "Bucket": "kb-prod",
            "Key": key,
            "ContentType": content_type,
        },
        ExpiresIn=300,
    )
    return {"url": url, "key": key}

// backend route: GET /api/upload-url?filename=foo.pdf
import (
    "context"
    "time"
 
    "github.com/aws/aws-sdk-go-v2/aws"
    "github.com/aws/aws-sdk-go-v2/credentials"
    "github.com/aws/aws-sdk-go-v2/service/s3"
    "github.com/google/uuid"
)
 
func newS3() *s3.Client {
    return s3.New(s3.Options{
        Region:       "us-east-1",
        BaseEndpoint: aws.String(os.Getenv("K3_ENDPOINT")),
        UsePathStyle: true,
        Credentials: credentials.NewStaticCredentialsProvider(
            os.Getenv("DODIL_ACCESS_KEY_ID"),
            os.Getenv("DODIL_SECRET_ACCESS_KEY"),
            "",
        ),
    })
}
 
func presignUpload(ctx context.Context, userID, filename, contentType string) (string, string, error) {
    key := "uploads/" + userID + "/" + uuid.NewString() + "/" + filename
    presigner := s3.NewPresignClient(newS3())
    req, err := presigner.PresignPutObject(ctx, &s3.PutObjectInput{
        Bucket:      aws.String("kb-prod"),
        Key:         aws.String(key),
        ContentType: aws.String(contentType),
    }, s3.WithPresignExpires(5*time.Minute))
    if err != nil {
        return "", "", err
    }
    return req.URL, key, nil
}

3. Upload from the browser

<input id="file" type="file" />
<button id="upload">Upload</button>
<script type="module">
  document.getElementById("upload").onclick = async () => {
    const file = document.getElementById("file").files[0];
 
    // 1. Ask backend for a presigned URL
    const r = await fetch(
      `/api/upload-url?filename=${encodeURIComponent(file.name)}` +
      `&contentType=${encodeURIComponent(file.type)}`
    );
    const { url, key } = await r.json();
 
    // 2. PUT directly to K3 — no server in the path
    const put = await fetch(url, {
      method: "PUT",
      headers: { "Content-Type": file.type },
      body: file,
    });
 
    if (!put.ok) throw new Error(`upload failed: ${put.status}`);
 
    // 3. Tell your backend the upload finished so it can record metadata
    await fetch("/api/upload-complete", {
      method: "POST",
      headers: { "content-type": "application/json" },
      body: JSON.stringify({ key, etag: put.headers.get("etag") }),
    });
  };
</script>

Showing upload progress

fetch can’t report PUT progress — use XMLHttpRequest instead:

function uploadWithProgress(url, file, onProgress) {
  return new Promise((resolve, reject) => {
    const xhr = new XMLHttpRequest();
    xhr.open("PUT", url);
    xhr.setRequestHeader("Content-Type", file.type);
    xhr.upload.onprogress = (e) => {
      if (e.lengthComputable) onProgress(e.loaded / e.total);
    };
    xhr.onload = () => (xhr.status === 200 ? resolve(xhr) : reject(xhr));
    xhr.onerror = () => reject(xhr);
    xhr.send(file);
  });
}

What happens after a successful upload

Once the PUT returns 200, K3’s S3 gateway:

1. Persists the object to durable storage.
2. Fires ingest discovery — if kb-prod has matching pipeline rules, the object is automatically routed to vector / warehouse / Scriptum pipelines (see Pipelines). You can check ObjectInfo.pipelineStatuses for per-rule progress.
3. Returns the ETag header — for a single PUT this is the MD5 of the object body; a multipart object gets S3’s composite <md5-of-part-md5s>-<partCount> form instead, which is not the hash of the file.

Don’t rely on flexible checksums for integrity. On the object endpoint the gateway drops x-amz-sdk-checksum-algorithm and every x-amz-checksum-* header before re-signing to the storage backend, so end-to-end checksum validation is not enforced there; the control-plane route re-signs with x-amz-content-sha256 instead. Either way, whichever headers the presigned URL committed to, the browser must send exactly those and no others — a checksum header your SDK signed in but fetch doesn’t send is the usual cause of SignatureDoesNotMatch. Check integrity against the returned ETag.

Your frontend doesn’t need to wait for ingest to complete — pipelines run async and your code can poll GetObjectInfo if it needs to know when indexing finishes.

Common gotchas

Security notes

• Always namespace keys server-side (e.g. uploads/<userId>/<uuid>/...). Never let the client choose the key — it’s a path-traversal risk.
• Keep expiresIn short (60–300 s). The URL is a bearer token for one PUT.
• Optionally bind to content type / size — pass ContentType (and consider ContentLength enforcement at presign time) so a client can’t lie about what they’re uploading.
• Don’t rely on Content-Type the browser sends for security decisions — sniff server-side if you care.

See also

• Multipart Large Files — for files over ~100 MB
• CORS — API Reference — full CORS rule shape
• S3 Compatibility — SDK setup details
• Pipelines — what happens to the object after upload