Multipart upload for large files

Goal: upload files larger than ~100 MB efficiently — parallel parts, resumable on failure.

Why: a single PUT ties up one TCP connection and aborts the whole upload on any failure. Multipart breaks the file into 5 MB+ parts that upload in parallel and retry independently.

K3 supports the full lifecycle: CreateMultipartUpload, UploadPart, ListParts, CompleteMultipartUpload, AbortMultipartUpload, ListMultipartUploads — see S3 Compatibility.

The cap that actually bites is the edge, not S3. K3 sets no part-size floor or ceiling of its own — the storage backend’s S3 rules apply — but every request body is buffered whole in memory at the edge before it is forwarded, and the public ingress caps a single request body at 5 GB on object.uk-lon-1.dodil.io and 1 GB on api.data.dodil.io. That bounds a single PUT and a single UploadPart. Leave your SDK’s default part size alone; it is already far inside both. Use multipart for files over ~100 MB, and mandatorily past the cap for your endpoint.

The easy path — let the SDK do it

Every official S3 SDK auto-multiparts above a configurable threshold. You almost never write the dance by hand. (<access_key_id> / <secret_access_key> come from your Dodil API key — see S3 Compatibility.)

# Multipart kicks in automatically above ~8 MB
aws s3 cp ./large-dataset.parquet s3://kb-prod/datasets/large.parquet

[profile dodil-k3]
s3 =
  multipart_threshold = 64MB    # start multipart above this
  multipart_chunksize = 16MB    # part size
  max_concurrent_requests = 10  # parallel parts

import boto3
from boto3.s3.transfer import TransferConfig
from botocore.config import Config
 
s3 = boto3.client(
    "s3",
    endpoint_url="https://object.uk-lon-1.dodil.io",
    aws_access_key_id="<access_key_id>",
    aws_secret_access_key="<secret_access_key>",
    region_name="us-east-1",
    config=Config(s3={"addressing_style": "path"}),
)
 
cfg = TransferConfig(
    multipart_threshold=64 * 1024 * 1024,   # 64 MB
    multipart_chunksize=16 * 1024 * 1024,   # 16 MB parts
    max_concurrency=10,
    use_threads=True,
)
 
s3.upload_file(
    Filename="large-dataset.parquet",
    Bucket="kb-prod",
    Key="datasets/large.parquet",
    Config=cfg,
    Callback=lambda bytes_xfer: print(f"  transferred {bytes_xfer:,} bytes"),
)

import { S3Client } from "@aws-sdk/client-s3";
import { Upload } from "@aws-sdk/lib-storage";
import { createReadStream } from "node:fs";
 
const s3 = new S3Client({
  endpoint: "https://object.uk-lon-1.dodil.io",
  region: "us-east-1",
  forcePathStyle: true,
  credentials: {
    accessKeyId: process.env.DODIL_ACCESS_KEY_ID!,
    secretAccessKey: process.env.DODIL_SECRET_ACCESS_KEY!,
  },
});
 
const upload = new Upload({
  client: s3,
  params: {
    Bucket: "kb-prod",
    Key: "datasets/large.parquet",
    Body: createReadStream("./large-dataset.parquet"),
  },
  partSize: 16 * 1024 * 1024,   // 16 MB
  queueSize: 10,                // parallel parts in flight
  leavePartsOnError: false,
});
 
upload.on("httpUploadProgress", (p) => {
  console.log(`  ${p.loaded?.toLocaleString()} / ${p.total?.toLocaleString()}`);
});
 
await upload.done();

Manual multipart — when you need resume / progress

Drop down to raw multipart when you need:

• Resume across process restarts — persist the UploadId and completed parts somewhere durable
• Per-part progress to surface to a UI more granular than the SDK gives you
• Browser uploads of files larger than the network round-trip will tolerate in one PUT

The shape (same as S3 — every part returns an ETag, complete with the list of (PartNumber, ETag) pairs):

import {
  S3Client,
  CreateMultipartUploadCommand,
  UploadPartCommand,
  CompleteMultipartUploadCommand,
  AbortMultipartUploadCommand,
} from "@aws-sdk/client-s3";
 
async function multipartUpload(s3: S3Client, bucket: string, key: string, file: Buffer) {
  const PART = 16 * 1024 * 1024;   // 16 MB
 
  // 1. Initiate
  const init = await s3.send(new CreateMultipartUploadCommand({
    Bucket: bucket, Key: key,
  }));
  const uploadId = init.UploadId!;
 
  try {
    // 2. Upload each part — parallelize this with p-limit / Promise.all in production
    const parts: { PartNumber: number; ETag: string }[] = [];
    for (let i = 0, num = 1; i < file.length; i += PART, num++) {
      const body = file.subarray(i, Math.min(i + PART, file.length));
      const res = await s3.send(new UploadPartCommand({
        Bucket: bucket, Key: key, UploadId: uploadId,
        PartNumber: num, Body: body,
      }));
      parts.push({ PartNumber: num, ETag: res.ETag! });
    }
 
    // 3. Complete
    await s3.send(new CompleteMultipartUploadCommand({
      Bucket: bucket, Key: key, UploadId: uploadId,
      MultipartUpload: { Parts: parts },
    }));
  } catch (err) {
    // 4. On any failure, abort to free server-side state
    await s3.send(new AbortMultipartUploadCommand({
      Bucket: bucket, Key: key, UploadId: uploadId,
    }));
    throw err;
  }
}

Resumability: persist (uploadId, parts[]) after every successful UploadPart. On restart, pass the same uploadId to CompleteMultipartUploadCommand with the parts list you saved. If you lost the list, ListParts (GET /:bucket/:key?uploadId) is supported and returns the parts the backend already holds, with their ETags — so you can rebuild the manifest and resume rather than restart.

Browser-side multipart with presigned URLs

The browser pattern presigns each UploadPart URL on the backend, hands them to the browser, and the browser PUTs parts directly to K3:

[browser]  --(1) POST /api/start-upload (filename, size)----->  [backend]
[backend]  -- CreateMultipartUpload ------------------------>  [k3]
[backend]  -- presign UploadPart × N ----------------------->  [k3]
[browser]  <--(2) { uploadId, partUrls: [...] }-------------- [backend]
[browser]  --(3) PUT partUrls[i] + chunk × N--------------->  [k3]   (parallel)
[browser]  --(4) POST /api/complete-upload (parts: [...])-->  [backend]
[backend]  -- CompleteMultipartUpload ---------------------->  [k3]

// Backend — presign each part URL
import { UploadPartCommand } from "@aws-sdk/client-s3";
import { getSignedUrl } from "@aws-sdk/s3-request-presigner";
 
async function presignParts(bucket: string, key: string, uploadId: string, partCount: number) {
  return Promise.all(
    Array.from({ length: partCount }, (_, i) =>
      getSignedUrl(
        s3,
        new UploadPartCommand({
          Bucket: bucket,
          Key: key,
          UploadId: uploadId,
          PartNumber: i + 1,
        }),
        { expiresIn: 3600 },
      ),
    ),
  );
}

Browser side, slice and PUT in parallel — collect ETags, send back to the backend for CompleteMultipartUpload. Combine this with the Browser Upload recipe for the auth flow.

Presign the part URLs against https://api.data.dodil.io. Per-bucket CORS is only evaluated on the control-plane route; the object endpoint carries a fixed fleet-wide origin allow-list and never reads your BucketCorsConfiguration, so cross-origin part PUTs fail there. Keep partSize under that door’s 1 GB body cap. Server-side multipart (everything above this section) has no such constraint — use the object endpoint and its 5 GB cap.

Common gotchas

See also

• Browser Upload — the auth flow for serving presigned URLs to a browser
• S3 Compatibility — what S3 multipart actions K3 supports
• API Reference — Objects — byte-plane endpoints