Mirror from AWS S3

Goal: copy objects from an existing AWS S3 bucket (or any S3-compatible store) into K3 — either as a one-shot migration or as an ongoing sync.

Why: every object you mirror in becomes a K3 object — it shows up in dodil data object list, supports presigned URLs, and enters K3’s pipeline plane (ingest rules fire on every uploaded object, so a mirror can be the entry point to auto-indexing your S3 data).

Mirroring transfers bytes + content-type. It does not transfer S3-specific metadata that K3 doesn’t model: object versions, lifecycle state, replication status, S3 tags, per-object ACLs. See S3 Compatibility — Not supported.

One-shot migration

aws s3 sync cannot cross two endpoints in one command. --endpoint-url is global to the invocation — there is no --source-endpoint-url — so a single sync between AWS and K3 would point both sides at the same host. Use rclone or mc, which model each side as a named remote, or stage through local disk with aws-cli. The tabs below are ordered accordingly.

# ~/.config/rclone/rclone.conf
 
[aws-prod]
type = s3
provider = AWS
access_key_id = <AWS_KEY>
secret_access_key = <AWS_SECRET>
region = us-east-1
 
[dodil-k3]
type = s3
provider = Other
access_key_id = <access_key_id>
secret_access_key = <secret_access_key>
endpoint = https://object.uk-lon-1.dodil.io
force_path_style = true
region = us-east-1

# Pre-create the K3 bucket on the admin plane — NOT `rclone mkdir`
dodil data bucket create migrated-from-s3 -d "Mirrored from production AWS"
 
# Dry run first — see what would copy
rclone sync aws-prod:my-aws-bucket dodil-k3:migrated-from-s3 \
  --transfers 16 --checkers 32 --dry-run
 
# Real run with progress
rclone sync aws-prod:my-aws-bucket dodil-k3:migrated-from-s3 \
  --transfers 16 --checkers 32 --progress

mc alias set aws-prod https://s3.us-east-1.amazonaws.com   <AWS_KEY> <AWS_SECRET>
mc alias set dodil-k3 https://object.uk-lon-1.dodil.io     <access_key_id> <secret_access_key> --api S3v4 --path on
 
# Pre-create the K3 bucket on the admin plane — `mc mb` would create a raw
# backend bucket with no K3 record: no ACL, no CORS, no quota, no pipelines,
# and it would not appear in `dodil data bucket list`.
dodil data bucket create migrated-from-s3 -d "Mirrored from production AWS"
 
# Mirror — preserves keys, content-type, modified times
mc mirror --preserve aws-prod/my-aws-bucket dodil-k3/migrated-from-s3

# ~/.aws/config
 
[profile aws-prod]
region = us-east-1
# credentials in ~/.aws/credentials [aws-prod]
 
[profile dodil-k3]
region = us-east-1
endpoint_url = https://object.uk-lon-1.dodil.io
s3 =
  addressing_style = path
# credentials in ~/.aws/credentials [dodil-k3]

# Pre-create the K3 bucket on the admin plane — NOT `aws s3 mb`
dodil data bucket create migrated-from-s3 -d "Mirrored from production AWS"
 
# 1. Down from AWS
aws s3 sync s3://my-aws-bucket/ ./stage/ --profile aws-prod
 
# 2. Up into K3 (endpoint comes from the profile). Multipart is automatic
#    above ~8 MB, so large objects are handled for you.
aws s3 sync ./stage/ s3://migrated-from-s3/ --profile dodil-k3

Verify the migration

# Object counts
echo "AWS:" && aws s3 ls s3://my-aws-bucket/ --recursive --profile aws-prod | wc -l
echo "K3: " && dodil data object list -b migrated-from-s3 -o json | jq '.objects | length'
 
# Spot-check a single key — sizes should match exactly
aws s3api head-object --bucket my-aws-bucket --key path/to/object --profile aws-prod
dodil data object show path/to/object -b migrated-from-s3 -o json
 
# Byte-compare a small file
diff <(aws s3 cp s3://my-aws-bucket/foo.json - --profile aws-prod) \
     <(aws s3 cp s3://migrated-from-s3/foo.json - --endpoint-url https://object.uk-lon-1.dodil.io)

Ongoing replication

If you need K3 to stay in sync with an upstream S3 bucket continuously, you have three patterns:

For most cases, a periodic rclone sync from a cron is the right answer — it’s idempotent, restartable, and stateless, and unlike aws s3 sync it can address both endpoints in one command.

What flows into K3’s pipeline plane

When aws s3 sync uploads each object, K3 treats it like any other PutObject — meaning ingest discovery fires on every transferred object. If you’ve set up pipeline rules on the destination bucket, your migration is also your first ingest run. See Pipelines for how to wire rules.

If you want to mirror cold without triggering ingest, mirror first to a bucket that has no rules, then apply the rules after the sync finishes and backfill explicitly with dodil data ingest trigger-discovery (dodil data ingest jobs and dodil data ingest watch <job-id> follow the run).

Common gotchas

See also

• S3 Compatibility — what S3 features carry over vs not
• Multipart Large Files — what aws s3 sync is doing under the hood for big files
• Pipelines — what happens to each migrated object