Mirror from AWS S3

Goal: copy objects from an existing AWS S3 bucket (or any S3-compatible store) into K3 — either as a one-shot migration or as an ongoing sync.

Why: every object you mirror in becomes a K3 object — it shows up in dodil k3 object list, supports presigned URLs, and enters K3’s pipeline plane (ingest rules fire on every uploaded object, so a mirror can be the entry point to auto-indexing your S3 data).

Mirroring transfers bytes + content-type. It does not transfer S3-specific metadata that K3 doesn’t model: object versions, lifecycle state, replication status, S3 tags, per-object ACLs. See S3 Compatibility — Not supported.

One-shot migration

aws-cli

Define two profiles — your existing AWS account and K3 — then aws s3 sync between them:


# ~/.aws/config
 
[profile aws-prod]
region = us-east-1
# credentials in ~/.aws/credentials [aws-prod]
 
[profile dodil-k3]
region = us-east-1
endpoint_url = https://k3.dodil.io
s3 =
  addressing_style = path
# credentials in ~/.aws/credentials [dodil-k3]


# Pre-create the K3 bucket
dodil k3 bucket create migrated-from-s3 -d "Mirrored from production AWS"
 
# One-shot sync
aws s3 sync s3://my-aws-bucket/ s3://migrated-from-s3/ \
  --source-region us-east-1 \
  --profile aws-prod \
  --endpoint-url https://k3.dodil.io

aws s3 sync between profiles does the right thing — it streams objects through aws-cli’s memory (no local disk round-trip), uploads them to K3 using SigV4, and uses multipart automatically for large files.

For very large buckets, split by prefix and run in parallel:


aws s3 sync s3://my-aws-bucket/2023/ s3://migrated-from-s3/2023/ ... &
aws s3 sync s3://my-aws-bucket/2024/ s3://migrated-from-s3/2024/ ... &
aws s3 sync s3://my-aws-bucket/2025/ s3://migrated-from-s3/2025/ ... &
wait

MinIO mc

MinIO’s mc mirror is purpose-built for this and reports progress nicely:


mc alias set aws-prod https://s3.us-east-1.amazonaws.com   <AWS_KEY> <AWS_SECRET>
mc alias set dodil-k3 https://k3.dodil.io                  <SA_ID>   <SA_SECRET> --path on
 
# Pre-create the K3 bucket
mc mb dodil-k3/migrated-from-s3
 
# Mirror — preserves keys, content-type, modified times
mc mirror --preserve aws-prod/my-aws-bucket dodil-k3/migrated-from-s3

Flags that matter:

Flag	Effect
`--preserve`	Copy mtime + access permissions where supported
`--remove`	Delete objects in destination that are gone from source (true mirror)
`--watch`	Stay running — replicate new objects continuously
`--overwrite`	Re-upload objects whose ETag differs

rclone

rclone is well-suited for one-time bulk migrations — it parallelizes aggressively and resumes cleanly.


# ~/.config/rclone/rclone.conf
 
[aws-prod]
type = s3
provider = AWS
access_key_id = <AWS_KEY>
secret_access_key = <AWS_SECRET>
region = us-east-1
 
[dodil-k3]
type = s3
provider = Other
access_key_id = <SA_ID>
secret_access_key = <SA_SECRET>
endpoint = https://k3.dodil.io
force_path_style = true
region = us-east-1


# Dry run first — see what would copy
rclone sync aws-prod:my-aws-bucket dodil-k3:migrated-from-s3 \
  --transfers 16 --checkers 32 --dry-run
 
# Real run with progress
rclone sync aws-prod:my-aws-bucket dodil-k3:migrated-from-s3 \
  --transfers 16 --checkers 32 --progress

Verify the migration


# Object counts
echo "AWS:" && aws s3 ls s3://my-aws-bucket/ --recursive --profile aws-prod | wc -l
echo "K3: " && dodil k3 object list -b migrated-from-s3 -o json | jq 'length'
 
# Spot-check a single key — sizes should match exactly
aws s3api head-object --bucket my-aws-bucket --key path/to/object --profile aws-prod
dodil k3 object show path/to/object -b migrated-from-s3 -o json
 
# Byte-compare a small file
diff <(aws s3 cp s3://my-aws-bucket/foo.json - --profile aws-prod) \
     <(aws s3 cp s3://migrated-from-s3/foo.json - --endpoint-url https://k3.dodil.io)

Ongoing replication

If you need K3 to stay in sync with an upstream S3 bucket continuously, you have three patterns:

Pattern	Latency	Ops complexity
`mc mirror --watch`	Seconds	Low — one long-running process
Periodic `aws s3 sync` cron	Minutes / hours	Lowest — just a cron job
S3 event → Lambda → K3 PUT	Sub-second	Highest — Lambda + IAM

For most cases, periodic aws s3 sync from a cron is the right answer — it’s idempotent, restartable, and stateless.

What flows into K3’s pipeline plane

When aws s3 sync uploads each object, K3 treats it like any other PutObject — meaning ingest discovery fires on every transferred object. If you’ve set up pipeline rules on the destination bucket, your migration is also your first ingest run. See Pipelines for how to wire rules.

If you want to mirror cold without triggering ingest, mirror first to a bucket that has no rules, then promote later — or apply rules after the sync finishes and use dodil k3 ingest to backfill explicitly.

Common gotchas

Symptom	Cause	Fix
`403 AccessDenied` reading from AWS	AWS credentials wrong / missing `s3:GetObject`	Verify with `aws s3 ls s3://source --profile aws-prod` first
`403 AccessDenied` writing to K3	K3 storage quota exhausted	Check your org’s quota
Files transfer but content-type is wrong	`aws s3 sync` between profiles re-infers from extension	Pre-set with `--content-type` per-file or use `mc mirror` (preserves)
Object versions / history lost	K3 doesn’t model versions	Flatten versioned objects to keys like `path/v1.txt`, `path/v2.txt` server-side before sync
Multipart parts visible in K3 (`?uploads`) after sync	Interrupted upload	`ListMultipartUploads` then `AbortMultipartUpload`; bucket quota counts in-flight parts