Skip to Content
We are live but in Staging 🎉
PipelinesAPI ReferenceSync

Sync — API Reference

Package: dodil.k3.ingest.v1 · Service: IngestService

Sync RPCs let you scan a source for objects and dispatch ingestion manually — useful for backfills, recovery, and testing. They complement the automatic trigger paths (direct S3 PUTs and scheduled source polls based on each source’s sync_interval_seconds).

RPCHTTP
TriggerDiscoveryPOST /:bucket/sources/:source_id/discover
TriggerIngestionPOST /:bucket/sources/:source_id/ingest
GetSyncStatusGET /:bucket/sources/:source_id/sync

gRPC setup — grpcurl, endpoints, reflection, and field-name casing — is covered once in Conventions → Using gRPC.

TriggerDiscovery

Scans the source for new/changed objects. With full_sync = true, K3 ignores the source’s etag checkpoint and rescans everything.

Request

Delta scan:

curl -sS -X POST "https://k3.dev.dodil.io/kb-prod/sources/src_a1b2.../discover" \
  -H "Authorization: Bearer $DODIL_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{ "bucket": "kb-prod", "sourceId": "src_a1b2..." }'

Full rescan:

curl -sS -X POST "https://k3.dev.dodil.io/kb-prod/sources/src_a1b2.../discover" \
  -H "Authorization: Bearer $DODIL_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{ "bucket": "kb-prod", "sourceId": "src_a1b2...", "fullSync": true }'

Response

{
  "jobId": "discovery-job-a1b2...",
  "accepted": true,
  "message": "Discovery scan started"
}

Discovery is asynchronous — track progress with GetSyncStatus.

TriggerIngestion

Dispatches already-discovered objects to ingest. Useful for replay after pipeline fixes (retry_failed = true) or targeted backfill (source_object_id or rule_id).

Request

Replay failed/partial:

curl -sS -X POST "https://k3.dev.dodil.io/kb-prod/sources/src_a1b2.../ingest" \
  -H "Authorization: Bearer $DODIL_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "bucket": "kb-prod",
    "sourceId": "src_a1b2...",
    "retryFailed": true
  }'

Scope to one rule:

curl -sS -X POST "https://k3.dev.dodil.io/kb-prod/sources/src_a1b2.../ingest" \
  -H "Authorization: Bearer $DODIL_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "bucket": "kb-prod",
    "sourceId": "src_a1b2...",
    "ruleId": "rule_a1b2..."
  }'

Response

{
  "accepted": true,
  "message": "Dispatched 1247 objects",
  "dispatched": 1247
}

GetSyncStatus

Snapshot of the source’s most recent and current sync.

Request

curl -sS "https://k3.dev.dodil.io/kb-prod/sources/src_a1b2.../sync" \
  -H "Authorization: Bearer $DODIL_TOKEN"

Response

{
  "bucket": "kb-prod",
  "sourceId": "src_a1b2...",
  "status": "SOURCE_STATUS_ACTIVE",
  "lastSyncAt": "1716843600000",
  "lastSuccessAt": "1716843600000",
  "nextSyncAt": "1716847200000",
  "objectsDiscovered": "1247",
  "objectsProcessed": "1247",
  "objectsIndexed": "1240"
}

current_job_id is set during an active discovery. objects_indexed ≤ objects_processed ≤ objects_discovered — gaps usually indicate ingest failures (inspect via Jobs).

See also

  • Jobs — inspect individual ingest jobs (ListIngestJobs, GetIngestStatus)
  • Sources — sync_interval_seconds controls the automatic poll cadence
  • Core Concepts → IngestJob — the full status enum and counters
  • grpcurl reference — full flag set + reflection-disabled fallbacks
RulesJobs