Sync — API Reference

Package: dodil.data.ingest.v1 · Service: IngestService

Sync RPCs let you scan a source for objects and dispatch ingestion manually — useful for backfills, recovery, and testing. They complement the automatic trigger paths (direct S3 PUTs and scheduled source polls based on each source’s sync_interval_seconds).

Discovery and ingestion are two separate stages, and both are needed. They are not interchangeable:

Stage	RPC	What it does	What it does not do
Discovery	`TriggerDiscovery`	Scans the source and records which objects exist / changed.	Does not run any pipeline.
Ingestion	`TriggerIngestion`	Dispatches already-discovered objects to their matching rules’ pipelines.	Does not scan — an object never discovered is invisible to it.

So a backfill is discovery then ingestion. If TriggerIngestion reports dispatched: 0 on a source you know has files, the usual cause is that discovery has not run, not that the rules are wrong.

Neither stage bypasses rules: an object with no matching enabled rule is discovered and then simply never dispatched.

RPC	HTTP
`TriggerDiscovery`	`POST /:bucket/sources/:source_id/discover`
`TriggerIngestion`	`POST /:bucket/sources/:source_id/ingest`
`GetSyncStatus`	`GET /:bucket/sources/:source_id/sync`

gRPC setup — grpcurl, endpoints, reflection, and field-name casing — is covered once in Conventions → Using gRPC.

`TriggerDiscovery`

Scans the source for new/changed objects. With full_sync = true, K3 ignores the source’s etag checkpoint and rescans everything.

Request

dodil data


dodil data ingest trigger-discovery -b kb-prod -s src_a1b2...
dodil data ingest trigger-discovery -b kb-prod -s src_a1b2... --full-sync
dodil data ingest trigger-discovery -b kb-prod -s src_a1b2... -r rule_a1b2...

HTTP

Delta scan:


curl -sS -X POST "https://api.data.dodil.io/kb-prod/sources/src_a1b2.../discover" \
  -H "Authorization: Bearer $DODIL_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{ "bucket": "kb-prod", "sourceId": "src_a1b2..." }'

Full rescan:


curl -sS -X POST "https://api.data.dodil.io/kb-prod/sources/src_a1b2.../discover" \
  -H "Authorization: Bearer $DODIL_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{ "bucket": "kb-prod", "sourceId": "src_a1b2...", "fullSync": true }'

gRPC


rpc TriggerDiscovery(TriggerDiscoveryRequest) returns (TriggerDiscoveryResponse);
 
message TriggerDiscoveryRequest {
  string bucket = 1;
  string source_id = 2;
  bool full_sync = 3;
  optional string rule_id = 4;                // scope to one rule's matchers
}

Delta scan:


grpcurl \
  -H "Authorization: Bearer $DODIL_TOKEN" \
  -d '{ "bucket": "kb-prod", "source_id": "src_a1b2..." }' \
  $K3_GRPC \
  dodil.data.ingest.v1.IngestService/TriggerDiscovery

Full rescan:


grpcurl \
  -H "Authorization: Bearer $DODIL_TOKEN" \
  -d '{ "bucket": "kb-prod", "source_id": "src_a1b2...", "full_sync": true }' \
  $K3_GRPC \
  dodil.data.ingest.v1.IngestService/TriggerDiscovery

Response

HTTP


{
  "jobId": "discovery-job-a1b2...",
  "accepted": true,
  "message": "Discovery scan started"
}

Discovery is asynchronous — track progress with GetSyncStatus.

`TriggerIngestion`

Dispatches already-discovered objects to ingest. Useful for replay after pipeline fixes (retry_failed = true) or targeted backfill (source_object_id or rule_id).

Request

dodil data


dodil data ingest trigger -b kb-prod -s src_a1b2... --retry-failed
dodil data ingest trigger -b kb-prod -s src_a1b2... -r rule_a1b2...
dodil data ingest trigger -b kb-prod -s src_a1b2... --source-object-id obj_a1b2...

HTTP

Replay failed/partial:


curl -sS -X POST "https://api.data.dodil.io/kb-prod/sources/src_a1b2.../ingest" \
  -H "Authorization: Bearer $DODIL_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "bucket": "kb-prod",
    "sourceId": "src_a1b2...",
    "retryFailed": true
  }'

Scope to one rule:


curl -sS -X POST "https://api.data.dodil.io/kb-prod/sources/src_a1b2.../ingest" \
  -H "Authorization: Bearer $DODIL_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "bucket": "kb-prod",
    "sourceId": "src_a1b2...",
    "ruleId": "rule_a1b2..."
  }'

gRPC


rpc TriggerIngestion(TriggerIngestionRequest) returns (TriggerIngestionResponse);
 
message TriggerIngestionRequest {
  string bucket = 1;
  string source_id = 2;
  optional string rule_id = 3;
  optional string source_object_id = 4;
  bool retry_failed = 5;
}

Replay failed/partial:


grpcurl \
  -H "Authorization: Bearer $DODIL_TOKEN" \
  -d '{
    "bucket": "kb-prod",
    "source_id": "src_a1b2...",
    "retry_failed": true
  }' \
  $K3_GRPC \
  dodil.data.ingest.v1.IngestService/TriggerIngestion

Scope to one rule:


grpcurl \
  -H "Authorization: Bearer $DODIL_TOKEN" \
  -d '{
    "bucket": "kb-prod",
    "source_id": "src_a1b2...",
    "rule_id": "rule_a1b2..."
  }' \
  $K3_GRPC \
  dodil.data.ingest.v1.IngestService/TriggerIngestion

Response

HTTP


{
  "accepted": true,
  "message": "Dispatched 1247 objects",
  "dispatched": 1247
}

`GetSyncStatus`

Snapshot of the source’s most recent and current sync.

Request

HTTP


curl -sS "https://api.data.dodil.io/kb-prod/sources/src_a1b2.../sync" \
  -H "Authorization: Bearer $DODIL_TOKEN"

Response

HTTP


{
  "bucket": "kb-prod",
  "sourceId": "src_a1b2...",
  "status": "SOURCE_STATUS_ACTIVE",
  "lastSyncAt": "1716843600000",
  "lastSuccessAt": "1716843600000",
  "nextSyncAt": "1716847200000",
  "objectsDiscovered": "1247",
  "objectsProcessed": "1247",
  "objectsIndexed": "1240"
}

current_job_id is set during an active discovery. objects_indexed ≤ objects_processed ≤ objects_discovered — gaps usually indicate ingest failures (inspect via Jobs).

Sync — API Reference

TriggerDiscovery

Request

dodil data

HTTP

gRPC

Response

HTTP

gRPC

TriggerIngestion

Request

dodil data

HTTP

gRPC

Response

HTTP

gRPC

GetSyncStatus

Request

HTTP

gRPC

Response

HTTP

gRPC

See also

`TriggerDiscovery`

`TriggerIngestion`

`GetSyncStatus`