A connector stops syncing documents without raising any exceptions. How would you detect this silent failure?

Production scenario: A SharePoint/Drive connector reports RUNNING or even SUCCESS, yet newly edited documents stop appearing in a RAG knowledge base. No exceptions fire because the worker is alive and its requests succeed, but the data is no longer progressing. The key distinction is liveness (the process is alive) versus freshness and progress (source changes have reached searchable storage).

1. What would you detect first?

1. Last successful sync versus last useful sync: Record the time the scheduler last completed, last new/updated source event observed, last committed destination checkpoint, and last document made searchable.
2. Expected source activity: Compare new source changes with changes detected by the connector. A zero-record sync can be perfectly healthy for an unchanged source; don't alert on zero output without checking upstream activity or source-change expectations.
3. Cursor progression: Track the source watermark, page/continuation token, CDC offset, and committed checkpoint. A cursor that never advances while the source changes is suspicious.
4. End-to-end freshness: Measure searchable_at - source_updated_at for each eligible document. Alert when the p95 or oldest pending source change exceeds your source-specific freshness SLO.
5. Heartbeat plus progress: A heartbeat detects dead workers; a progress watchdog checks whether meaningful work has moved forward. They catch different failure modes.

For example, Airbyte documents heartbeats for unresponsive sources and destinations, while Debezium describes both source lag and time-since-last-event signals. A heartbeat alone cannot prove that expected source changes reached your vector index.

2. Root-cause investigation

Start by reproducing with one known, authorized canary document, updating it at the source and tracing its version.

• Scheduler and worker: Was the job scheduled, claimed, started, and completed? Is the worker merely heartbeating while stuck on pagination or backoff?
• Credentials and permissions: Has a token scope narrowed, a folder moved, or a service account lost access? Some APIs return valid empty pages for inaccessible or newly excluded content.
• Incremental-sync state: Was a cursor accidentally advanced past records? Are timestamps rounded, timezones misinterpreted, or updates sharing the same cursor value omitted because the query uses > instead of a safe tie-breaking strategy?
• Pagination and API quotas: Are continuation tokens ignored, expired, or cyclic? Are 429 responses handled with bounded retries rather than silently treated as empty results?
• Filtering and transforms: Did a folder filter, MIME allowlist, deduplication rule, or parsing change start dropping all incoming records?
• Queue, destination, and indexing: Are source records observed but not acknowledged downstream? Check queue backlog, destination commits, dead-letter counts, document versions, and active-index visibility.
• Status reporting: Is SUCCESS set at the job level despite zero processed records, missing partitions, or uncommitted state?

Important: Never blindly reset or skip a source cursor before recording the last safely committed checkpoint and affected time range. You could create gaps or duplicates.

3. Beginner-friendly numerical example

A company expects SharePoint changes to become searchable within 30 minutes.

At 10:00, the source contains 120 updated documents since the 09:00 checkpoint, but the connector reports successful syncs at 09:15, 09:30, 09:45, and 10:00 with zero records loaded.

• Source changes observed independently: 120.
• Connector changes loaded: 0.
• Unreplicated changes: 120 - 0 = 120.
• Oldest outstanding update at 09:05 is 55 minutes old at 10:00, exceeding the 30-minute freshness target by 25 minutes.

These figures are illustrative. The independent source count and connector count must refer to the same scope, change window, and treatment of updates/deletes; raw event totals may not equal distinct document versions.

4. Engineering solution

1. Persist a durable sync-run ledger keyed by connector, source partition, cursor range, and run ID. Record examined records, emitted records, destination acknowledgments, committed checkpoints, and observed maximum source modification time.
2. Run a separate watchdog that compares source-side modification counts or a known canary with destination/index progress. Do not depend solely on worker-reported SUCCESS.
3. Expose metrics such as seconds_since_last_successful_sync, seconds_since_last_committed_progress, source_to_index_lag_seconds, cursor_stalled, records_emitted, records_committed, consecutive_empty_syncs, and dead_letter_count.
4. Create source-aware alert rules: an empty run is fine on a quiet source; a stale cursor with confirmed upstream changes, prolonged queue age, or a missing canary is not.
5. Recover using bounded overlap windows or a documented resnapshot/backfill starting at the last destination-acknowledged checkpoint. Use stable document IDs, source versions, and idempotent writes to tolerate redelivery.
6. Validate repairs by comparing a source manifest with indexed document/version IDs and testing authorized retrieval—not by checking connector status alone.

5. Trade-offs and prevention

• Polling source changes provides a strong independent signal but consumes API quota. Use sampled partitions, adaptive cadence, or event/CDC signals when available.
• Canary files detect breaks in the whole path but won't prove coverage for all folders, tenants, or file types. Combine them with periodic reconciliation.
• Aggressive no-progress alerts generate false positives on quiet sources and during approved maintenance. Use per-source activity baselines, thresholds, and suppression windows.
• Checkpoint safety: Advance the durable cursor only after the destination has acknowledged the relevant records. This prevents silent data loss across crashes.
• Maintain connector dashboards, on-call ownership, runbooks, and scheduled sync-resumption drills.

Interview takeaway: A connector can be alive, exception-free, and still stale. Monitor independent source activity, committed cursor progress, and source-to-search freshness; alert on unexplained lack of progress, then recover safely from a verified checkpoint.

References (additional research):

• Airbyte — Heartbeats: https://docs.airbyte.com/platform/understanding-airbyte/heartbeats
• Airbyte — Protocol and state/checkpointing: https://docs.airbyte.com/platform/understanding-airbyte/airbyte-protocol
• Debezium — Monitoring source lag and time since last event: https://debezium.io/documentation/reference/stable/operations/debezium-platform.html