One malformed document repeatedly crashes a processing worker. How would you implement poison-message handling and a dead-letter queue?

Production incident (illustrative): A RAG service ingests customer PDFs from a queue. One malformed PDF crashes the parsing subprocess. The message becomes visible again, another worker picks it up, and the same crash repeats. Valid documents wait behind repeated failures, the queue age rises, and a dashboard reports a generic ingestion failed error. This is a poison-message problem: an input repeatedly fails normal processing and needs controlled isolation.

1. Diagnose before retrying

• Identify the document: Correlate tenant_id, document_id, source version, message ID, processing stage, parser version, attempt count, and error fingerprint. Avoid logging confidential document contents.
• Find the failure boundary: Did the worker die at PDF validation, decompression, image rendering, OCR, chunking, or embedding? Inspect exit codes, OOM events, resource usage, stack traces, and container restarts.
• Check reproducibility: Replay the file in an isolated resource-limited environment using an approved test copy. Determine whether it fails deterministically, or only during a temporary resource/API outage.
• Inspect the delivery mechanics: Compare queue receive counts, visibility timeout, retry delay, acknowledgment sequence, and whether one problematic document prevents other messages from progressing.
• Classify the error: Corrupt PDF, unsupported encryption, decompression bomb, invalid format, or consistently failing parser input is likely permanent without intervention. An embedding 429, a temporary object-storage outage, or a brief database timeout is usually transient.

2. Implement poison-message handling

1. Create a dead-letter queue (DLQ): Route events that exceed an appropriate retry threshold from the primary ingestion queue to a restricted quarantine queue. In Amazon SQS, a redrive policy's maxReceiveCount controls this transfer; in RabbitMQ, dead-letter exchanges and delivery limits provide analogous mechanisms.
2. Use bounded retries: Retry transient failures with exponential backoff and jitter; cap attempts and total processing time. For confirmed permanent errors, quarantine immediately using a reliable broker-specific rejection or DLQ workflow rather than exhausting identical attempts.
3. Isolate untrusted parsers: Run PDF processing with CPU, RAM, runtime, page-count, rendered-pixel, and decompression limits. A crashing native parser should not take down the queue consumer or other jobs.
4. Preserve an investigation record: Store the safe source reference, checksum, document version, error class, stage, parser version, last attempt, and trace ID. Keep actual content in controlled object storage, not in broad-access queue diagnostics.
5. Update document state accurately: Mark the file QUARANTINED / FAILED_NEEDS_REVIEW, not SEARCHABLE or silently SUCCESS. Let other documents proceed.
6. Alert and triage: Notify operators on unusual DLQ growth, repeated crash fingerprints, or high-value tenant failures. Group matching signatures to avoid hundreds of duplicate incidents.
7. Replay after repair: Fix the parser, transform the input safely, or correct upstream data. Replay selected DLQ messages at a controlled rate, with deterministic document-version IDs and idempotent upserts. Confirm the source version is still current before publishing.

Broker nuance: DLQ movement and message acknowledgment semantics vary by platform. Set sufficient retention, verify destination accessibility, avoid a DLQ routing loop, and test whether a failed dead-letter publication can lose a message. For FIFO workflows, routing a message away can affect ordering; explicitly address any per-document sequencing guarantees.

3. Beginner-friendly numerical example

Suppose a batch contains 10,000 documents and 20 malformed PDFs. The queue retries every failing document 10 times, with each failed parse wasting 30 seconds of worker CPU time.

• Failed attempts: 20 × 10 = 200.
• Worker time wasted: 200 × 30 = 6,000 seconds = 100 worker-minutes.
• With an error classifier and a policy that quarantines each reproducibly malformed document after 2 attempts, failures fall to 20 × 2 = 40 attempts.
• Wasted time becomes 40 × 30 = 1,200 seconds = 20 worker-minutes.
• Illustrative saving: 100 − 20 = 80 worker-minutes and less contention for the 9,980 valid documents.

These are illustrative work totals, not wall-clock delays; parallel workers and actual broker behavior change elapsed time. Do not use a universal two-attempt threshold: configure limits by error type and infrastructure.

4. Practical recovery and operational checks

• Quarantine the identified document without deleting its source blob.
• Confirm the queue and worker fleet are healthy and good documents are draining.
• Inspect the file in a sandbox and determine whether the cause is malicious/malformed input, parser incompatibility, or a resource cap.
• Test a fixed parser or sanctioned conversion in staging using the same file signature.
• Replay one quarantined document version first, check chunk completeness, retrieval visibility, and that its status transitions to SEARCHABLE only after validation.
• Gradually replay the remaining matching failures with concurrency limits. Never blindly redrive a large DLQ into an unresolved defect.

5. Trade-offs and prevention

• Low retry thresholds: Fast quarantine and low wasted work, but can wrongly sideline recoverable transient errors.
• High retry thresholds: More chances of recovery, but repeated crashes consume capacity and increase queue latency.
• Immediate quarantine: Good for deterministic parser failures, but requires reliable error classification.
• Parser sandboxing: Adds startup overhead, but protects shared ingestion infrastructure.
• Long DLQ retention: Helps investigation, but increases storage and privacy obligations; follow retention and deletion policies.

Production signals: dlq_message_count, failures by stage and fingerprint, parser crash rate, attempt count, oldest unprocessed message, queue drain rate, quarantine age, replay success rate, and percentage of documents not searchable due to quarantine. Add fault-injection tests with corrupt PDFs, oversize images, encrypted PDFs, and crash-after-write cases.

Interview takeaway: Don't retry one unprocessable document forever. Classify failures, cap retry effort, sandbox parsing, quarantine poison messages with traceable metadata, and selectively replay after a verified fix—while allowing valid documents to continue.

References:

• Amazon SQS — Using dead-letter queues: https://docs.aws.amazon.com/AWSSimpleQueueService/latest/SQSDeveloperGuide/sqs-dead-letter-queues.html
• Amazon SQS — Capturing problematic messages: https://docs.aws.amazon.com/AWSSimpleQueueService/latest/SQSDeveloperGuide/capturing-problematic-messages.html
• RabbitMQ — Dead Letter Exchanges: https://www.rabbitmq.com/docs/dlx