A traffic spike causes embedding API rate limits and cascading retries. How would you introduce backpressure and retry budgets?

Production incident (illustrative): At 09:00, a customer-support RAG API receives a burst of 180 new questions per second after an internal launch. Each question triggers one embedding call before vector search. The organization’s embedding allocation is 6,000 requests/minute (100 requests/second averaged) and 1.2 million input tokens/minute; these values are fictional test settings, not provider limits. A typical question consumes 300 embedding tokens. Demand is 10,800 calls/minute and 3.24 million tokens/minute, exceeding both budgets. The embedding provider starts returning HTTP 429. The SDK silently retries twice; the RAG service also retries twice; the frontend retries failed requests. Work multiplies, queues grow, and p95 latency goes from 1.6 to 14 seconds. Users repeatedly resubmit while the system is still processing canceled attempts.

Root cause: Multiple independent retry loops turned a rate-limit response into retry amplification. The system lacked shared request/token admission budgets, a bounded waiting queue, deadline awareness and an explicit owner for retries. Increasing worker count made the provider overload worse. Fix this before tuning embedding similarity, vector indexing or model generation.

What this lesson covers: Q219 measured embedding latency, Q233 covered backpressure for offline ingestion, and Q276 established load-testing and bottleneck identification. Q277 focuses specifically on online query-embedding provider quotas, admission control, Retry-After, retry budgets and overload containment; Q278–280 will cover retrieval fan-out, slow dependencies and latency-budget allocation.

1. Diagnose whether this is a retry storm or a genuine quota change

• Gather one user-request correlation ID and all upstream/downstream attempt IDs: gateway → RAG orchestrator → embedding client/SDK → provider. Count total provider attempts per unique original user request, not just HTTP 429 responses.
• Inspect provider error status and structured error code. Some 429s represent temporary rate limiting; others can indicate project quota or spend-limit failures requiring operator action. Do not blindly retry every 429. Temporary 503 overload may be retriable, but invalid credentials, insufficient quota, unsupported models and permission errors are not transient backpressure signals.
• Record requested/remaining RPM and TPM, per-model or shared limits, Retry-After, rate-limit reset values where supplied, request timestamps, token estimates, model, project, account/tenant, effective quota and SDK retry configuration. Limits can apply at multiple organizational levels or to shared model pools.
• Measure offered originals/sec, accepted originals/sec, embedding attempts/sec, successful embeddings/sec, 429/503 rate, concurrent calls, queue depth and oldest queued age, per-attempt and end-to-end latency, deadlines exceeded, client cancels and spend.
• Correlate the start of latency growth with (i) provider limit exhaustion, (ii) repeated requests from clients, (iii) application retry middleware, (iv) SDK retries, and (v) a backlog persisting after incoming traffic falls.
• Pin a test traffic mix: normal and very long inputs, short/long prompt token estimates, cache-hit distributions, tenant identities and request bursts. RPM may be below the limit while TPM is exceeded, and vice versa.
• Capture one stable provider/model revision during replay and run a small controlled rate ramp with a test budget. Never recreate a retry storm against a shared production account.

2. Draw a single retry ownership boundary

• Pick one layer to own the embedding retry decision. Usually the RAG embedding adapter has the best knowledge of the total request deadline, quota, token cost and cancellation state.
• Disable SDK automatic retries when application retries are centrally managed, or explicitly count SDK attempts inside the same per-original budget. A wrapper with max_retries=2 surrounding an SDK with max_retries=2 may cause up to 3×3 = 9 embedding attempts for one logical call, before frontend retries.
• Client/browser retries must use the same request identity and be constrained by server-issued retry policy, especially if the previous request may still be running. Idempotency/deduplication is useful for request accounting but cannot make an embedding operation free.
• Classify errors as TRANSIENT_RATE_LIMIT, TEMPORARY_OVERLOAD, PERMANENT_QUOTA, AUTH_OR_PERMISSION, BAD_REQUEST, CANCELLED, or DEADLINE_EXPIRED. Never schedule retries for known permanent cases.
• Track a total retry budget: maximum attempts per logical request, absolute deadline, total allowed retry load relative to new traffic, and cost/token limits. An allowed attempt count is not permission to violate other budgets.
• If a provider reports a valid Retry-After of 30 seconds but the original user's request expires in 4 seconds, do not retry after 4 seconds. Return a clear rate-limit/unavailable response or route noninteractive work to a permitted deferred queue.
• Honor Retry-After as a minimum delay, including its valid HTTP-date or delta-seconds form when applicable; add jitter and avoid synchronized wakeups. If absent, use capped exponential backoff with jitter.
• Pass one absolute request deadline through every dependency and cancel delayed tasks when no longer useful. A per-attempt timeout is not the same as an end-to-end deadline.
• Use a single idempotent logical-work identifier across orchestration stages so metrics and budgets count originals versus attempts unambiguously.

3. Enforce admission control *before* embedding work

• Dual rate budget: meter requests and input tokens independently. Admit an embedding only when both counters/credits permit it. Respect provider-specific quota semantics instead of assuming universal rolling windows or token accounting.
• Concurrency gate: cap in-flight provider calls using a semaphore or adaptive limiter, independent of RPM/TPM. Even a request technically within the average rate budget can cause concurrency pressure during a burst.
• Token estimate: compute or conservatively estimate input tokens, reserve the expected cost at admission and reconcile actual usage when reliable provider accounting is available. Overestimating reduces utilization; underestimating produces avoidable TPM violations.
• Burst allowance: use a token-bucket or sliding-window limiter with bounded burst capacity. A token bucket can approximate a configured budget, but provider windows and shared quota enforcement may differ; calibrate against observed headers and 429s.
• Bounded queue: enqueue only work likely to finish before its deadline. Enforce maximum queue length and age. Do not create an unbounded async task for every rejected incoming user query.
• Load shedding: when no safe queue slot remains, quickly return a standard overloaded response (such as 429/503 as appropriate) with a bounded client retry hint, rather than consuming more memory and upstream attempts. Prefer an honest unavailable response to a fabricated answer without retrieval evidence.
• Fairness: reserve capacity for interactive user queries and prevent background bulk embeddings from exhausting the online quota. If bulk ingestion and online queries share credentials or a provider quota, impose explicit class budgets, priority and per-tenant caps.
• Cache with authorization: deterministic query-embedding caches can avoid repeat provider calls when text normalization, embedding model/version, configuration and data-handling requirements match. Separate embedding reuse from document retrieval authorization; cached embeddings must never bypass tenant ACLs.
• Batch carefully: where a provider supports batching, combine eligible offline or short requests to reduce request overhead, but track total token consumption; batching can lower RPM while leaving TPM unchanged and increasing interactive waiting time.
• Adaptive rate control: use provider limit headers, 429 trend, Retry-After and observed successes to decrease admission quickly under overload and increase gradually after recovery. Apply rate updates at the appropriate shared quota scope, not independently per API pod.

4. Retry scheduling algorithm and operational safeguards

• On each original request, compute deadline_at, attempts_remaining, tokens_reserved, quota_scope, and retry_budget_available.
• Before each attempt, check cancellation/deadline, then acquire both request and token credits plus a concurrency permit. Never hold scarce provider permits during a backoff sleep.
• On success, update rate-limit estimates, metrics and cache; release permits in a finally block. Record actual provider attempt count.
• On temporary 429/503, inspect Retry-After and structured error reason; backoff with random jitter, but retry only if the projected start+service duration fits the original deadline and global retry budget.
• On permanent quota/billing/auth errors, fail quickly and alert the appropriate owner. Retry loops cannot create budget or access.
• Under sustained high 429 rates, invoke client-side throttling/circuit-like admission reduction and stop retrying when overload is widespread. Avoid a circuit breaker that masks unrelated healthy scopes; key the controller to actual provider/model/quota scope.
• On timeout or cancellation, don't unconditionally resubmit a request whose provider-side completion status is unknown. Account for possible orphaned work and use safe operation-specific retry rules.
• Distinguish *retry attempts* from *unique failed requests*. A service can have fewer user-visible 5xx while consuming far more paid embedding attempts.
• Preserve source security and truthfulness: if embeddings cannot be created, do not silently use an outdated cached answer for a different user or return citations from unrelated tenant documents.

5. Beginner-friendly numerical production example

• Capacity: allocated request budget = 6,000 RPM = 100 RPS average. Allocated token budget = 1,200,000 TPM = 20,000 tokens/sec average. Typical input = 300 tokens. Token-limited sustainable request rate is 20,000/300 ≈ 66.7 RPS, *lower* than the RPM-derived 100 RPS. At 180 RPS, requested load is 54,000 tokens/sec, or 2.7× the token allowance.
• Retry multiplication: if each logical call can be attempted three times by SDK × three times by the RAG wrapper, a persistent failure could lead to nine embedding attempts per original request. At 180 originals/sec, the unconstrained worst-case upper bound is 1,620 attempts/sec before other client retries. This is an upper bound, not a measured steady-state rate, because timing, success, backoff and queueing affect observed attempts.
• Queue growth: even if the system perfectly throttles to about 66.7 successful 300-token embeddings/sec, new requests arrive at 180/sec. With no shedding, backlog grows at approximately 113.3 requests/sec, or 6,798 additional queued requests in 60 seconds. This approximation ignores burst allowance, failures and varying request sizes; it demonstrates why waiting alone cannot solve sustained excess demand.
• Admission policy (illustrative): reserve 60 RPS for online use at 300 tokens each (18,000 TPM? Correction: 60×300×60 = 1,080,000 tokens/min, below the 1.2M TPM ceiling) and leave headroom for estimation error/short bursts. A 6,000-RPM limit is not the bottleneck in this allocation; the 1.2M-TPM limit is.
• Retry policy: maximum two total provider attempts per original, a global retry budget no more than 10% of accepted original attempts for a measured interval, and 3-second end-to-end deadline. These are example engineering choices, not prescribed vendor defaults. Retry-After beyond the remaining deadline causes immediate deferral/failure, not an early retry.
• Steady burst outcome: of 180 incoming originals/sec, suppose 60/sec are admitted, 120/sec are quickly rejected or deferred, and of the 60 admitted 2/sec initially fail transiently. If all 2 retries are authorized under the budget and complete successfully, total provider attempts = 60 initial + 2 retries = 62/sec. That is a 3.3% retry ratio against originals and fits the 10% retry budget. Since retries also consume tokens, request/token reservations must account for 62×300×60 = 1,116,000 TPM, still under 1.2M TPM.
• Success rate accounting: accepted originals 60/sec can all succeed in this illustrative transient pattern, but only 60/180 = 33.3% of offered requests were served immediately; 66.7% were shed/deferred. Do not misleadingly report “100% success” solely among admitted users.
• Alternative control: if user-facing success rate needs to remain near 100%, provision greater authorized provider quota/capacity or reduce effective demand (validated caching, batching where appropriate, staged launches). Queueing more requests behind a hard 66.7 RPS token ceiling does not sustainably process 180 RPS.
• After mitigation: a hypothetical measured peak retry amplification declines from up to 9 attempts/original to ≤2, mean queue age is bounded, and p95 for *admitted* questions returns to 2.4s while rejected requests fail quickly. Report admitted and offered population latency, shed rate, cost, and grounded-answer quality separately.
• Figures are illustrative, not observed MachinoAI/provider data. The calculation presumes a typical 300-token embedding and simplified average quotas; real quotas, token variability, shared limits and burst windows must be measured.

6. Root-cause decision tree

• 429 coincides with TPM exhaustion while RPM remains available: reduce input tokens, cache identical embeddings, avoid repeated long query strings and tune token-based admission, not just request count.
• 429 coincides with RPM exhaustion but TPM is free: batch eligible requests if supported, increase request budget if authorized, or reduce duplicate calls with normalization/caching.
• 429 spikes immediately after a deployment or burst below average quotas: check burst/ramp limits, shared provider limits and distributed workers whose local rate limiters are unsynchronized. Coordinate a quota-scope controller and ramp gradually.
• Retry attempts exceed originals × configured max: inspect nested SDK, wrapper, proxy, queue worker and client retries. Confirm attempt counters propagate end to end.
• Provider 429s continue after incoming originals fall: look for retry backlog, scheduled retries, abandoned queue tasks and orphaned requests that were not canceled.
• Latency rises before provider errors: measure admission queue depth and age, semaphore waits, gateway backlog, connection exhaustion or worker event-loop stalls; rate limiting can be downstream but not the first bottleneck.
• High 503 and slow-down signals: treat as provider overload/ramp behavior per documented error codes and Retry-After, not necessarily an account quota cap.
• Budget exhausted but errors called “rate limits”: distinguish hard spending/account quota from temporary 429. Alert billing/owner rather than automatically resending.
• Cached answer recovers speed but is wrong/stale: validate source version, requester ACL and grounding separately. An embedding cache avoids computation; a cached *answer* must obey much stricter freshness/privacy conditions.
• One tenant causes cross-tenant starvation: enforce quota partitioning/fair scheduling rather than each tenant independently assuming the full provider allowance; later curriculum Q281 examines this in depth.
• Online ingest shares an embedding allocation with offline backfills: reprioritize or defer lower-priority indexing; Q233 addresses the separate ingest queue mechanics.
• A new retrieval source causes tail latency without embedding 429: inspect source fan-out/critical path under Q278, not indiscriminately lower embedding concurrency.

7. Engineering solution and rollout

• A. Trace originals and attempts: one root request ID plus unique attempt IDs across SDK/proxy/orchestrator; record status/error code, request and token budget state, retry count, Retry-After, queue wait and cost.
• B. Inventory effective limits: model/project/org RPM and TPM, provider burst allowances, quota sharing, request/token estimation and contractual capacity. Configuration should be versioned and operator-owned.
• C. Central admission broker: distributed request/token limiter keyed by actual quota scope; bounded online queue, deadline/cancellation checks and in-flight concurrency permit; explicit high-priority interactive and lower-priority batch lanes.
• D. One retry owner: disable uncontrolled SDK retries, set total attempts and wall-clock budget, parse Retry-After, use jitter and cap retry load per scope. Avoid nested retries in API clients, worker queues and frontend.
• E. Controlled rejection: respond quickly with a semantically correct 429/503 and safe retry information or create deferred work only where the product supports it. Never return unsupported RAG content as a “successful” fallback.
• F. Adaptive recovery: decrease admission on overload signals; ramp back up slowly after sustained successes, avoiding synchronized retry waves. Evaluate total token cost and limit-reset behavior.
• G. Capacity optimization: use authorized caching, input deduplication and carefully selected embedding batch sizes. If offered demand permanently exceeds capacity, purchase/add authorized capacity or reduce demand rather than masking the mismatch.
• H. Canary test: replay short bursts and sustained load with scripted 429, 503, Retry-After values, quota exhaustion, cancellation and multi-pod concurrency. Measure user-visible acceptance, completion quality, shed fraction, attempts/original and p95/p99.
• I. Rollback: feature-flag admission, retry counts and adaptive rules. Ensure rollback cannot recreate uncontrolled retries, disabled ACL checks or stale document responses.

8. Trade-offs and failure scenarios

• Bounded shedding vs waiting: shedding sacrifices short-term acceptance but prevents unbounded tails and protects healthy traffic; waiting may be reasonable for brief bursts if deadlines and queue age stay bounded.
• High retry allowance: may smooth rare transient 429s but amplifies an overloaded provider; a 10% global retry allowance is an *example* drawn from SRE practice, not a universal requirement.
• Very low retry allowance: can prematurely fail a request that would succeed after a short server-specified wait; calibrate against measured transient behavior and user deadlines.
• Static limits: easy to reason about, but may underutilize quota when provider allocations increase or overrun shared quotas when usage changes. Keep a provider-informed feedback loop.
• Adaptive controllers: respond to overload and changing capacity but can oscillate when all API pods update independently. Coordinate by real shared quota scope and use damped adjustments.
• Token forecasts: uncertainty can cause overadmission; include margin and reconcile actual token accounting.
• Local per-pod token buckets: each pod may admit the entire quota if there is no shared coordination. Avoid accidental N× budget multiplication on autoscaling.
• Caching: cuts identical query costs, but poor normalization can merge semantically different questions; keep model-version and intended text semantics.
• Batching: saves RPM and overhead but cannot evade token quotas; can worsen interactive tail latency.
• Provider SDK conveniences: automatic retries may be helpful in isolation but hide multiplied attempts when wrapped by more retrying layers.
• Backoff without jitter: synchronized clients retry at the same reset time, recreating the stampede.
• Premature retry vs Retry-After: violates temporary-limit guidance and increases rejection; if the hinted delay exceeds the request deadline, stop.
• False-green SLO: success rate measured only among accepted requests hides massive shedding. Report offered, accepted, completed, rejected and deferred groups.
• Failure isolation: embedding overload should not indiscriminately prevent safe nonembedding API routes; keep per-dependency budgets and request classifications.
• Security: do not bypass tenant/RLS or return cached retrieved facts from a different user's permission context under overload.

9. Prevention, alerts and on-call runbook

• Build a golden surge test with configured RPM/TPM, realistic token-length distribution, SDK retry configuration, per-tenant traffic, ramp spikes, constrained capacity, cancellations and provider response headers.
• Inject temporary and permanent 429, 503, malformed/missing Retry-After, rate-limit reset, slow responses, quota-scope changes, and callback cancellation. Verify bounded retries and no early retry.
• Alert on embedding_attempts/originals, retry-attempt ratio, 429/503 trend, available RPM/TPM, queue age, permit wait, shed fraction, provider cost and fraction of requests outside end-to-end deadlines.
• Track offered, admitted, completed, rejected, deferred RPS separately; for admitted requests also track citation validity, retrieval coverage and grounded-answer success.
• Set a safety condition that a canceled/expired user request cannot spawn another embedding attempt, and that known permanent quota/permission errors never retry.
• For an incident: identify whether RPM, TPM, ramp behavior or spend quota is binding → find first nested retry amplification → disable duplicate retries → cap admission before provider calls → bound waiting → honor Retry-After and jitter → coordinate scope-wide recovery → check accepted-versus-offered user outcomes.
• Rebaseline after provider quota changes, embedding model swaps, new prompt/chunk sizes, new API instances and shifts in online/offline traffic; do not assume quota is unchanged just because application code is unchanged.
• Review security and privacy for cache keys, embedding logs and test inputs; pressure must not become a reason to leak cross-tenant content.
• Keep release gates on user-visible success *and* honest failure, retry amplification, bounded queues, cost and groundedness—not just reduced 429 counts.

Interview takeaway: When embedding 429s create a retry cascade, the fix is rate and token admission control before the provider call, one retry owner, per-original and global retry budgets, Retry-After-aware jitter, deadline and cancellation propagation, and bounded queues/load shedding. Correctly distinguish RPM from TPM and temporary rate limits from hard account quotas. An admission system that safely rejects excess work is healthier than a seemingly successful service that endlessly retries and times out.

Research references

• OpenAI, *API rate limits and error mitigation*: https://developers.openai.com/api/docs/guides/rate-limits
• OpenAI Cookbook, *How to handle rate limits*: https://developers.openai.com/cookbook/examples/how_to_handle_rate_limits
• Google SRE, *Handling Overload* (client-side throttling and retry budgets): https://sre.google/sre-book/handling-overload/
• Amazon Builders Library, *Timeouts, retries and backoff with jitter*: https://aws.amazon.com/builders-library/timeouts-retries-and-backoff-with-jitter/
• RFC 6585, *429 Too Many Requests*: https://www.rfc-editor.org/rfc/rfc6585