How do you design token-aware GPU quota allocation and chargeback across teams and tenants in production LLM inference?

Token-aware GPU quota allocation controls how much LLM inference work each team or tenant is allowed to submit, run concurrently and consume over time. Chargeback attributes the actual cost of shared serving infrastructure to the teams that benefit from it; showback reports those costs without necessarily invoicing them. In a shared vLLM or comparable serving fleet, token accounting must be combined with GPU capacity, latency, KV-cache pressure, workload priority and accurate completion records. A count of requests alone is not enough: one 200-token chat and one 80,000-token long-context request have very different serving costs.

This lesson continues Question 251 (tenant fairness and priority), Question 260 (GPU cost per SLO-compliant request), Question 267 (inference observability), and Question 275 (mixed interactive/batch GPU scheduling). Its distinct contribution is quota policy, precise metering, usage reconciliation, fair shared-infrastructure cost allocation and financial governance.

1. Separate quota, admission control, rate limit and chargeback

• Entitlement/quota: a tenant's contracted or organizational allowance, such as 10 million input tokens/day, 2 million output tokens/day, 25 concurrent requests and four reserved GPU slots.
• Rate limits: protection against bursts: requests/minute, estimated input tokens/minute and maximum ongoing generated-token work.
• Admission control: decides whether a particular request fits its token budget, context-length limit, concurrency, memory and deadline. Rejecting an impossible request early is better than consuming GPU work before timing out.
• Scheduling/fairness: orders accepted work across tenants. Fair scheduling cannot enforce a monthly budget if no accounting and quota mechanism exist.
• Metering: measures what actually happened: attempts, prompts, cached/recomputed tokens, generated tokens, errors, retries, GPU-seconds and allocated GPU-hours.
• Chargeback/showback: applies a published policy for distributing shared costs to cost centers or customers.

Never confuse namespace GPU quotas with per-token consumption. Kubernetes ResourceQuota can limit a namespace's requested extended resource (for example \requests.nvidia.com/gpu\), but cannot directly meter one organization's tokens in a multi-tenant shared model process. That requires an authenticated inference gateway and a ledger.

2. What makes LLM quota accounting difficult?

• Input versus output: input prefill and autoregressive output decode have different GPU cost profiles. Pricing both as one undifferentiated “token” may systematically cross-subsidize some teams.
• Prefix caching: cached prompt tokens may avoid significant compute, but still consume memory and may need separate policy/pricing. The cache-hit count, provider billable cached tokens and compute reuse are different facts.
• Long context: higher memory/KV pressure can lower safe concurrency even when request counts or output tokens remain low.
• Streaming and cancellation: clients may cancel after some tokens; usage must reflect observed generated work, terminal status and agreed billing rules, not assume every started request was completed.
• Retries: a gateway may attempt the same logical user request multiple times. Avoid charging a customer twice by accident while still accounting for the real compute consumed.
• Model/hardware heterogeneity: 100k tokens on a small efficient model and 100k tokens on a very large model have radically different unit costs and latency envelopes.
• Multi-tenant isolation: trust verified auth-derived tenant identity; do not allow the caller to choose an arbitrary chargeback group in request headers.

3. Define a practical hierarchical quota

An organization may establish:

1. Organization budget: monthly currency spending cap or committed GPU-hour entitlement.
2. Team/project quota: daily token envelopes, maximum GPU-seconds or credit balance.
3. Tenant/application rate limits: input-token/min, generated-token/min, requests/min and concurrency.
4. Per-request limits: maximum input context length, maximum output tokens, deadline, approved models and region.
5. Class limits: strict interactive requests have protected capacity; batch tasks have deferrable queue budgets and lower priority.
6. Burst rules: an agreed small token burst bucket, paid overage, or documented rejection/429 policy.
7. Safety reserve: a fraction of the whole fleet retained for baseline SLO and node failures, not sold as simultaneously guaranteed to every tenant.

Use consistent hierarchical admission: a request must pass org, team, app and per-request constraints. A team quota is not an implicit permission to bypass model-safety, residency or hardware limits.

4. Numerical example: token quota and rate limits

Suppose Team A has:

• Daily allowance of 2,000,000 input tokens and 500,000 generated tokens.
• Token throughput allowance of 50,000 estimated input tokens/minute.
• Output generation allowance of 10,000 tokens/minute.
• Maximum 20 concurrent generations.

A new request contains 4,000 input tokens and sets \max_tokens=1,000\ output tokens:

• Admission reserves 4,000 input tokens and up to 1,000 output tokens against conservative quota/rate budgets, subject to the current policy.
• Actual response uses 4,000 input tokens and 300 generated tokens.
• Once final usage is committed, refund or release the unused 700 output-token reservation. The gateway must reconcile incomplete streams separately.
• If Team A had already used 499,000 output tokens today, reserving 1,000 leaves zero remaining headroom, so the next request requiring output must wait/reduce its declared maximum if approved or receive a clear limit response.

Why reserve before generation? Without estimated output reservations, 100 concurrent requests can all pass a simple “remaining daily output tokens” check, then overspend their shared allowance as decode proceeds. Use atomic reserve/commit/release updates or an equivalent distributed quota service.

Caution: max output-token limit is an upper bound on generation, not a prediction of actual output. Different models may expose visible and non-visible output/reasoning usage differently. Follow the deployed API's authoritative usage schema.

5. Numerical example: cost allocation with different token classes

Suppose a month of self-hosted inference incurs $12,000 in combined GPU infrastructure cost for one shared serving pool. Usage ledger totals:

• Team A: 20 million uncached input tokens, 5 million cached input tokens, 4 million output tokens.
• Team B: 10 million uncached input tokens, 15 million cached input tokens, 6 million output tokens.

To illustrate a transparent *internal weighted-token* policy, assume weights:

• Uncached input = 1 charge unit/token.
• Cached input = 0.2 charge unit/token.
• Output = 4 charge units/token.

Team A weighted units: \20 + (5 × 0.2) + (4 × 4) = 37 million units\. Team B weighted units: \10 + (15 × 0.2) + (6 × 4) = 37 million units\.

Total = 74 million units; each team gets 50% × $12,000 = $6,000 under this particular policy.

These weights are fictional policy choices, not a universal GPU cost model or provider prices. Real unit weights should come from measured model/GPU prefill/decode cost profiles, cache reuse, committed capacity and published contractual terms. You may choose to allocate a guaranteed reserved baseline by contracted GPU-hours and only allocate incremental variable cost by token weights, rather than sharing all GPU charges purely by token volume.

6. Chargeback policy options and trade-offs

Option A: Per actual GPU-second

• Use attributable compute duration and GPU device allocation where instrumentation makes it credible.
• Better alignment with time-based infrastructure invoices.
• Shared-batch kernels and overlapping requests make exact per-request GPU-seconds ambiguous; allocation must be approximate or use a documented cost model.

Option B: Weighted token cost

• Simple, explainable: \charge = uncached_input_tokens × rate_in + cached_input_tokens × rate_cached + output_tokens × rate_out\.
• Different model families and hardware SKUs need different rate cards; token ratios alone do not capture every long-context/low-concurrency memory cost.

Option C: Reserved capacity + usage

• Allocate contracted GPU-hours, dedicated replicas or minimum guarantees as a fixed fee; attribute dynamic burst costs by metered tokens or incremental GPU work.
• Often fairer for teams with very strict SLO guarantees and large standing capacity requirements.
• Requires clear policy for idle reserved GPU time.

Option D: Provider pass-through

• For hosted inference, apply model/provider bills and actual usage reports, including cached input and output/reasoning components where supported.
• Do not double-add cache-write tokens when already included in input-token categories. Pricing and usage schemas differ between vendors and model versions.

Recommendation: use a two-part internal chargeback for shared GPU fleets: capacity reservation charge for guarantees plus a metered variable charge for actual work. Show both to teams and disclose any shared overhead allocation rule.

7. Design the authoritative usage ledger

At request ingress, create an immutable logical request ID, verified org/team ID, model/config version, quota policy version, request class, region and admission decision.

The ledger should record:

• \logical_request_id\, \attempt_id\, \tenant_id\, \team_id\, \model_revision\, \region\, \request_class\, \cost_center\.
• Accepted/rejected status, start/end times, terminal outcome (completed, cancelled, deadline, error), and whether any stream produced visible output.
• Prompt tokens, cached input tokens when supported, generated tokens, any provider-reported reasoning/output detail, and whether values are measured or estimated.
• Versioned rate card, internal cost estimate, infrastructure allocation period, and separate adjustments/refunds.
• Idempotent commit key per attempt and separate logical-request reconciliation to prevent double billing.
• Durable audit trail for late-arriving metering records, corrections and disputes.

Important: vLLM exposes aggregate counters like \vllm:prompt_tokens_total\ and \vllm:generation_tokens_total\ via Prometheus. These are useful to reconcile fleet totals but are generally insufficient as a canonical tenant billing ledger, because aggregate series need not carry verified tenant identity or exactly-once logical request boundaries. Collect authorized request-level metering at the gateway/engine integration and retain sensitive identifiers only in controlled storage—not as high-cardinality Prometheus labels.

8. A safe two-phase quota accounting algorithm

\\\`text

1. Authenticate request; resolve tenant/team/cost center server-side.
2. Validate model, region, policy, prompt size, declared output cap.
3. Atomically reserve input/output budget and concurrency slot.
4. Admit to an eligible GPU/queue only if live capacity and deadline permit.
5. Record one attempt with a unique attempt ID.
6. As streaming progresses, update bounded provisional work metering.
7. At terminal completion/cancel/error, commit authoritative measured use.
8. Release unused reserved budget and concurrency.
9. Reconcile orphaned attempts from worker death with timeout and audit rules.
10. Deduplicate external customer billing by logical request ID and explicit retry policy.

\\\`

Do not write a distributed quota counter without atomicity. Concurrent requests can oversubscribe a quota during races. For financial records, use durable transactional ledgers or properly fenced reservation/commit operations and publish a documented policy for retries/cancellations. If a worker dies before reporting final usage, classify the record estimated/unresolved and reconcile rather than assuming zero tokens consumed.

9. Kubernetes quota example and limitations

For a dedicated team namespace requesting up to four GPU resources with a device-plugin resource named \nvidia.com/gpu\:

\\\yaml apiVersion: v1 kind: ResourceQuota metadata: name: team-gpu-quota namespace: team-a spec: hard: requests.nvidia.com/gpu: "4" \\\

This sets a namespace quota on requested GPU resources using Kubernetes extended resource quota rules. It does not limit per-request tokens, guarantee that four GPUs are physically available, allocate a fraction of a GPU to a shared inference service, or provide a bill. Dynamic Resource Allocation (DRA) quotas have their own resource names/rules; follow the cluster's allocation model.

If teams share a centrally deployed gateway and inference engine under one namespace, its GPU quota applies to the engine namespace, not automatically to end tenants. Enforce tenant quotas in the application control plane.

10. Design fairness and avoid quota gaming

• Rate-limit by estimated token work and active generation, not just requests/second.
• Budget input and output separately: output decode is often more expensive per token and less predictable.
• Consider cached-token discounts only if the inference cache saves real work under the chosen deployment and policy.
• Prevent a user from selecting an arbitrary team for billing. Authenticated identity and approved project membership must determine the cost center.
• Version rate cards and meter corrections; cost calculations must be reproducible.
• Give teams timely usage reports and dashboards: remaining allowance, daily burn, projected month-end spend, per-model cost and top workloads.
• Protect against retry amplification: a tenant may consume compute in repeated failed attempts even when customer billing applies only once.
• Enforce fairness for small tenants even when a large tenant has unused quota; unused budgets do not justify blocking all smaller tenants.
• Never include raw prompts, tool arguments or private data in financial telemetry unless absolutely necessary with authorization and retention controls.

11. Production incident: “free” retries create a surprise bill

Incident: Tenant A's application times out on long generations. The gateway retries requests twice without preserving a logical request ID. The metering system counts only successful responses and bills Team A for one completion, but the backend performed three partial decodes. GPU invoices rise 35% despite stable user-visible requests; other tenants' p95 TTFT degrades because retries fill KV memory.

Root cause:

• No linkage from logical request to downstream attempts.
• Failed and cancelled attempts omitted from capacity metering.
• Quota reserved only on the first attempt and not on aggregate compute budget.
• No tenant-level limits on parallel/retried generation.

Engineering fix: propagate logical request ID and unique attempt IDs, enforce bounded retry policy, meter all actual GPU work even on failure, and explicitly distinguish internal infrastructure chargeback from user billable completions. Add admission limits for token-work across retries and showback dashboards for expensive unsuccessful work.

Prevention: reconcile gateway request logs with vLLM aggregate token counters and cloud GPU invoices, test cancellation/retry races, and alert when billed infrastructure cost diverges from attributed usage by more than the documented tolerance.

12. Monitor reconciliation and business outcomes

Track:

• Per tenant/team: admitted/rejected requests; input/uncached/cached/output tokens; token-rate/active concurrency; quota reserves, commits, refunds and budget exhaustion.
• Shared fleet: GPU-hours by hardware/region, token throughput, KV occupancy/preemptions, actual idle capacity, failed decode work and SLO-compliant goodput.
• Financial: allocated versus unallocated GPU costs, unit cost by model/workload, reserved baseline charges, rate-card version, contested adjustments and invoice drift.
• Reliability: admission denial precision, p95 TTFT/TPOT by class, fair-share service ratios and timeout/retry amplification.

Reconciliation: compare (1) durable request-ledger token totals, (2) vLLM engine aggregate counters over the same time interval, and (3) infrastructure billing and reserved capacity costs. Expect some definitional differences from prefixes, retries, cancellation, cache and telemetry losses; document each reconciliation adjustment. Treat a mismatch as an investigation trigger rather than automatically modifying customer invoices.

13. Interview-ready answer

“I'd enforce hierarchical org/team/application quotas using an authenticated gateway, reserve estimated input/output-token budgets before admission, and settle actual usage from a durable per-request ledger. I'd track retries as distinct attempts under one logical request and reconcile cancelled/failed GPU work separately from customer billable output. Kubernetes GPU quotas can limit dedicated namespaces but cannot solve tenant token quotas inside a shared inference engine. For cost allocation I'd combine reserved GPU capacity commitments with weighted, model-specific actual token usage, expose showback dashboards, and reconcile ledger totals against vLLM metrics and cloud bills.”

Research references:

• Kubernetes — Resource Quotas and GPU Extended Resources
• vLLM — Production Metrics
• OpenAI — Prompt Caching Usage and Accounting
• OpenTelemetry — Generative AI Usage Conventions
• FinOps Foundation — Allocation