How do you reconcile LLM inference token usage, GPU infrastructure costs, and billing discrepancies across tenants?

LLM inference billing reconciliation verifies that three records of activity agree closely enough to explain, audit and correct the cost attributed to each tenant: (1) request-level usage events, (2) inference-engine operational telemetry, and (3) infrastructure/provider invoices. Their totals are not supposed to be identical units: request events may count billable tokens, engine counters measure work performed, and cloud bills measure time-based GPU resources plus storage, networking, commitments and discounts. A sound reconciliation system links these views through explicit definitions and maintains a documented unallocated-cost and variance ledger.

Prerequisite distinction: Question 276 designs hierarchical quotas, metering fields and chargeback policy. This question focuses on month-end and daily close operations, data-quality tests, exact discrepancy investigation, cost normalization, immutable financial corrections and invoice dispute handling.

1. Why totals differ even when nothing is broken

• Different units: input/output tokens are not GPU-hours; prefill and decode cost per token varies with context, batching, hardware, caching and utilization.
• Different scopes: the engine may serve batch jobs, shadow traffic, health checks, internal evaluations and retries that the customer ledger intentionally does not bill.
• Different clocks: gateway timestamps, inference-completion timestamps, cloud billing timestamps and invoice settlement can cross midnight or a billing period boundary.
• Different notions of success: a gateway may count one *logical request* while an engine performs multiple backend *attempts*; timed-out or cancelled work still consumes GPUs.
• Caching and token categories: input-token totals commonly include cached input as a subset. Reasoning output may be a subset of output tokens. Summing a subtotal with its parent double-counts.
• Hardware idle time: fixed or reserved GPU capacity accrues cost even when no tokens are processed.
• Shared costs: regional routers, replicas, object storage, networking, logging and observability often lack a unique owner.
• Pricing and credits: discounts, commitments, credit adjustments, taxes, support charges and amortized costs may arrive in different datasets or revisions.
• Telemetry gaps: restarts, missing final stream usage, lost events and poorly labeled metrics lead to real measurement uncertainty.

Do not force token totals to equal dollar totals. Reconcile each source within its own semantic domain, then apply a versioned, independently auditable allocation model.

2. Three independently controlled data sources

A. Financially authoritative request ledger For each authenticated original request, keep immutable logical-request ID, tenant/project/cost-center identity, model, API version, source region, submitted and completed times, billable input/output usage, cached/reasoning subsets where known, pricing/rate-card version, charge/refund and final status.

Distinguish each *backend attempt* using a separate attempt ID. Retried attempts can consume GPU work even if policy bills one completed logical request. Store provenance: server-measured, provider-reported, estimated, corrected or missing.

B. Operational usage and capacity telemetry Capture vLLM-compatible \vllm:prompt_tokens_total\ and \vllm:generation_tokens_total\ as counters, request success/finish reason, queue/cancellation metrics, model/region/pod revision, GPU running hours and fleet-idle capacity. Use the deployed engine's actual \/metrics\ exposition because names and labels vary by version.

Operational counters cannot automatically supply tenant billing events. Avoid tagging Prometheus time series with individual request IDs or unbounded tenant identifiers.

C. Supplier financial statement Ingest cloud provider detailed usage and cost exports or invoices with billing-account ID, resource ID, SKU, region/zone, usage period, price, discounts, commitments, credits, adjustment records and currency. Where supported, preserve raw versus amortized/effective cost semantics rather than quietly mixing them.

FinOps cost allocation guidance recommends consistent owner/account/project/tag metadata and transparent rules for shared costs; tags may be missing or introduced too late for prior charges.

3. Beginner-friendly numerical example: trace the mismatch

For one day, suppose a ledger contains:

• Team A billable: 1.0 million uncached input tokens, 0.2 million cached input tokens and 0.3 million output tokens.
• Team B billable: 0.6 million uncached input tokens, 0.4 million cached input tokens and 0.2 million output tokens.
• Therefore billable input = \1.0 + 0.2 + 0.6 + 0.4 = 2.2 million\, billable output = \0.3 + 0.2 = 0.5 million\.

Engine counters over the matching, normalized time window report:

• 2.5 million prompt tokens and 0.58 million generated tokens.
• Raw difference = 300,000 input and 80,000 output tokens.

A drill-down matches:

1. Shadow/integration-test attempts: 150,000 input, 40,000 output.
2. Client-cancelled or timed-out backend attempts: 100,000 input, 30,000 output.
3. Internal retried work on logical requests: 50,000 input, 10,000 output.

Input explanation: \150k + 100k + 50k = 300k\. Output explanation: \40k + 30k + 10k = 80k\.

The engine work reconciles in this simplified example: the customer billable ledger intentionally excludes these classes. However, a cancellation policy may bill some partial completed usage; inspect the actual contract rather than assuming every failure is free.

Infrastructure example: the cloud cost export shows $1,500 of GPU costs and $150 of other allocatable support/storage/network costs. Assume $300 in already-identified fixed guaranteed-capacity allocation is charged to Team A and the remaining $1,350 is split using approved workload weights:

• A weighted share 60% → \$810\ variable allocation + \$300\ fixed = $1,110.
• B weighted share 40% → $540.
• Check \$1,110 + $540 = $1,650\.

These are invented numbers and policies. They illustrate that matching engine tokens does not directly determine a unique dollar split: the documented contract and measured resource attribution choose the weights.

4. The close pipeline: raw, normalized, matched, explained, published

1. Collect immutable raw exports from gateway, worker/event queue, Prometheus rollups and provider billing datasets; record source revision, extraction time and checksum.
2. Normalize identity and time: map service account/resource to owner; fix UTC boundaries, request start versus completion semantics and late-arriving settlements.
3. Normalize token taxonomy: input total including cached, output total including reasoning if reported; preserve cache-read/cache-write and modality subsets *without adding them again*.
4. Dedupe and link: one logical user request → one or more backend attempts; link attempt IDs to worker/model/region and terminal status.
5. Measure three variances separately: ledger versus engine tokens, engine GPU time versus provisioned/billed GPU time, and allocated departmental costs versus provider total.
6. Explain variance in reason-coded buckets: retries, cancellations, shadow tests, late records, cache semantics, idle baseline, shared network, missing ownership or rate-card difference.
7. Apply corrections: append adjustment events rather than overwriting the closed ledger; preserve old invoice snapshots and audit who approved changes.
8. Publish a reconciled statement with residual unexplained amount, accepted tolerance, timestamps, source versions, cost allocation method and unresolved issues.
9. Reopen only under controlled process for late bills, refunds or confirmed defects.

A successful close needs an explicit residual. For example, “98.8% allocated, 1.2% still unallocated” is auditable; silently assigning the leftover to the biggest team is not.

5. Reconcile tokens using explicit equations

For a matched model, region and interval: \\\`text engine_input = billable_input + nonbillable_shadow_input + nonbillable_health_and_eval_input + retries_and_failed_attempt_input + boundary_or_provenance_adjustments + unexplained_input

engine_output = billable_output + nonbillable_shadow_output + nonbillable_health_and_eval_output + retries_and_failed_attempt_output + boundary_or_provenance_adjustments + unexplained_output \\\`

Warning: this identity only works if classes are mutually exclusive, the engine and ledger count tokens the same way, and all terms refer to the same model/time scope. A cached-token subtype is *inside* input totals; do not treat it as another top-level addend. Some backend counters may count *actual processed* prompt tokens differently from API-reported billable input, especially with prefix caching or partial failures; explicitly introduce separate definition adjustments rather than manufacturing equality.

Report absolute token variance and relative variance: \relative variance = abs(engine - explained ledger total) / max(engine, ε)\. Use tolerances determined by expected sampling, pipeline lag and engine semantics. A daily target like ≤0.5% unexplained could be used as an organizational gate, but is not a universal standard.

6. Attribute cloud spend without double counting

Create a cost fact table keyed by invoice month, account, cloud resource/SKU, region, effective cost basis and billing line. Separate:

• Direct inference GPU instance-hours.
• Guaranteed idle/reserved GPU baseline and commitment or amortized charges.
• Batch/spot GPU usage and interruption overhead.
• Shared ingress/router, storage, cross-zone/region data transfer, observability and recovery capacity.
• Provider adjustments, credits, taxes and unsupported-to-allocate lines.

Cost basis discipline: choose whether reporting uses *unblended*, *net*, or *amortized effective* cost, state it, and reconcile to the provider's equivalent invoice metric. Amortized commitment charges and usage records can double-count if blindly summed. Discounts and credits may be company-wide rather than tenant-specific; document whether they are passed through or retained centrally.

A simple allocation: \\\text tenant_allocated_cost = directly_owned_GPU_cost + guaranteed_capacity_fee + share_of_shared_GPU_pool + share_of_shared_network_storage_observability + approved_adjustments \\\ \\\text provider_in_scope_cost = sum(tenant_allocated_cost) + unallocated_or_central_cost + separately_recorded_out_of_scope_cost \\\

The equations must use the same cost basis and scope. A cloud invoice may include taxes or support items that should be excluded from inference chargeback; keep them visible in an out-of-scope reconciliation bucket, not silently ignored.

7. Handle out-of-order events and partial streams

A streaming LLM response can terminate without a final provider usage message. Use a state machine:

• \RECEIVED\ → \ADMITTED\ → \ATTEMPT_STARTED\ → \STREAMING\ → \COMPLETED | CANCELLED | ERRORED | TIMED_OUT\.
• Persist provisional token observations without treating estimates as authoritative settled amounts.
• At terminal transition, prefer engine/provider authoritative usage when available, with an explicit confidence/source flag.
• On worker death, create a missing final usage item, not a zero-cost completed request; reconcile against engine-level work and documented contractual rules.
• Use a reconciliation watermark (e.g., wait for an agreed late-event horizon before a day is final), but don't delay safety/overage limits on live traffic.
• Do not let a late completion from an expired backend attempt overwrite a more recent logical-request settlement.
• For retries, preserve multiple attempt records and a single logical request customer-billing policy. Prevent repeated side-effecting tools with separate idempotency protection.

Important: a customer-facing output that stops midstream can be *operationally incomplete* even when many tokens were generated. Determine billability by written policy and observed usage, not only HTTP 200 at first byte.

8. Example SQL reconciliation model

This query is illustrative and uses hypothetical tables; adjust to the real schema and time boundaries. The \usage_events\ table is an immutable fact table with one finalized record per backend attempt and explicit customer billing status.

\\\`sql WITH tenant_usage AS ( SELECT model_version, region, SUM(input_tokens) FILTER (WHERE customer_billable) AS billable_in, SUM(output_tokens) FILTER (WHERE customer_billable) AS billable_out, SUM(input_tokens) FILTER (WHERE NOT customer_billable) AS nonbillable_in, SUM(output_tokens) FILTER (WHERE NOT customer_billable) AS nonbillable_out FROM usage_events WHERE completed_at >= :start_utc AND completed_at < :end_utc AND source_quality = 'measured' GROUP BY model_version, region ), engine AS ( SELECT model_version, region, SUM(input_token_delta) AS engine_in, SUM(output_token_delta) AS engine_out FROM engine_usage_windows WHERE window_start >= :start_utc AND window_start < :end_utc GROUP BY model_version, region ) SELECT e.model_version, e.region, e.engine_in - COALESCE(t.billable_in,0)

• COALESCE(t.nonbillable_in,0) AS residual_input,

e.engine_out - COALESCE(t.billable_out,0)

• COALESCE(t.nonbillable_out,0) AS residual_output

FROM engine e LEFT JOIN tenant_usage t USING (model_version, region); \\\`

Pitfalls: actual backend attempt records may lack measured usage on crashes, and the time periods shown use event completion versus engine work-window boundaries. Account for these with a lag buffer and missing-usage categories. If a logical request has multiple attempts, ensure \customer_billable\ on an attempt record has well-defined semantics; ideally keep separate attempt-level physical usage and logical-request settlement to avoid charging multiple retry attempts unintentionally.

9. Production incident: a 27% unexplained GPU-cost spike

Scenario (fictional): Monthly GPU spend rises from $20,000 to $25,400 (+27%), but recognized customer billable tokens rise only 5%. Finance suspects tenant B is underbilled; engineers suspect GPU price changes.

Investigation:

1. Normalize invoices by region, GPU SKU, period, discounts, commitment amortization and usage hours.
2. Compare engine-generated token work and original request completions. Backend attempts rose 30%, including many retried long prompts after an engine timeout regression.
3. Worker logs reveal interrupted streams whose final usage was never recorded in the customer ledger; engine counters still include some of the work.
4. Reconcile hours: a new standby GPU deployment adds fixed spend but little new billable volume.
5. Separate causes: retry-amplified GPU work, underutilized standby capacity, and some missing usage events. Do not assume the 27% cost delta must be assigned entirely to tenant B.

Remediation: fix retry storms and timeout handling; improve terminal usage capture, log/metric reconciliation and immutable attempt linkage; allocate standby costs under the published reserved-capacity policy; issue approved invoice adjustments only when contractual billable usage is proven.

Prevention: daily anomaly tests for \GPU-hours per successful request\, \engine attempts per logical request\, \unexplained tokens by model\, and \unallocated spend\, plus a documented monthly close procedure.

10. Controls and alerts for financial integrity

Recommended checks:

• Data completeness: distinct logical IDs and attempt IDs, duplicate event rate, terminal-status coverage, missing usage share.
• Token taxonomy: cached input ≤ total input, reasoning output ≤ total output, no negative usage, valid model/tokenizer and modality.
• Aggregate drift: summed ledger tokens versus engine counters by model/region with explicit nonbillable and definition adjustments.
• Cost drift: provider charged instance hours versus scheduler-run hours, stable capacity versus idle time, unmatched resource IDs and tag coverage.
• Allocation coverage: % fully assigned cost, unallocated amount, shared-cost basis and changes in overhead.
• Freshness: lagged bills and missing report partitions, external credits, currency changes and backfills.
• Governance: versioned rate cards, separation of raw facts and adjustments, auditor read-access, approval trail and reproducible month-end snapshots.
• Privacy/security: financial records carry authenticated tenant/cost-center IDs; don't add raw prompts, retrieved content, secrets or request IDs to high-cardinality monitoring labels.

Set error thresholds proportionally to materiality, volume, expected pipeline lag and approved policy. A small absolute discrepancy can be material to a tiny tenant; a low percent mismatch in a large fleet can represent substantial dollars.

11. Source-system caveats

• vLLM: exposed \vllm:prompt_tokens_total\ and \vllm:generation_tokens_total\ are aggregate engine counters, not an itemized tenant invoice. Compute Prometheus counter rates with proper resets; never sum gauge samples to derive token counts.
• OpenTelemetry GenAI conventions: input-token totals contain cached subsets and output-token totals may contain reasoning subsets. Choose the documented version of semantic conventions; newer token metric families may be in development. Prevent parent-plus-subset double counting.
• FinOps: chargeback requires documented allocation for shared and unallocated costs, owner metadata and defensible treatment of unused commitments.
• Cloud invoices: detailed cost exports may be delayed or revised. Retain raw snapshots and corrections and define when a period is provisionally versus finally closed.

12. Interview-ready answer

“I'd reconcile three views: a durable per-attempt and per-logical-request ledger, engine token and GPU workload counters, and the cloud detailed cost export. I'd normalize model, region, UTC periods, cache-token subsets, retry/nonbillable work, and effective versus amortized costs. Then I'd explain residual differences with reason codes and explicitly carry unallocated costs. Streaming cancellations need estimated or missing-usage status, not fabricated zero usage. I'd close periods with immutable source snapshots and approved adjustments, and alert on unexplained token or GPU-hour drift.”

Research references:

• vLLM — Metrics and Prometheus Counters
• OpenTelemetry — GenAI Token Usage Semantic Conventions
• OpenTelemetry — GenAI Inference Span Usage
• FinOps Foundation — Cloud Cost Allocation Guide
• FinOps Framework — Cost Allocation