How do you measure the cost and latency impact of memory compaction across multiple AI agents?

Measure memory-compaction performance at the level of the entire multi-agent workflow, not merely by counting how many prompt tokens disappear. Compaction can reduce repeated input tokens but introduce summarization calls, validation, cache misses, retrievals, retries, and queue delays. The question is whether the *completed, correct task* becomes cheaper or faster.

In practice, compare the same workflow with compaction disabled and compaction enabled using consistent model versions, workload mix, cache policy, and quality checks.

Beginner-friendly production example

A LangGraph pipeline generates API test suites using a requirements agent, test generator, security reviewer, and publisher. The history grows as each agent adds tool results and findings. Without compaction, every subsequent model call rereads much of that history. With compaction, the coordinator builds a validated common summary and small role-specific context.

A team notices that input token usage falls sharply, but the API endpoint's p95 completion time rises because compaction jobs queue behind other Celery tasks. Looking only at prompt-token reduction would hide the regression.

Define workflow baseline → Instrument every stage → Run comparable workloads → Calculate fully loaded cost → Compare latency distributions → Check quality → Decide whether to retain compaction

Step-by-step numerical example

Assume one completed test-generation workflow makes 8 downstream LLM calls. The figures below are illustrative, not current provider prices.

Without compaction

• Each downstream call receives 20,000 history tokens: 8 × 20,000 = 160,000 input tokens.
• It generates 12,000 output tokens total.
• At an illustrative input price of $2 per million tokens and output price of $8 per million, model cost is 160,000/1,000,000 × $2 + 12,000/1,000,000 × $8 = $0.416.
• End-to-end duration is 80 seconds.

With compaction

• The downstream calls receive 6,000 history tokens each, or 48,000 tokens total.
• The compactor itself consumes 22,000 input tokens and produces 2,000 output tokens.
• Validation and fallback retrieval require another 10,000 input and 1,000 output tokens.
• The downstream agents still generate 12,000 output tokens.
• Fully loaded input is 48,000 + 22,000 + 10,000 = 80,000 tokens; output is 12,000 + 2,000 + 1,000 = 15,000 tokens.
• Model cost is 80,000/1,000,000 × $2 + 15,000/1,000,000 × $8 = $0.28.
• Model cost savings = ($0.416 − $0.28)/$0.416 ≈ 32.7%.
• End-to-end duration is 87 seconds because compaction, validation, and queueing add more wall-clock time than downstream calls save.

Conclusion: In this example, model spending drops by 32.7%, but latency worsens by 7 seconds (8.75%). If the user-facing latency SLO is strict, compaction is not an unconditional win. The calculation also excludes storage, CPU, telemetry, and infrastructure expenses; measure those for total cost of ownership.

Which metrics should you capture?

1. End-to-end cost per accepted workflow: Total model charges plus relevant compaction infrastructure, retrieval, vector-store, and retry expenses; divide by *accepted* completed tasks, not by attempts.
2. Model token usage: Input and output tokens per agent, compactor, validator, and fallback request; separate billable cached-input tokens if the provider exposes them.
3. Compaction overhead: Candidate-generation tokens, duration, validator calls, retries, rejected summaries, and queue wait time.
4. Context savings: (baseline repeated-context tokens − new repeated-context tokens) / baseline repeated-context tokens. Label this a gross context reduction, not a net billing saving.
5. Latency distribution: End-to-end p50, p95, p99, time to first useful response when applicable, and per-node service/queue durations.
6. Throughput and resource load: Accepted workflows per minute, concurrent executions, Celery backlog, and pressure on model and storage rate limits.
7. Fallback and quality: Extra retrieval requests, missing requirement rate, failed security checks, human rework, and successful completion rate.
8. Cache behavior: Prompt-cache hit and billed-token changes, because shortening prompts can sometimes reduce the value of existing caching.

How do you measure latency correctly?

Record the wall-clock times of workflow start, compaction queueing, compaction execution, validation, each downstream model/tool step, and final acceptance. For parallel branches, do not add all node durations to estimate end-to-end latency; overlapping nodes make summed durations larger than the critical-path wall time.

Example: requirements and security analysis run in parallel for 12 and 18 seconds. Together they contribute approximately 18 seconds, not 30 seconds, to the critical path if there is no additional serial work. A compactor blocking both branches can lengthen the critical path, whereas compaction running asynchronously ahead of a later handoff may not.

Watch p95 rather than only average latency: retries, queue spikes, and very long contexts often produce tail latency that averages conceal.

Set up a fair comparison

• Paired offline benchmark: Run the same set of API endpoints with full versus compacted contexts. Hold model versions, inputs, tool permissions, and acceptance rules constant. Repeat stochastic cases where feasible.
• Shadow evaluation: Generate compacted contexts without changing production decisions, then evaluate their correctness and expected token cost.
• Canary or randomized rollout: Compare production cohorts with similar endpoint complexity, workload, time of day, caching, and concurrency. Keep a fast rollback route.
• Stratify results: Measure short versus long conversations, high versus low agent count, security-heavy workflows, and low versus high context reuse.
• Include failures: Count compaction timeouts, failed validations, retries, and fallback paths. Excluding them biases the claimed savings upward.

Do not benchmark only workflows that successfully compacted. Use the complete eligible cohort, including failed or skipped attempts.

When does compaction pay for itself?

A simple break-even estimate is:

net_savings = (avoided downstream model charges) − (compaction + validation + fallback + infrastructure charges).

For example, if generating and checking a summary costs $0.04 and each future agent call saves $0.012 in model charges, then after 4 future uses, gross avoided charges are $0.048, producing $0.008 net savings before other overhead. After only 2 uses, savings are $0.024, so compaction loses $0.016.

This shows why a compaction policy should consider the expected number of future reuses, not just the length of the current transcript.

LangGraph and Celery implementation

In LangGraph, record a trace for the workflow and child spans for agent nodes, summarization, memory retrieval, validation, and model calls. Include workflow ID, summary version, source revision, and a compacted/not-compacted flag. Be careful not to expose sensitive prompt content or personal data in telemetry.

In Celery, separately time broker queue delay and worker execution time for compaction tasks. Collect retry counts, failure reasons, worker saturation, and time to publish a validated summary. Correlate task IDs back to the LangGraph run so queue time is not mistaken for model latency.

OpenTelemetry's evolving generative-AI conventions describe token usage and operation-duration measurements; tools such as LangSmith can provide agent traces and cost/latency analysis. Instrumentation completeness and actual billed cost should be verified against the chosen model provider.

Production decision rule

Adopt compaction only if it satisfies all required constraints:

• No unacceptable regression in test-suite correctness or critical security coverage.
• Lower fully loaded cost per accepted workflow for the intended workload.
• p95 end-to-end completion time remains within the application's SLO.
• Failure and fallback rates remain within agreed budgets.
• Audit and tenant isolation requirements stay intact.

A system optimized for batch generation might accept seven extra seconds to save money. An interactive assistant with a strict response-time target might choose fewer compactions or move them off the critical path.

Difference from earlier questions

• Q202: General strategies for reducing multi-agent workflow cost.
• Q203: General multi-agent performance metrics.
• Q229: Whether a compacted summary is accurate and useful.
• Q230: When compaction should trigger.
• Q235: Tailoring compacted context by agent role.
• This lesson: Measures the actual marginal cost, critical-path latency, queue impact, and break-even point of memory compaction in production.

In short

Count the whole workflow. Saving prompt tokens is valuable only after accounting for compactor and validator charges, retrieval fallbacks, queueing, caching effects, quality regressions, and end-to-end latency.

Further reading

• OpenTelemetry: Generative AI metrics
• LangSmith: Observability and evaluation
• LangGraph: Memory and summarization