How do you evaluate the quality of compacted memory in multi-agent systems?

Evaluating compacted memory means measuring whether a shortened working context helps multiple agents complete their tasks correctly while reducing token use, cost, and latency. A compact summary is not high-quality merely because it is shorter or readable. It must preserve essential facts, avoid unsupported claims, and yield reliable downstream decisions.

A useful evaluation combines content-level tests with task-level tests. Content-level tests check what survived compression; task-level tests check whether the agents can still do their jobs.

Beginner-friendly example: an API test-generation workflow

Four agents process an API specification: requirements extraction, test generation, security review, and publication. They repeatedly share a long history. A compactor replaces repeated chatter with a short context containing verified endpoint rules, artifact references, unresolved issues, and pending approvals.

Compare two versions on the same set of endpoint tasks:

• Baseline: Agents receive full, relevant historical context.
• Candidate: Agents receive compacted context plus access to original source records.

Keep the dataset, model version, tool permissions, and evaluation rubric comparable. Otherwise apparent improvements might come from unrelated changes.

Build reference cases → Define metrics → Evaluate preserved evidence → Run downstream agents → Compare quality and efficiency → Apply release gates

Numerical example: calculating quality and savings

Suppose a compacted context is evaluated against 40 required facts and contains 38 correctly preserved facts, 2 unsupported claims, and omits 2 required facts.

1. Critical-fact recall: 38 / 40 = 95%. Two facts were omitted.
2. Supported-claim precision: 38 / (38 + 2) = 95%. Two included claims lack authoritative support.
3. Unsupported-claim rate: 2 / 40 = 5%, where the denominator is all 40 extracted claims in the compacted context.
4. Compression ratio: Original context 20,000 tokens; compacted context 5,000 tokens. Ratio 20,000 / 5,000 = 4:1, with a 75% reduction in context tokens.
5. Task success: In 100 comparable endpoint cases, the full-context workflow passes 94, whereas the compacted workflow passes 90. Observed success rates are 94% and 90%, a difference of −4 percentage points.
6. Critical omissions: If either missing fact is a mandatory security rule, mark the candidate NO-GO, regardless of token reduction.

These are illustrative, sample-based metrics. A measured 4-point difference does not establish statistical significance without suitable paired-case analysis and uncertainty estimates.

Which quality metrics matter?

1. Requirement or fact recall: Fraction of reference facts retained. Separate mandatory security, business, and ordinary contextual facts.
2. Grounded precision or faithfulness: Fraction of summary claims supported by original evidence. Incorrect facts are different from missing facts and must be counted separately.
3. Contradiction rate: How often the compact context reverses a requirement, number, status, or decision.
4. Pending-work preservation: Whether unresolved issues, owner assignments, approval gates, and open tool operations remain explicit.
5. Source traceability: Fraction of material claims that point to valid document versions, record IDs, or evidence spans.
6. Task success and regression rate: Whether the test-generation and review agents still satisfy acceptance criteria after compaction.
7. Retrieval fallback quality: Whether agents can recover omitted but available details from authoritative records when necessary.
8. Efficiency: Input/output tokens, compaction overhead, actual monetary cost, end-to-end p50/p95 latency, and number of retrieval calls.

Do not combine these into one score without hard constraints. A 90% overall score can conceal a security-critical hallucination.

How should you build an evaluation dataset?

• Select representative short, long, and very long workflows, not just easy examples.
• Include normal API endpoints, authentication-sensitive endpoints, branching conversations, contradictory revisions, partial approvals, and recently changed requirements.
• Label the canonical facts, unresolved issues, and required source references directly from authoritative documents.
• Include adversarial cases where the earlier conversation contains wrong information later corrected.
• Add multi-agent handoff cases where one agent's omitted detail causes another agent to fail.
• Split development cases from a held-out regression set so the team does not tune only to remembered examples.

For high-impact facts, deterministic extraction and rule checks are especially valuable. LLM-as-judge or entailment checks can help detect unsupported claims, but should be calibrated against human-reviewed examples and not treated as infallible.

Practical paired evaluation

For each test case, run both full and compact contexts under controlled conditions. Store:

• Original and compact context revisions.
• Ground-truth requirement and issue IDs.
• Agent output artifacts and reviewer decisions.
• Missing, unsupported, and contradictory claims.
• Token consumption, latency, and retrieval behavior.

Use paired per-case differences, not just averages. If baseline succeeds and candidate fails on the same endpoint, inspect what evidence was lost or misrepresented. Re-run stochastic workloads or hold random seeds/configuration fixed where feasible to reduce noise.

Example release gate

A team could set illustrative gates:

• 100% preservation of known critical authorization facts and pending approvals.
• Zero known unsupported security claims or cross-tenant memory leakage.
• At least 99% factual coverage for other labeled required items.
• No unacceptable degradation in held-out endpoint-level task success.
• Measurable end-to-end cost reduction after including summarization and fallback retrieval.

The numerical example fails the recall and task-quality expectations even though compression saves tokens. The correct action is to revise the compactor and rerun the evaluation, not publish a lower-quality summary.

Monitor after deployment

Offline success does not guarantee safety in production. Canary the compact context for a small cohort, monitor missing-requirement incidents, clarification frequency, fallback retrievals, cost, and p95 latency, and compare against a suitable full-context control. Preserve the last validated summary revision and a quick routing rollback.

LangGraph and Celery considerations

With LangGraph, evaluate what is actually passed to the model, not only what is persisted in checkpoints. A checkpointer can retain full thread state while a node includes only the compact summary and selected recent messages in model input. Test handoff and resume behavior at real graph boundaries.

With Celery, run compaction evaluation as reproducible jobs keyed by workflow and source revision. Separate candidate generation from scoring, store per-case results, and block promotion if a revised source makes the evaluation stale.

Difference from nearby questions

• Q80: Evaluates whether agent memory is useful in general.
• Q227: Implements compaction for long-running multi-agent workflows.
• Q228: Protects required information from being lost.
• This lesson: Creates an *evaluation protocol* and quantitative release criteria comparing compacted and full-context workflows for fidelity, downstream accuracy, and actual efficiency.

In short

Measure factual coverage, groundedness, contradictions, task success, and end-to-end cost together. Never trade away mandatory constraints or approvals for smaller prompts; promote compacted memory only when held-out tests and production canaries demonstrate acceptable outcomes.

Further reading

• LangGraph: Add and manage memory
• Microsoft: Evaluation metrics for generative AI