How do you test and benchmark adaptive memory-compaction scheduling under realistic production traffic?

To benchmark adaptive memory-compaction scheduling, replay representative multi-agent workloads against a fixed-policy baseline and measure whether the adaptive controller improves accepted-workflow latency and cost without losing critical context, missing deadlines, or starving background tasks. A scheduler that looks good on a smooth synthetic queue may behave badly with bursty API imports, mixed task lengths, model throttling, and concurrent agent updates.

The key experiment is not “How fast does the compactor summarize?” It is “Does the complete, correct multi-agent workflow perform better under realistic traffic?”

Beginner-friendly production incident

A LangGraph service uses a requirements agent, test generator, security reviewer, and publisher to create API test suites. Celery runs background context-compaction jobs.

A new adaptive scheduler passes a quick benchmark with 100 identical short specifications. Production then receives several 200-endpoint specifications simultaneously: long summaries monopolize workers, urgent security reviews miss deadlines, and short workflows queue behind already reserved tasks.

Root cause: the benchmark used uniform jobs at a constant arrival rate; it did not model task-size distributions, bursts, dependencies, prefetch behavior, or provider quotas.

Engineering solution: build a repeatable traffic-replay harness with workload strata and controlled fault injection, compare identical cohorts under fixed and adaptive policies, and include correctness gates and tail metrics.

Capture workload → Define fixed baseline → Replay deterministically where possible → Inject realistic failures → Measure complete workflows → Check safety and fairness → Tune and retest

What does realistic production traffic include?

1. Arrival patterns: Quiet periods, peak hours, coordinated bursts, and sudden uploads that spawn many workflows together.
2. Workflow mix: Short, medium, and large API specifications; 5-endpoint jobs and 200-endpoint jobs; varying agent counts and handoff fan-out.
3. Context characteristics: Different token lengths, growth rates, role-specific views, frequency of source revisions, and percentage of already-valid summaries.
4. Compaction job mix: Short versus long summarization tasks, different validator runtimes, and candidates requiring retries or rejection.
5. Dependency topology: Serial and parallel LangGraph nodes, paused human approvals, and deadlines for security-critical handoffs.
6. Queue and infrastructure behavior: Celery routing, prefetch/reservations, worker restarts, cold scaling, broker delays, and shared LLM request/token limits.
7. Tenancy and fairness: A few high-volume tenants plus many small tenants, with scoped authorization and quotas.
8. Failure conditions: Timeouts, 429 responses, stale results, network interruptions, bad summaries, and validation failures.

Sanitize or synthesize production traces so the benchmark does not expose confidential prompts, credentials, or user data.

Beginner-friendly numerical benchmark

Imagine replaying 1,000 eligible API-test-generation workflows through two isolated runs with equivalent resources, model configuration, workload composition, and quality gates.

Fixed scheduling baseline:

• 900 workflows complete and pass all acceptance checks.
• p95 end-to-end completion time is 120 seconds.
• Total fully loaded cost is $180.
• 100 critical compaction handoff deadlines are missed.

Adaptive scheduling candidate:

• 950 workflows complete and pass the same acceptance checks.
• p95 completion time is 96 seconds.
• Total fully loaded cost is $165.
• 40 critical compaction deadlines are missed.

Calculate:

1. Accepted workflow rate: baseline 900/1,000 = 90% versus adaptive 950/1,000 = 95%, an improvement of 5 percentage points.
2. p95 latency reduction: (120 − 96)/120 × 100 = 20%.
3. Cost per accepted workflow: baseline $180 / 900 = $0.20; adaptive $165 / 950 ≈ $0.1737.
4. Cost-per-accepted-workflow reduction: ($0.20 − $0.1737)/$0.20 ≈ 13.2%.
5. Deadline-miss reduction: (100 − 40)/100 = 60%, assuming the same set of eligible critical handoffs was evaluated. Otherwise report misses per eligible handoff instead of comparing raw counts.

These results are illustrative, not measured production evidence. p95 must be computed from individual completion times; it cannot be inferred from averages. Report confidence intervals or repeated-run variability before claiming an improvement is robust.

Design a fair experimental comparison

• Control and treatment: Compare a known fixed scheduler against the new adaptive controller, changing as little else as possible.
• Identical inputs and order: Replay the same sanitized task arrivals, dependency graphs, source revisions, and tenant proportions. Use recorded arrival timestamps where feasible.
• Resource parity: Hold worker budgets, broker configuration, provider rate-limit assumptions, and per-token pricing consistent.
• Cache controls: Either warm both cohorts comparably or clearly distinguish cold-cache and warm-cache experiments.
• Model variability: Pin model versions when possible, preserve available tool responses, and repeat stochastic runs. Do not promise bitwise deterministic LLM output.
• No cross-contamination: Separate queues, caches, tenant state, result tables, and concurrency quotas for experimental cohorts. If both cohorts share an external LLM quota, contention may bias the result.
• Acceptance parity: Evaluate output test coverage, required OAuth checks, unresolved approval handling, and hallucinated requirements with the *same* rubric.

Do not exclude failed or timed-out workflows from the denominator. Benchmarking only successful compaction tasks creates survivorship bias.

Which experiments should you run?

1. Steady-state load: Replay representative traffic below sustainable capacity to establish baseline overhead and normal resource usage.

2. Burst and soak tests: Inject 5× arrivals for a short interval, then sustain a long run. Inspect backlog growth, drain time, memory growth, and controller oscillation.

3. Capacity saturation: Increase arrivals until throughput plateaus; identify whether the limiting factor is worker concurrency, CPU, queue reservations, model requests/minute, or model tokens/minute.

4. Priority and fairness tests: Keep urgent handoffs arriving while background compactions accumulate. Confirm critical deadlines and nonzero useful progress for background jobs and small tenants.

5. Fault-injection tests: Introduce 429s, worker crashes, lost acknowledgments, validator rejection, network partitions, and version conflicts. Confirm idempotent retries and that old candidates never replace newer validated summaries.

6. Context-safety tests: Generate high-token histories with mandatory authorization facts near the start, middle, and tail. Ensure no agent proceeds with a truncated or incomplete summary.

7. Controller-stability tests: Alternate load above and below congestion thresholds. Check hysteresis, cooldowns, scaling chatter, and rapid priority changes.

8. Regression tests: Repeat cases that previously caused missed security checks, stale event tails, or stuck Celery jobs after every scheduling-rule change.

Separate queue, execution, and workflow timing

Instrument these event boundaries for each compaction job:

• requested_at: workflow requests a compacted context.
• admitted_at: admission controller accepts the job.
• enqueued_at: message is published to the broker.
• received_at: worker reserves/receives it.
• started_at: actual compaction execution begins.
• validated_at: candidate passes checks.
• activated_at: the summary becomes the validated active version.
• consumed_at: downstream agent actually reads it.

Use these to distinguish admission delay, broker/reservation delay, execution and validation duration, activation delay, and agent-blocked time. For parallel LangGraph branches, use actual critical-path wall time rather than summing overlapping node durations.

Celery's task-event monitoring documents sent, received, started, succeeded, failed, and retried events. Its optimizing guidance also explains why prefetch settings affect apparent queue lengths. Instrument missing timestamps at the application level rather than assuming the broker exposes every boundary.

Suggested benchmark output

Record per experiment:

1. Accepted-workflow success rate and correctness guardrail failures.
2. End-to-end p50/p95/p99, critical handoff deadline-miss rate, and time blocked waiting for validated context.
3. Cost per accepted workflow, billable input/output tokens, compaction overhead, and infrastructure spend.
4. Unique compaction arrivals and validated completions, queue wait, oldest useful job age, backlog drain time, and dropped/deferred work.
5. Controller decisions: COMPACT_NOW, DEFER, JOIN_PENDING, SAFE_FALLBACK, and PAUSE, plus policy-state transitions.
6. Quality: Retained constraint coverage, wrong/stale source-version consumption, missing authorization checks, and unsupported test assertions.
7. Stability and fairness: Retry amplification, provider rate-limit errors, urgent/background service share, and per-tenant starvation.

Use traces and aggregate reports. Avoid storing sensitive full prompts in routine telemetry; record secure references and revisions where appropriate.

Example benchmark harness

PYTHON
def benchmark(policy, workload_trace, sandbox):
    sandbox.reset_to_known_state()
    results = []

    for arrival in workload_trace:
        sandbox.schedule_at(
            arrival.time,
            run_workflow,
            policy=policy,
            workflow=arrival.workflow,
            tenant=arrival.tenant,
            source_revision=arrival.source_revision,
        )

    sandbox.run_until_drained_or_timeout()

    for run in sandbox.workflows:
        results.append({
            "accepted": verify_artifacts_and_approvals(run),
            "latency_s": run.end_to_end_latency,
            "fully_loaded_cost": run.total_cost,
            "critical_deadline_misses": run.deadline_misses,
            "invalid_memory_reads": run.invalid_context_reads,
        })

    return aggregate_by_workload_and_tenant(results)

This is conceptual pseudocode: a real replay harness needs a deterministic virtual clock or controlled wall-clock orchestration, dependency ordering, isolated external side effects, reproducible provider mocks or quota-controlled real API calls, and clear timeout accounting.

Use deterministic mocked-model runs for high-volume stress and invariants; add a smaller real-model cohort for actual token, cost, semantic quality, and latency measurements. A pure simulation cannot establish real provider p95 performance.

Why both offline and online evaluation matter

Offline traffic replay makes scheduler comparisons repeatable and lets you inject crashes safely. It can miss real provider jitter, cross-service contention, and changing request patterns.

Shadow mode applies adaptive decisions without allowing them to control live workflows. It measures choices and predicted capacity impact but cannot directly measure all treatment effects on final latency.

Limited canary experiments test the actual production effect with strict abort criteria, such as missing approval checks, increased context-overflow failures, p95 latency regression, or tenant starvation. Protect against contamination between variants and monitor for statistically uncertain results.

Common benchmarking mistakes

• Reporting token reduction alone instead of accepted-workflow efficiency.
• Using constant arrival rate and identical compaction durations.
• Comparing unequal model versions, cache states, or worker quotas.
• Summing parallel agent durations and calling the total “workflow latency.”
• Ignoring tasks already reserved by Celery workers.
• Excluding 429s, failed validations, fallback paths, and retries.
• Tuning and reporting results on the same traffic traces without held-out workloads.
• Claiming reproducibility while reading changing external data during replay.
• Publishing confidential API requirements into unprotected observability logs.

Difference from earlier lessons

• Q205: Benchmarks a multi-agent system against a single-agent system.
• Q229: Evaluates compacted-summary content quality.
• Q236: Measures cost and latency of compaction.
• Q237–242: Designs scheduling, scaling, overload, and fairness controls.
• Q243: Implements a workload-adaptive compaction controller.
• This lesson: Builds a production-like, controlled evaluation harness that tests whether the adaptive scheduler improves complete-workflow outcomes under representative traffic and failures.

In short

Benchmark the adaptive scheduler with paired production-shaped workloads, realistic queueing and provider limits, fault injection, and safety-aware acceptance tests. Optimize accepted-workflow quality, p95 latency, total cost, and fairness—not compactor throughput in isolation.

Further reading

• Celery: Monitoring and Management
• Celery: Optimizing
• LangSmith: Evaluation types and benchmarking
• OpenTelemetry: Generative AI semantic conventions