How do you measure the performance of a multi-agent system?

Measure multi-agent performance by checking whether the entire workflow produces correct, useful results within acceptable time, cost, and reliability limits. A fast individual agent is not enough if its handoffs, retries, or final outputs are poor.

Beginner-friendly example: API test generation

A workflow uses a requirements agent, test generator, security specialist, and coverage reviewer. Over one evaluation batch it receives 100 API endpoints.

• 85 endpoints produce test suites that pass all required independent quality checks.
• 10 produce incomplete suites.
• 5 fail or time out.
• The system uses $12 in measured model/API charges for the batch.

The accepted-task rate is 85 / 100 = 85%. The model/API charge per accepted task is $12 / 85 ≈ $0.141. This is not the complete operating cost unless worker infrastructure, storage, and human review are also included.

Do not report 95% success by counting incomplete outputs as accepted. Define acceptance criteria before evaluating.

Which metrics matter?

1. Outcome quality: Correctness, requirement coverage, hallucinations, test validity, and independent acceptance rate.
2. End-to-end latency: Time from user request to validated final output, including queue waits and human approval where applicable. Monitor median and tail latency, such as p95.
3. Throughput: Accepted workflows or test suites per minute—not just agent calls made.
4. Cost efficiency: Total cost per accepted outcome, token usage, tool calls, retry expense, and human corrections.
5. Reliability: Deadline misses, errors, incomplete runs, recovery success, duplicate work, and unavailable agents.
6. Coordination overhead: Time and tokens spent on handoffs, repeated planning, consensus, or avoidable messages.
7. Resource utilization: Worker saturation, queue age, concurrency, provider quotas, and bottlenecks.

Keep quality and safety constraints visible: reducing cost or time is not a win if mandatory security tests are skipped.

Step-by-step numerical latency example

Suppose a completed workflow records:

• Requirements extraction: 2 seconds
• Test generation: 6 seconds
• Security review: 4 seconds
• Final aggregation: 1 second

If these run strictly sequentially with negligible queue and transfer overhead, end-to-end latency is 2 + 6 + 4 + 1 = 13 seconds.

If test generation and security review operate independently after extraction, the approximate critical-path latency becomes 2 + max(6, 4) + 1 = 9 seconds—assuming the review does not actually depend on generated tests. In a real test-suite *review* workflow, it often does, so the parallel schedule would be invalid. Always model true dependencies rather than inventing speedups by parallelizing required sequential steps.

How do you evaluate fairly?

• Build a representative test set of API documents and endpoint types, including difficult security and error-handling cases.
• Define objective acceptance checks and human-review rubrics for subjective output quality.
• Compare the same dataset, provider conditions, deadlines, and evaluation policy across versions.
• Measure each run end to end; keep agent-level traces so failures can be attributed to extraction, generation, coordination, or review.
• Report both averages and distributions; one exceptionally slow workflow can matter even when the mean looks good.
• Run ablations: remove or replace an agent to test whether its contribution justifies the added cost and complexity.
• Watch for data leakage and biased evaluators; do not use the generating agent's self-assessment as the only quality evidence.

LangGraph and Celery implementation

In LangGraph, instrument node spans, handoffs, model/tool calls, and final outcomes. Use stable workflow IDs to connect intermediate events to the accepted artifact.

With Celery, track queue waiting, execution, retries, worker failures, and result persistence separately. Successful Celery task execution is not proof that the generated test suite is correct.

A useful dashboard combines accepted outputs, end-to-end p50/p95 latency, cost per accepted output, failure rate, and queue age, with drill-down into each specialist.

In short

Judge a multi-agent system by validated end-to-end outcomes, not raw agent activity. Measure quality, latency, throughput, cost, resilience, and coordination overhead together to find improvements that actually matter.