How do you design production dashboards for monitoring multi-agent systems?

A production dashboard for a multi-agent system is an operational view that shows whether complete agent workflows are producing valid results, meeting deadlines, and using resources responsibly. Its purpose is to help an operator discover problems and investigate causes—not to display as many charts as possible.

Beginner-friendly example: API test generation

Suppose a requirements agent, test generator, security reviewer, and supervisor work together to create API test suites. The dashboard should answer:

• Are customers receiving validated suites, not merely text from individual agents?
• Are any workflows stuck waiting for a handoff, worker, or approval?
• Are errors coming from the model provider, a specific agent, a tool, or the queue?
• Are token usage and infrastructure costs rising faster than accepted outputs?

Workflow events → Metrics and traces → Aggregation by service and stage → Dashboard → Alert and investigation

Step-by-step numerical example

Over the last 15 minutes, a system records:

1. 200 eligible workflows accepted.
2. 180 deliver valid, reviewed test suites within a 5-minute deadline.
3. 12 finish late or invalid, and 8 remain past the deadline.
4. The on-time valid completion rate = 180 / 200 × 100 = 90%.
5. Workers execute 300 task attempts including retries, but this is *not* 300 completed workflows.

Suppose the dashboard also shows a p95 end-to-end latency of 7 minutes, and the review queue has 25 pending tasks. That combination points toward a possible review-stage bottleneck. Confirm using queue age, processing spans, worker availability, and traces before blaming the reviewer model.

For a time-window denominator, define how pending requests are treated: requests still within their 5-minute deadline should not be prematurely counted as failures. Here all 200 requests have either reached a terminal outcome or passed their deadline.

What should the dashboard contain?

1. User outcomes: Workflow requests, valid accepted outputs, failed/late/deferred outcomes, and incomplete or unsafe outputs prevented from publication.
2. SLO health: Success rate against the configured objective, end-to-end latency distribution, and error-budget burn.
3. Stage breakdown: Per-agent error rates, validation failures, duration, retry count, and handoff-related rework.
4. Queue and worker health: Celery queue age, pending task count, worker availability, active slots, and retry spikes.
5. Provider/tool dependencies: LLM/API failures, throttling, request duration, tool timeouts, and database errors.
6. Usage and cost: Input/output tokens, model and tool charges, cost per accepted suite, and remaining spend limits.
7. Coordination: Handoff duration, branch join waits, repeated decisions, and stalled-workflow counts.

Keep each chart tied to a diagnostic question. Overall metrics should lead to filters and trace-level details.

A practical dashboard layout

• Top summary: Valid completion rate, p95 latency, active incidents, and cost per accepted workflow.
• Reliability section: SLO trend, error-budget burn, and failures categorized by cause.
• Workflow map: Stage latency, retries, and time spent waiting at edges or joins.
• Capacity section: Queue depth and age, worker utilization, and provider rate limits.
• Investigation panel: Search by workflow ID to inspect related traces and sanitized artifact versions.

Use consistent time windows and clear units. Avoid comparing lifetime totals with five-minute rates without labeling the difference.

How do you collect data reliably?

• Assign a stable workflow ID to each user request; attempts and subtasks receive their own IDs linked to it.
• Emit structured events for acceptance, start/end of stage, validation, retries, handoffs, and final delivery.
• Instrument metrics, logs, and distributed traces with correlation IDs; adopt OpenTelemetry where suitable.
• Aggregate high-cardinality workflow IDs in trace or log searches, not as unbounded time-series metric labels.
• Deduplicate retried terminal events so one request does not count as several successes.
• Check missing telemetry and clock skew; an empty dashboard does not necessarily mean the service is healthy.
• Redact prompts, credentials, tenant data, and sensitive requirements before exporting telemetry.

LangGraph and Celery implementation

In LangGraph, instrument node spans and graph transitions, recording workflow state, relevant artifact versions, and validation outcomes. Link each specialist's activity to one parent workflow trace.

In Celery, collect task runtime, publish-to-start delay, retry count, worker status, queue age, and result delivery. Join those signals by workflow ID so queue delays are not confused with model inference delays.

For tooling, a combination such as OpenTelemetry + Prometheus/Grafana for numeric metrics and a compatible tracing backend is one possible architecture; choose based on operational needs rather than assuming one vendor is mandatory.

Dashboard pitfalls

• Reporting completed agent tasks as successful customer workflows.
• Showing average latency only, hiding slow tail cases.
• Displaying provider uptime while omitting invalid final outputs.
• Paging on every transient retry, creating alert fatigue.
• Using huge per-workflow labels that overload the metrics backend.
• Treating missing data as evidence that no failures occurred.

In short

Design dashboards around validated workflow outcomes and fast diagnosis. Start with success, latency, SLO health, and cost; then drill down into agents, handoffs, providers, and Celery workers to locate the real bottleneck.