How we design multi-agent systems for production, not demos

The demo problem

A multi-agent system in a demo is a beautiful thing. Agents hand off tasks to each other. A planner breaks work into steps. Specialists execute each step. A critic reviews the output. It all looks like it works.

Six weeks later, in production, the planner loops on an ambiguous task for twelve minutes before timing out. The critic hallucinates approval. The handoff between agents loses context. The system is down and nobody knows which agent caused it.

This is not a hypothetical. It is a pattern we've debugged at multiple clients who built demos that didn't survive contact with real users. The gap between a working demo and a working production system is where most multi-agent projects stall. Here's the architecture that closes it.

Start with the control surface, not the agents

The first decision in a multi-agent system is not "what agents do we need." It is "where does human control live, and how does a human stop, inspect, and override the system at any point."

This sounds obvious. In practice, teams skip it. They wire up agents and add observability later, after they've discovered that a runaway loop burned $400 in API credits overnight.

Before writing your first agent, define: what is the minimum unit of work that a human can review? What state needs to be checkpointed so a human can resume from a known-good point? What conditions trigger an automatic pause pending human review?

In LangGraph, this maps to interrupt nodes and persistent state channels. In a custom orchestrator, it's a state machine with explicit paused and awaiting-human-review states. The implementation varies. The requirement doesn't.

Type your handoffs

The most common failure mode in multi-agent systems is context loss between agents. Agent A knows something. It passes a message to Agent B. Agent B doesn't have the full context and makes a decision that's locally rational but globally wrong.

The fix is typed handoffs: structured payloads with explicit schemas that define exactly what one agent passes to the next.

interface ResearchHandoff {
  taskId: string;
  query: string;
  constraints: string[];
  priorAttempts: AttemptRecord[];
  confidenceThreshold: number;
}

Every field is intentional. priorAttempts is especially important — it prevents the next agent from re-trying approaches that have already failed. Typed handoffs also give you a natural place to add validation: if the payload doesn't conform to the schema, the receiving agent fails loudly rather than silently.

Keep the orchestrator dumb

A common architectural mistake is putting business logic in the orchestrator. The orchestrator decides which agents run and in what order. That's its job. The moment you start adding conditionals and domain knowledge to the orchestrator, you've created a second place where the system's core logic lives, and you've made both places harder to test.

The orchestrator should be a routing layer. Each agent should be stateless and self-contained. The shared state store — a persistent, checkpointed data structure — is where memory lives across the workflow.

In LangGraph, this means leaning on the state graph for routing and keeping node functions focused. A node function should do one thing, be independently testable, and have no knowledge of what comes before or after it in the graph.

Build the eval harness before the agents

Multi-agent systems have emergent behaviour that cannot be tested by unit-testing individual agents. The only reliable way to catch system-level failures before production is end-to-end replay testing with a representative corpus of real tasks.

We build the eval harness first: a test runner that can replay a library of tasks, record every agent action and state transition, and assert against expected outcomes. Then we build agents.

This discipline catches two categories of bugs that would otherwise only appear in production. First, interaction bugs: situations where two individually-correct agents produce incorrect behaviour in combination. Second, degradation bugs: cases where changing one agent's prompt or model causes a failure in a downstream agent that appears to be working correctly in isolation.

Observability is not optional

In a single-LLM system, debugging a failure means reading one trace. In a multi-agent system, a single user task might generate hundreds of LLM calls across a dozen agents over several minutes. Without structured observability, debugging is archaeology.

Every agent call should emit a structured trace event: agent name, input payload, output payload, tool calls made, tokens consumed, latency, and a correlation ID that links all events in a single task. Feed these events to a tracing system — we use Langfuse or Arize depending on the client stack — and build dashboards around the metrics that matter: task success rate, per-agent failure rate, average tokens per task, cost per task.

The teams that operate multi-agent systems well spend as much time on their observability stack as on their agents. It's not overhead. It's the instrument panel.

The production checklist

Before a multi-agent system goes live, it should pass all of the following:

• Every agent has an independent test suite with >80% coverage of its decision surface
• End-to-end replay tests cover the top 20 task types by volume
• Human-in-the-loop interrupts are defined and tested for all high-stakes actions
• Handoff payloads are schema-validated at ingestion
• Cost per task is measured and within budget at p99
• The runaway loop prevention timeout is set and tested
• Traces are flowing to the observability stack
• An on-call runbook exists with procedures for the top five failure modes

That checklist is the difference between a multi-agent system and a multi-agent demo.