Multi-agent orchestration: a production guide

What is multi-agent orchestration?

Multi-agent orchestration is the logic that decides which agent runs next, what it receives, and where its result goes. Most guides to multi agent orchestration stop at a list of patterns. In production the pattern is the easy part. The hard parts are the boundaries around it: how many helpers run at once, how long each may take, what each is allowed to touch, what happens when one fails halfway, and who pays for the tokens.

There are two ways to make the decision. The OpenAI Agents SDK documentation puts it plainly: let the model plan and delegate, or orchestrate in code, where the flow is fixed and the model fills in steps. Code is more predictable in speed, cost and behavior. The model is more flexible when the task is open ended. Most production systems mix the two: code owns the skeleton, the model owns the judgment inside each step.

This guide is about that skeleton. We run agents for other people on Orca, and delegation is the feature where a small design choice turns into a large bill or a run that never ends. What follows is the decision process we use, the boundaries we put around every helper, and the checklist we run before a multi-agent workflow goes live.

Why use multiple agents instead of one?

Three reasons hold up in practice: parallel work, clean context, and narrow tools. When a task splits into parts that don't depend on each other, helpers can run them at the same time. Each helper starts with only its own piece of the task, so its context isn't crowded with nine other files. And each helper can carry only the tools its piece needs, which keeps its instructions short and its mistakes contained.

The strongest published evidence is Anthropic's account of their research system. A lead agent on Claude Opus 4 with Claude Sonnet 4 subagents outperformed single-agent Claude Opus 4 by 90.2% on their internal research eval. They also found that token usage alone explained 80% of the performance variance on BrowseComp. Their reading of it is honest: multi-agent systems work mainly because they spend enough tokens on the problem.

That is also the catch. In the same post, a single agent used about 4 times the tokens of a chat, and a multi-agent system about 15 times. Delegation buys breadth with tokens. It pays off when the task is valuable and wide, like research across many sources, and it loses money when the task is narrow or tightly coupled.

What are the core components of a multi-agent system?

Whatever framework you use, the same parts show up. Naming them makes the rest of the design easier to discuss, because every production problem lives in one of them.

Lead

The agent, or the code, that receives the task, splits it, starts helpers, and writes the final answer. Also called a supervisor, manager, orchestrator or arbiter.

Helpers

Agents that each take one bounded subtask. They may be the same agent with a different task, or specialists with their own instructions and tools.

The contract

What a helper receives and what it must return: the task, its inputs, the shape of its result, and what counts as done or failed.

Shared state

Whatever more than one agent can read or write: files, a sandbox, a database, a conversation. Every shared thing is a place two agents can collide.

Limits

How many helpers at once, how deep delegation may go, how long each may run, and how much each may spend.

The trace

The record of who started whom, with what input, what tools each called, what each returned, and what it cost.

The lead is usually where teams spend their attention, and limits and the trace are usually where their incidents come from. A lead with no limits is an unbounded fan out. A system with no trace can't answer the first question anyone asks after a bad run: which helper did that?

What are the multi-agent orchestration patterns?

The names vary between frameworks, but the shapes are few. Semantic Kernel ships five: sequential, concurrent, group chat, handoff and Magentic. The OpenAI Agents SDK describes two core ones, agents as tools and handoffs, plus code patterns such as chaining, evaluator loops and running agents in parallel. Together they cover the multi-agent architecture patterns you will meet.

Sequential

A pipeline. Each agent takes the previous agent’s output as input: research, outline, draft, critique. Easy to reason about, and as slow as the sum of its steps.

Concurrent

Several agents work on the same input or on independent parts at once, and their results are collected and aggregated. Fast, and the aggregation step is where the quality is decided.

Supervisor

A lead calls specialists like tools and keeps ownership of the answer. The OpenAI Agents SDK calls this agents as tools. It is the default we reach for.

Handoff

An agent transfers control to a specialist, which then owns the rest of the turn. Good for routing, such as support triage, where the specialist should answer directly.

Evaluator loop

One agent produces, another judges against criteria, and the loop stops when the judge passes it. It needs a hard cap on rounds, or it is a loop with a bill.

Group chat

A manager picks which agent speaks next in a shared conversation, optionally with a human. Flexible, and the hardest shape to bound and test.

How to choose a multi agent orchestration pattern

Pick the pattern from the shape of the dependencies, not from the number of agents you would like to have. We ask four questions in order, and the first one that answers yes picks the pattern.

Two rules sit on top of the table. First, prefer the pattern your code can enforce. A sequential pipeline in code can't skip a step. A model told to run the steps in order sometimes will. Second, put the model's freedom where the uncertainty is. If the uncertainty is in routing, let the model route and fix the rest. If it's in research, let the helpers search freely and fix the merge.

Group chat and fully dynamic managers belong at the end of the list, not the start. They are the right tool when nobody knows the solution path in advance, and they are the hardest to bound, test and price. Get a supervisor working with measured results first, and you will know whether you need more.

How do agents communicate, discover each other, and hand off?

Inside one process, agents communicate through the lead: the lead writes each helper a task, and the helper returns a result. Keep that message a contract, not a conversation. Name the input, the expected shape of the output, a size limit, and what to return when there is nothing to report. "Return at most five findings, each with the line, the problem and a fix, or No findings" merges cleanly. "Have a look at this file" doesn't.

Discovery is how the lead knows which helpers exist. The AWS Well-Architected agentic AI lens recommends a capability taxonomy that drives deterministic routing, and warns against hard coding agent identifiers, because routing then breaks when an agent is replaced. In practice that means describing each helper by what it can do and letting the router match on that, with a named fallback when the preferred helper is unavailable. Across organizations, protocols such as A2A aim to standardize the same exchange.

A handoff moves control, so it must move enough context and no more. Pass the specialist the user's request, the facts gathered so far, and the reason for the handoff. Don't pass the whole transcript by default: it costs tokens on every later call and drags the previous agent's assumptions along with it.

Where do permissions and memory boundaries go?

Every helper should hold the least it needs. That sounds obvious until you notice how most frameworks delegate: the helper inherits the parent's tools and credentials, because that is the easiest default to build. Credal's guide makes the sharper point that an agent should inherit the permissions of whoever dispatched it, and that information passed between agents working for different users can leak what one user was never allowed to see.

Treat the edge between two agents like the edge between a model and a tool. The same MCP security checks apply: the model holds a handle, not a credential, each tool's schema is pinned, and risky calls pass a gate. A helper that only reads should get read tools only. A helper that writes should write to its own path, so two helpers never edit the same file at once.

Know whether your runtime's delegation is a security boundary, because often it isn't. On Orca, a subagent shares its parent's model, tools, sandbox and files, and the docs say directly that delegation is not a security boundary. What a helper does get is its own conversation: it sees only the task the parent wrote. That is the right default for splitting work. When you need a real boundary, use a separate agent with its own tools and session, and pass data between them in code.

How do retries, timeouts and idempotency work across agents?

Every rule from single tool calls still applies, and fan out multiplies the cost of getting them wrong. Our post on timeouts and retries covers the call level: a deadline per call, classify the error before retrying, and never repeat a write you can't prove never happened. Across agents, three more things matter.

Bound concurrency before you start, and know what happens at the limit. A lead that starts one helper per file will happily start two hundred. Give each helper a deadline that fits inside the run's. And key each subtask by its intent, the run plus the subtask, so a retried helper can find the first attempt's result instead of doing the work again. Here is the shape in plain Python, with no framework:

import asyncio
import hashlib
import random


MAX_WORKERS = 3          # helpers running at once
WORKER_DEADLINE = 30.0   # seconds one helper may take
ATTEMPTS = 2             # tries per subtask, for failures a retry can change




class Retryable(Exception):
    """A failure a second attempt can change: a reset, a 5xx, a read timeout."""




def task_key(run_id: str, subtask: str) -> str:
    # The key names the intent (this run, this subtask), not the attempt,
    # so a retry after a timeout can find the first attempt's result.
    return hashlib.sha256(f"{run_id}:{subtask}".encode()).hexdigest()[:16]




async def run_worker(subtask: str, key: str, done: dict) -> str:
    if key in done:  # the first attempt landed after we stopped waiting
        return done[key]
    await asyncio.sleep(random.uniform(0.1, 0.5))  # stands in for a model call
    if random.random() < 0.2:
        raise Retryable("upstream 503")
    done[key] = f"findings for {subtask}"
    return done[key]




async def supervise(run_id: str, subtasks: list[str]) -> dict[str, str]:
    gate = asyncio.Semaphore(MAX_WORKERS)
    done: dict[str, str] = {}  # in production: a durable store keyed by task_key


    async def one(subtask: str) -> tuple[str, str]:
        key = task_key(run_id, subtask)
        async with gate:
            for attempt in range(1, ATTEMPTS + 1):
                try:
                    async with asyncio.timeout(WORKER_DEADLINE):
                        return subtask, await run_worker(subtask, key, done)
                except (Retryable, TimeoutError) as error:
                    if attempt == ATTEMPTS:
                        return subtask, f"FAILED after {attempt} attempts: {error}"
        raise AssertionError("unreachable")


    results = await asyncio.gather(*(one(s) for s in subtasks))
    return dict(results)




if __name__ == "__main__":
    files = ["auth.md", "billing.md", "limits.md", "sessions.md"]
    report = asyncio.run(supervise("run-42", files))
    for subtask, outcome in report.items():
        print(f"{subtask}: {outcome}")

Three details carry the weight. The semaphore makes the limit a queue in your code, so a fifth subtask waits instead of failing. A failed helper returns a labeled failure instead of raising, so one bad file doesn't throw away the other three results. And the lead sees every outcome, including the failures, which is what lets it say "three of four reviewed" instead of quietly merging a partial answer.

When the model is the lead, the limit lives in the runtime instead, and you have to know how it behaves. On Orca, you turn delegation on with multi_agent, and max_concurrent_subagents sets the limit, 6 if you leave it out. The limit is not a queue: one helper over it is refused with "Session subagent concurrency limit reached", and the lead sees a failed call. Delegation is one level deep, so helpers can't start helpers of their own. On the Free plan, a turn can start only as many subagents as the pricing page lists. So the review team recipe builds the limit and the instructions from one constant and tells the lead to work in batches:

import os


import openai


client = openai.OpenAI(api_key=os.environ["ORCA_API_KEY"], base_url=os.environ["ORCA_BASE_URL"])


# Orca refuses a helper beyond this limit instead of queueing it,
# so the instructions tell the lead to work in batches of this size.
MAX_HELPERS = 3
INSTRUCTIONS = f"""You lead a document review. The documents are in /workspace/docs.
Give each document to its own subagent, with the file path and this task:
"Review this one file for unclear wording, missing steps, and claims without evidence.
Return at most five findings, each with the line it is about, the problem,
and a suggested fix. Return 'No findings' if there are none."
Run at most {MAX_HELPERS} subagents at a time. Any more are refused, not queued.
Start up to {MAX_HELPERS}, wait until every one of them has returned, then start the next ones.
Do not review the files yourself. When every document has been reviewed, merge the findings
into /workspace/outputs/review.md, grouped by file, most serious first."""


lead = client.beta.agents.create(
    model=os.environ["ORCA_MODEL"],
    name="review-lead",
    instructions=INSTRUCTIONS,
    multi_agent={"enabled": True, "max_concurrent_subagents": MAX_HELPERS},
)

Enabling multi_agent lets the lead delegate; it doesn't force it to. That is why the instructions say exactly when to delegate and what each helper returns, and why the recipe checks each helper's turns afterward: a closed helper is not always a successful one.

What should you trace, and how do you control cost?

Trace the tree, not the lines. Credal's list is a good minimum: a trace ID, a span per agent and action, decision logs including permission checks, data access, and cost and latency per step. The question you are building for is "who called whom, with what, and what did it cost". A flat log of model calls can't answer it once five helpers are interleaved.

Record each helper's outcome as a status, not a message. Completed, failed, refused at the limit, timed out, and cancelled are different events with different fixes. If the only record is the lead's summary, a helper that failed silently looks exactly like a helper that found nothing.

For cost, put the budget on the run, not on the agent, and count delegated work inside it. Every helper makes its own model calls, so a run that fans out to six helpers can cost several times the single-agent version for the same task. The pattern from our post on AI agent cost control applies unchanged: reserve before the expensive step, execute only after the reservation succeeds, and commit the actual result afterward. On Orca you can read cost per agent with orca usage --group-by agent, which adds up each session's cost under the agent it ran.

Finally, measure the multi-agent version against the single-agent version on the same tasks before you ship it. The harness that dispatches the calls matters as much as the pattern; our agent harness post shows how much of a run's cost and latency is decided below the model. If delegation doesn't win on your tasks, it isn't worth its tokens.

When is a single agent the better choice?

More often than the frameworks suggest. Anthropic's own post says that domains where every agent needs the same context, or where agents depend heavily on each other, are not a good fit for multi-agent systems today, and gives coding as the example: most coding tasks have fewer truly parallel parts than research.

One agent is also the better choice when the task is cheap, when latency matters more than breadth, and when the steps are a fixed sequence your code can run directly. A pipeline of plain function calls, with the model used inside each step, is often the most reliable "multi-agent" system there is. It also keeps the distinction from our post on agent infrastructure clean: the orchestration is your application's logic, and the runtime underneath should bound it, not decide it.

A production checklist for multi-agent workflows

We run through this list before a multi-agent workflow goes live. Every item is cheap to check and expensive to skip.

1. The pattern matches the dependencies. You can name which parts are independent and why the chosen pattern fits them.
2. Each helper has a written contract. Its input, its output shape, a size limit, and what it returns when there is nothing to report.
3. Concurrency and depth are bounded. You know the limit, and whether one over it queues or fails.
4. Every helper has a deadline that fits inside the run's, and one cancellation reaches all of them.
5. Retries are classified and keyed. Only failures a second attempt can change are retried, and each subtask has an idempotency key derived from its intent.
6. Helpers hold the least they need. Read-only helpers get read tools, writers write to their own paths, and credentials stay out of the model.
7. The trace is a tree. Who started whom, with what input, which tools, which outcome, at what cost.
8. Failures are visible in the answer. The lead reports which helpers failed instead of merging a partial result as if it were whole.
9. The run has a budget that counts delegated work, enforced before the call, not alerted after it.
10. It beats one agent on your tasks. You measured it on the same inputs, for quality, cost and time.

Frequently asked questions

What are multi-agent orchestration patterns?

The common ones are sequential (a pipeline), concurrent (several agents on the same input, results aggregated), handoff (one agent transfers control to a specialist), supervisor or manager (a lead calls specialists and owns the answer), and group chat (a manager picks who speaks next). Microsoft's Semantic Kernel ships sequential, concurrent, group chat, handoff and Magentic orchestration.

What are multi-agent orchestration examples?

A research agent that sends parallel helpers to search different angles and merges a cited report; a document review where each helper reviews one file; a support triage flow where an intent classifier routes to retrieval and remediation agents. Each splits into parts that can run without one another.

What are multi-agent orchestration tools?

Frameworks such as the OpenAI Agents SDK (agents as tools, and handoffs) and Semantic Kernel (five built-in orchestration patterns) give you the patterns in code. A runtime then has to run them: concurrency limits, deadlines, permissions, tracing and cost per agent.

What is a multi-agent architecture diagram?

A picture of which agent starts which, what each one receives, and where results go. For the supervisor pattern it is one lead agent, a set of helpers that each get one subtask with a clean context, and a single path back to the lead, which writes the answer.

When is a single agent better than multiple agents?

When the parts of the task depend on one another or need the same context, when the task is cheap relative to the extra tokens, and when you cannot yet measure whether delegation helps. Anthropic reports that multi-agent systems use about 15 times the tokens of a chat.