The quick answer: what does an MCP timeout and retry policy need?
An MCP timeout tells you one thing: the caller stopped waiting. It doesn't tell you what the tool did. So the policy we run has three parts. Every call gets a deadline that fits inside the run. Every failure gets classified before anything is repeated. And a write is never retried unless the tool can recognise the repeat, because the only thing worse than a slow tool is the same ticket created twice at three in the morning.
We run agents unattended on a schedule, so we've had every version of this go wrong: a tool server that answered after we'd given up, a retry loop that turned one outage into a burst, a provider that returned a 429 with a retry hint we ignored. This post is the policy we settled on, the code shape it takes, and what our own platform does and doesn't do at that boundary.
What does a timeout tell us, and what doesn't it?
A timeout is evidence about our side of the wire. Our process armed a timer, the timer fired, and we stopped reading, and that is the whole of what we know. The tool server saw a request arrive and may have finished the work a second after we closed the connection. It may have never received the request, or it may still be running as we read this. From where we sit those three cases look identical, and they aren't.
MCP makes this sharper, not softer. The protocol makes it easy to put a dozen tools behind one agent loop, and each tool brings its own latency, its own rate limits and its own failure shapes into that loop. A single reasoning step can now block on a database, a search API and a git host in sequence, and the model has no natural sense of how long any of them should take.
So the first rule we hold is that a timeout closes our wait but doesn't prove anything was rolled back. For a read, that distinction is cheap: reading twice costs time and nothing else. For a write, it's the whole problem. The retry that feels safe because the first attempt "failed" is the retry that duplicates the side effect.
We also record what we knew at the moment we gave up: the tool name, the run and attempt identifiers, the deadline we set, how long we actually waited, and the classification we assigned. Those fields are what let someone the next morning tell a provider outage from an exhausted retry budget, and a rate limit from a permission failure that a generic wrapper had dressed up as "transient".
Why does every call need its own deadline?
Because a tool without a deadline can hold the run hostage. A scheduled agent with a two hour budget and one hung tool call spends two hours doing nothing and bills for all of it. The deadline is the smallest boundary in the system, and it's the one that makes every larger boundary real.
We size deadlines from the tool, not from a global default. A lookup that returns in a second gets a few seconds. A repository mutation that clones and pushes gets more. What we don't do is give a tool a deadline longer than the time left in the run. If the run has ten minutes, a fifteen minute tool deadline is a lie, and nested deadlines are how the outer one stays honest.
We pass one cancellation through the whole call. The run's deadline, the tool's deadline and the backoff between attempts all derive from the same signal, so when the run is cancelled the tool stops and the sleep stops too. A backoff that outlives its run is a small leak that's easy to write and annoying to find, and we've found it in our own code more than once.
How do we decide whether a call can retry?
We classify before we repeat, every time, in a fixed order. The class of the failure decides the action, and the same six questions get asked before any second attempt. It sounds bureaucratic, and it's the opposite: it removes a decision the model would otherwise make on its own, badly, at two in the morning.
A 4xx stops the call on the first answer. Bad input, missing permission, a resource that isn't there: another attempt changes nothing, and each one costs a model turn. A 429 with a retry-after is the one case where the provider tells us the answer, so we wait exactly that long and try once. A 5xx or a connection reset gets a backoff and another attempt inside the budget. A timeout on a read retries, because reading twice is harmless. A timeout on a write goes to the key check, which the next section covers. A timeout on a write with no key stops the call and records the outcome as unknown, which is a real state and deserves a real word.
The budget is small and belongs to the call: three attempts is the number we use, all inside the run's deadline. Backoff has jitter, because ten runs that share a dependency and retry on the same schedule turn one outage into a thundering herd. And a call that would need to sleep past the run's deadline to retry doesn't retry at all; it stops and records why in the run.
type Outcome =
| { status: 'ok'; value: unknown }
| { status: 'stopped'; reason: string; attempts: number }
| { status: 'unknown'; reason: string; attempts: number }
async function callWithPolicy(tool: Tool, args: Args, run: RunContext): Promise<Outcome> {
const budget = 3
for (let attempt = 1; attempt <= budget; attempt++) {
const signal = AbortSignal.any([run.signal, AbortSignal.timeout(tool.deadlineMs)])
const result = await tool.call(args, { signal })
const verdict = classify(result, tool)
if (verdict.kind === 'ok') return { status: 'ok', value: result.value }
if (verdict.kind === 'stop') return { status: 'stopped', reason: verdict.reason, attempts: attempt }
if (verdict.kind === 'unknown') return { status: 'unknown', reason: verdict.reason, attempts: attempt }
// verdict.kind === 'retry': a reset, a 5xx, a read timeout, or a write whose key says it never landed
if (attempt === budget) return { status: 'stopped', reason: 'retry budget exhausted', attempts: attempt }
const wait = verdict.retryAfterMs ?? backoffWithJitter(attempt)
if (wait > run.remainingMs()) return { status: 'stopped', reason: 'no time left to retry', attempts: attempt }
await sleep(wait, run.signal)
}
return { status: 'stopped', reason: 'unreachable', attempts: budget }
}
Three things in that sample are load bearing. The classification runs outside the loop, so the loop never decides on its own what a failure means. The budget check comes before the sleep, so the last attempt never pays for a backoff it can't use. And unknown is a first class outcome, returned and recorded, not folded into stopped. Everything else in the sample is plumbing.
What makes a write safe to retry?
Only an idempotency key makes a write safe to retry, and nothing else comes close. The key is a value the caller chooses before the first attempt, sends with every attempt, and that the tool uses to recognise a repeat. When the key is there, a retry after a timeout is a question rather than a gamble: did the first attempt land? The tool answers by returning the existing result instead of creating a second one.
The key has to represent intent, not transport. A fresh request ID on every attempt is exactly the wrong key, because it makes every retry look new. We derive it from the run, the step and the target: this run, creating this comment, on this pull request. A reconnect from a different process reuses it, because the intent didn't change.
Most MCP servers don't expose an idempotency key today, and that's the honest limit of this whole post. Where the tool offers one, use it on every write. Where it doesn't but the provider exposes a query, the adapter checks for the side effect before retrying: does a comment with this marker already exist on this pull request. Where neither exists, the write isn't retryable after a timeout, full stop, and the run reports it as unknown for a human to resolve. That's slower than a blind retry, and it's also the only version that never creates the ticket twice.
What does the run around the call have to promise?
The call's policy only holds if the run around it has boundaries of its own. Three of them matter more than the rest. The run has a deadline that every tool deadline nests inside. The run has a retry budget that survives reconnects, so an overnight job can't make unlimited attempts while each individual request looks reasonable. And the run has an identity that a client can come back to.
That last one is how we handle work that outgrows a request. When a step can take longer than any sensible timeout, or contains several writes, or waits on a human, it moves behind a durable run boundary. The client gets a run ID and polls or subscribes instead of holding a connection open. A gateway timeout then stops being a duplicate factory, because the second submission can find the first run's state instead of starting again.
Circuit breaking handles the other failure shape, the dependency that keeps failing. A circuit records recent failures per tool or per server, opens after a threshold, and rejects new calls for a cool-down instead of letting every step in the loop discover the outage on its own. Then one probe call tests recovery. Without it, a five minute provider outage costs every retry budget of every run that touched that tool.
And we don't confuse a heartbeat with progress. A worker can emit a healthy heartbeat every second while its current tool call has been stuck for four minutes. What we watch is the age of the current attempt and whether expected outputs appear, which is the same instinct that runs through our headless Claude Code setup: a process being alive is not the same as work being done.
How do we do this in Orca?
Our agents don't call MCP servers directly. Tool calls go through a bridge that keeps full schemas server side and shows the model a small card, which is the design from our post on cutting MCP schema tokens. The bridge is also where the timeout lives, so it's worth being precise about what it does and what it leaves to the agent.
Every tool invocation through the bridge runs under a thirty second deadline that covers connecting to the server and the call itself. The bridge makes no retries of its own: the transport's retry setting is switched off, and the telemetry reporter that records each call will retry its own reporting up to three times but, in the code's words, never invokes or retries an MCP action. A result larger than eight mebibytes is refused rather than truncated. And every failure is mapped to one of four named classes before it reaches the agent: deadline exceeded, cancelled, response too large, unsafe endpoint. Anything else comes back as a generic provider failure.
That last sentence is the first honest gap. Four classes is fewer than the six in Figure 02. The bridge doesn't see HTTP status codes from the tool server, so a 429 and a 500 both arrive at the agent as "provider request failed", and the decision between waiting and stopping falls to the model. It usually decides sensibly, and usually isn't a policy.
The retry budget lives in the kernel the agent runs on, the same one described in the Orca Harness post. It has a tool retry extension that does what Figure 02 says: three attempts, 250 milliseconds of backoff, 5xx retried and 4xx left alone, and when attempts run out the model sees the last real failure rather than a synthetic one. It ships switched off, so a call today gets one attempt unless the operator turns it on. Model calls are different: those retry by default, with no attempt cap and a 200 millisecond base backoff, and a 429's retry-after is honoured as a minimum wait. We learned the value of that one the hard way this week, when a 429 from our model gateway ended a run four seconds in because the build we had in production predated the change.
Around all of it sits the spend boundary from our post on agent cost control. A run is admitted only after billing confirms a balance, with a three second deadline on that check and a refusal if it can't be verified. Each run holds twenty five cents plus ten minutes of the machine it asked for, released when it ends and expiring after six hours if it never does. Machine time is metered every sixty seconds while a run is active, and a watchdog checks the balance every thirty seconds and cancels runs once credits are gone. A retry loop that never converges still stops. It stops on money and time, which is the right backstop and the wrong first line.
So the gap, stated plainly: our platform bounds the call's time and the run's spend, and it doesn't bound the side effect. The bridge carries no idempotency key, so it can't tell attempt two from attempt one, and it can't check with the tool before a retry. The per call retry budget exists but is opt in. Both are on our list, and the key is the harder one, because it needs the tool server to take part. Until then, the rule from the writes section applies to our own agents: a timed out write with no way to check is reported, not repeated.
What should we build first, and in what order?
Deadlines first, on every call, sized per tool and nested inside the run. This is a day of work and it removes the failure that costs the most: the hung call that bills for hours. Make the deadline and the outcome visible in the run record before adding anything else, so the next steps have data to work from.
Classification comes second, and it is one function. Put the six questions from Figure 02 in one function, return a verdict, and make unknown a state the run can end in. This is where most retry wrappers go wrong: they retry on "any error" and hide a permission failure behind three identical attempts.
Keys come third, on one write before any other. Pick the mutation that would hurt most if duplicated, give it a key derived from intent, and make the adapter check before it retries, then widen to the next write. Every write without a key stays non retryable after a timeout until it has one, and the run says so.
Then the run boundary: a retry budget that survives reconnects, a durable run ID for anything that outgrows a request, and a circuit per dependency. Then the tests nobody enjoys writing: the server completes after the timeout, the retry returns a different answer, the circuit opens mid run, the worker dies after the write and before the checkpoint. A green happy path proves almost none of this, and every one of those cases has happened to us.
The version of this that works isn't clever. It's a deadline, a classifier, a key, and a budget, in that order, each one visible in the record the run leaves behind. The tool can still be slow. The agent can still be wrong. Neither of them can quietly do the same thing twice.
Frequently asked questions
What does an MCP timeout actually mean?
It means the caller stopped waiting when its deadline passed. It says nothing about whether the tool server finished, failed, or is still working, so the outcome of the call is unknown until something checks.
How many times should an MCP tool call retry?
Keep a small budget, two or three attempts per call, and only for error classes that a second attempt can change: connection resets, 5xx responses, a 429 with a retry-after, and timeouts on reads. Everything else stops on the first answer.
Why is retrying a write after a timeout dangerous?
The server may have completed the write after the client gave up, so a blind retry can create the same ticket, comment, or commit twice. A retry on a write is only safe once an idempotency key lets the tool recognise the repeat.
Should every MCP tool have the same timeout?
No. A lookup that answers in a second and a repository mutation that takes ten need different deadlines, and every deadline has to fit inside the deadline of the run that made the call.
How does Orca bound what a retry loop can spend?
Every run starts with a credit hold, machine time is metered every sixty seconds while a run is active, and a watchdog checks the balance every thirty seconds and cancels runs once credits are gone. Those bound the money, not the side effect.
When should a circuit breaker open on an MCP server?
When the same dependency keeps failing across calls. The circuit rejects new calls for a cool-down period so the agent stops adding load, then lets one probe through to test recovery.