Handling failures and retries

An agent has all the failure modes of a distributed system plus several of its own. Plan for them.

The failures you will actually hit

Rate limits. You will hit them, especially in bursts. Retry with exponential backoff and jitter, and respect any retry-after header.

Overload. Providers return 429 and 529 under load. These are transient. Retry.

Timeouts. A tool hangs, or generation stalls. Without a timeout the whole agent waits forever.

Malformed output. You asked for JSON and got JSON wrapped in prose, or with a trailing comma.

Context overflow. Input plus reply exceeds the window. On newer models generation can stop partway rather than erroring, which is easy to miss.

The unproductive loop. The agent retries the same failing call repeatedly. Not an error from the API's point of view, just expensive.

Retry the transient, not the wrong

Retry rate limits, overload, and network errors. Do not retry a 400 for a malformed request, or a 401. Those will fail identically forever and you are paying for each attempt.

Always cap retries. Unbounded retry against a failing dependency is how a small outage becomes a large bill.

Make errors instructive

This is the highest leverage change available, and it belongs in your tools rather than your prompt.

• Error: ENOENT produces the same call again.
• File not found at src/confg.ts. Use search_files to find the correct path. produces a different, better call.

The model can only recover if the error tells it what to try. Write tool errors as instructions to a colleague.

Break loops explicitly

Cap total turns per task and total tool calls. When the cap is hit, stop and report, rather than continuing.

Also detect repetition: if the same tool is called with identical arguments three times, something is stuck. Intervene rather than letting it run.

Validate structured output, then repair

If you need JSON, parse it. When parsing fails, do not fail the whole task immediately.

First try to extract the JSON from surrounding prose, which handles the most common case. If that fails, send it back with the parse error and ask for a correction. One repair attempt fixes most instances. Two is rarely better than one.

Where the API offers structured output or tool calling with a schema, use it. It is more reliable than asking nicely in a prompt.

Fail visibly, not silently

The worst outcome is an agent that reports success having done half the work.

Verify rather than trusting the report. If it says the tests pass, run them. If it says the file was written, check it exists. The claim and the fact are different things, and only one of them is checkable.

Degrade to something honest

When an agent cannot complete a task, the right output is a clear statement of what it did, what it could not do, and why.

That is genuinely useful. A confident summary that omits the failure is not, and it is what you get by default if you do not design for the alternative.