Latency and streaming

Agents are slower than ordinary software, and the reasons are structural. Some of it you can fix.

The two numbers

Time to first token is how long before anything appears. This is what users experience as responsiveness.

Total generation time is how long until the answer is complete. Roughly output length divided by tokens per second.

They are affected by different things. A long prompt raises the first. A long answer raises the second.

Why agents feel slower than chat

An agent does not answer in one call. It calls a tool, waits, reads the result, calls another, and eventually answers.

Five tool calls means five model round trips plus five tool executions. Each one is fine; the sequence is what takes twenty seconds.

The fix is usually fewer steps, not a faster model. A better tool that answers in one call beats three calls to worse ones.

What you can actually change

Stream the output. Does not make anything faster, transforms how it feels. Time to first token becomes the number the user experiences instead of total time.

Shorten the prompt. Input length affects time to first token directly. This is another reason to keep context lean.

Ask for shorter answers. Generation time is roughly proportional to output length. "Answer in under 100 words" is a latency fix.

Use a smaller model where it fits. Small models are substantially faster. For classification and extraction the quality difference is often invisible and the speed difference is not.

Run independent tools in parallel. If three tool calls do not depend on each other, do not wait for each in turn. Many harnesses support this and many implementations do not use it.

Cache. Prompt caching reduces time to first token as well as cost, because the cached prefix does not need reprocessing.

Show progress

For anything over a couple of seconds, show what is happening. "Searching the codebase" then "reading three files" then the answer.

A visible sequence feels dramatically faster than a spinner, and it also makes failures legible: the user can see it got stuck reading files rather than watching an endless spinner.

Measure the tail

Averages lie about latency. The average is fine and the complaints come from the ninety fifth percentile.

Track p95 and p99. Long conversations, large retrieved contexts, and tool timeouts all live in the tail and are invisible in the mean.

Set timeouts on tools

A tool call with no timeout can hang the entire agent indefinitely. An MCP server waiting on an unreachable network endpoint will wait forever unless you stop it.

Set a timeout, and return an error message the model can act on when it fires.

The honest limit

An agent doing real multi step work will take seconds, not milliseconds. You can make it feel responsive with streaming and progress, and you can remove wasted steps.

You cannot make it instant, and designing an interface that pretends otherwise sets an expectation you will not meet.