Measuring tool use reliability

Public benchmarks mostly measure reasoning and knowledge. Agents mostly fail at picking the right tool and calling it correctly. If you are building an agent, this gap is yours to close.

The four things worth counting

Selection. Given a task, did it choose the right tool? The most common failure, and almost always caused by tool descriptions rather than the model.

Arguments. Were the parameters correct and complete? Wrong paths, missing required fields, invented ids.

Sequencing. For multi step work, did it do things in a workable order? Reading before writing, listing before selecting.

Recovery. When a call failed, did it change approach, or retry the same thing?

Track these separately. A single "did the task succeed" number tells you something is wrong without telling you what.

Building the cases

A case is a task plus the expected tool behaviour:

{
  "id": "find-config",
  "input": "What port does the dev server run on?",
  "expect_tool": "search_files",
  "reject_tool": "read_file",
  "note": "Should search rather than guess which file to read"
}

reject_tool is worth having. Some failures are about doing the wrong thing rather than failing to do the right one.

Test recovery deliberately

Make a tool fail on purpose. Point at a file that does not exist, return a permission error, return an empty result.

Then check what happens next. Good behaviour is trying something different or asking you. Bad behaviour is the same call again.

This is the most valuable test in the set and almost nobody writes it, because it requires deliberately breaking something.

Count the calls

Record how many tool calls each task takes. A task that used to take four calls and now takes eleven has regressed, even if the final answer is right.

Call count is a good early warning. It moves before success rate does, because the agent is thrashing before it starts failing outright.

Fix descriptions, not prompts

When selection is wrong, the instinct is to add a line to the system prompt telling it which tool to use.

That works for one case and does not generalise. The description is what the model reads when deciding, and it travels with the tool everywhere. Fix it there. See tool design that agents can use.

Test with your real tool set

An agent with three tools picks correctly far more often than the same agent with twenty. If you test with a trimmed set and ship with everything connected, your measurements do not describe production.

This also demonstrates something useful: if adding tools drops your selection accuracy sharply, you have too many, or several overlap.

What good looks like

There is no universal number. Measure where you are now, change one thing, measure again.

The point is not hitting a benchmark. It is knowing whether last week's change to a tool description helped or hurt.