17 out of 20 versus 11 out of 20. That’s the gap in first-attempt tool accuracy when one Redditor (u/Possible-Fun-1122) swapped a code-dumping search skill for one that returns ranked file paths. Same repo, same test harness, fresh session every run, 20 multi-file debugging tasks covering async race conditions, schema drift and integration test failures.
Small sample, own codebase, single run per task. The author says so up front. But the direction is worth your attention if you build custom skills for Claude Code, because the fix is cheap and the effect showed up in every metric they tracked.
What the numbers say
Arm A was a monolithic retrieval skill. It searched project notes and past bug reports, then pasted full text matches and complete code blocks into the conversation. Arm B returned ranked file paths with one-line abstracts, and the agent had to read only what it needed. Everything else stayed constant, including the prompts, so the return format is the only variable that moved.
- First-attempt tool accuracy: 17/20 for B, 11/20 for A. In A, the agent hallucinated parameter names and repeated invalid shell flags after a big search dump. Once the early context filled with unrelated code, the details of the tool schema got buried, and the model started guessing.
- Context growth: cut by more than half on B. Inspecting request payloads with cost-xray showed A hitting context compaction by turn 8, which also killed the prompt cache discount. B stayed well under the threshold. Compaction is the expensive moment: you pay full price for tokens that were cheap a turn earlier, and the summary that replaces them is lossy.
- First-pass fix rate: 14/20 for B, 8/20 for A, without human help. The author’s explanation is that A’s agent kept trying to reconcile conflicting snippets from broad search hits. An old bug report that mentions a similar function can look like current evidence, and the agent treated it that way.
The pattern: a skill’s return payload is not free. Every line it dumps becomes context the model has to carry into every later tool call. A 4,000-token dump on turn 2 gets re-read on turns 3, 4, 5 and every turn after, whether it helps or not.
3 practical applications
🔹 Return pointers, not payloads. Make your search skill output a ranked list of paths plus a one-line abstract each. Something like src/queue/worker.ts: retry loop, owns the lock release is enough. Let the agent decide which two files deserve a full read. Include a line number or symbol name when you have one, so the follow-up read can be narrow.
🔹 Measure what your skills actually add. The author used cost-xray to see the request-time token distribution per tool call. Whatever tool you use, look at the payload before you guess where the bloat is. A quick version: log the character count of every skill response for one session and sort descending. The top two or three entries are usually responsible for most of the growth.
🔹 Watch for the compaction cliff. If your loop hits compaction around turn 8, you lose cache savings and context quality at the same moment. Count tokens per turn on a 10-turn task and find which tool call causes the jump. The culprit is often a single oversized result, and it’s usually a search or a log tail.
Tips and pitfalls
Pros of path-first retrieval: fewer conflicting snippets, lower context growth, higher first-pass success, and a cleaner trace to read when something goes wrong.
Cons: it costs an extra round trip. On a simple bug that needed a two-line helper fix, Arm B had to read the abstract on turn 1 and open the file on turn 2. Arm A’s instant dump was faster there. One commenter asked what B cost in extra round trips overall, and the post doesn’t give that number, so measure it on your own tasks before you trust the tradeoff.
A few guardrails:
- Don’t go path-only for tiny lookups. A hybrid works: return the snippet inline when the match is under a few lines, paths when it’s larger. A threshold of around 20 lines is a reasonable starting point to tune from.
- Write abstracts that let the agent skip files. A vague one-liner like “utility functions” just moves the cost to a wasted read. Name the behavior, the owner and the failure mode if you know it.
- Rank results and cap the list. Five to eight paths is plenty. Twenty paths with no ordering recreates the dump in a different shape.
- Keep your own test set honest. The author designed these tasks, so their habits shaped the results. Run your own 10 tasks, twice each, and write down the tasks before you change the skill so you can’t tune the test to the answer.
Your move
Pick the one skill in your setup that returns the biggest blob of text. Cap its output, switch it to paths plus abstracts, and run the same three bugs before and after. Record first-call accuracy, turn count and the turn where compaction kicks in. Then tell us what happened to your first-call accuracy. Numbers from your repo beat anyone’s anecdote, including this one.
Frequently Asked Questions
Q: Why would bigger search results hurt tool-calling accuracy?
Commenters who’ve seen this before point to context density. Once the window fills with raw code, the model starts mixing up parameter names or grabbing the wrong flag. Shorter, ranked results keep the agent focused on the call it’s making, which fits the gap in this test (17 of 20 first calls vs. 11 of 20).
Q: Doesn’t path-first retrieval mean more round trips?
Yes, it usually adds 2 to 4 read calls per task. The number that matters is cost per solved task, not cost per call. Track total tokens and wall time for the tasks that actually pass, since the extra reads could still come out lower overall.
Q: Is there a middle option between full snippets and bare file paths?
One commenter suggests returning file paths, line ranges, and a one-line summary, then hard-capping the output around 40 lines with a note like ‘truncated, N more matches’. They argue the cap matters more than the format. A short, ranked list signals that the agent should open the file itself instead of copying from the search output.
Q: Could pruning old tool results between turns fix the drop?
It might, if the failures cluster late in the 10-turn loop. That would point to accumulated context rather than the single big dump, and pruning stale results could recover most of the gap. The post doesn’t yet say when the Arm A failures happened, so check that timing before changing the skill.
I benchmarked 20 multi-file debugging tasks: monolithic tool search vs progressive path retrieval in agent skills
by u/Possible-Fun-1122 in PromptEngineering