Yesterday a clever build landed on r/PromptEngineering. It’s a Claude skill called Eval Suite Auditor, and the story behind it is the kind that makes you check your own dashboards.
The author’s eval passed a run at 0.9. The LLM judge had never executed. The judge model they’d hard-coded had been retired, the service swallowed the 404 and returned a default 0.5, and four deterministic checks carried the average. Green result, dead judge.
What’s new
Most eval advice says to write better evals. This skill goes the other way. You ask Claude to “audit my eval suite” or “can I trust this judge score,” and it checks the suite you already have for silent failure.
The skill is a single skill.md file. The repo link is at the bottom.
The twist
It doesn’t read your code and give you an opinion. Every finding needs a probe that actually ran against the real path. Every probe also gets a positive control, meaning the same path with a healthy judge and a good input, which must pass. If the control fails, your probe environment is broken and none of the findings count.
The author found that the dead judge was only the first case. Fixing it turned up more of the same shape:
- Three passing checks plus one dead judge still returned a pass.
[pass, pass, pass, FAIL]and[FAIL, pass, pass, pass]gave different verdicts.- A repair threshold sat inside the judge’s own noise.
- Some stored rows were placeholder scores that history views read as real failures.
That’s where the 14 failure classes come from.
The mini-workflow
🗺️ Map the pipeline. Every entry point that produces a score, which judge it uses, what it returns when the judge is down, and which components act on the result (gates, auto-repair, dashboards, CI). Anything it can’t run goes in a “not probed” list instead of being implied covered.
🔬 Probe the 14 classes. Dead judge, partial judge failure, unparseable reply, default-that-looks-like-a-score, aggregation that hides a failure, disagreeing thresholds, baseline and candidate graded by different instruments, same-family judge, oversized or injectable input, variance against the decision margin, score resolution, consumers acting on “unknown,” stored placeholder rows, and swallowed exceptions.
🧪 Fix probe-first, only if you ask. Each finding becomes an expected-failure test before any code changes. “Fixed” means a probe flipped, not that a diff looked right.
🎯 Check calibration honestly. With one human rater, it asks for a blind re-rating a day later. A failed comparison counts as inconclusive, not proof the judge is wrong. It also refuses to reuse the same labels to tune a threshold afterward.
The last step is a report. It’s a table with severity, the exact probe and result, and the exposure in real data, which is either a count from a query or “latent” plus the trigger condition. After the table come “verified sound,” “not probed,” and “open decisions” for the owner.
Pro tips
- Start with class 2, partial judge failure. Plenty of suites test “all judges failed” and miss “one of four judges failed,” which is the case where the average of the survivors looks fine.
- Run the variance probe. The skill’s example: a repair loop accepts a rewrite if it beats the original by 0.03. Score one prompt 10 times and say the standard deviation is 0.05. The spread of a difference between two single runs is about 0.07, so a 0.03 margin is inside the noise. The loop is committing changes on luck.
- Treat errors as their own state. One commenter’s fix is to put a status on the result type, separate from the score, so a failed call comes back as no score plus an error instead of a 0.5 that averages in.
- Query your stored rows. Look for suspicious constants and rows with error text in the explanation. Your history views may be reading placeholders as real failures.
The honest part
The author admits the joke writes itself. The skill was generated through their own pipeline, and their own judge gave it 0.0, 0.8, 0.0 and 0.5 on one rubric item across four runs. That’s exactly the variance class the skill describes. They only ran it four times, not the ten that class asks for, so they call it a smell rather than a measurement. They won’t cite a score and would rather you read it and say what it misses. I respect that.
One caveat from me: I haven’t run this skill. I’m reporting what the post and the skill file say. The 14-class list reads solid to me, but the real test is your own suite.
The thread already has people nodding along. One reader had a service silently fall back to a cached score from three weeks earlier. Another pointed out that “installed, read, and followed” are three different states that most skill evals collapse into one.
Your move
Grab the skill, point it at an eval suite you trust a little too much, and see which probe fails first. Then tell the author which of the 14 you’d run first, and whether you’ve hit a failure class that isn’t on the list. 🏴☠
Repo: https://github.com/nivlewd1/prompt-optimizer/tree/main/skill/eval-suite-auditor
The author also built a platform that generates skills like this from a goal description, with a free tier: promptoptimizer.xyz/about/context-engineer
Frequently Asked Questions
Q: My judge call failed. Why shouldn’t it just return a neutral score?
A fallback like 0.5 looks like a real grade, so it quietly pulls your average toward the middle. Return a status next to the score instead, so a failed call comes back as no score plus an error. Then make sure your batch stats count only completed evals, and have the suite fail loudly once too many judgments are missing.
Q: Is a partial judge failure really worse than a full outage?
One commenter argues it is, because a total outage is obvious and a partial one hides inside an average that still looks healthy. Report how many judgments actually completed alongside every score. Treat low coverage as a failed run, even when the number itself looks reasonable.
Q: How many samples do I need before I trust a repair loop?
One commenter suggests 10 or more, since a repair decision made inside the judge’s own noise is close to a coin flip. Running that many samples can feel expensive, though. Compare the spread of repeated judge scores against the gap your threshold requires. If the noise is wider than that gap, either take more samples or widen the margin.
Q: Why do my scores shift overnight when my code hasn’t changed?
Providers update judge models behind the same name, so the grader can change without any change on your side. Pin dated model versions wherever the provider offers them. Keep a fixed set of graded examples and rerun it on a schedule. If that set moves and your code didn’t, the judge changed, not your prompt.
Eval Suite Auditor skill.md: my eval passed at 0.9 while its judge had never executed
by u/Parking-Kangaroo-63 in PromptEngineering