Here’s a bold claim: most AI judges aren’t grading your work, they’re grading how well they can write about your work. Give a model a rating to justify and it’ll hand you a smooth paragraph every time, whether the rating is right or not. Nothing in that setup separates a correct call from a confident guess wearing a suit.
That’s the gap SpecJudge v0.2.0 just closed. SpecJudge is an MIT-licensed CLI that reads your project’s specs and tasks and tells you which AI model actually fits the job, quality versus price, instead of you eyeballing it. The judge used to return a rating plus an explanation. Now every rated dimension has to cite the exact fragment of your spec backing it up, and the tool checks, deterministically, that the fragment is real. Fake a citation and the whole assessment gets tossed, not just that one field. Can’t ground a dimension in the text? It comes back “unsupported” instead of quietly passing as easy, which is what used to happen and made thin specs look sturdier than they were. 🔍
Three things stood out once this shipped:
📊 The regression suite (12 reference projects, CI plus a local eval script) paid for itself immediately: it caught 8B judges, the most common local setup, failing on every single project. Not a reasoning failure either. The ratings and justifications were correct, but the model was dropping [true] where a citation ID belonged. Turns out “format: json” in Ollama guarantees valid JSON, not the JSON you asked for. Sending an actual schema flipped the result from 0 of 9 usable cases to 9 of 9.
🎯 Judge sampling is now pinned, so the same project produces the same recommendation on every run. Sounds minor until you remember this tool exists to answer one question: should you spend money on this model or not. A recommendation that flips between runs isn’t a recommendation, it’s noise with a UI.
⚙️ The breaking change is worth flagging if you’re already running this: it needs Ollama 0.5.0 or newer, so check your version before you upgrade.
If you’re running any local eval setup that grades on vibes instead of evidence, this is worth a look. pip install specjudge, code’s on GitHub at JoaquinRuiz/SpecJudge. Go make your judge show its work. 🚀
Frequently Asked Questions
Q: Why does SpecJudge require judges to cite specific parts of your spec?
It’s clever: by requiring citations and verifying they actually exist in your spec, the tool makes it impossible for the AI to fake confidence with well-written explanations that aren’t grounded. You’re not relying on model honesty , you’re structurally blocking overconfidence. One user nailed it: “You’re not asking the model to be honest, you’re just making it impossible for it to lie without getting caught.”
Q: What was the [true] bug that broke every 8B model?
Ollama’s “format: json” only guarantees syntactically valid JSON, not that it matches your schema , so 8B judges were returning `[true]` (valid JSON, wrong thing) instead of citation IDs. Once SpecJudge sent a proper schema instead of just “format: json”, it jumped from 0/9 to 9/9 passing. If you’ve wrestled with Ollama’s JSON mode, this is a deeply relatable pain point.
Q: Why does pinning the random sampling make such a difference?
Without it, you’d get different model recommendations every time you run , fine for exploring, useless for real decisions. Pinned sampling ensures consistent results, which matters when you’re seriously comparing models and deciding where to spend money. It’s what turns a research toy into a tool you can actually trust.
SpecJudge v0.2.0: the judge now has to cite evidence that actually exists — and a bug that broke every 8B model until it did
by u/jokiruiz in PromptEngineering