Here’s the pattern break: most people building an LLM-as-judge reach for the biggest model they can afford, because judging feels like it needs the same brainpower as doing the job. If the task is hard, the assumption goes, the judge has to be at least as capable as whatever produced the answer, otherwise how would it know what “good” looks like? Developer u/jokiruiz built one that works better with a small, local model, and the result quietly breaks that assumption. The catch that made it work has nothing to do with model size and everything to do with what you’re asking the model to decide.
The key idea
The project is a tool that reads a project’s specs and estimates which LLM it actually needs, an estimator model doing that judging job, running locally through Ollama. Feed it a description of the work, and it tells you whether a 7B model can handle it or whether you need something with more headroom. The obvious build hands the model everything: read the tasks, know the full model lineup, weigh the tradeoffs, pick a winner in one shot. That’s not what shipped.
Instead, the local model does exactly one narrow thing: rate how demanding the work is across a few fixed dimensions, reasoning depth, context size, domain specialization. Nothing about pricing, nothing about which vendor, nothing about which specific model exists this month. Just a difficulty profile. Turning that rating into a recommended model is deterministic rules in YAML. No model touches that step at all, so when the recommendation changes because a new model got added to the lineup, that’s a config edit, not a prompt rewrite and a fresh round of testing.
Old way vs new way
The old way (the one most eval pipelines default to): delegate as much reasoning as possible to the model, because it’s flexible and you don’t have to write the logic yourself. It feels efficient at first, one prompt handles intake, scoring, and recommendation together. The problem shows up later, when the output is wrong and you can’t tell which step in the chain broke. Was the difficulty assessment off, or did the model just pick a bad mapping from difficulty to model choice? With everything fused into one call, you can’t isolate the failure, you can only reroll the whole thing and hope.
The new way: split fuzzy from deterministic. The model only judges what genuinely needs judgement. Everything downstream of that judgement is code you can read, test, and debug like normal code. As u/jokiruiz put it, every inch of reasoning you hand to the model is an inch of variance you inherit. Keep that inch as small as possible and the variance shrinks to something you can actually reason about, and actually test with normal unit tests instead of vibes.
The other reframe worth stealing: a judge doesn’t need to be able to do the work. Estimating how hard a task is sits closer to a recruiter writing a job spec than to the engineer filling the role. A recruiter doesn’t need to write the code to know a role calls for someone senior. That’s the whole reason a small local model is enough here, not a compromise, the actual right tool. It also means the judging step runs free and fast, no API bill, no round trip latency, which matters a lot once you’re scoring dozens of specs a day instead of one.
Practical steps 🛠️
- 📋 Isolate the judgement call. Write down the one thing that actually requires the model to weigh ambiguous signals. Everything else becomes rules, not prompts. If you catch yourself asking the model to also format the output, pick a vendor, or apply business logic, pull those out and hardcode them.
- ⚙️ Move the mapping to code. Once you know the demand profile, translating it into a decision is deterministic. Put it in YAML or a config file you can diff, not another prompt. That way when the model lineup shifts, or your thresholds turn out to be too aggressive, you fix one file and move on instead of re-testing an entire prompt chain.
- 📦 Score in one pass, not item by item. Per-item scoring on a batch of tasks produces noise, and it costs you the full latency multiple, once per item, for no extra signal. Batch the specs, score them together, and the model has more context to calibrate against, which tends to produce more consistent ratings anyway.
- Make “not enough information” a real output. Models default to answering confidently even off three vague bullet points. Give insufficiency its own category and its own downstream handling, it’s a feature. A judge that can say “I don’t know yet” is more trustworthy than one that always hands you a confident number.
He also prints the reasoning behind every verdict, mostly for his own debugging, since an opaque LLM-as-judge is just guesswork with extra steps. Having the reasoning on hand turns a wrong verdict into a quick fix instead of a mystery you have to reverse-engineer from the final score alone.
Try it
The code’s open source if you want to see how far the fuzzy/deterministic split goes: github.com/JoaquinRuiz/SpecJudge. If you’re building any kind of LLM-as-judge setup, the exercise worth doing this week is simple: list every decision your judge currently makes, and ask which ones could be a config file instead!
Frequently Asked Questions
Q: How do you structure the prompt to get the model to say “not enough information” instead of guessing?
This is tricky because models are trained to provide answers. One effective approach is to explicitly tell the model that “I don’t have enough information” is a valid, valuable output, not a failure. In practice, include examples in your prompt where the right answer is “abstain” or “unsupported,” and design your output schema to make “not enough info” a first-class option. Some users also add an “evidence required” step: require the model to point to the exact spec fragments supporting its estimate, returning “unsupported” when it can’t, this forces honest reflection about gaps.
Q: Is evaluating the whole project at once better than evaluating tasks individually?
Whole-project evaluation cuts latency and reveals task relationships, but commenters raised a valid concern: 37 trivial tasks can dilute the estimate and mask the 3 hard ones that actually determine which model you need. A hybrid approach might work better: keep the single pass for relationships, but flag the hardest task separately and apply max/threshold rules (e.g., “if any task demands deep reasoning, project minimum is this model”). This keeps efficiency while catching the edge case.
Q: Can a small model really judge work it couldn’t do itself?
Yes, estimating difficulty is a different skill from doing the work, closer to writing a job spec than filling the role. A small model can see “this needs deep reasoning” without doing the reasoning itself. This principle is strongest when the spec clearly documents the challenge; it weakens when specs are vague or hide failure modes. Calibrate your judge against real completed projects to find where it breaks.
Q: How do you verify that the judge’s recommendations are actually correct?
Run the judge on completed projects where you know which model actually worked, then measure both accuracy and abstention quality. Include varied specs, vague, contradictory, adversarial, to stress-test. Some teams add evidence tracking: require the judge to cite which spec fragments support each dimension, then validate coverage in code. This makes judgment auditable and tells you whether the judge is reasoning or just confabulating confidence.
Building an LLM-as-judge with a small local model — the biggest win was taking judgement away from it
by u/jokiruiz in PromptEngineering