Someone on r/PromptEngineering dropped a tool yesterday that made me sit up: judgeDjudge, a way to evaluate the evaluator.
Quick context: LLM-as-a-judge is everywhere now. You ask one model to score another model’s output and call it done. But almost nobody stress-tests the judge itself. u/maylad31 built a rig to do exactly that.
Here’s the twist: it’s not about accuracy on some benchmark. It’s about consistency. The tool feeds a judge a task, a rubric, an ideal response, and a “negative” response that isn’t necessarily wrong, just less preferred under the rubric. Then it checks whether the judge can tell them apart every single time, even when you shuffle which answer comes first.
That’s the part that stings if you’ve ever shipped an LLM judge to production. Position bias (swap the order of two answers, get a different winner) and verbosity bias (longer answer wins for no good reason) are quietly wrecking your eval pipeline, and you probably never checked for either.
Mini-workflow to steal this weekend:
- 📋 Build your dataset: task + rubric + ideal response + a deliberately “less preferred” response
- 🔄 Run your judge model on each pair twice, once in each order
- 📏 Vary response length on the same content and re-run
- 🎯 Score consistency: does the judge pick the rubric-preferred answer regardless of order or length?
- Repeat across judge models (GPT, Claude, Gemini) to find your most stable judge
Pro tip: don’t grade your judge only on correctness. One commenter suggested going antagonistic, blinding the judge to everything except the goal prompt, references, and outputs. That removes the temptation for the judge to “reason” its way into a biased pick.
Second pro tip: keep your negative responses realistic. A response that’s obviously bad tells you nothing. The real signal is in near-misses your rubric should catch but your judge might not.
If you’re running evals on your own product, go check judgeDjudge on GitHub and run your judge through it before you trust another eval score 🔍. Worth the hour.
Frequently Asked Questions
Q: Should judges be blind to how responses were generated?
Good callout from a commenter: judges should see the objective, reference sources, and outputs, but *not* the intermediate reasoning steps. This stops judges from getting biased by “how” someone got to an answer rather than whether they actually hit the goal. It’s a solid stress test for whether your judge really understands the rubric or is just pattern-matching on explanation style.
Q: How do you catch position bias?
Swap the order of your ideal and negative responses and run the test multiple times. If the judge consistently picks the right one no matter where it appears, you’re in good shape. If position matters, you’ve found a real flaw that needs fixing before using this in production.
Q: Why test for verbosity sensitivity?
Because judges can get fooled by verbose answers that *sound* better even when they’re not actually better. Real users will submit answers of all different lengths, so you want your judge grading on actual quality, not word count.
Q: What counts as a “negative response” in your dataset?
It doesn’t have to be actually wrong, just less preferred under your rubric. Think “this is fine, but not ideal.” This teaches your judge nuance and quality distinctions, not just right-vs-wrong. That’s much harder and much more useful.
Evaluate the evaluator, judge the judge
by u/maylad31 in PromptEngineering