Yesterday I ran into a post that every person building an LLM-as-judge system needs to read. A small site called codedrills.jp teaches you to write instructions instead of code: you read a problem, write the prompt, paste back what the model gives you, and run the tests. The score lands on your instruction, not your code.
Here’s the twist. The reviewer scoring those instructions had been quietly rewarding people for making stuff up!
The review model only ever saw the user’s instruction and the resulting code, never the actual problem statement or its constraints. So on a task whose spec says “1 ≤ A, B, C ≤ 1000,” the reviewer would suggest “you should also specify behavior for inputs up to 10^9,” then raise the score the moment someone added that sentence. For a full week, the site was training people to pad prompts with plausible sounding noise. That’s the exact opposite of what the whole site exists to teach.
If you’re running any LLM-as-judge setup of your own, steal this mini-workflow. It’s how the founder (u/FirmCry2135) caught and fixed it:
- Audit what context your judge model actually receives. Missing the ground truth or the constraints means it will invent “gaps” to fill.
- Feed it the full spec, not just the output. The fix here was boring on purpose: pass the problem text in.
- Tell it explicitly that nothing outside the given constraints earns points. Left alone, judges default to “more thorough equals better.”
- Split “user under-specified” from “model slipped” in the output, with a confidence level, so the fix is obvious on sight.
Pro tip: the same site had a second bug worth borrowing the lesson from. The streak counter in the header counted login days. The one on the home page counted actual solving days. They disagreed, a user noticed, and of course the flattering number was the one on display. Any time you’re tracking a user-facing metric through two code paths, check they agree before someone else finds the gap first.
Second pro tip, straight from the top comment: sample a batch of past reviews and check whether the score delta between revisions actually correlates with test outcomes. Cheapest sanity check you’ll ever run on a judge model.
Worth the 3 minutes if you’ve got an LLM sitting in a grading seat anywhere in your stack: codedrills.jp/try?lang=en, no sign-up needed. 🏴☠
Frequently Asked Questions
Q: My scores changed after the updates, did I lose progress?
The updates improved how the reviewer works (it now sees the full problem before scoring your instruction), but we didn’t retroactively recalculate past scores. Going forward, feedback will be much more accurate, no more rewarding off-topic additions.
Q: How do I know if my instruction failed or if the model just made a mistake?
The new feedback explicitly tells you which one happened, with a confidence level. If it’s an instruction gap, you get a one-sentence suggestion to paste into your next prompt, plus a diff showing exactly what changed in your score.
Q: What’s different about streaks now?
Streaks now count actual learning, solving problems or running reviews, instead of just logging in. It’s stricter than before, but way more honest. Your streak actually reflects progress, not just open tabs.
Q: How do you catch scoring bugs before they hurt users again?
We’re now periodically spot-checking whether score improvements actually correlate with real test improvements, an early warning system for drift. User feedback still catches things too, so let us know if something feels off.
My LLM reviewer was rewarding users for adding requirements the task never had
by u/FirmCry2135 in PromptEngineering