Markdown won 70 to 81% of the time. That is how often five different LLM judges picked the markdown version of an answer over a plain-text version with identical content. The plain version got only 15 to 30%.
u/mrtac96 re-ran a style-bias test from the 2026 TMLR paper Judging the Judges on Claude Haiku 5.5, GPT-6 Luna, DeepSeek V4.1 Flash, Qwen3.8 Flash and Gemini 3.5 Flash-Lite. The setup: 50 pairs of the same answer, one formatted in markdown, one in plain paragraphs. Total cost: 1,880 calls. Nothing exotic was needed to run it, which is part of why the result is worth taking seriously. Anyone with an API key and an afternoon can reproduce it.
What the data says
Some of the paper’s pairs had an answer that was cut short, which could skew things. So the author also checked only the 24 pairs where both answers were complete. Markdown still won, 64% to 31%. The gap shrank a little, but the direction did not change. Bigger models don’t fix it either. In the original paper, Gemini 2.5 Pro and Claude Sonnet 4 leaned toward markdown even more.
The nastier result: the author planted one wrong fact in the markdown answer. The four reasoning judges never fell for it. Gemini 3.5 Flash-Lite picked the wrong answer 6 times out of 20, in all three runs. That means the failure was repeatable, not a fluke of one sampling run. In most of those cases it pointed out the error and then picked the markdown answer anyway. The judge saw the problem and let the headers outvote it.
Then the two fixes:
- “Ignore formatting” in the prompt: 240 verdicts, 180 of them ties. Of the remaining 60, markdown won 55 and plain won 5. So the instruction mostly creates ties, and the bias survives in what is left. Roughly 92% of the non-tied calls still went to markdown.
- Step-by-step judging: the judge assesses each answer on its own before comparing. The four reasoning judges gave 192 verdicts, most of them ties. Of the 40 that picked a side, markdown won 28 and plain won 12. Still a lean, but a much smaller one, about 70/30 instead of 92/8.
One commenter pointed out they have seen the same thing in client work. Structure makes an answer feel finished, even when the content is identical. Another asked a sharp question: did anyone strip the formatting from the pairs with the planted wrong fact? That is not in the write-up yet, and it would be the cleanest test of whether format alone caused the bad pick.
3 practical applications
🔹 Comparing prompt versions. If you A/B test two prompts and let an LLM pick the winner, a prompt that makes the model output bullets and headers may win just for looking organized. Normalize the formatting before you judge. A simple way: run every output through the same markdown-to-plain-text converter, then score. If the ranking changes, you were measuring looks.
🔹 Grading drafts and content. For client work, a markdown-heavy draft can beat a better plain one. Strip or standardize the format first, then score the content. If you grade support replies, newsletters or reports at scale, one pass of normalization keeps your scores comparable across writers who format differently.
🔹 Picking your judge model. Use a reasoning model. In this test the four reasoning judges resisted the planted error. The small non-reasoning-style judge did not. Budget matters, but a cheap judge that rewards formatting can cost you more in bad decisions than you save in tokens.
Tips and pitfalls
- Make the judge analyze each answer separately before it compares them.
- Add “ignore formatting” to the judge prompt, but count ties separately. Otherwise a pile of ties will make your results look cleaner than they are. Report three numbers: A wins, B wins, ties.
- Don’t trust a judge that spots the error and still picks that answer. Check its reasoning, not just its verdict. Spot-check a handful of verdicts by hand each time you change the judge prompt.
- Swap the order of the two answers on a second run. Position bias can stack on top of format bias, and you want to know which one you are seeing.
- Pitfall: the sample is small. 50 pairs, and only 20 trials on the planted-error test. Treat this as a strong signal, not a final law.
Prompt of the day
“First, evaluate Answer A on its own: list its factual claims and check each one. Do the same for Answer B. Ignore formatting, length and style. Only then compare the two on accuracy and completeness, and say tie if they are equal.”
Call to action
If an LLM judge decides which prompt ships in your workflow, run this check this week: take one answer, write it in markdown and in plain text, and see if your judge flips. If it does, you have found a hidden bias in your testing. Fix the judge prompt, rerun your last comparison, and see whether the winner changes. Full write-up and raw responses are linked in the original thread.
Frequently Asked Questions
Q: Should I strip the formatting before an LLM compares my drafts?
Probably, yes. Markdown makes a draft look finished even when the words are identical, so a judge can end up grading presentation instead of substance. This test didn’t measure stripping the markup itself, only adding an ‘ignore formatting’ instruction, so plain text is a safer default but not a proven fix.
Q: Does telling the judge to ignore formatting fix the problem?
Partly. With that instruction, 180 of 240 verdicts were ties, so the judges were listening. But when they did pick a side, markdown still won 55 of those 60 times. If you use this prompt, expect a lot of ties and don’t read them as a clean result.
Q: Which judge should I trust to catch factual errors?
Pick a reasoning model. A planted wrong fact fooled Gemini 3.5 Flash-Lite in 6 of 20 runs, while the four reasoning judges never fell for it. Even Flash-Lite often spotted the error and still picked the markdown answer, so a judge can notice a problem and still reward the prettier version.
Q: Does asking the judge to assess each answer first help?
Yes, noticeably. When the judge scored each answer on its own before comparing them, the reasoning judges called most verdicts ties. Of the 40 that picked a side, markdown won 28 and plain text won 12. The lean is still there, just much smaller, so this prompt reduces the bias rather than removing it.
I tested 5 LLM judges from 5 vendors. All preferred markdown when the content was identical, and one picked a wrong answer for its formatting.
by u/mrtac96 in PromptEngineering