New data: when one team reversed the order of Yes and No, GPT-6 Luna changed its choice in 12 of 132 tests.
The setup came from a study run through NM Signals. They gave the model 44 moral dilemmas, each in three wordings. That’s where the 132 comes from: 44 dilemmas times 3 wordings. Within each pair, the situation and the wording stayed fixed. Only the order of the answer options changed. If the model were judging purely on content, swapping the order should change nothing.
What the numbers say
The flip rate was 12 out of 132, or about 9%. That’s small enough to ignore and big enough to matter if you use an LLM as a judge. Run a thousand judgments through a pipeline with a 9% flip rate and you could have around ninety verdicts that depend on layout, not substance.
Two details are more interesting than the headline.
- All 12 flips covered just seven of the 44 dilemmas. The instability is concentrated, not spread evenly. One aggregate number hides that. A reader who only sees “9%” will picture a thin layer of noise over everything. The real picture is a handful of cases that wobble and a large majority that don’t.
- In all 12 flips, the model picked the first option in both orders. It wasn’t changing its mind about the content. It was leaning on position. When Yes came first, it said Yes. When No came first, it said No. The content of the case stopped deciding anything.
One commenter read this as the model telling you the case is a real toss-up. That’s a fair reading. When the model has no strong pull either way, position fills the gap. Either way, a flip tells you the answer depends on how you presented the options, not only on the case. That’s useful information if you treat it as a signal and not as a nuisance.
3 practical applications
🧭 Order-swap check for judging prompts. Run each case twice, once with Yes first and once with No first. Compare what the model actually chose, not how it phrased the reasoning. Map each answer back to the underlying option before you compare, so “Yes” in run one and “No” in run two can be recognized as the same pick if the labels were reversed. Review every disagreement before you use the output as a score. If you have a few hundred cases, a short script that logs the pair of answers side by side makes this a ten-minute job.
🧭 An “unstable” list instead of an average. One commenter keeps disagreeing cases out of the totals. They score them as a tie and file them separately. Averaging a flip gives you a 50/50 that looks like a real preference, and it isn’t one. A separate list also gives you something to act on. You can hand those cases to a human reviewer, rewrite the prompt for them, or drop them from a benchmark if they keep flipping.
🧭 Per-dilemma reporting. Split the flip rate by case, not just overall. If 12 flips sit on 7 of 44 items, those 7 are your fragile ones. Rerun them with each order more than once and see whether the pattern holds. If a case flips on every rerun, it’s a real toss-up. If it flips only once, you may have caught sampling noise. Those two situations need different fixes, and you can’t tell them apart from a single overall percentage.
Tips and pitfalls
- Compare choices, not wording. Two answers can sound different and still be the same pick. The reverse is true too: two answers with nearly identical reasoning can land on opposite options.
- Don’t treat a clean aggregate as proof. A 9% flip rate can hide a small cluster of cases that are wrong half the time.
- Don’t average the two runs. Mark the case as unresolved.
- Keep the sample honest. This is one model on 44 dilemmas, so it tells you to check your own setup. It doesn’t give you a rule about all models.
- Don’t assume your task is immune because it isn’t about ethics. Any prompt with ordered options can have the same pull toward the first one. Pass/fail graders, A/B response rankers, and multiple-choice classifiers all qualify.
- Randomize going forward. Once you know position matters, shuffle option order per case instead of always using the same layout, so any leftover bias spreads out and doesn’t pile onto one answer.
Pros of the check: it’s cheap, it needs no new tooling, and it shows you which cases to distrust. You can add it to an existing eval in an afternoon.
Cons: it doubles your calls, and it only catches position bias. It won’t catch other kinds of wrongness, like a model that picks the same wrong answer in both orders.
Your move
Pick one judging prompt you already run. Swap the option order on 20 cases this week and count the flips. If you get zero, good. If you get a handful, look at which cases they are before you trust the score. Write down the count and the case IDs, so next month you can check whether the same ones flip again.
How do you handle it when the two runs disagree? Tell me in the comments. I’m especially curious who is already balancing answer positions and what that looks like in practice.
Frequently Asked Questions
Q: Why do all 12 flips land on the first option?
One commenter thinks those are the real toss-up dilemmas, so a flip may be the model saying it’s unsure rather than a bias. Another saw the same pattern with this-or-that prompts and found that switching the order often just switches the bias. Check which dilemmas the flips come from before you decide which explanation fits your data.
Q: What should you do when the two orders disagree?
One approach is to skip averaging. Score the case as a tie and keep it in a separate unstable list, because a 50/50 average can look like a real preference in your totals. Another is to split the results per dilemma and rerun only the unstable ones, with each order a couple of times per side, to see whether the flip holds.
Q: Can prompt wording reduce the flips?
Some people got fewer flips by dropping the Yes/No labels. They asked the model to restate its choice in its own words, or to pick between the full option texts, and then mapped that back to yes or no in code. Putting the reasons for each side before the verdict, with the verdict on the last line, also helped one commenter. Neither fix brought the rate to zero, so keep running both orders on anything you use as a score.
Q: Are the unstable cases just the ambiguous ones?
A commenter suggests comparing the unstable dilemmas with the ones where humans split close to 50/50. If they overlap, the flip is measuring how ambiguous the question is, not only position bias. That check tells you which problem you actually have.
We reversed Yes and No: GPT changed its choice in 12 of 132 tests
by u/nymanmedia in PromptEngineering