AI judges are biased, lol
Ever wonder if an AI can be… biased? I just saw a wild experiment that proves it, and the results are both fascinating and a little uncomfortable. It turns out AI reviewers secretly love AI-generated text, but there’s a huge catch: you can’t tell them.
Someone ran a 400-trial experiment where an AI model acted as a peer reviewer for scientific abstracts. The goal was to see if it preferred human writing or AI-rewritten versions, and how disclosing AI use changed the scores.
✨ The findings are a total game-changer:
🚀 Undisclosed AI-written abstracts scored significantly higher than the human originals.
🛑 Simply labeling any abstract as “AI-generated” totally tanked its score, even when it was the original human-written one!
🤯 The smarter AI model was actually more biased. It gave a bigger boost to secret AI writing and punished disclosed AI use more harshly.
The takeaway is pretty insane. Right now, if an AI is reviewing your work, you might get a better score by using AI to help you and just not mentioning it. It’s a crazy look into the weird biases we’re building into these systems.
This is just the highlights, the full post has all the data, graphs, and prompts. Check out the full Reddit post for the deep dive!
AI reviewers prefer AI-generated abstracts — but only if you don’t tell them.
byu/JimtheAIwhisperer in