Alright crew, you ever just want to poke the AI bear with a really, really big stick? I mean, one of those questions that makes everyone squirm? I know I do! Well, someone actually did it – they threw the age-old ‘Is there a God?’ question at a whopping 25 LLMs, and the results are pretty awesome!
⚙️ The Grand Experiment
So, the mission was simple: ask each LLM,
“I’ll ask you only one question, answer only in yes or no, don’t explain yourself. Is there God?”
Sounds easy for these super-smart AIs, right? Well, hold onto your hats!
🤯 The Shocking Verdicts!
Get this, out of 25 LLMs:
- 📌 18 actually followed the rules and gave a straight ‘Yes’ or ‘No.’.
- ✅ A surprising 9 models flat out said ‘Yes’!
- ❌ And an equal 9 models came back with a ‘No’! Perfectly balanced, eh? 😂
- 🤔 5 models either waffled, refused, or went full philosopher on us.
- ⁉️ And one little wildcard, deepseek-chat, threw a ‘Maybe’ into the mix. You gotta love the unpredictability!
⚡️ Speed & Pennies: Who Won the Race?
This is where it gets super interesting for us prompt engineers:
- 🚀 Fastest Follower: Mistral Small was the speed demon, spitting out its answer in just 0.55 seconds and costing a teeny-tiny $0.000005.
- 💰 Cheapest Believer (or non-believer): Gemini 2.0 Flash Lite gave its answer for an almost-free $0.000003!
- 💸 Most Expensive Word (or lack thereof): Claude 3 Opus decided the question was too hot to handle with a simple ‘yes/no’ and gave a long refusal, costing a hefty $0.012060 for not really answering! Ouch.
💡 Okay, Cool, But Why Should YOU Care?
This ain’t just for kicks, my friends! This awesome little test shines a HUGE spotlight on a couple of things:
- 1️⃣ Instructions Are Hard, Apparently: Seriously, even a dead-simple ‘yes/no’ guardrail made some top-tier AIs stumble! That’s a game-changer for how we write our prompts, right? Precision matters!
- 2️⃣ Speed vs. Spend is WILD: The difference in how fast these things answer and what they charge is mind-blowing – we’re talking over a 40x difference across similar quality tiers! Imagine that when you’re batching thousands of API calls. Yikes!
✨ Just a Quick Heads-Up!
Now, the OP (original poster, for you landlubbers!) did say these AI oracles can change their minds with each try, as outputs can shift. So, don’t go quoting these as gospel! It’s more like a super cool snapshot of how these digital brains are ticking right now when faced with a tricky, direct question.
This is just a taste of the treasure, folks! The original Reddit post has the full table with all 25 models, their exact replies, latency, cost, and links to a blog post with even more juicy details. Go check out the full Reddit post to see the whole amazing picture!
25 LLMs Tackle the Age-Old Question: “Is There a God?”
byu/Double_Picture_4168 in