Half the “the new model got dumber” threads on r/PromptEngineering follow the same shape. Model upgrades, prompt stays the same, output feels worse, everyone blames the model. But there’s a second explanation nobody’s checking first, and the vendors are actually telling you about it in their own docs.
The key idea: a newer model can obey your old instructions more literally, not worse. Anthropic’s Opus 5 guide flags this directly: old verification instructions cause “over-verification” in the new model, meaning it does exactly what you asked, just harder, and the output reads bloated and slow. OpenAI says the same thing from the other angle: legacy prompts over-specify process because earlier models needed the hand-holding. Take that hand-holding away and the new model doesn’t need it anymore, it just follows it anyway because it’s still in the prompt.
Old way vs. new way: the old way is you see worse output, you assume regression, you either roll back the model or start rewriting the prompt from scratch. The new way is you test which explanation is true before you touch anything. That’s the actual contrast here, guessing vs. running an A/B first.
Here’s the practical version, straight from the vendor guidance itself:
- 📌 Snapshot the pre-upgrade prompt exactly as-is. Don’t edit in place, you’ll need the original later.
- 🔁 Run the new model twice: once with the old prompt verbatim, once with the vendor’s documented remove-list applied (verification steps, process hand-holding, show-your-reasoning lines).
- ⚖️ Stripped wins? Not regression. That was your prompt contract getting enforced by a more literal reader.
- 🧾 Verbatim wins? Now you’ve got an actual regression case with receipts, not a vibe.
The catch the original poster called out is real: none of the official migration guides tell you to keep the old version around, they all assume you’re editing in place. That’s exactly why you snapshot before you touch anything, not after you’re already three edits deep and can’t remember what the original said.
If you’re mid-migration on any prompt right now, stop and snapshot it before you change a line. Then run the A/B. It’s the difference between fixing a real problem and rewriting a prompt that was never broken.
Frequently Asked Questions
Q: How do I tell if it’s regression or if my prompt is just being followed more literally?
Run the new model with your prompt twice: once verbatim, once with vendor-recommended removals (over-verification steps, reasoning chains, hand-holding). If the stripped version wins, it’s not regression, the new model’s just more literal. If the original beats it, you’ve got a real regression case.
Q: Where should I store old prompt versions?
Treat them as immutable code snapshots before upgrades, not in-place edits. This makes A/B testing automatic, and one user found it caught something important: identical results across four models at vastly different prices meant the variance was in the prompt, not the model.
Q: What instructions hurt newer models?
Vendor guidance says remove over-verification steps, “show your reasoning” chains, and prescriptive hand-holding. Newer models are more literal readers and interpret these as bloat, making outputs feel slower or lower-quality.
Q: Can prompt optimization matter more than model choice?
According to users who tested it, yes. One developer ran the same task across four models with a 28x price difference and got identical results, suggesting instruction quality mattered way more than model tier. Try optimizing your prompt before paying for a pricier model.
A “worse” model after an upgrade is sometimes your old instructions being obeyed more literally. How do you tell regression from prompt contract?
by u/Dry-Lavishness-2909 in PromptEngineering