A single average score went up after a prompt rewrite, and for a moment that looked like the whole story. Split that same number by who was actually asking, and it fell apart. u/ImpossibleFood8242 posted the breakdown on r/PromptEngineering this week, and it’s a sharp reminder that one chart can hide more than it shows.
🔍 Quick Start: the original poster’s team rewrote a prompt, watched the aggregate score climb, then found the gain wasn’t shared evenly. Newcomers got clearer steps. Expert customers got longer answers that buried the useful part. A model update landed that same afternoon, so the blended chart couldn’t say which change actually helped. Prompt versioning, a frozen dataset, and cohort metadata untangled the mess, and you can copy the same setup on your next prompt test.
Here’s the core idea: an aggregate score only tells you the average moved. It says nothing about which segment moved it, or in which direction. Change two things at once, a prompt edit and a platform update on the same day, and that single number can’t separate the causes either. You end up guessing.
This isn’t just a prompt engineering nitpick. Teams use aggregate scores to decide whether a rollout is safe to ship. A number that hides a regression for your best customers is worse than no number at all. It hands you false confidence right before you ship.
The old way looks like this: rewrite a prompt, run it against live traffic, watch the top-line score, ship if it’s higher. That works fine when the prompt is the only thing changing. It stops working the moment a model update, dataset drift, or a shift in who’s actually using the product lands in the same window.
The new way this Redditor landed on treats the prompt itself as a versioned artifact. Braintrust let the team lock each prompt as an immutable version, so nobody could quietly edit it mid-test. They ran both versions against the exact same frozen dataset snapshot instead of live traffic that keeps shifting underneath you. Then they attached cohort metadata, tagging every result by user experience level, before comparing anything at all.
Splitting the results this way told two separate stories that one chart had flattened into one. The instruction change in the new prompt caused the extra detail that buried answers for expert customers. The model update, landing that same afternoon, improved citation accuracy across both groups. Neither fact showed up in the original blended score, and the team would have shipped a regression by mistake.
Want to try this on your own prompts? Here’s the process, step by step:
- Lock the prompt version. Save it as immutable the moment you’re ready to test, so it can’t be edited mid-experiment.
- Freeze your dataset. Run every version against the same fixed snapshot, not live traffic that changes day to day.
- Tag cohort metadata on every run. Experience level, account age, plan tier, whatever segment might respond differently to the same prompt. It costs almost nothing to add, and it’s the piece most teams skip until something breaks.
- Split the results before you celebrate. Check the score per cohort, not just the blended average.
- Isolate confounds on purpose. If something else changed that day, a model update, a data source swap, a UI tweak, test it separately. Otherwise you’re stuck reconstructing the afternoon from commits and chat threads later.
A few commenters pushed the idea further. One reader argued that newcomers and experts probably need entirely separate eval slices, since one average score is “asking for trouble.” Another asked whether the expert complaints showed up before or after the cohort split caught the regression. That’s a fair question about how most teams actually catch this stuff. A third said the post made them want to tag every eval case with the type of user it represents. That’s close to the whole lesson in one line.
I’ve watched this exact trap happen with a launch that looked great on paper and quietly annoyed the people who mattered most. The fix isn’t complicated. It’s just easy to skip when the top-line number already looks good.
If you’re running prompt experiments and only watching one aggregate number, try this on your next rewrite. Version the prompt, freeze the dataset, tag the cohort, then look twice before you ship. The full discussion has more on how the team built out their eval setup. Worth a read if you want the details straight from the source!
Frequently Asked Questions
Q: How do I spot when a prompt change helps some users but hurts others?
Split your evaluation results by user cohorts, newcomers vs. experts, different use cases, whatever divides your audience. One overall score hides important problems. In this example, a prompt change looked great in aggregate but made answers longer and less useful for experts. Evaluating per cohort catches these regressions before users complain.
Q: What metadata should I add to my experiments to catch hidden regressions?
Tag each eval case with user type, experience level, or relevant cohorts. Also track the prompt version, model version, and date, especially if multiple changes land around the same time. This metadata lets you slice results multiple ways and figure out whether gains came from your prompt tweak, a model update, or something else entirely.
Q: How do I know which change actually moved the needle, my prompt rewrite or the model update?
Version your prompts immutably and run both versions against the same fixed dataset snapshot. Compare apples to apples instead of trying to reconstruct what happened from commits and chat threads. Attach cohort metadata so you can see if one version helped newcomers but hurt experts. Tools like Braintrust with version lineage make this pretty straightforward.
Q: Why is one average score not enough for evaluating prompt changes?
Because it hides cohort-level problems. The post’s example is perfect: overall score went up, but experts got longer, less useful answers. If you only look at aggregate metrics, you’ll optimize for one group at the expense of another. Split results by user type and check each slice, that’s how you catch regressions early.
Versioning the prompt finally made our rollout chart mean something
by u/ImpossibleFood8242 in PromptEngineering