Your Prompt Changelog Is Lying to You: Read Per-Case Results Instead

Most teams track prompt edits by how the output looks. Cleaner, shorter, more polished. This approach tracks them by what a fixed test set says, and the two views disagree more often than you’d expect.

The idea comes from a write-up by u/ClickOk5811 in r/PromptEngineering. Take a prompt that turns meeting transcripts into JSON action items, which a script then uses to create tasks. Run 8 fixed test transcripts after every edit, three runs per case. Then compare what each edit looked like with what the test set showed. The gap between those two views is where bugs hide.

The old way vs. the new way

The old way: you edit, skim a few outputs, say “looks better,” and ship. It feels fast, and it works until the one case you didn’t skim quietly breaks. The new way: you edit, run the whole set, and read results case by case. It takes a few extra minutes per edit, and it replaces guessing with a record. Here’s how the five edits in the post played out.

  • 📝 v1.1, “Do not invent due dates; use null if none is stated.” It looked like a clear win because the invented date was gone. The test set still showed 5 of 8. “Next Friday” now failed, because the model had no meeting date to resolve it against. The rule fixed one problem and exposed another that had been hiding behind the invented dates.
  • 🔧 v1.2, add a meeting_date field to the input. It looked like a prompt tweak. Score: 6 of 8. But it was also a contract change. Every caller now has to send that field, so prompt version and input schema version need to be tracked together. A caller that forgets the field will fail in a way that looks like a model problem.
  • ✂️ v1.3, add “Be concise.” The outputs looked cleaner. Score: 5 of 8. A commitment mentioned in passing got dropped, and nothing about the tidy output hinted that a task was missing. Short outputs read as confident, which is exactly why this one is dangerous.
  • ✅ v1.4, replace “be concise” with “include every commitment, even if stated briefly; one sentence per task,” plus “unassigned, never guess” and “empty list if no items.” It looked like just a longer prompt. Score: 7 of 8. The last failure is two attendees with the same first name, which the transcript can’t resolve. It’s logged as a known limitation, so nobody wastes an afternoon trying to fix it later.

Then comes week 8, with no prompt change at all. Score: 6 of 8. The provider had updated the model behind the endpoint, and one case started returning dates in a different format. A commenter said they hit the same thing with a date field. Nothing in the prompt changed, and a case still broke.

Three lessons worth stealing

  1. Totals can hide movement. After v1.1 the total stayed the same, yet the set of failing cases changed. Only per-case results show a regression. Two edits can both score 6 of 8 while failing completely different cases, and a total alone will call them equal.
  2. Some edits aren’t prompt edits. Contract changes and model changes belong in the log too. Otherwise you’ll hunt through the prompt for a mistake that isn’t there.
  3. Write the reason next to each rule. That’s what lets someone who wasn’t there decide later whether a rule can be removed. “Unassigned, never guess” with a note like “added after the model assigned a task to the wrong person” is a rule people will keep. Without the note, it looks like clutter.

How to set this up this week

  1. Collect 8 real cases, including the awkward ones: relative dates, empty input, duplicate names. Pull them from actual transcripts your system has seen, not ones you wrote to be easy.
  2. Run each case 3 times after every edit. Models vary, and one run can mislead you. If a case passes twice and fails once, mark it as flaky rather than calling it a pass.
  3. Record pass/fail per case, not just the total. A simple grid of cases against versions is enough. A spreadsheet works fine, with cases down the side and versions across the top.
  4. Log every change that touches the system: prompt text, input schema, model version. Give each a version number.
  5. Write one line of reasoning beside every rule in the prompt.
  6. Re-run the set on a schedule even when nothing changed. Week 8 is why. A weekly run is enough to catch a silent model update within days instead of weeks.

Be honest about the limits

Eight cases only catch regressions you thought to include. Pass/fail is coarse, since a correct but badly worded output still passes. Three runs is a small sample. And the process costs time, so it’s not worth it for a prompt one person uses now and then. If other code depends on your prompt, though, it pays for itself fast.

The author notes that their full article, with a case-by-case matrix, mentions a paid resource near the end, so treat that link as optional.

Captain YAR’s take: build the grid first, then edit. Start with your three most fragile cases today, and when you add a fourth, you’ve started a real test set. Ahoy, and happy sailing!

Now your turn: when you edit a prompt that other code depends on, how do you check that you haven’t broken a case you weren’t looking at?

Frequently Asked Questions

Q: What else should I log with each test run?

Log the model name and the run date next to the prompt version. Providers can update the model behind an endpoint without you touching anything, and one commenter’s case started failing because the date format changed with no prompt edit. Without that record, you’ll end up searching your prompt for a mistake that isn’t there.

Q: Why track results per case instead of just the total?

Two versions can both score 5 of 8 and still be failing different cases. One commenter kept a small table of which cases passed in each version, and it showed an edit fixing one case while breaking another. It takes a few minutes to set up and shows you what the total hides.

Q: How do I tell a model change from a prompt regression?

Check the per-case table first. If the failing case changed but the prompt version stayed the same, look at the model and date log before you touch the prompt. A new failure with no prompt edit usually means the model moved underneath you.

Five edits to one prompt: what each one looked like vs. what the test set showed
by u/ClickOk5811 in PromptEngineering

Scroll to Top