A support rep reads a handoff summary before picking up a customer ticket. It says: “Charge waived.” Great, one less thing to worry about. They tell the customer the fee is gone.
Except nobody ever waived it. The original conversation said, “We could waive the charge.” Could. Not did.
That’s the exact slip a Redditor named u/PartyString7254 caught on r/PromptEngineering while testing a prompt for AI-generated handoff summaries. Everything else in the summary was accurate. It read smoothly. It just changed the meaning by one verb tense.
Why this matters 🧐
This kind of error is sneaky because it doesn’t look like an error. There’s no weird phrasing and no made-up fact. The summary sounds like a perfectly normal summary.
But “could happen” and “already happened” are very different things in the real world. Think about:
- A refund that was offered, not issued
- A discount that was discussed, not approved
- A deadline that was suggested, not agreed
- A customer who “might cancel” turning into a customer who “canceled”
Now picture what happens next. The rep promises the customer something the company never agreed to. The customer screenshots it. Someone in finance has to untangle it a week later, and the customer feels misled even though nobody lied. The summary was the only link in the chain that got it wrong, and it was the one everyone trusted.
The original poster has been testing prompt changes in Braintrust with similar examples. Some versions keep the uncertainty. Others still flatten “could” into “did.” So this isn’t a one-off fluke. It’s a pattern you have to test for on purpose.
How to catch it: a few fixes from the thread 🔧
The comments gave some practical ways to attack this. Here they are as steps you can steal.
- Make the model quote instead of reword. For anything involving money or a promise, tell the summary to quote the original sentence rather than paraphrase it. u/ChemicalExcellent463 and another commenter both backed this. If the source says “We could waive the charge,” the summary carries that exact line. No room to quietly upgrade it. A simple instruction works here, something like: “For refunds, discounts, deadlines, and commitments, copy the exact sentence from the conversation in quotation marks.”
- Force a status field per item. u/According-Stable4487 said this is what made it stick for their team. Instead of free-flowing prose, every action item gets a field like
state: offered | agreed | done | declined. Now the model has to pick a box. “Offered” and “done” can’t blur together because they’re literally different values. As a bonus, the next person reading the handoff can scan the states in two seconds instead of parsing paragraphs. - Add a cheap hedge-word test. u/Moth__leaf suggested a simple automated check. Pull the hedge words out of each source item (could, may, might, would, if, unless, pending, offered). Then assert that each one either survives in the summary or is covered by the item’s status. It’s a small script, and it catches exactly this class of bug. It won’t be perfect, but a rough check that runs on every prompt change beats a perfect check nobody writes.
- Check your own brevity instructions. u/Kallini-Cliona asked a sharp question: are you telling the model to be brief or punchy? Instructions that push for short output can make the model drop words like “could” to save tokens. Your “keep it tight” line might be the culprit.
Tips & Tricks ⚓
- Build test cases around hedges on purpose. Write examples with “could,” “might,” “pending,” and “if approved.” Run every prompt change against them.
- Compare prompt versions side by side. The poster found some changes kept the uncertainty and others didn’t. You only see that if you test variants on the same examples, which is what tools like Braintrust are for.
- Treat money and promises as high-risk. Those are the places where a wrong tense costs real trouble. Put your strictest rules there.
- Don’t trust “reads fine.” Fluent output is not the same as faithful output. The whole point of this story is that the summary read perfectly.
- Keep a “greatest hits” file of failures. Every time a summary gets a tense wrong, save the source and the bad output. Over a few weeks you’ll have a test set built from your own real mistakes, which is worth more than any generic example.
Your turn 🏴☠️
Pull up whatever prompt you use for summaries, notes, or handoffs. Write three test examples where something was only offered, suggested, or pending. Run them. If any come back as “done,” you’ve found your bug before a customer does.
And if you’ve got your own “could became did” story, I’d love to hear it. Bonus points if you found a fix that isn’t on this list.
Frequently Asked Questions
Q: Should you quote the source or tag each action with a status to keep “could” from turning into “did”?
Commenters split on this. One approach is to quote the original sentence word for word for anything about money, permissions, or promises, with who said it and when, so the summary only points at the quote. Another is to give each action a state field like offered, agreed, done, or declined, and have the model pick that label before it writes the sentence, so the tense follows the label instead of drifting toward the confident version.
Q: How can you test for this without reading every summary?
Pull the hedge words out of each source item (could, may, might, would, if, unless, pending, offered) and check that each one either survives in the summary or flags the item as changed. Then build a small set of 20 to 30 snippets full of hedges like “could,” “if they confirm,” and “pending approval,” and check that none of them come back marked as done. Both checks are mechanical, so they catch the regression even when the prose reads perfectly fine.
Q: Could asking for a brief summary be why the hedges disappear?
One commenter suspects so, since instructions that push for brevity can lead the model to drop words like “could” to save space. It’s worth checking your system prompt for that kind of language before you change anything else. If you find it, try a version that treats keeping hedges as a required part of the output, then compare the two versions on your eval set.
A handoff summary changed ‘could waive’ to ‘waived’
by u/PartyString7254 in PromptEngineering