Zero out of 90. That is how often AI agents noticed a failed save when nothing in their tool responses showed the problem.
A tester on r/PromptEngineering ran 20 airline tasks from τ-bench with Claude Sonnet 5.5, GPT-6.1 Sol and DeepSeek V4.1 Flash. The twist: a write tool said “success” but changed nothing. Would the agents catch it? Airline tasks are a good test bed because they involve real state changes, like booking, modifying and cancelling reservations, where a phantom save leaves the customer with a wrong itinerary.
The numbers
- 21 of 28: the agent caught the failure when a later step’s tool response happened to show the booking.
- 0 of 90: the agent caught it when no response showed the booking.
- 11 vs 10: catches at high reasoning effort versus low. That is basically no difference.
So the agent’s thinking budget wasn’t the lever. What it could see was. The catches were accidental. A later read, such as looking up the reservation to answer a follow-up question, happened to reveal the real state, and the agent noticed the mismatch. Without that, it believed the “success” message every time. Notice the split: when the evidence was on screen, the agents caught the problem 75% of the time (21 of 28). When it wasn’t, they caught it 0% of the time. That gap is the whole story, and it has nothing to do with which model you picked.
Why this matters
An agent can only reason about the text in front of it. If the tool says “saved” and nothing else disagrees, there is nothing to reason about. More thinking just builds a more confident story on top of a false claim. As one commenter put it, the model can only work with what it sees.
This also explains why the problem is so easy to miss in your own logs. The agent’s final message reads fine. The conversation looks clean. The customer hears “your booking is confirmed,” and the only place the truth lives is a database row that was never written. You will not find it by reading transcripts. You find it when a user complains, and by then the trail is cold.
3 practical applications
🔹 Return the saved record. Have your write tools send back the full record they just wrote, not a bare “ok.” The agent can then compare it to what it asked for. For a flight change, that means returning the new flight number, date, seat and passenger list, so a wrong date jumps out immediately.
🔹 Read back fresh. After a write, query the database again and return that result. A fresh read is evidence. A status flag is just a claim. If your write goes through a queue or a cache, read from the source of truth, not from the layer that just accepted the request.
🔹 Fail loudly on mismatch. If the stored data doesn’t match the request, return an error. Don’t let the tool say success when the state says otherwise. A hard error gives the agent something concrete to react to, and most agents handle errors well because they are trained to retry or escalate on them.
Tips and pitfalls
- Don’t reach for “think harder” settings to fix silent failures. In this test, they didn’t help. The 11 versus 10 split shows extra reasoning effort buys you roughly nothing here, and it costs more tokens and time.
- Don’t rely on a lucky later step to expose the problem. That is how you get 21 of 28 in the good case and 0 of 90 in the bad one.
- Keep tool outputs small and specific. Return the fields that matter, so the mismatch is easy to spot. A 400-line JSON dump buries the one wrong value, and the agent may skim past it.
- Test your own tools by faking a failure. Make a write silently do nothing in a staging environment and see whether your agent notices. If it doesn’t, you have found your bug before a customer does.
- Caveat: this is 20 tasks and three models, so treat it as a strong signal, not a law. The pattern is still clear enough to act on.
Prompt of the day
Add this to your agent’s system prompt: “After any write action, compare the returned record to what you intended to save. If they differ, say so and retry before reporting success.” It only works if your tool actually returns the record, so fix the tool first. The prompt is the seatbelt, but the tool output is the road. Without real data coming back, the instruction gives the agent nothing to compare against.
Your move
Open your agent’s write tools today and check what they return. If the answer is “success” and nothing else, change it. Start with the write that would hurt most if it failed quietly, like payments, bookings or account changes, and fix that one first. The write-up and the code with transcripts are linked in the original Reddit post if you want to dig into the data yourself.
Frequently Asked Questions
Q: How would an agent know a save failed if the tool said ‘success’?
It usually wouldn’t. The agent only works with what the tool sends back, so a bare ‘success’ gives it nothing to doubt. Have write tools return the saved record, or an error when the saved data doesn’t match what was requested.
Q: Should I read back after every write?
That’s the habit one commenter picked up from building small automation scripts: trust but verify. Fetch the record fresh from the database after the write, so you check what was actually stored instead of what the write claimed. For agents, that read-back also lands in the tool response, where the model can see it.
Q: Does this only matter for AI agents?
No. The same gap shows up in plain scripts, where a write reports success while the data is still wrong. The fix is the same in both cases: make the evidence visible, then check it. A few lines of read-back code cost far less than finding out a booking never saved.
AI agents only caught failed saves by accident
by u/TruthDeep8659 in PromptEngineering