Two AI models looked at the same message and made the identical mistake. The message read: “there is a payment that is not mine,” which is a fraud report, not a billing question. Reddit user Nir777 ran this test while building a support router and shared the fix on r/PromptEngineering.
Both a general chatbot model and a second model called Jev sent that message to the Payments team. Not Security, where a fraud report actually belongs. The routing list only gave the models team names to pick from, nothing else. Once the creator added one short line describing what each team handles, both models switched their answer to Security immediately.
That’s the finding in one sentence: a label is not a description. Give a model a bare noun and it pattern-matches on the nearest keyword. Give it a sentence of context and it reasons about intent instead.
Quick Start 🧭
- ✅ Old setup: routing options were just team names (Payments, Security, Billing).
- ✅ Problem: models matched on keywords like “payment,” not on what actually happened.
- ✅ Fix: one line per option explaining what that team actually handles.
- Result: both models routed the fraud message to Security, every time.
The old way: names carry no meaning
A list of team names looks obvious to a human, because a human already knows what each team does. A model doesn’t have that shared context unless you write it down.
When the only signal is “Payments,” a model treats the message like a bag of words. “Payment” appears in the sentence, so Payments wins the match. The word “fraud” or “not mine” never gets weighed against what each team is actually for.
That’s the trap in most routing setups. The list looks complete because it’s short and clean. It’s actually missing the one thing that makes routing decisions correct: intent.
The new way: one line per option
The fix here was small on paper. Next to each team name, the creator added a single sentence describing its scope. Something like “Payments handles billing questions and failed transactions” next to “Security handles unauthorized activity and account takeover.”
With that context in place, both models re-read the same message and picked Security. Nothing else changed: same message, same models, same list of teams. Only the descriptions were new.
This matters more than it sounds. It means the models were never confused about the message. They were confused about the choices.
Steps to copy this
- List every routing option you currently use (teams, categories, intents). This is where most setups already fall short, because the list often exists as a config file nobody has reread since it was written.
- Write one line per option describing what it covers, in plain language. Skip jargon. Write it the way you’d explain the team to a new hire on their first day.
- Re-run your hardest or most ambiguous test messages through the model. Ambiguous messages are where a bare label fails first, so that’s where the fix shows up fastest.
- Check whether the routing decision changes once descriptions are added. If the model was already reasoning correctly, nothing changes, and that tells you the labels were fine all along.
- If the decision does change, keep the descriptions. Ship them into production instead of the bare team list.
Don’t stop at one test message. The creator validated this against roughly 3,000 messages from Banking77, a public intent-classification dataset built from real banking queries. That’s the step people skip: testing a fix on one example and calling it done.
The scale test turned up something else worth stealing. When Jev reported high confidence in its own answer, it was correct 98 out of 100 times. That confidence signal is usable on its own, separate from the routing fix itself. Route the confident calls automatically, and send the uncertain ones to a human or a second model for a check. That combination, better options plus a confidence gate, is basically a cheap safety net for the cases a single model still gets wrong.
A few things to watch before you copy this:
- Keep each description to one line. Longer descriptions blur back into noise.
- Update the descriptions when a team’s scope changes, or the routing drifts again.
- Confidence scores are only trustworthy after you’ve measured them against a real dataset, not a handful of test messages.
The lesson holds for any classification setup, not just support routing. Ticket triage, intent detection, agent handoffs, order-routing bots. All of it runs on the same underlying choice: pick option A or option B from a list. If a model has to pick between options, describe the options. Naming them is not the same as explaining them.
It’s a small change with an outsized payoff, and that’s rare enough to be worth flagging. This Redditor shared a full video walkthrough of the test, worth a watch if you want to see the before-and-after side by side. Go check out the discussion on r/PromptEngineering and see how the same fix plays out on your own routing list.
both models sent a fraud report to the wrong team until we added one line per option
by u/Nir777 in PromptEngineering