Rewriting the prompt is the usual first move when a RAG answer misses a condition or an exception. A Reddit post in r/PromptEngineering suggests a different first move: before you touch the prompt, check where the supporting evidence disappeared. Most teams treat a wrong answer as a wording problem. This approach treats it as a location problem, and it is usually faster to settle.
The key idea
A wrong answer doesn’t tell you where the failure happened. The model can only work with what reached its context window. If the passage with the exception never got there, no prompt wording will bring it back. Think of a customer support bot that says refunds are allowed within 30 days, when the policy has a clause excluding digital goods. If that clause never reached the model, the answer is a reasonable summary of incomplete material. You can add “always mention exceptions” to the prompt, and the model will still have nothing to mention.
That is why the same symptom can come from very different causes. A missed exception might come from a chunk boundary that split the clause away from its heading, a metadata filter that excluded the right document, or a ranker that pushed the passage to position 14 when only the top 8 were kept. Each of these needs a different fix, and none of them is a prompt edit.
The old way vs. the new way
The old way: the answer is wrong, so you tweak the instructions, rerun, and hope. You end up with a pile of prompt versions and no idea which one helped or why. Worse, a prompt change can seem to work on one question by luck while quietly making another one worse, and you only find out later.
The new way: take one failing question and trace the passage you expected to see. Where it vanished tells you which part of the system to look at. You stop asking “how do I phrase this better?” and start asking “at which stage did the evidence get lost?” The second question has an answer you can check.
How to trace it, step by step
- Pick one failing question and name the document or passage that should have supported the right answer. Write down the exact sentence or clause, not just the file name. If you can’t point to it, you aren’t ready to debug yet.
- Look for that passage in your saved retrieval results, then in the actual context the model received. These are two different places. Retrieval can return a passage that a later step trims, merges, or reorders before the model ever sees it, so check both.
- Read the result:
- 🔎 Missing from the saved retrieval results: look at retrieval, filters, and corpus coverage. Is the document in the index at all? Did a filter on date, source, or tag exclude it? Does the query wording differ so much from the passage wording that the embedding never matched?
- 📉 Present in the results but below the context cutoff: look at selection and ranking. Check your top-k setting, any reranker, and any token budget that cuts the list short. A passage sitting at rank 9 with a cutoff of 8 is a ranking problem, not a prompt problem.
- 📄 Present in the actual model context: retrieval doesn’t explain the failure. Inspect generation separately, and the prompt is a fair suspect now. This is the case where rewriting instructions, asking for step-by-step reasoning, or requiring citations can genuinely help.
- ❓ No trusted expected evidence: your diagnosis is still uncertain. Fix that first by deciding what the right source actually is. Ask a domain expert, or read the source documents yourself, until you can say with confidence which passage should have answered the question.
- Run the same questions before and after a retrieval change. Keep a small fixed set, even 15 to 20 questions is enough to start. You get a simple review: which cases lost evidence, which recovered, and which are still unresolved. A spreadsheet with one row per question and one column per stage is plenty. It also stops you from celebrating a fix that helped three cases while breaking two others.
Know the limits
The original post is upfront about this, and it’s worth repeating. A saved top-k export only shows what’s in the export. If a passage is missing there, you can’t conclude it was never retrieved further down the ranking. And this check alone doesn’t prove the root cause. It narrows where to look, and that’s all it promises. Treat the result as a pointer, then confirm with a targeted change: widen the filter, raise k, or fix the chunking, and see whether the passage now shows up.
Why this beats guessing
It turns a vague “the AI got it wrong” into a specific question: did the evidence arrive or not? That keeps you from spending a week polishing a prompt for a problem that lives in your ranking. It also gives your team a shared vocabulary. Instead of arguing about whether the prompt “feels” better, you can say “this case is a ranking miss” and everyone knows what to open next. Over time the pattern counts become useful too. If most of your failures land in the same bucket, you know where your effort pays off.
Your turn
Pick one failing question this week and trace its evidence through your pipeline. Then tell us: has a “prompt problem” ever turned out to be missing or poorly ranked evidence? What check helped you tell the difference? Share it, because the best debugging habits come from people who’ve been burned by this!
Before rewriting a RAG prompt, I would check where the supporting evidence disappeared
by u/SeaAstronomer9044 in PromptEngineering