Here’s the moment every RAG team hits: the system spits out a vague answer, and the whole team reaches for the same three levers, swap the embedding model, rewrite the prompt, bump Top-K. Nine times out of ten none of those fix anything, because nobody diagnosed what actually broke first. The sprint gets burned, the dashboard barely moves, and the same vague answer shows up again next week with a different customer’s name attached to the complaint.
A Redditor going by u/Critical-Elephant630 posted a framework in r/PromptEngineering that flips the whole approach. Instead of guessing at a fix, separate every bad answer into one of three failure shapes first. Diagnose, then fix. It sounds almost too simple to matter, but that’s exactly why most teams skip it. Diagnosis feels slow when the answer is already wrong in front of a user. The instinct is to act fast, not think first. The framework is a bet that the fifteen minutes you spend classifying the failure saves you the two days you’d otherwise spend re-tuning a system that was never broken where you thought it was.
Old way vs new way
Old way: bad answer comes in, someone says “let’s try a better embedding model” or “let’s rewrite the prompt,” and the team burns a sprint on a change that may not even touch the real failure point. Worse, these changes often “fix” the one example everyone was staring at while quietly breaking ten others nobody tested. Now you’ve got a new embedding model in production, a new set of regressions, and still no idea what actually caused the original bug.
New way: run the answer through three checks, in order, before touching anything.
- 🔍 Retrieval absence: did the required fact even make it into the context? If the gold document never shows up in Top-K, no prompt tweak on earth saves you. Check query rewriting, filters, chunk boundaries, and embeddings instead. This is the cheapest check to run and the one most teams skip, because it means admitting the problem might be upstream of the part they already understand. Pull the raw retrieved chunks for the failing query and just look. If the fact isn’t there, stop debating prompt wording and go fix the retrieval path.
- ⚖️ Evidence competition: the right passage IS there, it just lost to a louder, more plausible-sounding distractor. The post’s legal example nails this: a precise “90 days” clause ranked second while generic termination language ate up the context window. Test it counterfactually, run the gold passage alone, then with one distractor at a time, then with the order flipped. That tells you whether it’s a ranking problem, a weighting problem, or one specific bad block. This layer is the sneakiest of the three because the system looks like it’s “almost right.” The answer sounds confident and on-topic, so teams assume the model just needs a firmer instruction. Usually it needs the distractor demoted or removed, not a sterner prompt.
- 📋 Answer-contract failure: the evidence survived, but the generator was never forced to use it right. “Advance written notice is required” sounds grounded and still fails to answer “how many days.” Fix this with a field-level contract: the answer must contain a quantity and a unit, or it has to abstain. This is the layer prompt engineering actually helps with, but only once you’ve confirmed the first two layers are clean. Otherwise you’re bolting a stricter output schema onto a model that never had the right number in front of it to begin with.
How to actually run this
- Feed the model the gold passage alone first, before touching retrieval at all
- If it still fails there, the problem is downstream (prompt, schema, or evaluator), not retrieval
- Build a separate regression fixture for each of the three layers, don’t lean on one end-to-end score to tell you which layer broke
- Keep a running log of which layer each past failure traced back to. Patterns show up fast, and most teams find that one layer accounts for the majority of their bad answers, which tells you exactly where to invest first
- When a fix ships, rerun all three fixtures, not just the one you were targeting, so you catch a retrieval fix that quietly hurt the answer-contract layer
Commenters mostly agreed this beats the usual swap-and-pray approach, especially the counterfactual test for competition failures. One RAG builder said it saved them from chasing a context-ordering bug they’d have blamed on the embedding model. Another pointed out that once they started tagging failures by layer, retrieval absence turned out to be the cause almost 60% of the time on their dataset, and they’d spent the previous month tweaking prompts instead.
If your RAG system keeps failing and you keep turning the same three knobs with no better luck, stop guessing. Run the three-question diagnostic first, then fix exactly what broke!
Frequently Asked Questions
Q: When should I swap my embedding model vs. fixing context ordering?
Start by testing whether the gold passage alone answers your question correctly. If it does, your embedding model is probably fine, the real issue is likely context ordering, chunk boundaries, or competing distractors. Jump to embedding swaps only after ruling out these structural problems. Many teams waste time tuning embeddings when the fix is reranking or removing noisy context.
Q: How do I know if my problem is retrieval or generation?
Use the gold passage test: feed only the correct passage to your generator and see if it produces the right answer. If it works in isolation, your generation layer is fine and retrieval tuning will help. If it still fails, you have an answer-contract problem, the model is allowed to produce incomplete or imprecise answers. Fix that first.
Q: Why do faithfulness scores sometimes miss the real problem?
A faithfulness score tells you if an answer is grounded in text, but not whether it’s complete. For a duration question, “advance written notice is required” is faithful but useless, it’s missing the number of days. Add field-level contracts (e.g., “duration answers must include quantity + unit”) to catch incomplete answers that slip past generic scoring.
Q: What should I do when the right passage is present but gets drowned out?
Test counterfactually: gold passage alone (should pass), then gold passage + one distractor at a time. If a specific distractor causes failure, your options are reranking to boost the gold passage, compressing the distractor, or removing it entirely. The fix depends on whether the distractor contradicts or just dilutes focus.
Three RAG failures can produce the same bad answer. They need different fixes.
by u/Critical-Elephant630 in PromptEngineering