Ask a model to answer from your notes and include a supporting quote, and you will usually get both. Most people stop there and call the prompt done. A Reddit user on r/PromptEngineering suggests a better check: run three small cases and see what the model does when the notes get awkward.
The key idea
A quote sitting next to an answer proves nothing by itself. The model may have written the answer first and then found a line that sounds close enough. The quote is only worth something if the answer actually depends on it.
So the test is not “does the prompt return a quote?” It is “does the quote behave correctly when the sources are clean, conflicting, or silent?” Think of it like checking a receipt. Having a receipt is nice, but you only care that the numbers on it match what you actually bought.
The old way vs. the new way
Old way: Write the prompt, try one easy question, see a quote, ship it. The trouble is that one easy question is the single case almost any prompt passes. You learn nothing about the messy situations your real notes will create.
New way: Use one question and three note sets. Each set tests a different failure.
The question is the same every time: When does the office close on Friday?
- 📌 Case 1, one clean note. The note says “Friday hours are 9 a.m. to 5 p.m.” A good answer says 5 p.m. and quotes that note.
- 📌 Case 2, two conflicting notes. Add a second note saying “Friday hours are 9 a.m. to 3 p.m.” Neither has a date. A good answer shows both quotes and flags the conflict instead of quietly picking one.
- 📌 Case 3, no relevant note. Give it only “Monday to Thursday hours are 9 a.m. to 5 p.m.” A good answer says the notes don’t specify Friday hours. It should not stretch Monday to Thursday to cover Friday.
Case 1 is the easy one. Cases 2 and 3 are where weak prompts fall apart. In real use, your notes will conflict and go silent far more often than you expect, because notes get written at different times by different people.
How to run it
- Open three fresh chats. Fresh matters. A reused chat carries earlier context and muddies the result.
- Paste one note set per chat and ask the same question. Keep the wording of the question identical so the only thing changing is the notes.
- Check each answer against the “good answer” description above. Did it quote, flag, or admit the gap? Write down a simple pass or fail for each case so you can compare later.
- Change one instruction only. Make one prompt ask for the answer, then the supporting quote. Make the other ask for the supporting quote first, then the answer. Leave everything else, including tone and formatting instructions, exactly as it was.
- Run both versions through all three cases. That gives you six results to compare side by side.
The order change is the interesting part. Quote-first forces the model to commit to evidence before it states a conclusion. Answer-first lets it commit to a conclusion and then justify it. Whether that difference shows up when sources disagree is exactly what the original poster wanted to find out. I haven’t got a verdict for you, and the thread doesn’t have one either. That is why you run it yourself.
Make the test more trustworthy
Two commenters added fixes worth stealing:
- Score two things separately. Grade answer correctness on its own, and grade quote support on its own. An answer can be right with a quote that doesn’t back it up, or wrong with a perfectly real quote. A two-column tally makes this easy to see at a glance.
- Repeat each ordering across several fresh chats. One lucky run can make an ordering look better than it is. Models vary from run to run, so a single pass tells you very little. Five runs per case is a reasonable starting point if you want a rough signal without spending your whole afternoon.
Another commenter uses a second model to check whether a quote really supports the claim drawn from it. That is a heavier setup, but it is worth it if the output feeds something important. One slip in their pipeline let drafts skip that check, and they threw out the whole batch. Treat the verification step as part of the system, not an optional extra.
Quick scorecard
- Case 1 passes if the answer and quote match.
- Case 2 passes if both quotes appear and the conflict is named.
- Case 3 passes if the model admits the notes are silent.
If Case 3 fails, your prompt probably needs an explicit line allowing “the notes don’t say.” Models tend to fill gaps unless you tell them gaps are acceptable. A short sentence such as “If the notes do not answer the question, say so instead of guessing” is often enough to fix it. Run Case 3 again after adding it and see whether the behavior changes.
Your turn
Take a prompt you already use for note-based answers and run these three cases today. Try both orderings, repeat each a few times, and score answer and quote separately. Then tell me which order held up when your sources disagreed! I’m curious whether it holds for you.
Frequently Asked Questions
Q: Does putting the answer or the quote first actually change the result?
Commenters haven’t settled this, and one said they haven’t tested the order at all. The safer approach is to score answer correctness and quote support as two separate checks. Then repeat each ordering across a few fresh chats, so one lucky run doesn’t make a layout look better than it is.
Q: How do I check that a quote really supports the answer?
One commenter sends the answer and its quote to a second model and asks whether the quote backs up the claim. That check caught a bug that let unverified drafts through, so it’s worth adding if you trust quotes in any pipeline. Using a different model for the check helps it catch mistakes the first model would miss.
Q: Does the formatting around the prompt matter, or just the wording?
Formatting can matter a lot. Sclar et al. (ICLR 2024) changed only colons, spacing, and field labels, and on LLaMA-2-13B the accuracy gap reached 76 points. He et al. (2024) found GPT-3.5 shifted by up to 40% across plain text, Markdown, JSON, and YAML, while GPT-4 was steadier. No single format won for every model, so test your layout on the model you actually use.
Q: What’s a good layout to test first?
One suggestion is to split the prompt into three labeled parts: Task, Context, and Answer format. Keep the notes and question the same, then compare answer-first and quote-first inside that layout. Once the labels are fixed, any difference you see is easier to trace to the ordering itself.
A three-case test for prompts that ask for a supporting quote
by u/MeringueNew4583 in PromptEngineering