Picture this: forty minutes into a client call, someone finally asks their AI assistant for a clean summary of the transcript. Five tidy sections come back, formatted, confident, ready to paste into a report. Two of those sections describe things nobody in the meeting actually said. That’s the exact situation u/vedantk21 described in a post on r/PromptEngineering, and it’s a problem a lot of us have run into without naming it.
Here’s why it matters: a rough transcript dump gets read with suspicion. Everyone knows it’s messy, so everyone double-checks it. But the second that same content gets cleaned up into five neat headers, it stops looking like a draft and starts looking like a decision. Nobody questions a formatted doc the way they question raw notes, including the person who wrote the prompt in the first place. The redditor’s own workflow ends with the cleaned summary going straight into Gamma to build a client-facing version. Once fabricated sections survive that trip, they’re basically locked in. Someone on the client side reads a polished five-section deck, nods along, and starts making decisions based on a “commitment” that was never actually made on the call.
Why It Matters 🧭
The root cause here isn’t a bad model. It’s a prompt that lets the AI treat “plausible” and “true” as the same thing. Ask for a five-section summary of a long call, and the model will happily generate five sections whether or not the transcript actually supports all five. It’s not lying on purpose. It’s filling gaps the way any autocomplete would, with whatever sounds coherent. Ask for exactly five sections and you’ll get exactly five sections, even if the actual conversation only justifies three. The model isn’t built to say “there were only three real themes here.” It’s built to satisfy the shape of the request, and a confident, complete-looking output is part of that shape. That’s the trap. The format itself creates pressure to invent content, because an obviously incomplete summary looks like a worse answer than a fabricated but polished one.
The Two-Source Fix 🔧
Quick start: this is a two-step constraint you bolt onto any summarization prompt, in this specific order.
Step 1: Quote before you summarize. Force the model to pull verbatim lines from the transcript first, before it’s allowed to build any section. If it can’t find a real line to quote, it can’t build a section around it. This alone kills a huge chunk of invented content, because fabrication requires the model to skip straight to “conclusion” without ever touching “evidence.” Making the quote step happen first and separately matters more than it sounds. If you ask for quotes and sections in the same breath, the model can still work backward from a section it already wants to write and cherry-pick something that sort of fits.
Step 2: Two-source minimum. A section only survives if at least two separate statements in the transcript back it up. One passing mention isn’t a theme, it’s someone thinking out loud, and that distinction is where most of the fabricated structure was sneaking in. Say a client mentions budget once, in passing, thirty minutes into the call. Under a single-quote rule, that comment is enough to justify a whole “Budget Concerns” section. Under the two-source rule, it isn’t, unless someone else in the meeting also raised it, or the same person circled back to it later. That second requirement is what separates an actual recurring theme from a stray comment getting inflated into a headline.
The actual prompt language, straight from the post: “first extract verbatim quotes relevant to the discussion. then propose sections, and for each one cite the two or more quotes it rests on. drop any section you can’t support twice.”
One commenter, u/No_Steak_131, called the second constraint “the real killer,” and that tracks. Requiring one quote catches the worst fabrications, the sections built entirely out of thin air. Requiring two catches the subtler ones, the sections that sound structurally reasonable but only rest on a single offhand comment stretched to fill a header.
Tips & Tricks 💡
- Run the quote-extraction step as its own pass before asking for sections. Don’t let the model do both in one breath, or it’ll skip straight back to guessing.
- Expect some drift to remain on genuinely ambiguous topics. This method removes confident fabrication, not every judgment call.
- Treat this as a pre-formatting checkpoint, not a final QC step. Clean the content before it gets dressed up in a polished doc, because once it looks finished, nobody reads it skeptically anymore, including you.
- This same quote-then-verify pattern works beyond call transcripts. Try it on long PDF summaries, research digests, or anything where an AI is condensing a big source into a short structure.
- If a section gets dropped for lacking two sources, don’t just delete it silently. Ask the model to note what was cut and why. That single line becomes a fast sanity check next time you skim the output, and it tells you exactly where the transcript was thin.
- For recurring meeting types, like weekly client check-ins, save the working prompt somewhere you’ll actually find it again. The two-source rule only helps if it’s applied every time, not just the one time you got burned.
Final Word 🎯
The fix costs one extra prompting step and saves a client-facing document from shipping with sections nobody said. Worth stealing for the next transcript that lands on the desk. Drop the two-source rule into your next summarization prompt and see how many sections quietly disappear.
Frequently Asked Questions
Q: Why is two sources better than one?
One mention is usually someone thinking out loud, not a pattern. Two sources mean it’s something people actually discussed repeatedly. As the commenter learned the hard way, a single offhand comment can balloon into an action item that nobody agreed to, and clients won’t forget that mistake even after it’s corrected.
Q: What’s the actual implementation?
Extract verbatim quotes first, then build sections around those quotes. For each section, cite at least two separate statements. If you can’t find two quotes backing a section, drop it. This keeps your summary grounded in what was actually said instead of what you inferred.
Q: Does enforcing two sources slow things down?
Yes, but invest it upfront. The alternative is shipping summaries with hallucinations embedded in polished documents. Once they’re formatted nicely, they rarely get questioned, even when they’re wrong.
Q: What if something important only got one mention?
If it doesn’t have two sources, it wasn’t actually a pattern, just one person’s passing thought. The approach still allows some wiggle room on ambiguous stuff, but kills most confident fabrications. If it’s truly important, it’ll probably come up more than once.
The two-source rule that stopped my transcript summaries from inventing sections
by u/vedantk21 in PromptEngineering