Everyone does the same thing when a prompt gives flaky output. They swap the model, nudge the temperature, or add “IMPORTANT” in capital letters. This approach goes the other way: audit the prompt first, because the fix is often sitting in the text you already wrote.
Credit where it’s due. This comes from u/ClickOk5811 on r/PromptEngineering, and it’s one of the cleanest prompt checks I’ve seen.
The key idea
Pretend you’re a new colleague who just received your prompt. You can’t ask anyone anything. You can’t peek at the Slack thread where the prompt was born, and you don’t know what the last version looked like. You only have the text. Now try to answer five questions from that text alone. Wherever you get stuck, the model gets stuck too, and it fills the gap with a guess. Different guesses on different runs is what you see as inconsistency.
The nice part is that this reframes the problem. A flaky prompt isn’t mysterious. It’s a prompt with open decisions, and open decisions can be found by reading.
Old way vs. new way
The old way: the output is wrong, so the model must be the problem. You tweak settings, try a bigger model, and add more adjectives. Sometimes it works by accident, which is the worst outcome, because you learn the wrong lesson and carry it into the next prompt.
The new way: the output is inconsistent, so look for what the prompt never decided. You add the missing decisions as plain, short rules. The model stops guessing because there’s nothing left to guess. The fix also stays put when you change models later, since the rules live in your text and not in one model’s habits.
The five questions
- 🔍 What happens when the input is missing or empty? If the prompt is silent, the model may invent a plausible value. A blank customer name becomes “Valued Customer” on one run and “John” on the next. Add one line: “If X is missing, return null and don’t guess.”
- ⚖️ What happens when two instructions conflict? “Be concise” and “explain your reasoning in detail” can sit in the same prompt, often added weeks apart by different people. Pick a winner: “If length and completeness conflict, completeness wins.”
- 🎯 What does “good” look like? Words like “professional” or “high quality” have no edges. A short example of an acceptable output, or a hard limit like “max 3 sentences, no bullet lists,” beats another adjective. If you can, show one example of a bad output too, so the model sees the boundary from both sides.
- 🚧 What am I not allowed to do? Boundaries often live only in your head. If it can’t give legal advice, can’t use outside knowledge, or can’t change the user’s wording, write that down. The model has no way to know about a rule you never stated.
- Output destination: what does the answer feed into? If a program parses it, the format is part of the task. “Return JSON” is weak. Name the fields, the allowed values, and what to do when a field is unknown. A human reader forgives a stray sentence. A parser crashes on it.
A quick before and after
Before: “Summarize the customer message and suggest a reply. Keep it short and friendly.”
After: “Summarize the customer message in at most 2 sentences. Then draft a reply of at most 4 sentences. If the message contains no question or request, write ‘No reply needed’ instead of drafting one. Don’t promise refunds or dates. If tone and length conflict, keep the length limit.”
The second version isn’t smarter. It just answers the five questions, so the model has less to improvise. Notice what the “before” left open: how short is short, what to do with a message that’s just “thanks,” whether a friendly reply may promise a refund, and which wins when friendliness wants more words. Each of those is a coin flip on every run.
How to run the audit
- Open your flakiest prompt and read it like a stranger would. Better yet, hand it to an actual teammate and ask what they would produce.
- Write one answer per question. If you can’t answer from the text, that’s a gap. Jot the gaps in a list so you can see the pattern.
- Turn each gap into one short line in the prompt. Don’t rewrite everything. Small edits are easier to test and easier to undo.
- Test on a handful of real inputs, not just one. Include an ugly one: an empty field, a rambling message, something off topic. A prompt that works once proves very little.
- Check the length. Longer prompts can bury important rules, so keep each added line tight, and put the rules that matter most near the top or the bottom where they’re hard to miss.
What this won’t fix
The author is upfront about this, and I like that. Some inconsistency really does come from the model or the sampling settings. This audit removes the prompt as a suspect. It doesn’t promise a perfect run every time. If you’ve closed every gap and the output still wobbles, that’s the moment to look at temperature or the model, and you’ll do it with a clear conscience.
Your turn
Pick one prompt you use every week and run the five questions on it today. It takes about ten minutes. Then tell me which question your prompt fails most often. My bet is number 2, because conflicting instructions sneak in whenever a prompt gets edited more than twice!
A 5-question ambiguity audit for your prompt (before you blame the model)
by u/ClickOk5811 in PromptEngineering