Most people fix a flaky prompt by rewriting it. This approach leaves the words alone and tests the prompt harder, and it saves you hours.
Here’s the situation. You run a prompt on Monday and get a clean answer. You run the exact same text on Tuesday and get a mess. So you start tweaking, adding a line here, swapping a word there, chasing a ghost. A Redditor on r/PromptEngineering described this as maddening, and I think a lot of us have been there. The worst part is that every edit feels productive. You change a phrase, run it once, get a good result, and decide you fixed it. Two days later the problem is back, and you have no idea which edit mattered, or whether any of them did.
The Key Idea
The prompt usually isn’t the problem. The problem is that it only works when the model happens to take the right path. Language models sample their output, so the same input can wander down different routes on different runs. If your prompt succeeds only when the dice cooperate, it isn’t done. It’s lucky.
Think of it like a recipe that says “add some salt and cook until it looks right.” A seasoned cook gets a great dish most nights. A beginner gets a different dish every time. The recipe never changed. It just left too many decisions open. Your prompt is the recipe, and the model is a cook who fills every gap with a guess.
The Contrast: Old Way vs. New Way
The old way: Run the prompt once. It looks good, so you ship it. Later it fails, so you edit the wording and run it once more. You are judging a probability with a single sample. If the prompt works 70 percent of the time, one run will look fine seven times out of ten, and you will never see the problem until it lands in front of a customer or a client.
The new way: Treat the prompt like code. You don’t trust a function because it passed once, and you shouldn’t trust a prompt that way either. As one commenter put it, prompt engineering is slowly turning into software testing. They have a point. You write the prompt, you run it many times, you look at the failures, and you tighten the weak spots. The edit only counts if the failure rate drops.
Another common reaction is to lower the temperature and hope. That helps a little, but it hides the weakness instead of fixing it. A prompt that only works at zero randomness is still fragile. Change the input slightly, or switch models, and the problem comes right back. The goal is a prompt that works across runs, not one that works because you pinned the dice.
Practical Steps
- Specify the format explicitly. Say exactly what the output should look like: sections, length, field names, order. Every open choice is a place for the model to drift. For example, “Summarize this email” leaves length, tone, and structure open. “Write three bullet points, each under 15 words, starting with the action required” closes all three.
- Add a worked example. One good input and output pair works like a map instead of a hope that the model guesses the right direction. Pick an example that is typical, not a perfect edge case, so the model copies the pattern you actually want.
- Constrain the output structure. Use a fixed template, a JSON schema, or labeled sections. The less room to wander, the less variance you get. If a downstream tool reads the output, a strict structure also saves you from parsing surprises.
- Run it several times on the same input. Try five to ten runs before you trust it. If even one looks off, the prompt still has slack in it. Paste the runs side by side and look for the spots where they disagree. Those spots show you exactly which instruction is too loose.
- Test on different inputs too. Repeating one input only proves consistency on that one case. It won’t tell you whether your worked example made the prompt too narrow. Keep a small set of varied inputs aside while you revise. Include one short input, one long and messy one, and one odd case that breaks your usual assumptions.
- Pin the model version. Another commenter pointed out a cause beyond sampling: if you call an alias like “latest”, the same text can hit a different snapshot from one day to the next. Use the dated model version so you know what changed when results shift.
A Quick Checklist 🧪
- Format spelled out? ✅
- One worked example included? ✅
- Output structure fixed? ✅
- Ran it multiple times on the same input?
If any box is empty, the prompt isn’t ready to rely on yet.
Why This Matters
A prompt that works once teaches you almost nothing. A prompt that works ten times in a row, on several different inputs, on a pinned model, is something you can build a workflow on. That’s the difference between a lucky roll and a reliable tool. It also changes how you spend your time. Instead of rewriting a prompt every week when it misbehaves, you do the testing once, fix the loose spots, and move on to the next problem.
Your Move
Pick the one prompt you depend on most and run it ten times on the same input today. Count the misses. Then fix the weak spot with a tighter format or a better example, and test again. Share what you find with the crew, because I’d love to hear how many of your “working” prompts turn out to be lucky ones!
Frequently Asked Questions
Q: Why does the same prompt give different results on different days?
Two things can change: sampling randomness, and the model behind an alias like ‘latest’ can quietly switch to a new snapshot. Pin the dated model version while you test so the model stays fixed. Then the prompt is the only thing you’re changing.
Q: How many runs do I need before I trust a prompt?
One good run tells you very little. Run the same input 5 to 10 times and count how many pass, since 9 out of 10 is a much stronger signal than one lucky result. Repeat that check whenever you change the prompt or the model.
Q: Does adding a worked example actually help?
Some commenters see an example as a map: it shows the model the exact format and direction you want instead of making it guess. Others warn that an example can make the prompt too narrow, so the model copies the example’s shape even when the input is different. Test with inputs unlike the example to see which side your prompt lands on.
Q: What other inputs should I test besides the one that works?
Keep a few different inputs aside while you revise, not just the one you keep repeating. For an email deadline extractor, try a message with no deadline, one with two conflicting dates, and a forwarded old thread. Write down the expected result for each before you run it, then reuse those same cases after every change so a tidier output can’t hide wrong facts.
Why my prompt worked yesterday and failed today, same text
by u/ChanceWheel646 in PromptEngineering