Most people treat a weak prompt like a weak sentence. Reword it, swap a verb, add “be concise and thorough,” and hope the model finally gets it. Maybe you throw in “think step by step” because someone told you it helps. Maybe you add three more adjectives to the instruction, or move the important part to the top because you read somewhere that models pay more attention to the first line. A thread on r/PromptEngineering from u/Future_AGI makes the case that this whole approach is aimed at the wrong target, and once you see it, you can’t unsee it.
The key idea: the exact words in your prompt matter way less than what you feed it and how you check the output. Wordsmithing feels like progress because you can see yourself doing it. You typed something, it looks different, surely that counts as work. Measuring feels slower because it doesn’t feel like writing at all. It feels like homework. But only one of them actually moves the needle, and it’s not the one that feels like writing.
Old way vs. new way
The old way: write a prompt, run it once, read the output, decide if it “feels” better, ship it. That’s tuning on vibes. You’re grading your own homework with a sample size of one. You read the output, it sounds more confident or more polished than the last version, and you call it done. Nobody ran the old version and the new version against the same twenty cases side by side. Nobody wrote down what “better” even means before judging it.
The new way: treat the prompt like a system you evaluate, not a sentence you polish. The post’s author admits they rewrote prompts, liked how the new version read, and then scored it against saved test cases only to find it performed the same or worse than the original. The wording changed. The results didn’t. That’s the uncomfortable part. It’s not that wording never matters, it’s that you can’t tell whether it mattered until you check, and most people never check. They ship on the strength of one good-looking output and move on to the next prompt.
What actually moved the results, every time, according to the post:
- Context. The model can’t infer what you never gave it. Half the prompts people “fix” with sharper phrasing were just missing input the model needed, like the actual policy document, the format of a real past example, or the constraint that made the task hard in the first place. No amount of clever phrasing replaces the information the model simply doesn’t have.
- Contrastive examples. Two or three good-and-bad outputs teach the model more than a paragraph explaining what “good” means. If the same mistake keeps showing up, paste it right next to the correct version. Telling a model “be more precise” is vague. Showing it one sloppy answer and one precise answer, side by side, gives it something to pattern-match against instead of guess at.
- A scoreable rubric. Write down exactly what a correct answer must contain, then run the prompt across ten or twenty real cases and check each one against that list. One nice-looking output is not evidence. A rubric turns “this feels better” into “this passed 17 out of 20,” which is the only kind of statement you can actually act on.
How to actually do this
- Pull 10-20 real cases you already have, not hypothetical ones. Old support tickets, old emails, old outputs you already judged by hand. Real data exposes edge cases a made-up example never will.
- Write a short rubric: what must a correct answer include, what disqualifies it. Keep it specific enough that two different people would score the same output the same way.
- Score your current prompt against all of them, not just the one you remember liking. This is the step people skip because it’s tedious, and it’s also the step that actually tells you something.
- When you see a recurring mistake, add a contrastive example (wrong version next to right version) instead of adding another instruction sentence. One good pair of examples usually fixes what a paragraph of extra instructions can’t.
- Re-score after every change. If the number didn’t move, the wording didn’t matter, no matter how much better the new prompt sounds when you read it out loud.
One commenter put it well: for support-ticket routing, “classify this carefully” does almost nothing, but showing the model a refund request next to a billing question does the actual work. Another commenter added a similar case for summarization prompts: telling the model to “keep it concise” barely changes anything, but pasting one bloated summary next to one tight summary shows exactly what concise looks like in practice.
This is closer to prompt evaluation than prompt engineering, and that’s the part that holds up as the model underneath you keeps changing. Wordsmithing is guessing. A rubric and a batch of test cases is how you actually know, and that method survives every model upgrade because it doesn’t depend on which version you’re running, only on whether the output still meets the bar you wrote down.
Next time you catch yourself rewriting a prompt because the new version “feels sharper,” stop. Build the rubric first. Then find out if it’s actually better, or if you just got lucky on one run.
Frequently Asked Questions
Q: How do I add examples to my prompts without spending forever on it?
Don’t overthink it. Paste one example of output you don’t want, then one showing exactly what you do want, side-by-side. That’s contrasting examples. One commenter had formatting chaos in summaries, and no instruction rewording fixed it, but a single pair of examples solved it overnight. You don’t need perfect examples; contrasting ones teach faster.
Q: How do I avoid “overfitting” my prompt to my test cases?
Keep two groups of test cases: one for developing (your tuning set) and one you don’t touch until the end (your evaluation set). While tweaking, score only against the evaluation set. This catches the trap where a prompt sounds better but actually performs the same on fresh data. It’s especially important before model updates, which change what works.
Q: Give me a concrete example of examples beating instructions.
Support ticket routing. Saying “classify this carefully” does almost nothing. But show the model three boundary cases (a refund request, a billing question, a technical issue that mentions price, an ambiguous message needing “needs_review”), and it learns what “careful” means. Examples do the work instructions couldn’t.
Q: How do I define “good” for scoring if I don’t have a rubric yet?
Pick 3-5 concrete criteria. For summaries: “no random bolding,” “under 50 words,” “one main idea per bullet.” Then score 10-20 real outputs against those. It’s tedious upfront, but way faster than endlessly rewriting prompts hoping one sounds right. And you’ll catch the ones that sound good but actually don’t perform better.
Wordsmithing your prompt is guessing. Measuring “good” is the actual engineering.
by u/Future_AGI in PromptEngineering