One bad trace is usually all it takes. You watch a model wrestle with an instruction, you rewrite the prompt on the spot, and you feel like you fixed something. u/drunk-at-noon, posting in r/PromptEngineering, called out the gap in that reflex: a trace where a model fights an instruction is a good reason to edit a prompt. It’s not evidence the edit works anywhere else.
That’s the whole problem with prompt engineering by vibes. You see one failure, you patch it, and you never check if the patch holds up on the next ten tasks. The original poster pointed to a project called GEPA, built by Reef Infra, that treats those two jobs as separate steps instead of one impulsive edit.
Here’s the quick version. GEPA keeps an archive of prompt candidates instead of just the current best one. It picks a parent prompt, looks at where that parent failed on a small batch of training tasks, and asks a separate reflection model to rewrite exactly one part of the prompt. Nothing gets promoted until it earns it.
🆚 Old Way vs New Way
The old way looks like this: something breaks, you rewrite the whole prompt based on gut feeling, you ship it, and you find out weeks later it only fixed your one example. There’s no checkpoint between “this feels better” and “this is now live.”
The new way adds two checkpoints. First, the edited prompt has to beat its parent on the same minibatch of training tasks before it earns anything else. Second, only then does it get a full validation pass across a wider set of tasks. The version that actually gets used is picked by its mean validation score, not by how convincing the edit sounded.
One detail matters more than it looks: the model doing the actual task never changes. Only the text around it does. That makes this approach useful specifically when the problem is a contradictory rule, a vague instruction, or a skill description that’s actively working against the model. It won’t help if the model is simply missing a capability it doesn’t have.
🔧 How to steal this for your own prompts
You don’t need Reef Infra’s codebase to use the idea behind it. Here’s the version you can run this week:
- Keep every prompt version you write, even the ones you think are bad. 🗂️ An archive beats a single “current” file, because you’ll want to compare later.
- When something fails, don’t rewrite the whole prompt. Pick the one instruction or rule that actually caused the fight, and change only that.
- Before you trust the edit, run it against a small batch of similar tasks, not just the one that broke. Five to ten examples is enough to start.
- If the edit doesn’t beat the old version on that batch, throw it out and try again. Don’t promote a tie.
- Once it wins on the small batch, test it on a separate, larger set of tasks the edit has never seen. ✅ This is the step most people skip.
- Score everything the same way, every time, and let the score pick the winner. Not your gut, not how clever the wording feels.
One commenter, u/Better-Track-6784, put the core insight better than most explanations do: most people see a bad trace and immediately rewrite the whole prompt, then wonder why only their one example got better. Separating “why did this fail” from “does this fix work elsewhere” is the entire trick.
There’s a real limit here worth saying out loud. The repository’s example runs on AIME math problems with a fixed train, validation, and test split, and a set search budget. That held-out test set stays separate from the one used to pick the winning candidate. That separation matters, because an optimizer given too many shots at the same evaluator can learn to write answers that flatter the grader instead of answers that are actually better.
None of that proves this works for a coding agent or a support bot. If you’re editing prompts for those workflows, the process transfers but the AIME result doesn’t. You’ll need your own task set and your own scorer before the win means anything.
The reusable part isn’t the AIME benchmark. It’s the two-step habit: let a failed trace suggest what to change, and let a different set of tasks decide whether the change actually travels. Go read the original thread if you want the full recipe, and try running one prompt through both checkpoints before your next rewrite.
Frequently Asked Questions
Q: Why should I keep validation separate from held-out eval?
If you pick your best prompt based on the same eval set you used to select it, you’re essentially overfitting to that data, your prompt looks good on that set but might fail on new tasks. The held-out eval confirms your improvement actually generalizes, not just that you’ve memorized the validation examples.
Q: Reflection model or human review, which is better?
It depends on scale and cost. Humans catch nuances but don’t scale; reflection models automate analysis at lower marginal cost but might miss context. If you’re iterating constantly, automation wins; for one-off improvements, a human review of traces is probably smarter and faster.
Q: Why not just rewrite the entire prompt when something fails?
Full rewrites risk breaking other cases you weren’t monitoring. The parent-child structure keeps you honest, each candidate must beat the parent on the failing batch before getting validated further, preventing over-correction that works locally but breaks elsewhere.
Q: How do I know if my prompt improvement is real or just got lucky on validation?
Watch both validation and test scores, they should move together if the improvement is genuine. If validation improved but test stayed flat, your prompt probably memorized the validation set. Real improvements generalize across both.
Use failed runs to generate prompt edits, then judge the edits on fresh tasks
by u/drunk-at-noon in PromptEngineering