Yesterday, a single markdown file quietly reset how prompt testing should work. Step 2 of its seven-stage process is the part that stopped me mid-scroll: it never lets a good vibe overrule a failed check. u/Parking-Kangaroo-63, the original poster over on r/PromptEngineering, built the whole thing as one skill file for Claude Code. It answers a question most prompt advice skips entirely: how do you actually know a prompt works?
Most guides stop at “make it more specific.” That helps for about five minutes, right up until it hits missing data, an adversarial input, a schema violation, or an untested model swap. The skill, called prompt-evaluation-engineer, treats every prompt like a behavior contract instead of a paragraph you tweak until it “feels right.” No backend, no API key, no package. Drop the file into .claude/skills/prompt-evaluation-engineer/SKILL.md and Claude Code picks it up the next time you ask it to evaluate, test, benchmark, or compare a prompt.
The twist
Here’s the part that got me. The skill splits every check into two buckets, deterministic and semantic, and runs them in a strict order that never reverses.
Deterministic checks go first: JSON parsing, regex matches, schema validation, required and forbidden strings, length bounds. Only after those pass does a semantic rubric even get a turn, scoring things like tone, relevance, and completeness. A polite, well-written support answer that leaks an account secret still fails, no matter how helpful it sounds. That ordering is the whole trick: it stops a fluent, wrong answer from sneaking through as a “pass” just because it read smoothly.
There’s also a section most eval write-ups skip entirely: integrity rules. The skill bans grading a prompt with the same model that generated the response, unless that’s disclosed upfront. It bans quietly dropping a failing test case just to make an average look cleaner after the fact. Every reported result gets treated as permanent data, not something you get to touch up once the numbers look bad.
The mini-workflow
The skill walks Claude through seven stages, but the practical loop for anyone testing a prompt today looks like this:
- 📝 Write the contract first: objective, inputs, required and forbidden outputs, acceptance criteria, and what you’re explicitly not scoring.
- 🧩 Build a test matrix: golden cases, boundary cases (empty, huge, malformed, multilingual), adversarial cases, contrast pairs, and regressions from prior failures.
- 🔍 Run the deterministic checks, record exactly what breaks, and only then look at the semantic scores for diagnostic color.
- 📋 Preserve the raw evidence: exact prompt, model settings, unedited output, per-case pass or fail. Never average failures into one clean-looking number.
Then it reports a plain decision (pass, fail, inconclusive, or not executed) plus next steps, instead of a vague “looks good to me.”
Pro tips
If you’re comparing two prompt versions, freeze the test matrix and settings between runs. The skill is explicit about this. Change the matrix or the evaluator mid-comparison, and you’re not running an A/B test anymore. You’re just generating noise that looks organized.
For anything customer-facing, score refusal and redaction separately from helpfulness. The skill’s own example nails it. A support answer that stays polite while revealing a secret needs to fail outright, not get smoothed into a decent-looking average.
Most prompt-eval setups assume you already have a harness: a scoring backend, a dashboard, maybe a hosted judge model wired in through an API. This one skips all of that. It’s a single file that turns Claude Code itself into the harness. It runs the same seven-stage checklist every time, instead of whatever loose process happens to live in someone’s head that day.
One thing worth knowing before you install it: this is a protocol, not a test runner. It tells Claude how to structure the contract, matrix, and report. But someone still has to run the prompt against a real model for the results to count as anything more than a proposal. The skill actually says this out loud, which is more honesty than most eval tooling bothers with.
A commenter on the thread already picked up on this. They pointed out how rare it is to see deterministic and semantic checks kept apart, instead of blended into one fuzzy “looks good” score.
I’ve watched plenty of prompts look great in three cherry-picked examples and fall apart the moment real users touch them. If that sounds familiar, this is worth the five minutes it takes to install. Grab the file, drop it into .claude/skills/, and run your next prompt through the contract before you ship it. 🚀 The full writeup and the r/PromptEngineering thread are worth a look if you want to see the reactions it’s already pulling in.
Frequently Asked Questions
Q: Do I really run all 7 stages every time I tweak a prompt?
Nope. For quick iterations, just run the deterministic checks (things like format breaks, missing required fields, or forbidden content). Save the full protocol for before you ship something or make important decisions. You don’t need ceremony for every small change.
Q: What’s the actual difference between deterministic and semantic checks?
Deterministic checks are the hard stuff: Does the output have the right format? Is sensitive data leaking? Are required fields present? Semantic rubrics are the softer judgments: Is the tone right? Is the answer actually helpful? Use deterministic checks constantly; add semantic ones when you care about quality, not just correctness.
Q: How do I know which edge cases to test?
Start simple: empty inputs, weird characters, boundary values. If your prompt touches money, health, or security decisions, go deeper with adversarial tests. If it’s low-stakes, basic checks usually catch the worst problems; you can always add more tests later if things break in production.
Q: Do I have to use the whole framework?
No. Pick what helps. Start with an evaluation contract and deterministic checks, add semantic rubrics later if you need them. No backend means no commitment; remix it however makes sense for your workflow.
Prompt-Evaluation-Engineer skill.md (Claude Code)
by u/Parking-Kangaroo-63 in PromptEngineering