A refund-policy prompt can return the correct dollar amount for the wrong reason, and you’d never catch it by reading the answer. That’s the gap skillEd goes after, and it’s a neat one.
What’s new
skillEd is an open-source toolkit for evaluating reusable agent skills. You keep test cases right next to a skill, change the instructions, and compare saved results before and after. No more squinting at two versions of a prompt and guessing which one is better. Because the cases live in the same folder as the skill, they travel with it. If a teammate edits the instructions next month, the same cases are sitting right there, ready to run, and the history of past results tells you whether the change helped or quietly broke something.
The twist
The creator’s point is that “follow the refund policy” is too broad to test as one instruction. Split it into checks that can fail on their own:
- Use the correct eligibility window
- Exclude shipping
- Apply the required rounding
- Explain the decision in a way that matches the calculated result
A polished answer can land the right amount with a bad explanation, or give a lovely explanation with the wrong amount. Reading prose alone misses both. Splitting the checks makes each failure show up by name. Instead of one vague “this feels off,” you get a report that says the rounding check failed while the window check passed. That tells you exactly which sentence of the prompt to go fix.
Picture a customer who bought a $49.99 item plus $6.00 shipping and asks for a refund on day 29. The right refund is $49.99, not $55.99. A prompt might say “refund approved, $49.99” and look perfect. But if its explanation says “since shipping is refundable within 30 days, we’ve included it,” the number is right and the reasoning is wrong. The next customer, with a different order, gets the wrong amount. A single end-to-end read would have waved this through. Separate checks would not.
The tool uses two kinds of checks. Exact assertions or scripts handle the mechanical conditions (the amount, the window). Model-based judges handle the softer stuff (does the explanation actually match the math). Quick caveat from the author, and I like the honesty: your cases and judges are only as good as you write them, so a passing score is not proof the policy is fully covered.
Mini-workflow to try
- 📋 Pick a small policy-based skill, like refunds, returns or discounts. Smaller is better here. Three or four rules is plenty for a first pass.
- 🧪 Write boundary and counterexample cases BEFORE you touch the prompt. Day 30 vs day 31. An order with shipping. An amount that needs rounding, like a total that lands on a half cent. Writing these first keeps you honest, because you can’t unconsciously shape the cases around what the prompt already does well.
- 🔍 Run the cases and save the results as your baseline. Treat this file like a snapshot you can always return to.
- ✏️ Fix only what the failures actually support. Resist the urge to rewrite everything. One failing rounding check means you edit the rounding instruction, nothing else, so you can tell whether that edit was the thing that helped.
- 🛡️ Test again on separate holdout cases, so you’re not just fitting the prompt to the exact examples you already saw. A good holdout set uses different amounts, different dates and a different customer situation, but the same underlying rules.
The repo’s refund example walks through this with real mistakes around eligibility, shipping and rounding, plus before/after results, so you can see the loop end to end.
Pro tips
- Hunt for cases where the output passes an assertion but still breaks the intended rule. That’s where your test is too loose, and the author is asking for feedback on exactly this. Tighten the assertion or add a judge for the reasoning. A quick way to find these: read five passing outputs out loud and ask “would I have approved this if I were the support lead?”
- Keep holdouts truly separate. If you peek at them while editing the prompt, they stop being a test. Store them in their own file and only open it when you’re ready for the final check.
- Live runs use your configured model engine, so they cost normal usage. Start with a handful of cases, then grow the set only when a real failure teaches you something new.
- When a judge disagrees with your own reading of an answer, don’t just trust one or the other. Reread the judge’s instructions, because a fuzzy judge prompt is the most common reason for a confusing score.
Call to action
Grab the repo at https://github.com/DeadHead04/skillEd, point it at one small policy skill, and see which rule your prompt quietly breaks. Then tell the creator which “passes but wrong” cases you found. That’s the feedback they want, and a GitHub star helps too. ⭐
Frequently Asked Questions
Q: How do I catch a wrong eligibility window when the final amount still looks right?
Write a pair of cases for each boundary: one on the last valid day and one a day past it, each with its expected decision. Assert the eligibility decision as its own check instead of relying only on the final amount, so a lucky total can’t hide a wrong window. Include at least one negative case where the math would pass if the window were ignored, because that’s the case that catches the silent failure.
Q: What should I test when rounding and shipping exclusion interact?
Pick inputs where the amount after removing shipping lands exactly on a half cent, then assert the rounding rule your policy names. Check the order of operations too, since rounding before the shipping exclusion can produce a different total than rounding after it. Run the same case several times to confirm the output stays stable.
Q: How much should I trust a model judge?
Commenters agree the judge is the weak point, so don’t treat its verdicts as ground truth. Some builders keep judges for softer behavior only and rely on exact assertions for anything with a clear right answer. Others use judges more broadly but spot-check their verdicts against a handful of human labels first. Either way, keep the rubric narrow and tied to one rule, since a vague rubric can pass the same mistakes you’re trying to catch.
skillEd: comparing policy-prompt revisions against cases instead of judging the wording
by u/MiserableFlatworm337 in PromptEngineering