Ship a tone change to a skill you maintain and you have no real way to know if it helped. Most builders just glance at a couple of outputs and call it a vibe check. u/zszzszzsz maintains a skill called vc-teardown. He ran an actual controlled test instead of a vibe check, and hit two setup mistakes that would have quietly wrecked the whole thing.
Here’s the short version before the details. Claude Skills are keyed by the name field. Upload a new version and it overwrites the old one instead of sitting next to it. Fix that by renaming the copy before you upload it.
Then run each version in its own separate, incognito conversation. A shared chat lets version two read version one’s answers before it even responds, which quietly contaminates every result that follows. Get either of those two things wrong and your “test” just measures whichever version ran last, not which prompt actually works better.
The old way most people test a skill change looks like this: edit the skill, reupload, run a prompt, read the output. Then decide it “feels better.” There’s no comparison. There’s no separate baseline, and usually no separate conversation either.
Run the old version and the new version back to back in the same chat, and the second one already saw the first one’s answer. It’s sitting right there in context. That’s not a test. It’s just an echo of whatever came before it.
The approach the original poster used flips that around completely. Before running anything, he wrote down what he expected to see. On a startup idea with nothing going for it, the two skill versions should return nearly identical, harsh feedback.
On a genuinely strong idea, they should diverge a lot. The whole point of the tone change was to be less brutal when brutal isn’t warranted. That’s a test built so it can actually fail, and that’s exactly what separates a real evaluation from wishful thinking.
🔧 How to run it properly
- Unzip your current skill folder and open SKILL.md.
- Edit the name field so it’s different from the live version, then rezip it and upload it as a second skill. He built a build flag for this exact step. Running “bash scripts/package.sh yesbut” spits out a renamed, version-suffixed bundle straight from the source, without touching the original files.
- With both versions installed, switch between them per conversation from the Skills menu at the bottom of the chat.
- Open a fresh, separate conversation for each version you’re testing. Never run both in one chat, since the model reads everything written above the current message.
- Use Incognito windows for these test chats too, so nothing from an earlier session leaks in.
- Pick two test cases at opposite ends of the quality spectrum, then write your expected result before you run anything.
- Run both versions against both test cases, then compare the actual outputs against what you predicted going in.
The results matched the prediction closely. There was near-zero change on the weak idea. On the strong one, “concession-led” openers jumped from roughly 0-of-12 to 7-of-13. The softer version actually started conceding good points instead of attacking on reflex every time.
⚠️ One warning worth remembering
This setup still missed a bug. The author changed how challenges open but forgot the summary line. It still led with “dead on arrival,” the exact phrase that started the whole rewrite in the first place.
His A/B test only measured openers, so the leftover phrase sailed straight through untouched. A commenter in the thread made a related point about conversations. People treat a chat like a notebook and forget the model is reading everything typed above it. That’s basically the same failure mode as the missed grep, just one layer up.
The fix for that last one is simple, and easy to skip anyway. When you change a prompt’s tone, grep the whole file for the exact phrase you’re trying to remove. Don’t just check the section you edited, because old phrasing has a habit of surviving in the one paragraph nobody thought to reread. A/B testing catches drift in the part you changed. It says nothing about the parts you forgot to touch.
If you’re maintaining any skill or reusable prompt, borrow this shape for your next change. Rename before you duplicate. Test in separate incognito chats, and write your expected result down before you look at a single output. Check out the original poster’s full breakdown and the vc-teardown project itself for the complete numbers and every logged output from all four test runs.
Frequently Asked Questions
Q: Why can’t I test two skill versions in the same conversation?
When you run both versions in one chat, the second version reads the first’s output and bases its response on it , they’re no longer independent. Run each version in a separate conversation (ideally in Incognito mode) to keep them truly isolated and avoid silently invalidating your results.
Q: What if I change the skill’s tone but miss updating other parts?
This happens more often than you’d think , you might update how the skill challenges ideas, but forget the summary line or outro still uses the old phrasing, undercutting the whole effect. After making changes, grep your entire skill for the old wording and update every instance, not just the main section. A single overlooked phrase can make an A/B test pass when it should fail.
Q: How do I install two versions of the same skill if keys are based on the name field?
Rename one version first by unzipping, editing the `name:` field in SKILL.md, and reziping before upload. Many maintainers use a build flag (like `bash scripts/package.sh yesbut`) to auto-generate a version-suffixed copy without touching the source. Once both are installed, toggle between them per conversation from the Skills menu.
Q: How do I design a test that actually catches regressions?
Pick test cases where you expect very different outputs (a genuinely good idea) and nearly identical outputs (a clearly bad idea), then write down your hypothesis first. If the softer version comes back warmer on a bad idea, that’s your signal the change is causing problems , the test designed this way can actually fail.
How to actually A/B test a Claude Skill change (two gotchas that will silently ruin it)
by u/zszzszzsz in PromptEngineering