63,000 Calls Later, “You Are an Expert” Still Doesn’t Make Models Smarter

63,000 calls. 7 models from 2026. $10.11 total. And the result for the classic “You are an expert…” opener? Basically nothing.

Reddit user u/mrtac96 is running a series where they take a prompting question, find the paper that tested it, and re-run the test on current models. This round re-tests a 2024 EMNLP paper (Zheng et al., “When ‘A Helpful Assistant’ Is Not Really Helpful”) on MMLU questions, with reasoning on and off. The hope was that newer models would use a role better than the 2024 ones. They don’t.

The setup is worth a second look. MMLU is a multiple-choice benchmark spanning dozens of subjects, so every question has one clean right answer and scoring is automatic. Running each condition with reasoning on and with reasoning off also checks whether extra thinking time soaks up the effect of a persona. Neither setting showed a reliable lift. With 63,000 calls spread across seven models, small quirks in any single question wash out, which is why the flat result is believable.

What the numbers say

  • A matching expert role (“You are a lawyer” on a law question) moved accuracy by -1.0 to +1.0 points. Asking the same questions twice with no role at all moved it by about the same amount. That’s noise, not signal. Models are not perfectly deterministic, so two identical runs rarely score exactly the same, and that gap is your yardstick for everything else.
  • “You are a helpful assistant” never lifted accuracy by more than 0.3 points. On Claude Haiku 5.5 with reasoning off, it actually dropped accuracy by 1.9 points versus no role. That is a reminder that even the most common default line can cost you something on smaller, faster models.
  • A silly role like “You are a 4-year-old toddler” mostly broke the answer format, not the model’s knowledge. With reasoning on, it barely mattered, because the model had room to think the question through before answering in character.

So the model still knows the law. Telling it to be a lawyer doesn’t unlock extra knowledge. It just changes how the answer sounds. The knowledge was already in the weights, and a job title in the first line doesn’t move it.

3 practical applications

🏴‍☠️ Use roles for voice, not for brains. Want a friendly support reply, a blunt code review, or a bedtime-story tone? A role is great for that. Try “You are a support agent who writes short, warm replies and never uses jargon” and you will see the style change immediately. Want a more accurate answer? Spend your words elsewhere.

⚓ Spend the saved tokens on real context. Give the model the facts, the constraints, the format you want, and an example of a good answer. For a contract question, paste the relevant clause, name the jurisdiction, and say what the reader needs to decide. That beats “You are a senior attorney” every time. One of the commenters (u/bytezvex) made the same point from client work: once the model has enough context, the role line mostly changes the vibe.

🧭 Run your own A/B test. If a role line lives in your production prompt, test it against a version without it. Use at least 20 to 50 real examples, score them the same way both times, and compare. If the accuracy gap sits inside the run-to-run noise, delete the line and enjoy a shorter prompt. Shorter prompts also cost less and are easier to maintain.

Tips and pitfalls

  • Pro: roles are cheap, easy to write, and good for tone consistency across a team or a product.
  • Con: they can give you false confidence. A “world-class expert” persona doesn’t make the answer more correct, it just makes it sound more sure. Confident and wrong is the worst combination to ship.
  • Watch the format. The toddler example shows that odd personas can break your output structure. If you parse the answer with code, keep personas boring, and ask for a fixed format like “Answer: B” at the end.
  • Measure against noise. Run the same prompt twice with no changes first. If your “improvement” is smaller than that gap, you haven’t found anything.
  • Fair caveat from the comments: role-based loops, like having a “web designer” agent critique your site, may behave differently. This test measured multiple-choice accuracy, not critique style, so that question is still open.

One thing to try today

Take your longest prompt, cut the first line that says “You are an expert…”, and run it on 20 real examples. If nothing gets worse, you just saved yourself a line forever. If something does get worse, keep the line and note which kind of task it helped. That is a real finding, and far more useful than a hunch.

The author’s full write-up and code are linked in the original Reddit post on r/PromptEngineering, and next up is a needle-in-a-haystack test at 32K tokens. Got a prompting question you’ve always wondered about? Drop it in the thread and they’ll hunt for a paper to re-test.

Frequently Asked Questions

Q: Does a role still matter when the model is judging something in a loop, like a web designer critiquing a site?

One commenter wondered this, and the test here doesn’t fully answer it. It measured accuracy on multiple choice questions, not the quality of a design critique. If you run a critic loop, a role can set a useful lens, but the bigger gain usually comes from listing the exact checks you want, like contrast, mobile layout, and button labels, and asking for each issue with a severity rating.

Q: If the role doesn’t do much, where should my prompt effort go?

A commenter who works with client workflows says prompt structure and examples matter far more than the opening line. Spell out the task, the audience, and the output format, and put hard constraints in one clear place. One or two worked examples of a good answer often do more than any persona.

Q: Did anyone test this before the 2026 models?

The original paper this post re-tests (Zheng et al., EMNLP 2024) came out of that early period, and a commenter recalled a similar study from around the same time. Its finding, that a “helpful assistant” line doesn’t reliably improve answers, held up on the newer models here. That makes the older research a solid guide for rewriting your prompts.

I always started prompts with “You are an expert…”. I tested it on 7 models from 2026 with 63,000 calls. It didn’t help.
by u/mrtac96 in PromptEngineering

Scroll to Top