The number that stopped me: 1,165. That’s how many input tokens a Codex user added to a run by loading a skill meant to save tokens. Mean input went from 39,223 to 40,388, about 3% more, and mean output (reasoning included) only fell from 268 to roughly 245.
That’s the data from a r/PromptEngineering post by u/Muted-Vermicelli-180. It’s a small experiment, but it shows a trade-off most of us skip over. We tend to judge a tool by what it promises to remove, and we rarely check what it costs to carry.
The setup
The author built three synthetic project histories of about 20k tokens each. Each run asked the model to recover five current-state values from one history. Think of the kind of facts an agent needs mid-project: which branch is active, which test is failing, what the latest config value is. That makes 15 runs in total, all on Codex 0.162 with gpt-6-astra (high).
The configurations were cumulative, so each row builds on the one before it:
- Original request: 49,621 mean input tokens
- Plus shorter rules: 45,023
- Plus an rtk log replacing a saved test log: 39,223
- Plus the Caveman skill: 40,388
- Plus TOMC history selection: 28,800
Every configuration recovered 15/15 state values. That matters, because it means the savings in the other rows did not come at the expense of getting the answers right.
What the numbers say
The skill’s instructions are input. They ride along with every request, whether the reply needs them or not. The skill asks the model to be terse, but the visible replies were already short, so there was almost nothing left to trim. You pay about 1,165 tokens up front and get back about 23 tokens of output. That’s roughly 50 tokens of added input for every token of output saved.
The big wins came from somewhere else. Shorter rules cut about 4,600 tokens. The rtk log swap cut about 5,800 more. Selecting relevant history cut another 11,600 or so. All three shrank what goes in. The skill tried to shrink what comes out. When the task is “read a long history and return five values,” the input side is where the weight sits, so that is where the savings live.
One caveat from the author: they wrote TOMC, the history-selection tool, so read that last row with that in mind. They say so openly in the post.
The follow-up
The author then shortened the skill from 1,166 to 462 cl100k tokens and reran the same three histories on Codex 0.163. Full versus short averaged 40,482 versus 39,786 input tokens. Both recovered 15/15 values. That’s a drop of about 700 tokens from a skill that was cut by about 700 tokens, so the arithmetic lines up. Cached input differed between conditions, so the rerun doesn’t establish a billing saving.
3 practical applications
- 📏 Audit skill size before you install it. Count the tokens in any always-loaded instruction file. If it’s over a thousand, it needs to earn that on every single call. A quick way to do it: paste the file into a tokenizer, note the count, and multiply by how many calls your agent makes per day.
- 🔍 Trim input first. Replace raw logs with condensed output, tighten your rules, and select only the history the task needs. Those moves produced the largest drops here. For example, swapping a full saved test log for a short summary of failures saved more than the whole brevity skill could ever return.
- 🧪 Measure your own baseline. If your replies are already short, a brevity skill has little to cut. Check mean output before you add one. If your average reply is a few hundred tokens, as it was here, the ceiling on savings is small from the start.
Tips and pitfalls
- Cached tokens count in the input figure here. Billing was not measured, so don’t read these numbers as dollar savings. Cached input is usually priced lower, so the real cost gap may be smaller than the raw counts suggest.
- Each stage changed several instructions at once, so you can’t credit any single tool for its step.
- 15/15 recovery on 15 runs is a sanity check, not proof that quality holds in general.
- One fresh answer per history per configuration means noise is possible. Treat the 3% as a direction, not a constant.
- Change one thing at a time when you test your own setup. Otherwise you will end up with the same attribution problem.
Pros and cons of brevity skills
- Pro: they can help when your outputs are long and verbose, such as chatty explanations or repeated recaps after every step.
- Con: they add fixed input cost to every request, whether or not the reply needed trimming.
Do this next
Pull up the largest instruction file your agent loads on every call and count its tokens. Then compare that cost to the output you expect it to save. If the math doesn’t work, shorten it and rerun, which is exactly what the author did.
What’s the biggest always-on instruction block in your setup? Drop the token count in the comments.
Frequently Asked Questions
Q: Why does a brevity skill still add tokens when the replies get shorter?
The skill text goes in as input on every call, so you pay for it before the model writes anything. In this test the skill saved about 23 output tokens but added about 1,165 input tokens. A brevity skill only pays off when normal answers are long enough for the shorter output to outweigh the instructions.
Q: Does trimming the skill fix the overhead?
Mostly, but not completely. The author cut the skill from 1,166 to 462 tokens, and average input fell from 40,482 to 39,786. That is still above the 39,223 from the earlier no-skill run, which used a different Codex version, so treat that comparison as rough.
Q: Does prompt caching make the skill cost disappear?
Caching can make repeated instruction text cheaper, but these runs did not measure billing. Cached input also differed between the conditions, so total token counts alone don’t show what you would pay. Check the cached and uncached numbers in your provider’s usage data on your own workload before you count it as a saving.
Why did a token-saving skill add ~3% to my Codex input?
by u/Muted-Vermicelli-180 in PromptEngineering