TL;DR: A Reddit user turned their worst LLM failures into a short set of plain-language rules you paste into custom instructions. The rules make your assistant flag guesses, stop inventing specifics, and lead with bad news. The trade-off is drier, slower answers.
The problem it targets
AI assistants sound just as confident when they’re wrong as when they’re right. You can’t tell from the tone which answer you got. A made-up statistic and a well-established fact arrive in the same smooth, assured voice, so you end up checking everything or trusting everything. Neither is a good way to work.
u/ChartumX, an everyday heavy user and not a researcher or programmer, got tired of this. Each time an assistant failed them in a way that mattered, they wrote down a rule. The rules that held up stayed. The ones that didn’t were cut. That origin is part of the appeal: these aren’t theory, they’re scar tissue from real use.
The result is the Confidence Compact, published on GitHub under CC BY 4.0, which means you can copy it, adapt it, and share it as long as you credit the author.
What the rules ask for
The post describes five core behaviors:
- Label guesses as guesses, and say plainly when it is confident. A good answer sounds like “I’m sure about the first part, but the second part is my best guess.”
- Never invent specifics like links, prices, names, or version numbers without checking or flagging them. If it can’t confirm a price, it should say so instead of handing you a number that looks plausible.
- Put the unwelcome answer first, instead of reassuring you and saving the bad news for your second question.
- Hold its ground on judgment calls unless you give new information, instead of folding because you pushed back. Pushing back harder is not new information.
- Treat its own claims about its own features as unreliable until checked. Models often describe settings or tools they don’t actually have.
Two versions, plus two shortcuts
The Lite version is about 1,200 characters. That fits most custom-instruction boxes, and it’s the easiest place to start. The Full version has 15 rules, and each one is labeled by how long it has been in use: long-running, newer, or experimental. That labeling helps you decide which rules to trust and which to treat as works in progress.
There are also two optional research shortcuts. “Big look at X” runs broad, multi-angle research and reports what it couldn’t reach. “Little look at X” does close image inspection using code tools, which is handy when you want the model to examine a screenshot or chart instead of glancing at it and summarizing.
The honest costs
The author is upfront about this. Answers get slower and drier. People who tried it called it “less personal, even clinical.” It is built for people who use AI to make decisions, not for casual chat. If you like your assistant chatty and warm, you’ll probably miss that. Nothing in it has been rigorously benchmarked. The author calls it “a well-tested hunch” that builds on existing work like Chain-of-Verification. It is also published as-is, with no promise of maintenance.
What the comments add
The thread raised a few fair points worth keeping in mind:
- “Unwelcome answer first” is hard to enforce. The model doesn’t always know which answer you’ll find unwelcome until it knows your stakes. Telling it what’s riding on the task helps.
- A model can’t truly measure its own confidence. A rule like this counteracts some RLHF habits, but “I’m confident” is still just output text.
- One commenter suggested tagging claims as prompt-provided, externally checked, or inferred. That makes a sourceless confident answer easier to spot.
- Someone asked whether the rules eat context window space. That’s a real cost with the Full version, which is why the Lite version exists.
Use cases
- Budgeting or pricing work, where an invented tax rate or fee can quietly wreck a plan.
- Research, where fake links and version numbers waste your time. One bad citation can cost you twenty minutes of digging.
- Decisions where you need to hear “this won’t work” before “here’s how to try.”
- Any long chat where you push back and the model caves just to be agreeable.
Prompt of the Day
This is my own short sketch in the spirit of the Compact, not the author’s text. Start with the real repo for the tested version.
“Mark each factual claim as confirmed, inferred, or guessed. Never give links, prices, names, or version numbers unless you can verify them. If you can’t, say so. If the answer is bad news, lead with it. If I push back without new information, keep your position and explain why. Don’t trust your own claims about your own features.”
Call to action
Grab the Lite version from github.com/chartummang/confidence-compact, paste it into your custom instructions, and run it for a week on a task where being wrong costs you something. Keep a quick note each time it flags a guess you would have otherwise believed. Then tell the author what held up and what didn’t. They asked for forks and improvements, so take them up on it.
Frequently Asked Questions
Q: Does ‘put the unwelcome answer first’ get in the way of multi-step work?
Commenters didn’t reach a clear consensus, but from what they described, the rule mainly changes how each reply opens, so the steps after it should still flow. The catch is that the model can’t tell which answer is unwelcome until it knows what’s at stake for you. Add a line about your situation, like ‘I’m budgeting a freelance project, so flag anything that changes my numbers,’ so it has something to work from.
Q: If the assistant can’t judge its own confidence, what’s the point of labeling it?
One commenter pointed out that a model’s answers come from patterns in its weights, so its sense of ‘I’m sure’ isn’t a readout of accuracy. The labels still help in a different way. Another commenter suggested tagging each factual claim as prompt-provided, externally checked, or inferred. That won’t make the model more accurate, but it shows you which answers have no source behind them, and those are the ones to verify before you act.
Q: Does the Compact use up room that could go to your own context?
A commenter asked this, and the thread didn’t settle it. The Lite version is about 1,200 characters, which works out to a few hundred tokens, so it takes a small slice of most context windows. The Full version is longer. If space is tight for you, start with Lite and see whether it does the job.
Q: Will this make the assistant less warm?
Probably, and the author says so. Testers described the answers as ‘less personal, even clinical.’ One commenter noted that the Compact pushes back against the agreeable, reassuring habits that come out of training, which is the point. For casual chat that trade may not feel worth it. If you use AI to make decisions, the drier tone may be a fair price for answers that lead with the bad news.
The Confidence Compact: A plain-language ruleset that lets you truly trust your LLM, built from real failures, not theory
by u/ChartumX in PromptEngineering