Jev doesn’t write anything. It just scores. Every request gets rated on a handful of questions: which route fits, is this irreversible, is it a coding task, how big is the blast radius if it goes wrong. That’s the entire job. A developer named Chestaum built this scoring layer and paired it with a routing system, then shared the whole thing on r/PromptEngineering this week.
Here’s what’s new: most “smart” AI systems call the model for everything, then hope the prompt handles the edge cases. This one flips that. The Jev model scores confidence first. Chestaum’s code reads those scores and picks a branch it already knows how to run, a deterministic handler, a call to the LLM, a check-in with the user, or a full handoff to a human. The LLM only gets involved when the situation actually earns it. That’s a meaningful shift in cost too, since every call you route around a deterministic handler is a call you’re not paying tokens for, and at scale that adds up fast.
Now the twist, and it’s the part that makes this worth stealing for your own projects. High confidence does not mean “go ahead and do it.” It means “route to the code path that’s already been vetted.” That distinction matters. A lot of confidence-gating setups quietly treat a high score as permission. Chestaum’s system treats it as a signpost. The code still owns the decision of what’s safe to execute, the model just tells it where to look. Anything uncertain or high-stakes falls back to the LLM or a human by default, and every single decision gets logged so the thresholds can be recalibrated later instead of guessed at. Think of it like a triage nurse instead of a doctor. The nurse doesn’t diagnose and treat, the nurse decides who sees a doctor right now, who waits, and who gets sent home with instructions. The actual treatment decision still sits with someone qualified to make it.
How the routing actually works
- 🧭 A request comes in and Jev scores it across the four dimensions: route, reversibility, task type, blast radius.
- 🔀 The router code reads those scores against thresholds you set yourself, not fixed defaults.
- ⚙️ Low-risk, high-confidence stuff goes straight to a deterministic handler. No model call needed.
- 🤝 Anything genuinely ambiguous gets bumped to the LLM, or to a human if the stakes are high enough.
- 📝 Every decision, every score, every branch taken gets logged for later review.
One Redditor, u/No_Internet3543, flagged the real risk with this kind of system: gates like this tend to overfit to whatever handful of test prompts the builder used to tune them. Their take was that the logging is what saves it, because you can actually see where the router picked wrong and adjust the thresholds instead of just trusting the gate blindly. That’s the difference between a demo and something you’d run in production. It’s also the difference between a system you can defend to a skeptical teammate and one you’re just hoping holds up. If someone asks “why did it do that,” you want a log line, not a shrug.
Pro tips if you’re building your own version
- Set per-action thresholds instead of one global confidence cutoff. A coding task and a data-deletion task should never share a bar.
- Log the misses, not just the hits. The value here is entirely in seeing where the gate guessed wrong.
- Start conservative. Route more to the human or the LLM than feels necessary at first, then loosen the gate as your logs prove it’s safe.
- Treat “high confidence” as a routing signal, never as authorization. Keep that check in your code, not in the model’s output.
- Run a shadow period before you flip the gate live. Let it score real requests and log what it would have done, without actually routing anything, so you can see how often it would agree with a human before you hand it the wheel.
The repo is open source and still early, Chestaum said as much in the post. The core idea, though, already solves a real problem: most people either trust the LLM with everything or lock it down so hard it’s useless. A confidence gate that only escalates what actually needs escalating splits the difference, and it does it with a paper trail you can actually learn from. It’s a small piece of infrastructure, but it’s the kind that quietly saves you from the worst failure mode in agent design, an agent that’s confidently wrong and nobody notices until it’s expensive.
If you’re running any kind of agent that touches real systems, coding tools, file operations, anything with a blast radius, this pattern is worth testing this week. Go grab the repo, wire up your own thresholds, and watch the logs for a few days before you trust it.
Frequently Asked Questions
Q: How is this different from other LLM routing systems?
Most approaches either always call the LLM or hard-code rules. This one uses a lightweight scorer that evaluates your request first (without executing anything), then your code decides which handler to use: deterministic logic, ask the LLM, prompt the user, or escalate. It’s faster and more intentional about when you actually need the model.
Q: Won’t this overfit to my test prompts like other confidence gates?
That’s what happens to most systems, but the logging here changes everything. By tracking every decision and where the router gets it wrong, you can recalibrate thresholds without redeploying code. You avoid the trap of optimizing for just a handful of test cases.
Q: What does the confidence scorer actually evaluate?
It answers questions like: which route is this, is it irreversible, is it a coding task, and what’s the blast radius? It’s assessing risk and context to help your code pick the right branch, not making the final call itself.
Q: Can I use different thresholds for different actions?
Yes, the system supports per-action thresholds. A read-only query might have a higher threshold for calling the LLM than a destructive operation would, since the stakes are completely different.
I built a confidence-gated router (Jev + LLM) that only calls the LLM when it’s worth it
by u/Chestaum in PromptEngineering