Sixty-six out of sixty-six on paper. Nineteen out of sixty-six once real models touched the pipeline. That’s the gap this Reddit user just published on r/PromptEngineering, and the breakdown is worth studying before you ship your next AI agent.
The creator built a deterministic verification engine for an AI financial system. It’s the part of the pipeline that checks claims against evidence and decides pass or fail. Tested against clean, structured “canonical” inputs, the engine hit a perfect score: 66 out of 66. Every single benchmark case passed, with zero exceptions.
Then the author ran the full pipeline end to end. Instead of feeding the verifier pre-cleaned data, a live model generated the inputs itself. The score dropped to 19 out of 66. Same verifier, same rules, completely different outcome.
That’s a 47-point collapse between “logic tested in isolation” and “logic tested in the real world.” It’s the exact gap most AI builders never bother to measure, because it’s easier to publish one clean number and move on. Most demos only show the 66 out of 66 part. This author showed both halves of the story.
What’s Actually Happening
This isn’t a story about a broken verifier. The deterministic logic is rock solid. It caught every case when the inputs were already well-formed and structured the way it expected.
The real story is that most of the failure happens before verification even starts. Somewhere between “the model generates an answer” and “the verifier checks it,” things go sideways: transport hiccups, malformed parsing, schema mismatches, normalization errors.
The author is now restructuring the benchmark to isolate the “first invalid state.” That’s the exact stage where a given case goes wrong, instead of just reporting a single pass or fail number for the whole pipeline. That change turns a vague verdict into something specific.
“The model is bad” becomes “parsing broke on step three.” Or “schema validation choked on step five.” Each failure gets a location, not just a score.
One commenter, u/dopey_sang, summed it up well. The pipeline builder now “knows where the bodies are buried.” Splitting failures by their first invalid state is “the move,” the commenter added. Most of these systems fail in a parsing step long before the model ever deserves the blame.
Three Ways to Use This
- 🔍 Separate your verifier’s accuracy from your pipeline’s reliability. Benchmark your deterministic logic on canonical, hand-built inputs first. Only add a live model into the loop once that core logic already scores near perfect.
- 🧩 Add stage-level tracing to your own agent pipeline. Break it into named stages: transport, parsing, schema validation, normalization, claim binding, evidence graph construction, verification, final output. Log exactly which stage a case dies at, every single time it fails.
- 🎯 Stop blaming “the model” by default. If a case fails, check whether it broke in parsing or normalization first. Only blame the generation step once you’ve ruled out everything upstream of it.
Tips and Pitfalls
- Tip: Build your canonical, structured test set before you touch a live model. If your core logic can’t pass clean inputs, no amount of prompt tuning downstream will save it.
- Tip: Track pass rate by stage, not just by final outcome. A single number like “19/66” hides exactly where the real problem lives.
- Tip: Publish both numbers, the isolated logic score and the end-to-end score. The gap between them tells a more honest story than either number alone.
- Pitfall: Don’t assume a low end-to-end score means your core logic is flawed. It might mean nothing more than a fragile parsing step upstream of a perfectly good verifier.
- Pitfall: Testing only on canonical inputs and calling the project done. Real-world inputs generated by a live model will always be messier than anything in your test set.
- Pitfall: Treating “first invalid state” as a nice-to-have metric. It’s the single fastest way to find out whether you have a model problem or a plumbing problem.
Worth Following
This project is still mid-benchmark, and that’s part of what makes it worth watching. The next version promises full stage-level attribution across the whole pipeline instead of one aggregate score.
That kind of rigor is rare on this subreddit, where most tool posts stop at “look what I built.” This one keeps going into “here’s exactly where it breaks, and here’s how I’m proving it.” That’s the habit worth stealing for your own agent projects.
Head over to the original thread on r/PromptEngineering to see the full breakdown and follow along as the next benchmark version drops.
Frequently Asked Questions
Q: Why the huge gap between canonical (66/66) and live (19/66)?
Canonical tests run on perfect, hand-crafted inputs. Live evaluation includes all the real-world chaos: data transport hiccups, parsing edge cases, schema mismatches. The comment nails it: half the time these pipelines fail in parsing and the model takes the blame. The new benchmark splits failures by stage, so you’ll know exactly where the 47 failures live and whether it’s infrastructure or model.
Q: How does splitting by “first invalid state” actually help?
Right now you know a test failed. With stage attribution, you see which step broke it first: transport, parsing, validation, normalization, or verification. That tells you whether the failure is upstream infrastructure (fixable) or downstream model behavior (different kind of fix). Way more actionable than “test failed, investigate everything.”
Q: Is 29% end-to-end success a red flag?
Not necessarily. The deterministic verifier itself is bulletproof (66/66). The gap comes from upstream infrastructure and live model behavior, not the verification engine. Once you have stage-level data, you can identify which fixes matter most: input robustness, parsing, or model tuning.
Our deterministic verification engine passed 66/66 benchmark cases on canonical structured inputs.
by u/MuhammadMujtaba21 in PromptEngineering