One developer just admitted the quiet part out loud: nobody has a clean way to compare AI models yet. Every team building with LLMs eventually hits the same wall, and this Redditor decided to actually do something about it. Posting as u/Spare_Strike9940 in r/PromptEngineering, the creator is building an AI agent testing tool that lets developers test and compare their models side by side. The ask is simple: real testers, pure feedback, no fluff.
That’s a smart move. Shipping something half-finished to real users beats polishing alone in a vacuum for six more months. The post reads like someone who wants the ugly truth before the pretty launch.
Worth noting upfront: the post doesn’t name the tool, share a link, or list which models it currently supports. That’s fine at this stage. Early builds looking for testers usually trade polish for speed, and the real substance shows up once people start poking at it and reporting back.
If you’ve used eval frameworks like promptfoo or LangSmith before, this sounds like it’s chasing the same problem from a lighter, more developer-friendly angle. Those tools are powerful but come with real setup overhead: config files, test suites, sometimes a whole eval pipeline before you see a single result. A simpler testing agent built for quick comparisons, not full production monitoring, fills a real gap for developers who just want a fast answer.
Here’s the twist: the hardest part of this project isn’t the tool itself. It’s deciding what “better” even means.
One commenter, u/Illustrious-Arm8035, asked the question that actually matters. Are you tracking output quality, or also tracking latency? That single question exposes the real problem behind every “compare your models” idea. Quality is subjective. Latency is a number. Cost per token is another number, and consistency across runs is a fourth axis nobody talks about until production breaks. Building a comparison tool means choosing which of these to measure first, and that choice shapes everything downstream.
Why this matters if you build with AI
Run the same prompt through two models and eyeball the difference, and you already know how messy this gets. You end up with ten open tabs, a spreadsheet nobody updates, and a gut feeling instead of a metric. A dedicated testing agent that tracks this stuff automatically would save real hours every week.
This is exactly why the Reddit comment matters more than the original post. A tool that only checks “did the answer look right” misses half the picture. Teams shipping AI features in production care just as much about how fast a model responds and what it costs per call. A testing agent that ignores those numbers gives you a partial answer dressed up as a full one.
A mini-workflow you can run today
You don’t need to wait for a finished tool to start comparing models the right way. Here’s a lightweight version you can run this week:
- 🎯 Pick one real prompt from your product, not a toy example
- 🔁 Run it through every model you’re considering, same settings each time
- Log latency and token cost for each response, side by side
- 📝 Score output quality on a simple 1-5 rubric, not a gut feeling
- 📊 Repeat three times per model to catch inconsistency, not just one lucky run
That gets you most of the value of a dedicated tool with a spreadsheet and twenty minutes.
Pro tips
Decide your metrics before you run a single test. Skip that step and you’ll just pick the model that sounds best in the moment. That’s the exact bias a real testing process should remove.
Keep subjective and objective scores in separate columns. Mixing “this reads better” with “this took 400ms longer” hides tradeoffs you actually need to see.
Track cost per successful output, not cost per call. A cheaper model that needs three retries to land a usable answer isn’t actually cheaper.
Save every run, even the bad ones. The failures teach you more about a model’s weak spots than the wins do. You’ll want that history the next time a provider ships a “better” version.
Where this goes next
The tool is still early, and that timing works in your favor. Beta testers get a say in what gets built, not just a finished product to accept or reject. If model comparison has been a recurring headache on your team, this one is worth a look before it locks into a direction.
Head over to the discussion in r/PromptEngineering and drop your take on which metrics actually matter for testing AI models. 🚀
AI Agent Testing Tool
by u/Spare_Strike9940 in PromptEngineering