9.8%. That’s the completion rate for the worst model in a new benchmark that tests something overlooked: whether generated UI actually works. The original poster, u/1glasspaani on r/PromptEngineering, built it and called it GenUI Bench. The best model in the same test hit 99.5%. I didn’t expect a spread that wide on a task most of us already assume AI has nailed. We ship AI-built UI to real users pretty casually these days.
What GenUI Bench Actually Tests
The setup is stripped down on purpose. Here’s the shape of it:
- 46 screen briefs, 2 to 18 numbered requirements each, spread across 5 size bands
- One shared 70-component catalog, so every model builds from the same pieces
- 4 generations per brief, same temperature, same token ceiling, so results are actually comparable
- 30 models run through OpenUI’s format, plus a 6-model showdown across three formats: OpenUI Lang, Google’s A2UI, and Vercel’s json-render
- Every run checked by its own format’s SDK, then one shared pass: parse, render a root, resolve references, reach every component
On OpenUI’s own leaderboard, grok-4.6 topped the field at 99.5%, with 183 of 184 runs landing clean. gemini-3.7-flash and claude-sonnet-5 followed close behind at 98.9% and 98.4%. Reasoning effort didn’t behave the way you’d expect either. The same qwen model scored 87.0% on low reasoning, then dropped to 85.8% on medium. More thinking tokens don’t automatically mean a more reliable render.
That’s the whole point of the benchmark: it measures whether the output is structurally sound, not whether it looks good. It doesn’t yet check whether every requirement in the brief actually got satisfied, and the original poster flags that limitation directly.
The mid-pack results are where it gets interesting. Google’s own gemini-3.5-flash-lite landed at 78.3%, well behind Anthropic’s claude-sonnet-5 at 98.4% on the same three formats. deepseek-v4-flash and one of the qwen reasoning settings tied at 85.8%. That alone tells you brand and size don’t predict structural reliability nearly as well as you’d hope. The one true outlier was ling-3.0-tiny, a local model run through llama.cpp, which finished at 9.8% and dragged the whole floor down.
This isn’t the first attempt at scoring generative UI either. The method builds on Mobile Reality’s MDMA benchmark, and the original poster extended that groundwork into three live SDKs instead of one. If you’ve seen MDMA before, GenUI Bench is the version that checks whether real production formats hold up, not just a single reference implementation.
3 Ways to Actually Use This
- 🎯 Pre-flight model picker. Before you wire generative UI into a product or an agent, check the leaderboard logic instead of trusting vibes. A 10-point gap between two “good” models is real money in failed renders.
- 🧪 Reuse the scoring layer. The shared completeness check runs five tests: parses, renders a root, references resolve, every component reachable, props valid. That’s a solid template for anyone building an eval harness on structured LLM output, even outside UI.
- 💸 Question your reasoning-effort defaults. If you’re paying extra for “high” reasoning assuming it buys reliability, check first. The qwen results here show effort and completion rate don’t move in a straight line.
Tips & Pitfalls
- Structural pass doesn’t mean semantic correctness. A UI can validate clean and still miss half the brief’s requirements, since that check isn’t built yet.
- You don’t need API keys to check the numbers yourself. The scorer runs offline against the committed raw outputs, so you can audit before trusting any leaderboard row.
- Every raw generation and every scored verdict lives in the repo. That’s the real value: you can diff your own reruns against the published numbers instead of taking anyone’s word for it.
- Don’t assume more reasoning effort buys a better structural pass rate. This benchmark says check before you scale the spend.
One more thing worth knowing before you rely on this: failed API calls get retried at generation time and never count against a model. Empty responses still score as blanks rather than passes. That distinction matters if you’re comparing a flaky provider against a stable one, since it keeps infrastructure hiccups out of the reliability number.
GenUI Bench is fully open on GitHub, raw outputs and all, so pull it and run your own model through it. New models can be scored with one command, no waiting on a maintainer to add them. The original discussion on r/PromptEngineering has more context straight from the person who built it. Go poke at the raw data yourself, it’s more interesting than another leaderboard screenshot!
Building a benchmark for Realtime UI Generation
by u/1glasspaani in PromptEngineering