Someone ran a head-to-head coding build-off with twelve AI models, and the results scramble a few assumptions about who’s on top. The experiment, which hit the front page of Hacker News, pitted the big names against open-weights challengers on four small apps, five attempts each. No single model swept the board. That’s the headline.
The crew was deep: GPT-5.6 in three tiers (Sol, Terra, Luna), GPT-5.5, Claude Opus 4.8, Claude Fable 5, Grok 4.5, Meta’s surprise coding model Muse Spark 1.1, plus open-weights entries GLM-5.2, Qwen 3.7 Plus, DeepSeek V4 Pro, and Kimi K2.6 served via Fireworks.
How the test worked
The method was deliberately hands-off. Each model got the same prompt for each app and ran it five times. The reviewers counted how many of the five actually worked, linked their favorite build, and published every raw attempt so readers could run them and judge for themselves.
The authors are blunt that this isn’t science. As they put it on Hacker News, “This isn’t objective… We are not handing down a scientific verdict.” What stands out is the five-attempt design. It exposes something a single run hides: these models swing hard from attempt to attempt. Consistency, not peak performance, becomes the real story.
The apps and the winners
Three tasks are detailed in the writeup:
- Doom-style raycaster maze: a first-person walkable maze with shaded walls and collision.
- 3D Rubik’s Cube: scramble and solve buttons with visible animation.
- Calculator: digits, operators, and correct order of operations.
The raycaster favored GPT. GPT-5.6 Sol and Luna both went 5-for-5, and Grok 4.5 also hit 5/5 at just $0.27 for all five runs. Claude underperformed expectations here. Muse Spark was the wildcard: three of five builds broke, but the working ones rivaled the best.
The Rubik’s Cube flipped the script. GPT stumbled despite its 3D lead in the maze. Claude Fable 5 carried Anthropic with a clean 5-for-5, while Claude Opus 4.8 couldn’t land a single flawless solve. Grok managed 3/5.
The calculator was the great equalizer. Grok 4.5 and Claude Opus 4.8 both went 5-for-5 with clean, simple results, while GPT-5.5 lost points for over-engineering, adding extra buttons and 3D rendering that got cut off.
Cost tells its own story
Price per five runs varied wildly, and cheaper often meant faster:
- Grok 4.5: $0.27 on the maze, strong across all three tasks. Genuinely usable at its price point.
- GPT-5.6 Luna: $0.15 and 23 seconds on the maze with a 5/5 score, though weak on the cube.
- Open-weights: Qwen ran as low as $0.07 and looked great when it worked, but consistency was thin. GLM-5.2 was the cautionary tale, rendering nice detail but going 0/5 on both the maze and the cube.
- Claude Fable 5: the priciest at $2.03 to $2.35 per five runs, but delivered the cleanest cube.
What practitioners can take from this
A few practical reads:
- Match the model to the task. GPT owned 3D rendering in the maze but faltered on the cube. There’s no universal best.
- Run more than once. The swing between attempts was large enough that a single run tells you almost nothing. If you’re shipping AI-generated code, generate several and pick.
- Cheap models are real contenders. Grok 4.5 and GPT-5.6 Luna delivered top results at a fraction of the cost. Don’t assume the priciest tier wins.
- Open-weights are close but streaky. Qwen produced beautiful output on its good runs, but you can’t count on it landing every time.
The big caveat, which the authors flag themselves, is that this is one person’s eyeball judgment on four toy apps. Small sample, subjective scoring, no production complexity. Treat it as a signal, not a leaderboard.
Still, the takeaway holds up: the gap between the frontier models and the cheap or open ones is narrowing fast, and consistency is now the battleground. You can find every raw build and run them yourself at the original source.