Yesterday a 3 trillion parameter open weights model dropped, and the benchmark charts look unreal. Step 2 of the testing is where it gets interesting. I just watched a full hands-on breakdown from Matthew Berman, and the creator did what most of us skip: he paid for a Kimi subscription, downloaded the desktop app, and pushed Kimi K3.1 Pro through a long list of hard, real builds.
The result is a model that crushes leaderboards but still needs a few nudges in practice. Here’s what you need to know.
What’s new
Kimi K3.1 Pro is a frontier-class open weights model. That means you can download it, run it on your own servers or US data centers, or just use it through Kimi directly.
The specs, according to the author’s walkthrough of the official blog post:
- 2.8 trillion total parameters, 104 billion active. It’s a “mixture of experts” model, so only part of the network wakes up for each prompt. That keeps it fast and cheaper to run.
- 1 million token context. You can feed it whole codebases or long documents.
- Multimodal. It takes images for sure, and probably audio too.
- Strong claims: full project coding (Unreal, Unity, Godot games), parametric 3D modeling in Blender and CAD, and finance modeling with hallucinations down 65%.
The pricing is where it really stands out:
- Regular speed: $2 per million input tokens, $10 per million output, 20 cents for cached input.
- High-speed: $4 / $20, same cached price, and 2 to 3 times faster.
On Artificial Analysis it scores 55 on the Intelligence Index. That puts it ahead of Fable 5.1, GPT6 Astra Max, and Google’s Argon. Its cost per intelligence task is $2.11, compared to $7.63 for Fable 5.1. The expert called this one of his favorite metrics, because a cheap model that burns 10x the tokens isn’t actually cheap.
The twist
The benchmarks and the hands-on tests tell two different stories.
On paper, Kimi sits right next to Opus 5.5. Terminal Bench: 65.7 vs 66.4. GDPval (real-world knowledge work): 1846 Elo vs 1866. On SWE Marathon it even beats Opus, 51.7 to 46.3. OSWorld computer control comes in at 79.6, right in line with the leaders.
In real builds, the results were all over the place:
- 3D Rubik’s Cube: a clean pass. Open models usually fail this one badly. Colors vanish or rotations break. Kimi built a realistic cube with light reflections, keyboard controls, a timer, and a real solving algorithm (not just reversing moves), plus confetti at the end. The creator called it one of the best he’s seen.
- iPhone Duo in WebGL: pretty, but wrong. It took 28 minutes and 45 seconds, searched the web, and still gave a phone that didn’t look like the real device.
- Bowline knot in Blender: a fail. This new test mixes computer control, web search, and spatial reasoning. The knot shape was close but not actually tied or attached to the harness.
- Knot-tying motion graphic: weak. A 28-second 1080p clip in five and a half minutes, but you couldn’t see where the rope went over or under. GPT 6.1 did the same task, and you could actually learn the knot from it.
- Fall Guys clone (“Bean Blitz”): decent. 48 minutes, 12 racers, obstacles, sound effects. Colors were too pastel and the collisions were shaky.
- Website redesign: okay. It followed the brand guidelines but felt like generic AI design.
- A parody “YTP” video: surprisingly solid. It used Kimi’s own video generation API, Python with NumPy for the soundtrack, and FFmpeg for the edits.
The part that stuck with me most: when asked for a benchmark chart, Kimi made up the numbers. The author asked if they were real, and Kimi admitted they were “invented placeholders.” That’s a big deal if you’re using it for research or reports.
His verdict? “It is a very good model,” but not near Opus 5.5 or GPT 6.1 in daily use. He also hinted it might be a bit benchmaxed, meaning tuned to score well on tests.
Step-by-step mini-workflow for testing Kimi yourself
This is the testing method the creator uses, and I think it’s a smart way to check any new model:
- 🧪 Start with short, simple prompts. He keeps prompts short on purpose. The model has to make its own guesses, so you see its real taste and skill.
- 🎯 Pick tests that combine skills. The knot test needs web search, computer control, and spatial reasoning all at once. One prompt shows you three skills.
- 🔍 Ask “Is that accurate?” Kimi will admit it made something up, but only if you ask. Make it a habit.
- 🔁 Nudge, then nudge again. “List the specific benchmarks.” “Find the actual numbers.” “Now make the image.” Kimi needs more of these follow-ups than top closed models.
- ⚖️ Run the same prompt on a closed model. That side-by-side knot comparison made the gap obvious in seconds.
Pro tips
- Add detail only when you care about the outcome. The expert’s rule: short prompts are fine, but if you want specific colors, animations, level layouts, or a specific tech stack, spell it out.
- Use the reasoning dial. He ran everything on “high” effort. “Max” is available if a task keeps failing.
- Pick high-speed for agent loops. Paying double for 2 to 3 times the speed is a good deal when a single build takes 30 to 48 minutes.
- Play to its strengths. Terminal tasks, long coding runs, computer use, and some 3D work look strong. For anything with numbers or facts, check every figure.
- Watch for the safety warnings. Kimi added “not a substitute for instruction” notes to the climbing content, much like ChatGPT and Claude do.
I love that a model this big is fully open and this cheap. It’s not quite at the frontier yet, but it’s closer than any open model I’ve seen so far, and you can run it yourself.
👉 Check out the full video to see the Rubik’s Cube solve, the side-by-side knot comparison, and all the benchmark charts up close.