OpenAI’s GPT-5.6 Sol has finished a full game of Factorio on its own. It played on peaceful mode, which removes the hostile creatures, and launched the rocket after 13.5 hours of in-game time. A developer who has spent months testing AI agents on the factory-building game ran the experiment, and the write-up drew a big response on Hacker News. The win is real. How Sol got there tells you more about where AI agents stand today than the win does.
What the researcher did
The setup was simple. Sol ran inside Codex, OpenAI’s coding agent, with its thinking level set to low. It didn’t see the screen the way a human player does. A game manager gave it a small Python library for controlling the game through code.
The library still held Sol to the same rules a human faces in the normal game interface:
- Reach limits: the character can only touch things close to where it stands
- Fog of war: Sol can’t see map areas it hasn’t explored
- No shortcuts: code access didn’t let it skip actions a human would have to perform
This matters because it keeps the test honest. Sol had to plan, explore and build like a player, not command the game engine directly.
How the run unfolded
The researcher had planned to stop after blue science, the mid-game research tier where he expected Sol to hit scaling problems. Sol handled that stage well enough that he let it keep going.
The scaling problems did show up later. They slowed Sol down a lot in the late game. Sol never properly connected some of its production lines, so it filled the gaps by hand, carrying supplies from one part of the factory to another over and over. The researcher said the rocket launch took “significant ‘manual’ effort on Sol’s part” because of those failures.
Here’s how Sol compares with the earlier model:
- GPT-5.5: didn’t use transport belts at all. Result: didn’t finish.
- GPT-5.6 Sol (low thinking): used transport belts, but left some systems unconnected. Result: rocket launched in 13.5 game hours.
That’s a big jump in one model generation. Transport belts are the most basic way to move items in Factorio. A model that skips them can’t build a real factory.
Why Factorio is a useful test
What stands out to me is the researcher’s argument for why Factorio works as a benchmark. He says it’s “analogous to programming” but “much more legible to casual observers.”
I think he’s right. A factory is basically a system design problem. You break a goal into parts, link them together, find the bottlenecks and scale up. Software engineering works the same way. The difference is that anyone can watch a timelapse and see a tangled belt or a starved assembly line. Few people can spot the same problems in a codebase.
Sol’s failures in the late game look a lot like what developers see from coding agents today:
- Strong early progress: quick, capable work on contained, well-defined tasks
- Integration gaps: trouble connecting pieces once the system gets big
- Brute-force patches: repeated manual workarounds instead of fixing the design
If you’ve watched an agent add a fifth special case rather than restructure a function, you’ll recognize the ferrying behavior.
What practitioners can take from this
The experiment gives teams that use AI agents some useful lessons:
- Expect scale to be the weak point. Agents look great on small projects and then struggle as more parts need to fit together. Plan reviews around the integration points.
- Watch for hard-working workarounds. An agent that hits the goal through repeated manual fixes may be hiding a broken design. Check how it got there, not just whether it finished.
- Low thinking settings can go further than you’d expect. Sol did this on low. Try cheaper settings before you assume you need the most expensive one.
- Rules make tests meaningful. Holding the agent to human limits, as this library did, gives a more honest read on what it can do.
Limitations
This was one run on peaceful mode, with one model at one thinking level. That’s a single data point, not a controlled benchmark. Hostile creatures add defense and resource pressure, which would make the game much harder. The researcher hasn’t yet compared Sol against other models under identical conditions.
What’s next
The researcher plans to test other models and thinking levels, try runs with enemies turned on, and publish a commentary video on the full recording. He has run similar experiments before, including getting Claude Code to play Factorio and building a test environment called Faketorio. A head-to-head comparison could become one of the easiest-to-follow ways to see how frontier agents differ. You can find the full timelapse and production charts in the original post linked on Hacker News.