A developer just squeezed a 28.9 million parameter language model onto an ESP32-S3, a microcontroller that costs about $8, and got it generating text on the chip itself. According to Hacker News, where the project climbed to 175 points, the model writes each word to a small screen wired to the chip at roughly 9.5 tokens per second, with nothing sent to a server. What stands out here is the scale jump. The last language model people ran on a chip like this held 260 thousand parameters. This one holds about a hundred times more.
The trick that makes it fit
The hard part is memory. The ESP32-S3 gives you just 512KB of fast SRAM. Normally the whole model has to be reachable from there, which is exactly why previous efforts stayed stuck with tiny models.
The workaround is clever: stop putting most of the model in fast memory at all. Most of a language model’s parameters live in an embedding table, which the model reads from rather than computes on. So the builder left that 25 million row table sitting in slow flash storage and pulled only the handful of rows each token needs, about 450 bytes at a time. The small part that does the actual thinking stays in fast memory.
The result is that the big model costs almost nothing to run, because you never load most of it. This idea is called Per-Layer Embeddings, and it comes from Google’s Gemma 3n and Gemma 4 models. The new part is running it on the memory layout of a microcontroller instead of a phone or a GPU. As the author put it, as far as they can tell, nobody had tried it on a chip this small.
The numbers
Here’s how the setup breaks down, per the write-up on Hacker News:
- Parameters: 28.9M total, with 25M stored in a flash lookup table
- Chip: ESP32-S3, about $8, with 512KB SRAM, 8MB PSRAM, 16MB flash
- Speed: about 9.5 tokens/second end to end (9.7 of pure compute)
- Model size: 14.9MB at 4-bit
- Connectivity: none, everything runs on the device
The memory split is the whole story. Fast SRAM holds the thinking core used on every token. Medium PSRAM holds the output head and working memory. Huge, slow flash holds the 25M parameter table, sampled about six rows per token.
What it can and can’t do
Don’t expect a pocket ChatGPT. The model was trained on TinyStories, a dataset of short synthetic stories simple enough that a small model can still learn to write coherently. So it writes short, simple stories and mostly keeps them on track. It will not answer questions, follow instructions, write code, or know facts. That limit comes from the small reasoning core, and the memory trick doesn’t change it.
The author is upfront about this. The interesting result is the architecture, fitting a large model onto a tiny chip, not what a 28.9 million parameter model has to say.
Why it matters
This is significant because it reframes what “too big for the device” means. If the bulk of a model can live in cheap flash and get sampled a few rows at a time, the fast-memory ceiling stops being the hard wall it used to be. That’s a real path toward larger on-device models for the flood of cheap microcontrollers already in the world, all running fully offline with no cloud round trip and no data leaving the chip.
A note for practitioners who want to poke at it: the builder left the messy history in the repo on purpose, including a bug in their own parameter accounting that had inflated an early number, plus the corrected result after the fix. The firmware and flashing steps, the training and quantization code, and the full method with ablations and on-chip measurements are all documented. The project also credits Andrej Karpathy’s llama2.c as the reason many people believe you can train a tiny model and run it in plain C at all.
The takeaway is straightforward. The bottleneck for edge AI was never just raw compute. It was memory layout, and this shows there’s more room to play than most assumed. Full details, ablations, and the on-chip measurements are available at the original source.