AMD’s MI355X just delivered better performance per dollar than Nvidia’s B300 while running Kimi K3, the new 2.8-trillion-parameter open source model, according to a technical writeup on Hacker News from the team at Wafer. The headline number: 48 tokens per second per dollar on the MI355X, versus 33 on the B300. That’s the kind of gap that makes infrastructure teams rethink their GPU orders.
What stands out here is why AMD won. It wasn’t raw speed. It was memory capacity paired with a much lower price.
Why Kimi K3 broke the usual setup
Kimi K3 is huge. At 2.8T parameters, it needs over 1.5TB of VRAM before you even allocate a KV cache for 1M tokens of context. As Hacker News reports, that size breaks the standard playbook. A full B200 node of 8 GPUs can’t even fit the model.
That leaves two options:
- Serve on B300s, which pack 288GB of VRAM per GPU.
- Chain two B200 nodes together (TP16) just to hold the weights.
AMD’s MI355X also carries 288GB per GPU. And it runs roughly 2.4x cheaper than a B300 and 1.7x cheaper than a B200. Same memory headroom, a fraction of the price.
The benchmark numbers
The team tested on a 1,024-token input, 400-token output workload. Here’s how the three setups compared:
- 8x MI355X (TP8): 118 tok/s per stream, 952 tok/s peak aggregate, 48 tok/s/$
- 2×8 B200 (TP16): 90 tok/s per stream, 498 tok/s peak aggregate, 7 tok/s/$
- B300 (TP8): 172 tok/s per stream, 1,568 tok/s peak aggregate, 33 tok/s/$
Pricing assumptions were $2.50/GPU-hr for the MI355X, $6.00 for the B300, and $4.25 for the B200.
Read that carefully. The B300 still wins on raw throughput, about 1.65x more aggregate tokens. But at 2.4x the cost, it loses badly on value. The MI355X crushed the two-node B200 config on nearly every axis, partly because the B200 setup pays a cross-node all-reduce penalty on the decode path.
The engineering catch
The well-known knock on AMD is software. Slower kernels, less day-zero support. This case was easier because AMD shipped day-zero support for Kimi K3, so the model ran out of the box. But squeezing out top performance still took real work.
Two fixes mattered most:
- Speculative decode. The ROCm build hit a crash because a top-k renorm function was simply undefined. The team’s reflex was to assume they needed a custom kernel. They didn’t. It was a missing definition, and the fix was a single small PyTorch function (sort, mask, divide). That change delivered roughly 2.2x faster single-stream performance and pushed peak throughput to much higher concurrency.
- Prefill speed. This is the honest part of the report. The MI355X struggled on time-to-first-token, the delay users actually feel. A 172k-token cold prefill took about 51 seconds on the MI355X versus 23 on the B300. The cause was a shape mismatch that forced a slow fallback kernel. Padding the attention head count from 12 to 16 let the fast AITER kernel load, cutting prefill time by 2 to 3x.
Why it matters for practitioners
This is significant because it’s the first clear case where AMD’s bet on HBM capacity translates into a real, measured edge. If you’re serving frontier-scale open models, the lesson is direct:
- Memory capacity, not just FLOPs, decides whether a model fits at all.
- Cheaper GPUs with comparable VRAM can win on cost even when they lose on peak speed.
- Many ROCm gaps are missing definitions or shape mismatches, not missing kernels. Check before you write custom code.
The caveat is worth repeating: the team was honest that time-to-first-token remains a weak spot on AMD, and heavy cold-prefill workloads can still stall a fleet if you don’t tune for it.
The broader signal is that the AMD software gap is closing, and agent-driven kernel optimization may close it faster still. For teams tired of paying Nvidia premiums, that’s worth watching closely. Full technical detail is available at the original source.