OpenAI just gave the world its first real look at Jalapeño, the custom inference chip it’s building with Broadcom, and the early numbers are strong. At the Hot Chips conference on Tuesday, the company shared benchmark results for the new system, according to TechCrunch AI. Tested on SemiAnalysis’ InferenceX benchmark, Jalapeño posted both more tokens per user and more throughput per kilowatt than today’s leading inference processors.
That’s a big claim, so let me unpack what it actually means.
What OpenAI announced
Jalapeño isn’t a training chip. It’s built for inference, the part where a trained model actually answers your prompts and serves responses to users. Two things matter there: how fast you get a reply (latency) and how much work you can squeeze out of every watt of power (efficiency). OpenAI says Jalapeño wins on both.
“The bottom line is that the results show a very, very significant performance advance over state of the art,” said Richard Ho, OpenAI’s head of hardware, on a press call reported by TechCrunch AI. “Jalapeño can serve more AI work per unit of power, while also returning responses more quickly. It’s very efficient to serve a lot of customers, but it can also be very low latency.”
Here’s the important context. The comparison ran against an Nvidia Blackwell system, the current gold standard. Beating Blackwell on a benchmark is notable. But timing matters.
The catch on timing
Jalapeño won’t ship at scale for a while. Ho estimated deployment at the end of 2026 “in very small volumes,” with meaningful rollout in 2027. By then, Nvidia will have moved on to newer hardware. So the benchmark is a snapshot against today’s competition, not tomorrow’s. That’s worth keeping in mind before anyone calls this a Blackwell killer.
Jalapeño was first announced last October. What stands out is how OpenAI built it: in close collaboration with Broadcom, and with OpenAI’s own models helping in the design process. The company plans to make Jalapeño a multigenerational platform, where models, chips, and memory all get developed together.
Why the full-stack approach matters
That co-design is the real story here. Because OpenAI controls the models and the silicon, it could target specific choke points in inference that generic chips struggle with.
Two bottlenecks in particular:
- Prefill. Before a model generates a single word, it has to process your entire prompt. On long inputs, that setup phase eats time. Jalapeño is designed to cut those delays.
- Communication. Big models spread work across many chips, and shuffling data between them adds lag. OpenAI built Jalapeño to minimize that data movement.
The technical trick centers on the KV cache, the running memory a model keeps while writing a response. “We designed Jalapeño to minimize data movement and communication delays,” OpenAI wrote in a blog post shared by TechCrunch AI. “This means that model state, including the KV cache used while generating a response, can be explicitly placed and kept local while the system activates the right combination of compute, memory, and networking for each inference phase.”
In plain terms: keep the working data close, skip the expensive trips across the network, and light up only the hardware each phase needs.
Why this matters
OpenAI spends enormous sums renting Nvidia capacity to serve ChatGPT and its API. Custom silicon tuned to its own models is a bet on cutting that cost and controlling its own supply. It also puts OpenAI in the same club as Google, Amazon, and Meta, all of which build in-house AI chips.
For practitioners, don’t expect Jalapeño in your stack soon. This is a 2027 story. But the direction is clear: the companies running the biggest models increasingly want their own chips, and inference efficiency is becoming the battleground. Full details are available in the original TechCrunch AI report.