AI stopped being one big compute problem. It quietly became a memory and storage problem, and most data centers weren’t built for it. That’s the argument running through a new MIT Tech Review piece on architecting infrastructure for the AI era, and it’s backed by Jim McGregor, founder and principal analyst at Tirias Research.
His framing is blunt. “We tend to think of AI as a single workload, and it’s not. It’s thousands, it’s millions, it’s billions of different workloads,” McGregor says. Once you accept that, the whole optimization game changes.
What stands out here is the shift from training to inference. Training was a sprint. You threw massive compute at a model, finished, and moved on. Inference is the opposite. It runs constantly, in real time, across distributed systems, and it hammers infrastructure in ways older enterprise setups never planned for.
McGregor puts it plainly. “Data centers must now support continuous, distributed, and increasingly real-time AI services, none of which are a single workload.” Each one has its own demands for latency, data movement, and utilization.
That’s why memory and storage move to the center of the story. For years they were supporting cast, the stuff you bought after picking your chips. MIT Tech Review reports that inference flips this. Sustained inference means constant data retrieval and caching, so the bottleneck often isn’t raw compute. It’s how fast you can move and serve data.
This is significant because it kills a comfortable assumption. Enterprises used to treat performance as the only scoreboard. Now the scoreboard has four columns: performance, efficiency, cost, and scalability. Overbuild for peak demand and you burn money. Underbuild and growth stalls the moment traffic spikes.
The winners, per the analysis, will be the organizations that squeeze more performance per watt, shrink their environmental footprint, and clear memory and storage bottlenecks before those limits bite.
🔮 Why this matters now
Three forces are converging in 2026. Agentic AI is going into production, not just demos. Inference costs are becoming the real budget line, not training. And power and cooling constraints are turning “performance per watt” from a nice metric into a hard ceiling.
McGregor’s advice is refreshingly specific. “You have to optimize the entire network, and that includes memory and storage, around the types of workloads you plan on running,” he says. “You have to really have a detailed understanding of what those workloads are going to be.”
Translation: you can’t buy generic infrastructure and hope. You have to know your workloads first.
🛠️ What practitioners and businesses should do
- Map your real workloads. Chatbots, RAG pipelines, and autonomous agents have wildly different memory and data-movement profiles. Audit before you buy.
- Treat memory and storage as first-class design decisions, not afterthoughts. Build a pipeline that can ingest, clean, move, and serve data fast.
- Add efficiency to your benchmarks. Track performance per watt and cost per inference alongside raw speed.
- Don’t overbuild for peak. Architect for flexibility so you can scale without paying for capacity you rarely touch.
- Stop shoehorning AI into legacy systems. McGregor’s point is that purpose-built beats retrofit, and the gap widens as workloads grow.
🚀 Where this goes next
Expect the next two years to reward companies that rethink infrastructure from the data up. As agents get more autonomous and inference runs nonstop, the memory and storage layer becomes the quiet decider of who scales cleanly and who hits a wall. The compute arms race grabbed the headlines. The next edge is architecture.
You can find the full breakdown in MIT Tech Review’s report on architecting memory and storage for AI.