Microsoft’s AI diagnostic system got complex medical cases right 83% of the time. Generalist physicians got 36%. The results come from a study published Thursday in Science, according to TechCrunch AI. Microsoft AI CEO Mustafa Suleyman calls it a step toward “medical superintelligence.”
That headline number is striking, but the details matter more. The model underneath does most of the work, and the extra layer Microsoft built on top helps a lot in some cases and barely at all in others.
🩺 How the System Works
The system is called MAI-DxO, or DxO for short. It’s an orchestration layer: several AI agents work together like a panel of doctors. They question a simulated patient, order tests, and argue their way to a diagnosis.
The researchers tested it on complex cases from the New England Journal of Medicine. They compared it with primary care doctors and general internists, who worked alone but could use reference materials.
One caveat stands out. Cases this hard usually go to specialists or teams of physicians, and neither group was part of the comparison. So this was AI against generalists on specialist-level problems.
📊 The Numbers Behind the Headline
Microsoft ran DxO on models from OpenAI, Google, Anthropic, Meta, and DeepSeek. It also measured the estimated cost of running the AI on each case, not just accuracy. Here’s how the main results compare:
| Model | Case type | On its own | With DxO |
|---|---|---|---|
| GPT-5 | Complex | 78.9% (~$8,400/case) | ~83% (~$3,000 cheaper) |
| GPT-5.5 | Complex | 72% | Nearly 87% (slightly cheaper) |
| GPT-5.5 | Emergency dept. | ~85% | ~85% (similar cost) |
| Gemini 2.5 Flash | Complex | 46.4% | Nearly 69% |
On emergency department cases, DxO added little to GPT-5 or GPT-5.5 on either accuracy or cost.
🔍 What the Data Actually Says
What stands out here is how honest the study’s own data is. Most of the improvement comes from stronger underlying models. DxO adds gains on top, but the gap between weak and strong models was bigger than any boost the orchestration provided.
A few patterns stand out:
- Orchestration helps weaker models most. Gemini 2.5 Flash jumped about 22 points with DxO.
- Complex cases benefit, routine ones don’t. On emergency cases the boost mostly disappeared.
- Cost matters as much as accuracy. With GPT-5, DxO’s biggest win was cutting the cost by about $3,000 per case. Accuracy rose only about four points.
The researchers note that gains naturally shrink as models get close to the theoretical maximum. A Microsoft spokesperson framed DxO as a tool for “optimization both accuracy and cost.”
💡 Why Practitioners Should Care
This study is useful well beyond healthcare. If you’re building agent systems, here’s what to take from it:
- Upgrade the model first. A better base model beat clever orchestration across the board.
- Use multi-agent setups where the problem is actually hard. On simpler tasks you mostly add complexity without gaining accuracy.
- Track cost per task, not just accuracy. Orchestration can pay off by making the work cheaper, even when accuracy barely moves.
- Small models plus structure can close part of the gap. That’s handy when budget or latency rules out frontier models.
🏥 From Simulation to the ER
Microsoft now joins Google and OpenAI in claiming AI that matches or beats doctors. The next test is real patients.
Through a new partnership with Beth Israel Deaconess Medical Center in Boston, patients will get standard care as usual. Those who consent will also interact with DxO, and researchers will compare its recommendations with what the human doctors decided. “This technology is ready for real-world evaluations,” said Dominic King, VP of Health at Microsoft AI. He expects it to be “the biggest study of its kind,” covering both emergency and urgent care.
Google recently published a similar Beth Israel study in The Lancet. It tested its AMIE diagnostic AI in real urgent care visits while physicians watched in real time.
🔮 What Comes Next
Suleyman says Microsoft is working with the Mayo Clinic on a healthcare foundation model, with release planned for next year. He expects healthcare models to get “an order of magnitude better in capability within the next 24 to 36 months.” He’s betting on a flywheel: put a model into production, find its gaps through real use, and fix them, the same way coding AI took off.
The Beth Israel trial will show whether simulation results hold up in a busy Boston ER on a Friday night. The full story is available at TechCrunch AI.