83%: Microsoft’s AI Panel Outperforms Doctors in Complex Diagnostics

A multi-agent artificial intelligence system achieved an 83% diagnostic accuracy rate on complex medical cases, a figure that more than doubled the performance of human generalists in a simulation published this week. The results, detailed in a study released in the journal Science, indicate that the technology, known as MAI-DxO, can navigate difficult clinical scenarios with a level of precision that currently exceeds that of individual primary care providers and general internists.

The data reveals a stark contrast in performance metrics. While the AI system correctly identified the diagnosis in 83% of the simulated cases, the human participants, who worked individually with access to reference materials, were correct only 36% of the time. It is important to note that these complex cases are typically managed by specialists or collaborative physician teams in real-world settings, groups that were not included in this specific comparative analysis. The study utilized complex cases derived from the New England Journal of Medicine to test the limits of current diagnostic tools.

MAI-DxO operates by coordinating several AI agents that function similarly to a panel of consulting physicians. These agents interact with the patient, order necessary tests, and engage in a debate regarding the diagnosis before arriving at a conclusion. Microsoft AI CEO Mustafa Suleyman described the findings as a significant step toward what he terms "medical superintelligence." Dominic King, vice president of Health at Microsoft AI, emphasized the significance of the gap in capability, noting that the current healthcare system faces a growing mismatch between the demand for medical expertise and the available supply.

The study indicates that much of the performance improvement is attributable to the underlying power of the AI models used, with Microsoft’s orchestration layer providing additional gains. The system was tested with models from various providers, including OpenAI, Google, Anthropic, Meta, and DeepSeek. The highest results were achieved using OpenAI’s GPT-5 and GPT-5.5 models. However, the cost-benefit analysis shows that the orchestration system reduced the estimated cost per case by approximately $3,000 while adding only about four percentage points to the accuracy score when using GPT-5. For emergency department cases, the improvement in both accuracy and cost efficiency was minimal.

The advantage of the orchestration system becomes more pronounced with less capable models. For instance, Gemini 2.5 Flash scored 46.4% on complex cases on its own, but its performance rose to nearly 69% with the assistance of MAI-DxO. Despite these gains, researchers noted that the gap between weaker and stronger base models remained larger than the boost provided by the orchestration layer. A Microsoft spokesperson explained that the system helps optimize both accuracy and cost, with accuracy gains naturally diminishing as models approach their theoretical maximums.

Suleyman, who leads Microsoft’s frontier AI efforts, stated that the company is collaborating with the Mayo Clinic to train a foundation model specifically for healthcare, with a release scheduled for next year. He believes this infrastructure will enable substantial leaps in medical reasoning. Suleyman, a former co-founder of DeepMind who joined Microsoft in 2024, predicted that by 2027, the industry will see a "coding style breakthrough" in medical records, allowing for highly accurate predictions of patient outcomes. He views this foresight as invaluable for clinicians.

To validate these capabilities in real-world settings, Microsoft has established a partnership with Beth Israel Deaconess Medical Center in Boston. This collaboration will evaluate the clinical reasoning of MAI-DxO across various hospital environments. Patients will receive standard care, including triage and specialist consultation, while those who consent will also interact with the AI system. The recommendations from the AI will be compared against the decisions made by human doctors. King described this as potentially the largest study of its kind, covering both emergency and urgent care settings.

The publication of this study aligns with a broader manifesto from Microsoft outlining the requirements for achieving medical superintelligence, defined as surpassing contemporary human-led health systems in reasoning and coordination. The company aims to move beyond simple benchmarking tasks like answering exam questions and instead integrate with clinicians to improve overall outcomes. Suleyman stated that Microsoft is committed to accelerating the introduction of this technology, viewing it as a key differentiator compared to other frontier labs. The company already sells dictation and radiology tools to clinicians and sees this diagnostic system as a natural expansion of its existing portfolio.