Microsoft’s AI Compared With Doctors: A Guide To The Paper

🔍 Read the full analysis: Microsoft’s AI Compared With Doctors: A Guide To The Paper on ThorstenMeyerAI.com

Age 18–24?Offer from Amazon

Prime made for students and young adults

• Fast, free delivery for dorm and study essentials
• Prime Video and Amazon Music included
• Member-only deals

Try Prime for Young Adults →Free trial for eligible 18–24 year olds

As an affiliate, we earn on qualifying purchases.Hide

Microsoft reported that its MAI-DxO diagnostic system reached 83.2% accuracy on 304 difficult medical case records, compared with 36.1% for generalist physicians using textbooks and the internet, in a paper published October 22 in Science. The peer-reviewed results describe performance on a historical, text-only benchmark, not a trial diagnosing patients in clinical care, so they do not establish that the system is safer or more effective than doctors treating patients.

The paper, titled Sequential diagnosis with language models, evaluates diagnosis as a series of decisions rather than a single multiple-choice answer. Each case begins with a short summary. The system or physician must request additional information, such as a question or test result, from a simulated case file. The benchmark scores both the final diagnosis and the tests used to reach it. Microsoft calls this the Sequential Diagnosis Benchmark.

On a separate set of 325 emergency department cases, Microsoft reported accuracy of 80.3% for MAI-DxO and 46.8% for physicians. The physician comparison was with generalists working individually, rather than specialist teams. The case records were historical and presented as text; participants did not examine patients or use real-time imaging. The results therefore measure performance in a constructed testing environment, not routine clinical outcomes.

The system coordinates a language model through several rounds of proposed diagnoses, information requests and internal critique. Microsoft’s results also show that the underlying model accounts for much of the score: GPT-5 alone scored 78.9% on the complex cases, according to the source material. MAI-DxO’s orchestration added roughly four percentage points, while Microsoft reported lower estimated testing costs. Those cost figures rely on modeled prices, not observed hospital spending.

What the Scores Can—and Cannot—Show

The findings matter because diagnosis often depends on choosing what to ask and test next, not simply recognizing a condition from a complete list of facts. A benchmark that makes each request a decision offers a different test of medical AI from an exam-style prompt with the full case already supplied. It can help researchers examine whether systems gather information efficiently as well as whether they reach the recorded diagnosis.

But the reported accuracy gap should not be read as evidence that MAI-DxO can replace doctors. The comparison was between a coordinated AI system and individual generalists, while difficult cases in practice may be referred to specialists or discussed by teams. The benchmark also focused on cases selected for complexity. It did not include healthy people or benign explanations, according to the source material, limiting what the study can say about false alarms in a broader patient population.

For hospitals, insurers and patients, the distinction is practical: high performance on curated records does not by itself establish patient benefit, safety, affordability or suitability for a particular workflow. The paper provides evidence about a test setting. Whether a system improves decisions or outcomes when used alongside clinicians remains an open question.

How Microsoft Tested the System

The work builds on an earlier version posted to arXiv in June 2025, before peer review. That preprint used OpenAI’s o3 model and reported an 80% score for the AI against 20% for doctors on its comparison. The later Science paper is the peer-reviewed publication and reports different figures and a broader comparison that includes physicians with reference materials. The source material says the full journal text is paywalled, so some methodological details available here come from the earlier preprint and may not be identical to the final paper.

In the preprint’s physician sample, 21 U.S. and UK generalists took part: 17 in primary care and four hospital generalists. They had a median of 12 years’ experience. They were tested on a subset of 56 held-out cases, while the AI was evaluated on 304. The final paper reports physician averages of 22.3% without references and 36.1% with textbooks and internet access on the complex-case comparison.

The benchmark also uses models and estimates that affect interpretation. A separate model grades diagnoses against the published answer; physicians reviewing a sample agreed with those judgments well, but not perfectly. If a requested result was absent from the record, a gatekeeper model generated a synthetic result. Test costs were estimated using billing codes and a 2023 U.S. hospital price list, alongside a fixed physician-visit charge. The authors described the cost estimates as first-order approximations.

“The system was reported as 1.7 times as accurate as physicians on emergency department cases, more than twice as accurate on complex cases against physicians using reference materials, and nearly four times as accurate against physicians without them.”

— Microsoft, in its blog post, “Building Medical Superintelligence”

Questions Beyond the Benchmark

The study does not show how MAI-DxO performs with patients who have common, mild or no serious conditions, or how often it would raise unnecessary alarms. It also does not compare the system with specialist physicians or multidisciplinary teams. The results cannot establish whether clinicians would accept its recommendations, whether its test requests would fit local practice, or whether using it changes treatment or patient outcomes.

Microsoft’s figures are the company’s results, even though the paper has been peer-reviewed. Peer review evaluates the study’s methods and reporting; it does not mean independent groups have reproduced the findings. The source material also notes that the complete Science article is paywalled, leaving some details here based on the earlier preprint. Independent replication and the prospective study will be needed to address questions that a historical-record benchmark cannot answer.

Testing MAI-DxO Alongside Care

A prospective study is planned at Beth Israel Deaconess Medical Center in Boston. According to the source material, consenting patients will interact with the system alongside their normal care. That setting could provide evidence about how the tool performs with real patients and how it fits into clinical workflows, though the available information does not specify the study’s size, timeline, endpoints or whether clinicians will see the system’s suggestions while making decisions.

Until those details and results are available, the firm conclusion is limited to the published benchmark: MAI-DxO scored higher than the participating generalists on selected case records under the study’s conditions. The next key evidence will be whether that performance holds in prospective care and improves decisions without adding unacceptable risks, costs or delays.

Key Questions

What did Microsoft’s MAI-DxO study find?

In a paper published in Science on October 22, 2026, Microsoft reported that MAI-DxO scored 83.2% on 304 complex case records. Physicians using textbooks and the internet scored 36.1% in that comparison. These were benchmark results, not outcomes from a patient-care trial.

Was the comparison with specialist doctors?

No. The comparison described in the source material was with generalist physicians working individually. It did not include specialist teams, even though complex cases in clinical practice may be referred to specialists or discussed by teams.

Did the AI diagnose real patients in this study?

No. The benchmark used historical case records presented as text. It did not involve physical examinations or real-time imaging. A separate prospective study involving consenting patients alongside their normal care is planned at Beth Israel Deaconess Medical Center.

Does peer review confirm that the system is ready for hospitals?

No. Peer review means the paper underwent journal review; it does not independently reproduce the results or establish clinical safety and benefit. The reported study measures performance under benchmark conditions, and further real-world evidence is still needed.

Source: ThorstenMeyerAI.com