What The Science Paper Says About Microsoft’s AI And Doctors

🔍 Read the full analysis: What The Science Paper Says About Microsoft’s AI And Doctors on ThorstenMeyerAI.com

Buying for a business?Offer from Amazon

Get business pricing on tech for your team

• Business-only prices and quantity discounts
• Tax-exempt purchasing
• Multiple users, one account, clear invoices

Create a free Business account →Free to register

As an affiliate, we earn on qualifying purchases.Hide

Microsoft’s MAI-DxO diagnostic system scored 83.2% on 304 complex medical cases, compared with 36.1% for generalist physicians using textbooks and the internet, in a study published October 22 in Science. The peer-reviewed results show strong performance on selected case records, but do not establish how the system would perform with real patients or alongside specialist teams.

The paper, titled Sequential diagnosis with language models, also reported results from 325 emergency department cases: MAI-DxO scored 80.3%, while physicians scored 46.8%. Microsoft describes the system as a group of virtual physician agents that propose diagnoses, request further information or tests, and evaluate one another’s reasoning over multiple rounds.

The test used a sequential format rather than giving participants every case detail at once. Each case began with a short summary; participants had to request additional details from a simulated gatekeeper, with tests assigned costs. The benchmark therefore assessed both the final diagnosis and the information-gathering process. The results were produced by Microsoft, and all 15 listed authors were affiliated with the company, including Microsoft AI chief executive Mustafa Suleyman.

Microsoft’s reported comparison for the complex cases included physicians working with reference materials and a separate group without them: 36.1% and 22.3%, respectively. The study also reported that GPT-5 alone scored 78.9% on the complex cases, indicating that most of MAI-DxO’s diagnostic score came from the underlying model; the orchestration system added about four percentage points, according to the source material. These scores describe performance on the study’s cases, not measured clinical outcomes.

What the Scores Mean for Care

The paper offers evidence that language models can perform well on a structured diagnostic task that requires selecting questions and tests, rather than simply choosing from a list after receiving a complete case history. That could inform future research into clinical decision-support tools. It does not show that MAI-DxO can safely diagnose patients, reduce costs in hospitals, or improve health outcomes.

The difference between the benchmark and routine care matters for patients, hospitals and insurers. The comparison was between an AI system operating as a team and individual generalists, not specialist teams or clinicians consulting colleagues. The cases were selected as difficult teaching examples, and the study did not include healthy patients or benign explanations. Those limits leave open how the system would handle common complaints, competing diagnoses, incomplete records and the need to avoid unnecessary testing.

For health systems, the results are a reason to examine the system in clinical research, not a basis by themselves for adopting it. A real-world evaluation would need to test its accuracy and safety with actual patients, measure effects on clinical decisions and costs, and establish how responsibility is shared between the tool and medical staff.

How the Benchmark Was Built

The study’s approach reflects a difference between exam-style medical AI tests and diagnosis in practice. Instead of providing all details at once, the Sequential Diagnosis Benchmark required participants to decide what to ask or test next. MAI-DxO used several language-model agents to make and critique those choices, while the benchmark assigned costs to requested tests.

Some methodological details described in the source account come from the team’s June 2025 arXiv preprint, which was not peer-reviewed; the full Science paper is behind a paywall, and details may differ in its final version. In the preprint, a model graded diagnoses against the cases’ published conclusions. When a requested result was missing from a case record, the gatekeeper generated a synthetic result. Test costs were estimates based on billing codes and a 2023 U.S. hospital price list, plus a fixed fee for a physician visit. The authors described these costs as approximations.

The preprint’s physician group included 21 generalists from the United States and United Kingdom, with a median of 12 years of experience. They were barred from using search engines or language models, and physicians and AI were not assessed on identical sets of cases in that earlier version. The Science report gives updated figures, including an open-book physician comparison. The published result should be read on its own terms, while recognizing that the detailed methods available in the source material are drawn partly from the earlier preprint.

“I really believe that in 2027, we will be on the verge of a coding style breakthrough for medical records.”

— Microsoft AI chief executive Mustafa Suleyman, speaking to TechCrunch

Limits Before Patient Use

The study does not report a trial in which MAI-DxO made or guided diagnoses for real patients. It therefore does not establish whether the system improves care, causes harmful delays or false alarms, or changes the total cost of treatment. The benchmark also lacked healthy patients and harmless explanations, so it could not measure how often the system would flag a serious disease when none was present.

Other questions concern the evaluation itself. In the preprint, a model graded answers, and human re-grading showed good but imperfect agreement. Missing test results could be generated synthetically, and cost values were approximations. The Science paper is peer-reviewed, but peer review does not mean that an independent group has reproduced the findings. The available source does not establish whether those methods changed in the final paper.

It is also not clear how the system would compare with specialists, multidisciplinary teams or doctors using their usual tools and time. Nor does the reported accuracy alone show whether the system’s suggested tests would be appropriate in practice. These are open research questions, not evidence that the system has failed or succeeded in clinical use.

A Planned Hospital Study

Microsoft’s next reported step is a prospective study at Beth Israel Deaconess Medical Center in Boston. According to the source material, consenting patients are expected to interact with the system alongside their usual care. That setting could provide evidence beyond historical case records, although no study results are available yet.

Details still needed include the study’s timing, enrollment, clinical safeguards, outcomes and how clinicians will use or review the system’s suggestions. Until those results are reported, the Science paper remains evidence from a benchmark rather than a demonstration of performance in hospitals. Readers should watch for whether future research measures patient outcomes, safety, testing decisions and costs, and whether independent teams validate the findings.

Key Questions

What did Microsoft’s MAI-DxO score in the Science study?

Microsoft reported 83.2% accuracy on 304 complex medical cases and 80.3% on 325 emergency department cases. These were results on case records, not diagnoses made in a clinical trial.

Did MAI-DxO outperform doctors in the study?

It scored higher than the generalist physicians in the reported comparisons. On the complex cases, physicians scored 36.1% with textbooks and the internet and 22.3% without those resources. The comparison did not include specialist teams.

Does the paper show that MAI-DxO is safe for patients?

No. The results came from historical, text-based cases in a simulator. The study did not establish safety, better health outcomes or effectiveness in routine patient care.

What happens next for the system?

A prospective study is planned at Beth Israel Deaconess Medical Center in Boston, where consenting patients are expected to interact with MAI-DxO alongside usual care. The source material does not provide results or a detailed timeline.

Source: ThorstenMeyerAI.com