|
Microsoft's diagnosis AI outscored generalist physicians on a set of hard benchmark cases in a peer-reviewed study published Thursday in Science. The results come from a controlled test built on past case records, and Microsoft's next step is a study with patients at a Boston hospital.
The benchmark draws on 304 complex case records from The New England Journal of Medicine. Running on OpenAI's GPT-5, the Microsoft AI Diagnostic Orchestrator, or MAI-DxO, reached the correct diagnosis 83.2% of the time. Generalist physicians, who worked through a 56-case subset, scored 36.1% with search and reference tools and 22.3% without them. On 325 emergency department admissions drawn from the MIMIC-IV-ED database, the system scored 80.3% against 46.8% for physicians, Medical Economics reported from the paper.
The setup differs from the preprint Microsoft posted in June 2025. In that version, the system ran on OpenAI's o3 and scored 80% while doctors averaged 20%, but the physicians had been told not to use search engines, chatbots or other online sources of medical information. The Science paper tests doctors both ways and adds the emergency cases, which come from routine hospital records. The gap narrowed, though it is still wide.
Several AI agents act like a panel of doctors. They pick questions and tests one step at a time and argue against their own leading theory before they commit to an answer. Each test carries an estimated price, so the benchmark, called SDBench, grades cost too. |
