A Harvard-led study just dropped a result that will make every ER doctor’s stethoscope twitch.
In a head-to-head against attending physicians at a Boston hospital, OpenAI’s o1-preview reasoning model diagnosed patients more accurately at the moment of triage, just when information is sparse, time is short, and getting it wrong can kill someone.
The Numbers
Researchers handed the AI and two attending physicians the same starter pack: vitals, demographics, and a nurse’s brief note explaining why the patient showed up. Across 76 cases at Beth Israel Deaconess:
• AI: 67% exact-or-very-close diagnosis at initial triage
• Doctors: 50% and 55%
When more information was layered in, the AI hit 82% versus 70–79% for the humans (a gap that wasn’t statistically significant).
On longer-term care planning across five clinical case studies, the AI scored 89% versus 34% for 46 doctors using conventional resources like search engines. Published in Science. Independent experts called it “a genuine step forward” in AI clinical reasoning.
Why This Matters
The triage moment is medicine at its hardest: messy electronic health records, ambiguous symptoms, and a clock running. If an AI can passively scan that wall of noise and flag missed diagnoses before they happen, that’s not a robot doctor, that’s a tireless second opinion.
As one of the researchers noted, doctors already get better outcomes when they consult human colleagues; AI just makes that consultation available at 3 a.m. on a Tuesday. A 2025 Elsevier study already found 1 in 5 clinicians worldwide were quietly using AI this way.
Now The Analysis
Before we hand o1 a white coat, a few things deserve a hard squint:
Two doctors is not “the medical profession.” The Boston ER comparison pitted an AI against a pair of attendings on 76 cases. That’s a tight sample with high variance — swap in two different doctors and the gap might shrink, vanish, or flip.
The “89% vs 34%” treatment-planning number is more dramatic, but the humans were limited to “conventional resources” like search engines. Letting doctors Google things while the AI runs on its full reasoning stack is not a fair fight; it’s a benchmark designed to flatter the model.
Diagnosis is not treatment. Harvard’s own Arya Rao flagged it: clinical reasoning isn’t moral reasoning. Choosing the right antibiotic regimen is one thing; deciding when to stop one is another.
The study tells us a model is good at pattern-matching symptoms to labels — it tells us very little about whether it should be trusted to weigh a frail patient’s quality of life against an aggressive intervention.
“Eclipsed most benchmarks” is a tell. When researchers say an LLM has beaten the benchmarks, the next question should be: are the benchmarks any good?
Medical exam questions and curated case vignettes reward the exact thing LLMs are built for, recall and pattern matching over text. They underweight the parts of medicine that aren’t text: the way a patient looks, the half-sentence a family member mutters in the hallway, the gut feeling that something is off.
Where’s the failure analysis? A model that’s right 67% of the time is wrong 33% of the time. In an ER, the shape of those errors matters more than the average. Are the misses random?
The Honest Take
This is a real result, not hype. AI as a passive co-pilot scanning EHRs for missed diagnoses is probably coming, and probably soon and probably good.
“AI outperforms doctors” is the kind of headline that gets administrators excited about replacing people, when the actual finding is closer to “AI is a useful second opinion in the first ten minutes.” Those are very different products. Only one of them is supported by 76 cases.
The doctors aren’t hanging up the scrubs. But they clearly need to incorporate AI into their daily workflow.
Dr Bob, Brain Mechanic 