Is AI Better at Diagnosing Than Doctors?
Sometimes I browse new studies in the evening, and usually I just nod: "nice, another model, another benchmark." This time, I nodded twice.
Microsoft AI published the test results for its diagnostic system MAI-DxO in Science. On simulated, genuinely difficult clinical cases, the system nailed the diagnosis in 83.2% of cases. Family physicians and internists, working solo with access to reference materials – 36%.
- AI
- Doctors
Show data
| (%) | AI | Doctors |
|---|---|---|
| Difficult cases | 83.2 | 36.1 |
| Common diagnoses | 80.3 | 46.8 |
On routine cases, the advantage is there too, though smaller. You don't need to be a statistician to see the difference. But before we start packing hospitals into the cloud, it's worth taking a closer look at what exactly was measured.
How does this even work?
MAI-DxO is not another brain, that is, a language model. It's more of a control layer that can be laid on top of any large model. The system simulates a panel of doctors – a virtual medical board. Some "doctors" propose diagnoses, others order tests, and still others challenge the leading hypothesis and keep an eye on costs. Sound familiar? Yes, that's more or less what morning rounds on an internal medicine ward look like, minus the coffee and the complaining about night shifts.
What's more interesting is that the test wasn't your typical medical quiz. Standard benchmarks hand the model a ready-made package: history, exam, results – and ask "what is it?" Real diagnostics looks different. The patient says: "my belly hurts," and the doctor has to decide what to ask, which test to order, and when to stop searching.
And that's exactly what was tested on 304 cases from the New England Journal of Medicine. The system received a short case description and had to gather information step by step – ask questions, order tests, update hypotheses. Along the way, it turned out it also picked a cheaper set of tests than the doctors did.
The small "buts"
Now for the part Microsoft would happily relegate to a footnote.
Difficult cases in real life don't land with a family physician working alone. They go to specialists or entire consulting teams. And those weren't included in the study. So the comparison is methodologically fair – and at the same time a bit unfair to the people who deal with exactly these kinds of puzzles every day. It's a bit like a chess grandmaster beating someone who plays recreationally. A real win, but one that says little about how the other side would play with a proper setup.
Then there's the issue that the editor of Science put aptly: patients don't walk into the office with a neatly written case description. They show up with chaos, half-sentences, and "well, it's been kind of like this for a week." A simulation is a simulation.
Why it's still worth paying attention
Because even after subtracting all the asterisks from the result, something significant remains.
Diagnostics is a process – gathering information, forming hypotheses, revising them. Previous AI tests skipped exactly that stage: they could analyze a ready-made description, but nobody checked whether the system could drive the diagnostic process. Whether it asks the right questions. Whether it orders everything under the sun "just to be safe." Whether it knows when enough is enough.
MAI-DxO passed that test. And it did it cheaper than humans – which in medicine isn't a technical detail, but the real difference between a test being available or unavailable.
The head of Microsoft AI called the result a step toward "medical superintelligence." Well, lab bosses are entitled to enthusiasm – it's part of their job description. I'd be more careful with the adjectives, but I do think the direction itself is justified.
My humble prediction
I don't think that in a few years I'll be talking to a computer instead of a doctor. I do think, however, that in a few years my doctor will have a tool at hand that suggests a hypothesis they forgot about, or warns them against an unnecessary test. And that on difficult cases, such a board – human-machine – will perform better than either of its parts alone.
And if anyone has a problem with that picture, let me remind you: a medical board is, at its core, also just a bunch of specialists arguing about a single diagnosis. AI simply does the same thing, only faster and without lunch breaks.
That the stakes in this game are real, and that a misdiagnosis is not some abstract statistic, is something Christina Applegate – known from Married... with Children and Don't Tell Mom the Babysitter's Dead – learned firsthand. She announced she had multiple sclerosis in 2021. For five years, she lived with the diagnosis of an incurable disease. Now she's been told it was a mistake. How can you treat a disease for years that the patient doesn't even have? And who bears responsibility for an error like that?
So the question at the end is simple: would you rather have a diagnosis with a 36% accuracy rate or an 83% one? I know my answer.
