If AI alone > (AI + Human) > Human alone, what is that telling us?
Recent studies of AI in medicine, particularly in imagery and diagnostic reasoning, have surfaced an alarming trend if we are to believe in AI as a way to augment human abilities: an expert aided by AI is not as good as AI on its own. The figure here is from @eric topol and @pranav rajpurkar’s blog post (link), which includes the text of their @new York times OpEd (link). It shows the performance gap in task performance between AI and AI + physician input, which can sometimes be large. Here is Topol and Rajpurkar’s comment on these findings:
“What explains these counterintuitive findings? They could simply reflect that physicians haven't been well grounded in using A.I., or have "automation neglect" (bias against A.I.), or that the studies are relatively small and contrived—attempts at simulating medical practice but a far cry from the complex, messy world of how we diagnose and care for patients. But there may be a more fundamental consideration: we may need to rethink how we divide responsibilities between human physicians and A.I. systems to achieve the goal of synergy (not just additivity, i.e. 1+1 = 5).”
All of these things may contribute to the observed gap, but their last point subsumes a lot of the issues and deserves our attention: the current state of AI-human interaction leads to sub-optimal outcomes. These findings are about specific clinical or medical management tasks and how they generalize is unknown, but diagnostic reasoning, for example, is a commonly required skill across many disciplines: the ability to accurately assess a situation. The 2024 article (link) by @ethan goh, @robert gallo and colleagues provides an excellent starting point: 50 physicians (family medicine, internal medicine, or emergency medicine) were asked to review 6 clinical vignettes in 60 minutes and “randomized to either access an LLM in addition to conventional diagnostic resources or conventional resources only”. Lots of caveats, obviously, e.g., clinical vignettes are not the same as spending time with a patient. But when given this information, the respective median diagnostic reasoning scores for physician with conventional resources, physician with conventional resources and LLM and LLM alone are 74%, 76% and 90%. In other words, the LLM “augmented” the physicians by a small amount (2%). But it could also be said that the physician diminished the LLM’s scores by 14%. For that to happen, the physicians had to dismiss the LLM’s output and favor their own in situations where they differed. The authors of that paper suggest “that further development in human-computer interactions is needed to realize the potential of AI in clinical decision support systems.” Indeed.
The reason I like the diagnostic reasoning case, in addition to its general applicability, is that its psychology has been studied for a long time in the medical space. Numerous cognitive biases have been identified, e.g., availability and self-confidence, as playing an outsized role in diagnostic errors. While self-confidence is something to work on in medical training, availability biases should be addressable by better AI-human interfaces that promote exploration progressively further away from the physician’s comfort/familiarity zone. The progressive aspect is essential for the physician to keep an open mind: anything that is too far from their comfortable center will be perceived as spurious, ridiculous, noise. Think of it as levels in a game: you can only play the next level if you master the current one.
@spencer dorn, @chris cassel, @robert watcher, @jay Parkinson, @thomas wolf, @cassie, @anorld Milstein, @jason maude
https://www.nytimes.com/2025/05/14/technology/ai-jobs-radiologists-mayo-clinic.html