ChatGPT’s accuracy varied across the steps of a clinical case

A 2023 study from IO Lab found that ChatGPT performed unevenly as clinical cases progressed from an initial presentation to a diagnosis and treatment plan. A 2023 study in the Journal of Medical Internet Research found its strongest results in final diagnosis and weaker performance when it had to propose a range of possible explanations early in a case. The findings helped define a question that remains central to medical AI evaluation: how performance changes as clinical information accumulates.

Arya Rao and colleagues tested the January 2023 version of ChatGPT on 36 clinical vignettes from the MSD Manual. They supplied information in stages and asked about differential diagnosis, diagnostic testing, final diagnosis and management. Each case ran in three independent sessions. Two reviewers scored the answers against the case materials.

Overall accuracy was 71.7%. Accuracy reached 76.9% for final diagnosis and 60.3% for differential diagnosis, the process of considering several plausible causes of a patient’s symptoms. These results describe performance on structured written cases. They provide a way to examine which parts of a clinical workflow deserve closer scrutiny before a tool is used in practice.

That distinction matters because early decisions shape the information available later. A suggested test can narrow a diagnosis, and an incomplete list of possibilities can leave relevant conditions unexplored. An overall score may therefore conceal differences between tasks that carry different consequences. Evaluating each stage separately can help researchers identify where additional safeguards and clinical oversight need testing.

The study had several limits. It used a specific early version of ChatGPT, so its numerical results should not be applied to newer systems. Most cases were publicly available before the model’s training cutoff, creating a possibility that training data influenced performance. Questions requiring image interpretation were excluded. The experiment also measured answers to educational cases without examining patient outcomes or how clinicians would use the responses.

For developers and health systems, the work offers a framework for testing a model across a sequence of decisions. Further studies would need to examine unfamiliar cases, real clinical settings and the interaction between clinicians and AI. Evidence from those settings would help establish whether improvements in case accuracy translate into useful and reliable support for care.