Breast imaging pilot finds GPT-4 closer to guideline recommendations

A 2023 study from IO Lab found that GPT-4 more closely matched breast imaging recommendations than GPT-3.5 on several tasks. The pilot appeared in the Journal of the American College of Radiology. Performance depended on the clinical scenario and the way researchers asked the question. The findings illustrate why evaluating a medical chatbot requires attention to both the model and the task it is expected to perform.

Arya Rao and colleagues compared the models’ responses with the American College of Radiology Appropriateness Criteria for breast cancer screening and breast pain. They used open-ended prompts and questions that required selecting every appropriate option. Each prompt was repeated three times. The experiment examined recommendations about imaging, using written scenarios.

For screening questions in the select-all format, accuracy was 98.4% for GPT-4 and 88.9% for GPT-3.5. For breast pain questions in the same format, the corresponding figures were 77.7% and 58.3%. Both models averaged 1.83 out of 2 points on open-ended screening responses. GPT-4 also scored higher on the open-ended breast pain questions.

The variation is relevant to decisions about how a tool might fit into clinical work. Selecting an imaging examination involves the patient’s symptoms, risk factors and previous findings. A system that performs well in one narrowly defined setting still needs evaluation in other settings. Question format also influences what a score captures. An answer may sound reasonable while omitting an appropriate option or including an unnecessary examination.

The study’s scope limits what can be concluded. It was a pilot involving selected guideline scenarios, and it did not test whether the models could interpret mammograms or ultrasound images. It also did not examine patient outcomes, actual imaging orders or the effect of giving recommendations to practicing clinicians. Its results concern the model versions evaluated in 2023.

For radiology research, the comparison supports testing against explicit clinical standards and reporting results by task. A useful next step would be to assess broader clinical scenarios and examine errors that could change care. Studies of clinician use would also need to measure whether a recommendation is understood, checked and acted on appropriately. Those questions determine how a guideline-based benchmark might inform decisions about clinical decision support.