Skip to content
IO Lab
IO LAB / NEWS

AI-generated clinical exam cases earn high ratings and reveal areas for faculty review

Citation: Esmail KP, Rao AS, Lee RS, Ebada A, Whitehead DC, Meehan C, Cheung A, Succi MD. Extending synthetic medical education: generating OSCE cases with LLMs. npj Digital Medicine. Published online October 19, 2026. doi:10.1038/s41746-026-03421-8

Online first: October 19, 2026.

A new study from the IO Lab finds that a large language model can draft complete clinical examination cases that physicians rate favorably, while identifying specific areas requiring faculty review. Published in npj Digital Medicine, the feasibility study examines how AI could support the preparation of Objective Structured Clinical Examinations, or OSCEs.

OSCEs assess clinical skills through structured encounters, often involving people trained to portray patients. Creating a station requires instructions for students, patient scripts, scoring criteria and learning objectives. Faculty must keep these elements consistent and appropriate for the learner’s experience. The study addresses the challenge of producing this full set of materials together.

IO Lab researchers developed a tool that generates each station through a single structured request to a language model. They created 24 cases covering eight internal medicine diagnoses at three training stages: before clinical clerkships, during clerkships and during a sub-internship. The conditions included acute coronary syndrome, chronic kidney disease and rheumatoid arthritis. Three board-certified internists independently evaluated every case using a 12-item questionnaire.

All 24 cases met the tool’s structural requirements. Across 864 individual ratings, 95.4% were four or five on a five-point scale. Communication behaviors received a mean score of 4.64, alignment with core professional activities scored 4.62, and overall educational value scored 4.61. Those professional activities describe clinical tasks that trainees should become able to perform with appropriate supervision.

Reviewers also identified weaknesses. Agreement with clinical guidelines received the lowest mean rating, 4.24, followed by alignment of difficulty with the learner’s level at 4.25. Written feedback described clinical inaccuracies, missing information, cases pitched above the intended training stage and imprecise assessment criteria. Differences in how strictly the physicians scored cases contributed to only fair agreement between raters.

These findings support further evaluation of AI as a drafting aid within a faculty review process. Educators could use generated materials as an initial version, then check clinical accuracy, difficulty, completeness and the instructions given to patients and examiners. The study did not measure how much faculty time this process would save.

The cases have not yet been administered to trainees, and the study did not compare their quality with faculty-written stations. It also examined one model, a small group of internal medicine reviewers and a limited set of diagnoses. Further research will need to test learner performance, compare authoring time and assess consistency when the same case is generated repeatedly. Those steps would help determine how the approach could contribute to routine medical education.

Get IO Lab announcements

New research, courses, events, and opportunities, by email.

Subscribe