AI in Medicine | Student Interest Group Seminar

Title: Diagnostic accuracy and clinician-rated reasoning quality dissociate in an evaluation of large language models in psychiatry

Abstract: Evaluations of large language models (LLMs) in medicine rest almost entirely on diagnostic accuracy. As frontier models converge on benchmark performance, it is unclear whether accuracy still distinguishes systems under consideration for clinical use, or whether structured clinician assessment of model reasoning carries information that accuracy does not. Psychiatry, where diagnosis depends on narrative interpretation and clinical judgment under uncertainty rather than confirmatory biomarkers, is a demanding setting to stress-test this. We evaluated four frontier LLMs on 196 clinician-curated psychiatric case vignettes (135 screened from published case reports; 61 authored by clinicians and never published, to mitigate memorization effects). Diagnostic accuracy was measured across ranked five-diagnosis differentials. On a 30-vignette subset, five psychiatrists, blinded to model identity, independently rated all 120 reasoning traces for data extraction and diagnostic reasoning quality on a 0-4 rubric adapted from the ACGME Psychiatry Milestones, and independently adjudicated diagnostic correctness, yielding 600 ratings. Models were compared pairwise within-case on both measures; equivalence in accuracy was assessed by two one-sided tests. We additionally tested whether three computable features of reasoning traces (length, number of ICD-10 codes named, and self-correction phrases) could substitute for clinician rating, comparing between-model associations to within-model associations. Clinician-adjudicated accuracy separated zero of six model pairs, while reasoning quality separated five. Two systems were identical on accuracy (0.827 each), with the true difference bounded within ±0.06, yet differed by 0.70 points in reasoning quality (95% CI 0.50 to 0.90, Holm-adjusted p < 0.001). All five raters, working independently and blinded, produced the same rank ordering of the four systems. Clinicians agreed about reasoning quality at least as closely as about diagnostic correctness. Trace features that tracked reasoning ratings across four model averages were null or reversed within-model across 120 traces, reflecting how systems formatted their differentials rather than rated quality. Accuracy estimates were themselves unstable: regenerating outputs under identical prompts changed the top-1 correctness verdict for 14.3% of cases, and re-adjudicating identical outputs moved top-5 accuracy by 5.1 percentage points on average. Systems ranked equivalent by diagnostic accuracy differed consistently in clinician-rated reasoning quality, and readily computable reasoning trace features did not recover the difference. Where an LLM is intended to support clinical reasoning rather than to classify, structured clinician assessment of reasoning traces supplies information that accuracy benchmarks do not, and therefore belongs in pre-deployment evaluation.