ATOMIC CLAIM VERIFICATION FOR CLINICAL DIAGNOSTIC REASONING: DECOMPOSING AND VALIDATING LLM REASONING STEPS AGAINST MEDICAL EVIDENCE

Abstract

Large Language Models (LLMs) are increasingly evaluated for, and piloted in, clinical settings for diagnostic support, and a growing literature uses LLMs themselves to grade other LLMs' medical reasoning. This dissertation set out to build an atomic-claim verification pipeline for LLM medical reasoning and to evaluate that pipeline against clinician-authored reasoning statements in the MedCaseReasoning corpus (Wu et al., 2025; N = 200 cases, 620 clinician statements). The principal result is a negative methodological one: across a rule-based decomposer, a single primary LLM verifier, and five additional vendor verifiers, no single-verifier configuration is operationally reliable as a stand-alone judge of medical reasoning quality. Seven empirical findings establish this conclusion. (1) DeepSeek-R1:14b reproduces only 25.5% of clinician reasoning statements (95% bootstrap CI [21.3, 29.8]) under a stricter qwen2.5-based gold-standard judge, against Stanford's published 64% baseline. (2) Reasoning recall and effective diagnostic accuracy are statistically dissociated (Pearson r = 0.09), so reaching the correct diagnosis does not require addressing the clinician's reasoning steps. (3) Rule-based atomic-claim decomposition achieves precision 0.88 but recall 0.15 against an independent qwen2.5:32b annotator, with Cohen's kappa = 0.02 for five-type classification, at chance for the four non-observation types. (4) A marginal-matched permutation null shows that the Agreement Bias Index is mathematically determined by the verifier's support rate: 0 of 11 verifier-by-prompt conditions are distinguishable from a random labeler with the same marginal. (5) Cross-verifier agreement on SUPPORTED versus NOT_SUPPORTED is negative, with Fleiss' kappa = -0.134 (standard prompt) and -0.109 (adversarial prompt) and 95% bootstrap confidence intervals of [-0.155, -0.109] and [-0.133, -0.081]. (6) Prompt wording flips identical-claim verdicts: Opus-Sonnet pairwise agreement collapses from kappa = 0.66 (standard) to kappa = -0.02 (adversarial) on the same 250 claims. (7) The dominant LLM failure mode is exclusion failure (63.3%) and factual omission (36.1%), together 99.4% of unrecalled clinician statements and invariant to whether the final diagnosis was reached. Findings 1 through 7 establish that single-verifier LLM-as-judge evaluation is not a defensible methodology for medical reasoning assessment. The contribution of this dissertation is the systematic demonstration of this limitation, the Agreement Bias Index as a descriptive marginal-rate index (not a bias quality metric), and a gold-standard-grounded evaluation alternative built against clinician-authored reasoning statements. Knowledge-grounded verification using UMLS and PubMed is positioned as the methodological response motivated by these findings, with empirical evaluation deferred to future work.