A Lower Score Is Not a Failed Transfer: What the Claim That Static Medical Benchmarks Do Not Carry Over to Interactive Clinical Encounters Measures, Why Level, Rank and Selection Come Apart, and the Harness Controls the Interactive Benchmarks Owe

Abstract

Life of Research

Interactive clinical benchmarks now routinely report that a language model's score on static medical questions "does not translate" to an encounter in which it must take a history, order tests and manage a patient over time. A June 2026 benchmark built from 1,638 standardized-patient teaching cases is a clear recent case: the best of seven models completes 60.4 percent of expert rubric items, medically tuned models trail general ones, and extra test-time compute adds under one point. In this analysis the author reviews and synthesises published results to show that the verb "translate" carries three different claims. Level transfer, whether absolute scores carry over, fails almost everywhere and largely by construction. Rank transfer, whether the order of models carries over, mostly holds: recomputed from one published paired table of 15 models, the Spearman correlation between full-information and interactive accuracy is 0.90 and the top model is unchanged, although close pairs reverse and two reasoning-distilled models fall far below their static rank. Selection transfer, whether either score picks the system that will serve real patients and clinicians, is the claim the evidence supports least, because every interactive score is also a measurement of its simulator, budget, judge and harness: the same model's accuracy moves from 52 to 25 percent when its question budget is halved, and simulated users outperform real ones while tracking them only weakly. The paper separates four manipulations that "dynamic" names, brings 31 rows of published and recomputed values together in Table 1 and Figure 1, gives a step-by-step reading procedure for any non-translation claim, and specifies the paired, simulator-swapped and human-anchored designs that would settle what remains open.

About the author

Pranay Mahendrakar is an ai specialist and large language model engineer based in Bengaluru, India. He builds production artificial intelligence systems and publishes open-access research on how those systems fail. See all 145 papers by Pranay Mahendrakar, or his ORCID record.