Hospital BenchIndependent benchmark analysis / 2024 paper; dataset metadata v1.1
MIMIC-CDM reconstructs diagnostic tasks from records for four abdominal conditions. Its two information settings make a useful contrast: an interactive model requests findings, while a full-information model receives a prepared case. We examine the implications of that contrast without treating a four-condition sample as a general emergency-department population. The paper’s per-class accuracy also matters: its denominator is patients with the target disease, not every patient for whom a positive diagnosis might be made. Our independent analysis connects the selection rules, disease balance and result labels to the claims a hospital reader can reasonably make.
01 / What is being tested?
Data origin. 2,400 deidentified MIMIC-IV v2.2 patients selected for appendicitis, cholecystitis, diverticulitis or pancreatitis; all required modalities must be present. [4][5][6]
Four selected abdominal pathologies, not an all-comer emergency cohort.
Methods [5]Hospital module and clinical notes underpin the derivative.
Background [5]Cases with multiple selected target pathologies are excluded.
Methods [5]The two settings offer different amounts and timing of evidence.
Results; Methods [4]July 2024 release adds pathology IDs and corrects procedure columns.
Release Notes [5]Require a target primary diagnosis and the needed clinical modalities; remove diagnosis leakage from selected text.
[5]Provide findings on request, or use the full-information comparator.
[4][5]Save the diagnosis, proposed treatment and information-gathering path.
[4]Assess the final diagnosis and other behaviors against references, with pathology-specific denominators.
[4]Dataset anatomy
Selected cohort.
Selected cohort.
Selected cohort.
Selected cohort.
Published cohort counts; do not interpret as population prevalence. Bar lengths use the largest listed count as their reference; they are not percentages of a shared population.
02 / Measurement
Higher is better within a fixed setting and cohort.
Correct diagnoses divided by the number of evaluated cases with that disease; headline means average across the four pathology classes.
Accuracy_c = correct diagnoses in class c / evaluated patients in class c; macro = Σ accuracy_c / 4
This restricted cohort cannot supply general emergency-care specificity or positive predictive value. Full-information and interactive results are separate settings. [4]
03 / Measured evidence
Paper-reported results / selected rows
Hager et al., July 2024; models gather information in the MIMIC-CDM simulation.
Paper-reported means across pathologies; no current-frontier inference. Do not combine with full-information rows.
Source: Results accompanying Figure 3 [4]
Paper-reported results / selected rows
Same paper, MIMIC-CDM-FI setting; relevant case information is provided up front.
These are the paper’s comparison values, not new reruns or an estimate of benefit from a particular clinical intervention.
Source: Results accompanying Figure 3 [4]
04 / Our original analysis
All three selected models have lower mean accuracy in the interactive setting. [4]
An all-information test leaves the ability to choose useful next information largely unmeasured. The observed gap is a setting contrast, not a causal attribution to one component.
Cases are selected for four known primary pathologies. [4]
These results cannot supply the false-positive burden for a hospital seeing the complete differential diagnosis.
The data card retains only the first laboratory instance for each test. [5]
This configuration is poorly suited to claims about longitudinal response to treatment. A larger record volume does not restore omitted temporal behavior.
05 / Scope of the evidence
The selected four conditions do not represent all abdominal pain or the full hospital diagnostic workload. [4][5]
Requiring available imaging, tests and physical examination selects cases with particular documentation and care pathways. [5]
The 2024 comparison cannot settle performance of later model generations or different agent systems. [4]
Patient-derived data remains credentialed and governed by PhysioNet terms. [5]
Evidence trail
Hager, Jungmann, Holland et al.; Nature Medicine. Original MIMIC-CDM study; historical models and restricted disease cohort.
Hager, Jungmann and Rueckert; PhysioNet. Official data card; 2,400 patients, disease counts, access and release notes.
Paul Hager and collaborators. MIT code license; source data remains credentialed.
Benchmark authors retain authorship of their work. This publication provides independent analysis; published rows are not Arcophos evaluation runs. Editorial method.