MedEval-1: held-out evaluation for medical AI | NakedSignal

published scores re-derived by our own audit: 48 of 48
domains scored by one instrument: 9

The held-out track

Mint, seal, score, rotate, audit

Step 1: Mint
A test set is drawn against published targets. Its size, window and expiry are public; its contents are not.
Step 2: Seal
Every case, the set and the answer key are hashed into an append-only ledger before any model is scored.
Step 3: Score
By design, submitters never see the cases and the operator never sees the weights. In this demo NakedSignal played both roles on one computer, with the separation enforced in code. One scored run per model version.
Step 4: Rotate
Each set expires on its published date and is replaced from a reserved pool, with a shared anchor for comparability. In this demo the accrual order and the calendar are simulated.
Step 5: Audit
Every published score can be re-derived from its manifest and a de-identified bundle.

What it is

MedEval-1 is an evaluation harness and a public leaderboard. It scores a model on more than accuracy: how well its confidence matches its error rate, how it holds up when the way an image or record was acquired changes, how it fails under corruption, how it does for subgroups where the data allow it, and how it does with little training data. Each score is tied to a manifest that records what was run on what.

The same instrument scores 9 domains, inside and outside medicine, without renaming a dimension. The public leaderboard runs on public data and is labelled as such; it is also the control arm for the held-out track below.

Why a held-out track

A public benchmark has a half-life: once published, it drifts into the next generation of training data, and its scores start to measure memory as much as ability. The held-out track answers that by keeping the test set secret and replacing it on a schedule. It runs today on a public image corpus [2], so a model may already have seen these images in training; cases no model can have memorised need private data, which this demo does not have.

What has been checked

48 of 48 published scores were re-derived by our own audit from de-identified bundles and manifests.
A submission deliberately trained on a sealed set was flagged by the contamination check, whose threshold was written into the ledger before any set was minted. After one rotation, almost all of the leaked model’s advantage was gone on the fresh set.
A repeat scoring of the same model version, and a rotation before the published expiry, were both refused and logged.
Anyone holding one case from a sealed set can check that it was in the set when the set was sealed, without learning anything about the other cases.

How to repeat it

The held-out page of the demo (opens in a new tab) replays one real run step by step: the ledger, the two sealed generations, the contamination check, the rotation and the calibration across generations. The audit code, manifests and bundles are available on request; with them, an auditor can re-derive each published score on their own machine. Ask for them.

Instrument change is dataset shift

A model trained on laboratory values inherits every instrument that produced them. Clinicians and developers have been warned that dataset shift can come from changes in technology, such as a new assay, as well as from changes in populations and practice [1].

An analyser change is one of the few such shifts that is documented and dated. The same method this site uses for laboratories sizes how far a model’s inputs moved at the thresholds the model relies on, against measurement noise. Held-out evaluation and input-change sizing are the two halves of keeping a model reliable.

Guide: Measurement change in trials and medical AI

Sources

Finlayson SG, Subbaswamy A, Singh K, Bowers J, Kupke A, Zittrain J, Kohane IS, Saria S (2021). The Clinician and Dataset Shift in Artificial Intelligence. New England Journal of Medicine. pmc.ncbi.nlm.nih.gov/articles/PMC8665481/
Yang J, Shi R, Wei D, Liu Z, Zhao L, Ke B, Pfister H, Ni B (2023). MedMNIST v2: a large-scale lightweight benchmark for 2D and 3D biomedical image classification. Scientific Data. doi.org/10.1038/s41597-022-01721-8