MedEval-1

What it is

MedEval-1 is an evaluation harness and a public leaderboard. It scores a model on more than accuracy: how well its confidence matches its error rate, how it holds up when the way an image or record was acquired changes, how it fails under corruption, how it does for subgroups where the data allow it, and how it does with little training data. Each score is tied to a manifest that records what was run on what.

The same instrument scores 9 domains, inside and outside medicine, without renaming a dimension. The public leaderboard runs on public data and is labelled as such; it is also the control arm for the held-out track below.

The held-out track: mint, seal, score, rotate, audit

A public benchmark has a half-life: once published, it drifts into the next generation of training data, and its scores start to measure memory as much as ability. The held-out track answers that by keeping the test set secret and replacing it on a schedule.

1. Mint. A test set is drawn from an accrual window against published targets for class and setting mix. Its size, targets, window and expiry are published; its contents are not.
2. Seal. Every case, the whole set and the answer key are hashed and written into an append-only ledger before any model is scored against them, so the set cannot be edited afterwards without it showing.
3. Score. A submitter never sees the cases, and the operator never sees the weights. Each model version gets one scored run.
4. Rotate. Each set expires on its published date and is replaced from fresh accrual. A small shared anchor keeps scores comparable across generations. A retired set is republished in full as a dated, known-exposed benchmark.
5. Audit. Every published score can be re-derived from its manifest and a de-identified bundle, without the underlying data.

What has been checked

• 48 of 48 published scores were re-derived by our own audit from de-identified bundles and manifests.
• A submission deliberately trained on a sealed set was flagged by the contamination check, whose threshold was written into the ledger before any set was minted. After one rotation, almost all of the leaked model’s advantage was gone on the fresh set.
• A repeat scoring of the same model version, and a rotation before the published expiry, were both refused and logged.
• Anyone holding one case from a sealed set can check that it was in the set when the set was sealed, without learning anything about the other cases.

How to repeat it

The held-out page of the demo (opens in a new tab) replays one real run step by step: the ledger, the two sealed generations, the contamination check, the rotation and the calibration across generations. The audit code, manifests and bundles are available on request; with them, an auditor can re-derive each published score on their own machine. Ask for them.

What it is not

• Not independent of us. NakedSignal built it and ran the audit with its own pipeline. The re-derivation is a reproduction, not a re-implementation, so a bug shared by both would survive it.
• Not yet on private data. The held-out track runs on a public image corpus as a labelled stand-in [3]. It shows the machinery works; it cannot show a test set no model has ever seen, which needs clinical data that has never been published.
• Time is attested by us. The order of ledger entries is fixed by the hash chain, but the absolute time of each seal is not yet countersigned by an outside timestamping service.
• Some settings are simulated. The second generation’s imaging setting is simulated and declared as such.
• No outside submissions or data. No model developer has submitted to the held-out track, no data holder has contributed a test set, and MedEval-1 is not a certification or a regulatory clearance.

The demo site is kept unchanged as a dated record and does not name NakedSignal. Where its wording differs from this page, this page is the current description.

Instrument change is dataset shift

A model trained on laboratory values inherits every instrument that produced them. Clinicians and developers have been warned that dataset shift can come from changes in technology, such as a new assay, as well as from changes in populations and practice [1], and that models are best built and checked with the shifts they will meet in deployment in mind [2].

An analyser change is one of the few such shifts that is documented and dated. The same method this site uses for laboratories sizes how far a model’s inputs moved at the thresholds the model relies on, against measurement noise. Held-out evaluation and input-change sizing are the two halves of keeping a model honest.

Guide: Measurement change in trials and medical AI

Sources

1. Finlayson SG, Subbaswamy A, Singh K, Bowers J, Kupke A, Zittrain J, Kohane IS, Saria S (2021). The Clinician and Dataset Shift in Artificial Intelligence. New England Journal of Medicine. pmc.ncbi.nlm.nih.gov/articles/PMC8665481/
2. Subbaswamy A, Saria S (2020). From development to deployment: dataset shift, causality, and shift-stable models in health AI. Biostatistics. doi.org/10.1093/biostatistics/kxz041
3. Yang J, Shi R, Wei D, Liu Z, Zhao L, Ke B, Pfister H, Ni B (2023). MedMNIST v2: a large-scale lightweight benchmark for 2D and 3D biomedical image classification. Scientific Data. doi.org/10.1038/s41597-022-01721-8

Each link was opened and checked on 26 September 2026.