Reagent and calibrator lot changes: verifying with patient samples at decision limits

Why a lot change can move results

Every new lot of reagent or calibrator is made to the same specification, but not to the same result. Lot-to-lot variation limits a laboratory’s ability to produce consistent results over time, and it has well-documented clinical consequences [Thompson and Chesher, 2018]. Most lot changes are small. The ones that matter are those that move results near a decision limit, where a small shift changes what the clinician does.

Verifying each new lot is part of monitoring the long-term stability of a measurement procedure. In practice it is constrained by resources and by uncertainty over which design and statistics suit an individual laboratory [Loh et al., 2023].

Why quality control may not show it

The obvious check is to run quality control or external quality assessment material on both lots. The problem is commutability: that material often does not behave like patient samples, so a difference seen on it may not be present in patient serum, and a real difference in patients may not show on it. Native patient samples are preferred [Thompson and Chesher, 2018]. The same review cites one large study across several platforms and analytes in which control material and patient serum gave significantly different results in 40.9% of lot-change events.

What EP26 asks

The Clinical and Laboratory Standards Institute guideline EP26, now in its second edition (2022), sets out how a laboratory can use patient samples to detect clinically important changes from the current lot [CLSI EP26]. It works in two stages. Once per analyte, the laboratory fixes its acceptance thresholds and its tolerance for risk: a critical difference tied to allowable total error, and the statistical power it wants. Then, for each new lot, it runs a small number of patient samples on both lots. In the first edition, as reviewed by Thompson and Chesher, the mean difference at each target concentration is compared with a rejection limit set as a share of the critical difference [Thompson and Chesher, 2018].

The design’s strength is that it is fixed in advance. The laboratory states what size of change matters before it looks at the data, and the number of samples follows from that and from the method’s imprecision.

Where it leaves gaps

• Sample numbers. The required number of samples depends on imprecision and on the critical difference, and can be large. One laboratory that compared the first edition of the protocol with its own 20-sample procedure across six thyroid assays found the two agreed on 9 of 12 lot evaluations; the protocol needed more samples for 4 of the 6 analytes, and its rejection limits were hard to set [Katzman et al., 2017].
• Power. Published lot-evaluation protocols differ in statistical rigour, and some may be underpowered to detect a clinically meaningful change [Thompson and Chesher, 2018].
• Cumulative shifts. Each lot is compared with the one before. Current protocols, the CLSI one included, will not detect small shifts that accumulate in one direction over several lots [Thompson and Chesher, 2018].
• Patients rather than means. A pass or fail on the mean difference says whether the lot is acceptable, but not how many patients near a limit are now classified differently.

Patient samples at decision limits

A lot comparison is a method comparison on a small scale, and the same habits apply. Choose patient samples spread around each limit the laboratory reports, not only across the analytical range. Estimate the difference at each limit, not once for the whole range: a proportional shift that is small on average can be large at a high limit, and a constant shift matters most at a low one. Then place each pair in the categories the limits define and count how many changed category, at each limit and in each direction.

Simulation makes the case for looking at classification directly: the share of patients reclassified rises with both imprecision and bias, and at larger biases bias dominates [Loh et al., 2024]. A lot shift is a bias, so counting patients adds exactly the information the mean difference leaves out.

Small studies and the repeat-testing baseline

A lot comparison is usually small, and samples chosen near limits are, by design, the ones most likely to change category on a simple repeat. A handful of crossings in a small study can be entirely ordinary. To tell, compare the count with the number expected from repeat testing alone, computed from the method’s imprecision and each result’s distance from the limit. The guide to crossings by chance sets out that calculation.

Grade the result by study size. In our run on public method comparisons, 5 of the 34 comparisons, with 56 to 89 pairs, were graded indicative, and the three with 21 pairs got no re-test band at all. A lot study of 3 to 12 samples [Tao et al., 2020] is smaller still: below 20 pairs our program computes no crossing count, so a single lot is judged on the mean difference at each limit.

Drift across several lots

Because each verification compares a new lot with the one before, a series of small shifts in the same direction can pass every check and still move results a long way. Moving averages of patient results over several lot changes can reveal that drift, given software, careful tuning and a stable patient population [Thompson and Chesher, 2018]. Collaborative verification between laboratories and patient-based monitoring are likely to improve detection further [Loh et al., 2023]. Keeping every lot comparison in one record, with the difference at each limit, makes the running total visible.

A public example

Public data

The closest public case is a comparison of two creatinine chemistries, enzymatic and kinetic Jaffe, run on the same samples on one analyser, in an open-access deposit [Schmidt et al., 2015, PLoS ONE]; read as a switch from one to the other, it asks the same question a lot verification asks: how many patients would the switch move across the limits, compared with what repeat testing alone would move at the laboratory’s measured imprecision?