Reagent and calibrator lot changes: verifying with patient samples at decision limits

Why a lot change can move results

Every new lot of reagent or calibrator is made to the same specification, but not to the same result. Lot-to-lot variation limits a laboratory’s ability to produce consistent results over time, and it has well-documented clinical consequences [Thompson and Chesher, 2018]. Most lot changes are small. The ones that matter are those that move results near a decision limit, where a small shift changes what the clinician does.

Verification of each new lot is part of monitoring the long-term stability of a measurement procedure. It is limited by the resources it needs and by uncertainty over the design and statistics that suit an individual laboratory [Loh et al., 2023].

Why quality control may not show it

The obvious check is to run quality control or external quality assessment material on both lots. The problem is commutability: that material often does not behave like patient samples, so a difference seen on it may not be present in patient serum, and a real difference in patients may not show on it. Native patient samples are therefore preferred [Thompson and Chesher, 2018]. The same review cites one large study across several platforms and analytes in which control material and patient serum gave significantly different results in 40.9% of lot-change events.

What EP26 asks

The Clinical and Laboratory Standards Institute guideline EP26, now in its second edition (2022), sets out how a laboratory can use patient samples to detect clinically important changes from the current lot [CLSI EP26]. It works in two stages. Once per analyte, the laboratory fixes its acceptance thresholds and its tolerance for risk: a critical difference tied to allowable total error, and the statistical power it wants. Then, for each new lot, it runs a small number of patient samples on both lots. In the first edition, as reviewed by Thompson and Chesher, the mean difference at each target concentration is compared with a rejection limit set as a share of the critical difference [Thompson and Chesher, 2018].

The strength of this design is that it is decided in advance. The laboratory states what size of change matters before it looks at the data, and the number of samples follows from that and from the method’s imprecision.

Where it leaves gaps

• Sample numbers. The required number of samples depends on imprecision and on the critical difference, and can be large. One laboratory that compared the first edition of the protocol with its own 20-sample procedure across six thyroid assays found the two agreed on 9 of 12 lot evaluations; the protocol needed more samples for 4 of the 6 analytes, and its rejection limits were hard to set [Katzman et al., 2017].
• Power. Published lot-evaluation protocols differ in statistical rigour, and some may be underpowered to detect a clinically meaningful change [Thompson and Chesher, 2018].
• Cumulative shifts. Each lot is compared with the one before. Current protocols, the CLSI one included, will not detect small shifts that accumulate in one direction over several lots [Thompson and Chesher, 2018].
• Patients rather than means. A pass or fail on the mean difference says whether the lot is acceptable. It does not say how many patients near a limit would now be classified differently.

Patient samples at decision limits

A lot comparison is a method comparison on a small scale, and the same habits apply. Choose patient samples spread around each limit the laboratory reports, not only across the analytical range. Estimate the difference at each limit, not once for the whole range: a proportional shift that is small on average can be large at a high limit, and a constant shift matters most at a low one. Then place each pair in the categories the limits define and count how many changed category, at each limit and in each direction.

Loh and colleagues’ simulations make the case for looking at classification directly: the share of patients reclassified rises with both imprecision and bias, and at larger biases bias dominates [Loh et al., 2024]. A lot shift is a bias, so it is exactly the case where counting patients adds information the mean difference does not.

Small studies and the repeat-testing baseline

A lot comparison is usually small, and samples chosen near limits are, by design, the ones most likely to change category on a simple repeat. A handful of crossings in twenty samples can therefore be entirely ordinary. The way to tell is to compare the count with the number expected from repeat testing alone, computed from the method’s imprecision and each result’s distance from the limit. Our guide to crossings by chance sets out that calculation.

The count should also be graded by size, not hidden. In our own run on public method comparisons, 5 of the 34 comparisons were too small for a full re-test band and are shown as indicative. A lot study will usually fall in that range, and should be read the same way: a direction and a rough size, not a precise estimate.

Drift across several lots

Because each verification compares a new lot with the one before, a series of small shifts in the same direction can pass every check and still move results a long way. Moving averages of patient results over several lot changes can reveal that drift, though they need software, careful tuning, and a stable patient population [Thompson and Chesher, 2018]. Collaborative verification between laboratories and patient-based monitoring are likely to improve detection further [Loh et al., 2023]. Keeping every lot comparison in one record, with the difference at each limit, makes the running total visible.

The nearest public example

Public data

We have no lot-change results of our own. No public data set we hold is a reagent or calibrator lot change. The nearest case is a change of creatinine chemistry on one analyser, from an enzymatic to a kinetic Jaffe method, in an open-access deposit [Schmidt et al., 2015]. It is a method change, not a lot change, but it shows the question a lot verification should answer: did the change move more patients across the limits than repeat testing alone would?