How many results cross a decision limit by chance?

The question

When a laboratory changes analyser, reagent or method, the paired results tell it how many patients now fall in a different category. That count alone does not say what the change did, because some of those patients would have changed category on a simple repeat, with nothing changed at all. The useful number is the difference between what was observed and what repeat testing alone would produce.

The same arithmetic answers a question laboratories ask without any change in view: if imprecision rose, or a small bias crept in, how many patients would be classified differently? Simulation studies of fixed interpretive limits show the share of patients reclassified rising with both imprecision and bias [Loh et al., 2024].

One result near a limit

Take a result x, a decision limit L and the analyser’s repeat-test imprecision as a coefficient of variation (CV). A repeat measurement of the same sample varies around the first. Because the first result and the repeat both carry imprecision, the spread of the difference between them is larger than either alone by a factor of √2. Treating that difference as normally distributed:

spread of a repeat = √2 × CV × x

chance of crossing L = Φ(−|L − x| ÷ spread)

Φ is the standard normal distribution function. The chance depends on one ratio: how far the result sits from the limit, measured in units of that spread. A result exactly at the limit has an even chance of landing on either side. One spread away, the chance is about 16 in 100; two spreads away, about 2 in 100. So only results near a limit contribute much, and how near counts as near is set by the CV.

From one result to a count

Summing that chance over every result in the comparison gives the number of results expected to change category on repeat testing alone. Where a laboratory reports several limits, the chance for each result is the probability of leaving its own category, above or below. The sum is an expected value, not a prediction for any one patient, and it can be a fraction.

Two things drive it: how many results sit close to the limits, and how large the CV is there. A comparison in which most results cluster around a limit will show many crossings from imprecision alone, even with a perfectly matched new method. That is why a raw count of crossings is not a verdict on the change.

Bias against imprecision

Imprecision scatters results in both directions, so its crossings roughly cancel in the totals above and below a limit. Bias moves every result the same way. A new method that reads a few percent lower pushes all results just above a limit to just below it, and those moves add up in one direction.

In simulation, the effect of imprecision on reclassification lessens as bias grows, and at large biases bias becomes the dominant cause [Loh et al., 2024]. In practice this means a small change with little bias usually shows crossings close to the repeat-testing baseline, while a larger bias shows an excess that is mostly in one direction. Counting crossings separately for each limit and each direction makes that visible.

Measured or assumed imprecision

The expected count is only as good as the CV behind it. A larger CV widens the spread, so more results count as near a limit and the baseline rises; a smaller CV lowers it. An assumed CV that is too large makes a real change look like noise, and one that is too small makes noise look like a change.

Where the laboratory has replicate measurements of patient samples, or duplicate or quality-control data at concentrations near its limits, the CV should come from those. Where it does not, a default can be used, and the result should say so. CV also varies with concentration, so the value at the limit matters more than a single figure across the whole range.

How this relates to reference change values and performance specifications

Reference change values. The reference change value asks whether two results from one patient differ by more than analytical and within-subject biological variation would explain. Its formula carries the same √2 factor, because it too compares two results that each carry variation [Fraser, 2011]. Estimates of within-subject variation for many measurands are collected in the [EFLM Biological Variation Database]. The baseline here uses analytical imprecision only, because the question is what a repeat of the same sample would do, not what the patient would do over time.

Analytical performance specifications. The 2014 consensus of the European Federation of Clinical Chemistry and Laboratory Medicine sets out three models for deciding how good a method must be: the effect of analytical performance on clinical outcomes, biological variation, and the state of the art [Sandberg et al., 2015]. The first model includes indirect outcome studies of the effect of performance on clinical classifications, for example by simulation. Counting results that change category against the repeat-testing baseline is a direct, local version of that question, run on the laboratory’s own paired results rather than on a simulated population.

Worked examples on public data

Public data

We ran open-access method comparisons through our program as if a laboratory had exported them. Fig. 1 shows four of the 34 comparisons, chosen to show both outcomes: changes that moved more results than repeat testing alone would, and changes that moved fewer. The first is the public CD4 sample, where 535 results changed category against about 238 expected from repeat testing [Coetzee and Glencross, 2017]. The imprecision there is assumed, not measured.

In the second, a change of creatinine chemistry on one analyser, 23 results changed category against about 41 expected from repeat testing [Schmidt et al., 2015]. On these pairs, the change moved fewer results than a simple repeat would have, and the honest report is that there is nothing extra to manage at these limits.

The last two are haemoglobin meters compared with a laboratory analyser in a community clinic, the only public series we hold with imprecision measured from replicate patient samples [Jaggernath et al., 2016], data [Gelderblom, 2016]. With similar numbers of crossings (36 and 37), one meter sits far above its repeat-testing baseline and the other below it. The baselines differ because the imprecision measured for each meter differs. The crossing count alone would have ranked them the same.