How many results cross a decision limit by chance?

The question

When a laboratory changes analyser, reagent or method, the paired results show how many patients now fall in a different category. Some of them would have changed category on a simple repeat with nothing changed at all. The useful number is the difference between what was observed and what repeat testing alone produces.

The same arithmetic answers a question laboratories ask with no change in view: if imprecision rose, or a small bias crept in, how many patients would be classified differently? Simulation studies of fixed interpretive limits show the share of patients reclassified rising with both imprecision and bias [Loh et al., 2024].

One result near a limit

Take a result x, a decision limit L and the analyser’s repeat-test imprecision as a coefficient of variation (CV). A repeat measurement of the same sample varies around the first. Because the first result and the repeat both carry imprecision, the spread of the difference between them is larger than either alone by a factor of √2. Treating that difference as normally distributed:

spread of a repeat = √2 × CV × x

chance of crossing L = Φ(−|L − x| ÷ spread)

Φ is the standard normal distribution function. The chance depends on one ratio: how far the result sits from the limit, measured in units of that spread. A result exactly at the limit has an even chance of landing on either side. One spread away, the chance is about 16 in 100; two spreads away, about 2 in 100. Only results near a limit contribute much, and the CV sets what counts as near.

From one result to a count

Summing that chance over every result in the comparison gives the number of results expected to change category on repeat testing alone. Where a laboratory reports several limits, the chance for each result is the probability of leaving its own category, above or below. The sum is an expected value for the population of results, and it can be a fraction.

Two things drive it: how many results sit close to the limits, and how large the CV is there. A comparison in which most results cluster around a limit will show many crossings from imprecision alone, even with a perfectly matched new method. A raw count of crossings is never a verdict on the change by itself.

Separating the crossings seen at a decision limit into the part imprecision alone would give and the part due to the change is a published procedure (Schmidt et al., 2015, Clin Chim Acta), which the creatinine study we use applies. What we add is a program that runs it on a laboratory’s own comparison and precision data at its own decision limits, keeps the re-test list in the laboratory and releases files designed to carry only aggregates. [Schmidt et al., 2015, Clin Chim Acta]

Bias against imprecision

Imprecision scatters results in both directions, so its crossings roughly cancel in the totals above and below a limit. Bias moves every result the same way. A new method that reads a few percent lower pushes all results just above a limit to just below it, and those moves add up in one direction.

In simulation, the effect of imprecision on reclassification lessens as bias grows, and at large biases bias dominates [Loh et al., 2024]. In practice a change with little bias shows crossings close to the repeat-testing baseline, while a larger bias shows an excess mostly in one direction. Counting crossings separately for each limit and each direction makes that visible.

Measured or assumed imprecision

The expected count is only as good as the CV behind it. A larger CV widens the spread, so more results count as near a limit and the baseline rises; a smaller CV lowers it. An assumed CV that is too large makes a real change look like noise, and one that is too small makes noise look like a change.

Where the laboratory has replicate measurements of patient samples, or duplicate or quality-control data at concentrations near its limits, take the CV from those. Otherwise use a stated default. CV varies with concentration, so the value at the limit matters more than one figure for the whole range.

How this relates to reference change values and performance specifications

Reference change values. The reference change value asks whether two results from one patient differ by more than analytical and within-subject biological variation would explain. Its formula carries the same √2 factor, because it too compares two results that each carry variation [Fraser, 2011]. Estimates of within-subject variation for many measurands are collected in the [EFLM Biological Variation Database]. The baseline here uses analytical imprecision only, because the question is what a repeat of the same sample would do, not what the patient would do over time.

Analytical performance specifications. The 2014 consensus of the European Federation of Clinical Chemistry and Laboratory Medicine sets out three models for deciding how good a method must be: the effect of analytical performance on clinical outcomes, biological variation, and the state of the art [Sandberg et al., 2015]. The first model includes indirect outcome studies of the effect of performance on clinical classifications, for example by simulation. Counting results that change category against the repeat-testing baseline is a direct, local version of that question, run on the laboratory’s own paired results rather than on a simulated population.

Worked examples on public data

Public data

We ran open-access method comparisons as laboratory exports. Fig. 1 shows four of the 34 comparisons, chosen to show how much the imprecision used decides the reading. The first is the public CD4 sample, where 535 results changed category against about 238 expected from repeat testing, with a default imprecision [Coetzee and Glencross, 2017].

In the second, two creatinine chemistries run on the same samples on one analyser and read as a switch from enzymatic to Jaffe, 23 results changed category against about 12 expected from repeat testing [Schmidt et al., 2015, PLoS ONE]. That expected count uses the 1.8% imprecision the study itself measured; at that imprecision the count from repeat testing alone falls between 6 and 17 with 95% probability, so the 23 crossings are about twice the expected count, and by the exact Poisson-binomial test 23 or more from repeat testing alone has probability 0.00017. That 1.8% is also our program's laboratory-analyser default (1.79%, this study's own reported precision), so the default gives about the same count; the program's cautious point-of-care default of 6%, which it uses when a point-of-care device is involved and no precision data are supplied, would give about 41, which overstates repeat-testing noise for these laboratory analysers. At the upper end of the study's own precision estimate, 2.34%, about 15 would be expected and 23 or more has probability 0.014. A laboratory’s own imprecision decides its baseline.

The last two are haemoglobin meters compared with a laboratory analyser in a community clinic, the only public series we hold with imprecision measured from replicate patient samples [Jaggernath et al., 2016], data [Gelderblom, 2016]. With similar numbers of crossings (36 and 37), one meter sits far above its repeat-testing baseline and the other below it. The baselines differ because the imprecision measured for each meter differs. The crossing count alone would have ranked them the same.