Guide
How many samples does a method comparison need? CLSI EP09 versus EP15
When a laboratory changes analyser it plans two experiments and usually sizes both from habit. This guide sets out what CLSI EP15 and EP09 are each for, going by their titles and public summaries, how many patient samples the comparison line needs, and how many pairs it takes before a count of the patients who changed category can tell you anything.
Two documents, two questions
EP15, User Verification of Precision and Estimation of Bias, is the short protocol a laboratory runs to confirm that a procedure performs here as its maker says. Its public summary describes a single experiment that fits in 5 working days and verifies precision and bias together, with the bias estimated from 25 or more measurements over 5 or more days on materials whose concentrations have been assigned [CLSI EP15]. The summary is open that the precision part verifies a claim rather than establishing one, and has limited power to reject a claim [CLSI EP15].
EP09, Measurement Procedure Comparison and Bias Estimation Using Patient Samples, is about the comparison itself: the same patient samples run on two procedures, inspected with difference and scatter plots, and summarised by bias estimates and regression, including Deming and Passing-Bablok, with bias at clinical decision points and confidence intervals for every parameter [CLSI EP09]. It is written for manufacturers as well as laboratories, and it leaves random error to EP05 and EP15 [CLSI EP09].
So the two are not alternatives. EP15 asks whether the new system is as precise as claimed and how far it sits from assigned values. EP09 asks how the new system’s results relate to the current one’s on your own patients’ samples. A laboratory changing analyser usually needs an answer to both.
EP15 and EP09 side by side
| EP15 | EP09 | |
|---|---|---|
| Question | Is the new system as precise as claimed, and how far is it from assigned values? | How do the new system’s results relate to the current one’s on patient samples? |
| Material | Materials with assigned concentrations, the same ones across days | Patient samples run on both systems |
| Size in the public summary | 5 working days; bias from 25 or more measurements | No number of samples stated |
| Gives you | Within-laboratory imprecision, which later sizes what repeat testing alone would move | Bias across the range and at each decision point, with intervals |
| Does not give you | Bias against your current system on your patients | Imprecision, which it leaves to EP05 and EP15 |
Read from the public CLSI product pages [CLSI EP15], [CLSI EP09]. The documents themselves are sold by CLSI; nothing here reproduces them.
How many samples for the line
Neither public summary gives a number of patient samples for the comparison. The figure most laboratories plan around comes from practice, and a sample-size study for regression-based comparisons put the conventional 40 to 100 samples in question. To detect one standardised slope deviation took 544 samples at a range ratio (highest result over lowest) of 2, and 64 at a range ratio of 10 [Linnet, 1999]. The spread of the samples matters more than their number, so cover the whole reporting range and add samples near each decision limit, where EP09 reads the bias [CLSI EP09]. The guide to method comparison regression covers the line and its checks.
That number is for estimating the line. Counting the patients a change moves across a decision limit is a different question, and it has a much larger answer.
How many pairs for a patient count
A change can pass every accuracy check and still move patients across a limit, because results crowd there. Counting the results that changed category, against the number repeat testing alone would move at the measured imprecision, is a published way to see it [Schmidt et al., 2015, Clin Chim Acta], and the guide to reclassification from imprecision and bias sets out the expected count. The planning question is how many pairs the count needs.
Public data
The crossing count tells a lab how many patients a method change moves across its decision lines; it is not a sensitive alarm. In our simulation on a public creatinine set, on a typical 20 to 60 sample verification it usually stayed quiet even when the method had shifted, and other analytes and patient groups will differ, so it is most useful on large comparisons; after go-live, a separate before-and-after test on routine results is the better check for a small shift.
In a simulation on the public creatinine distribution [Schmidt et al., 2015, PLoS ONE], at lines 1.2 and 1.5 mg/dL (5,000 runs per cell), a proportional shift of one times the assay’s CV at the line (about 1.8%) is flagged 2% to 4% of the time on 20 to 60 pairs, and a shift of two times the CV 6% to 14%. 80% power takes about 800 pairs for a two-CV shift and about 200 for three; for a one-CV shift, power is still 71% at 5,000 pairs. A quiet count on a verification set is no evidence either way.
| Shift at the line | 20 pairs | 40 pairs | 60 pairs | 120 pairs | 250 pairs | 500 pairs | 1,000 pairs |
|---|---|---|---|---|---|---|---|
| None (false alarm) | 1% | 2% | 2% | 3% | 4% | 5% | 6% |
| 1 × CV | 2% | 3% | 4% | 6% | 9% | 14% | 22% |
| 2 × CV | 6% | 10% | 14% | 25% | 45% | 72% | 94% |
| 3 × CV | 18% | 29% | 40% | 66% | 90% | 99% | 100% |
Simulation, not any laboratory’s results: a pure proportional shift with repeat noise at a CV of 1.82%, our recomputation from the study’s own precision workbook (the paper prints 1.8%), flagged by the exact test. Source distribution: Schmidt RL, Straseski JA, Raphael KL, Adams AH, Lehman CM (2015). A Risk Assessment of the Jaffe vs Enzymatic Method for Creatinine Measurement in an Outpatient Population. PLoS One. doi:10.1371/journal.pone.0143205 (opens in a new tab); PMC4657986 (opens in a new tab). Licence: CC BY 4.0 (opens in a new tab). Re-analysed by NakedSignal.
The report counts how many results changed category and compares that with what repeat testing alone would do. It is a measure of patient impact, not a sensitive test: in our simulation on one public creatinine data set, on 20 to 60 samples it usually did not flag a shift of one or two times the assay's CV, and other analytes and patient groups will differ. Treat a quiet count on a small set as no evidence either way. Reported with its interval, the count still describes patient impact honestly at any size, but at 20 to 60 pairs, for a one-CV shift, the 95% interval around the share of patients who changed category beyond what repeat testing alone would move is typically 8.6 to 16.1 percentage points wide.
The real creatinine change, resampled
A pure proportional shift is the simplest case. The real comparison behind the simulation, enzymatic against Jaffe creatinine on the same samples [Schmidt et al., 2015, PLoS ONE], moved 23 of its 529 results across the two lines against about 12 expected from repeat testing alone at the 1.8% imprecision the study measured. Drawing smaller verification sets from those real pairs (10,000 draws per size, at a 1.8% CV), the count flags the change in 17% of sets of 30, 27% of sets of 60, 38% of sets of 100 and 93% of sets of 529 (the study’s full size). These rates are for the paired count on sets drawn from the study’s own pairs; the go-live figure below is for a different test, on routine results. Where nothing changed, false alarms stayed at or below 5%. A quiet result on a small set is not reassurance.
After go-live: routine results
For a small shift the pairs from a verification will rarely be enough, but the patients tested after go-live are many. In the same simulation study, a different test, on unpaired routine results rather than the paired count above, detects a one-CV shift: compare routine patient results before and after go-live. With 500 creatinine results a day, 14 days of daily means detect it 93% of the time if the patient mix is stable day to day (56% if daily means wander by 2%), with false alarms at 5%, against 28 days before the change.
That comparison is between different patients before and after, so it can detect a shift but cannot tell a method change from a change in who is tested, and it cannot name the patients who moved. It complements the count on paired samples rather than replacing it. How much the daily patient mix wanders was assumed in the simulation, not measured; a laboratory’s own history would set it. The change check on this site does not monitor routine results.
A plan that answers both questions
- Precision first, on the new system. An EP15-style experiment over several days [CLSI EP15] gives the within-laboratory CV that sizes what repeat testing alone would move. A same-run CV from duplicates understates it.
- Size the patient comparison for the line, spread across the reporting range with extra samples near each decision limit [Linnet, 1999], and report bias at each limit with its interval [CLSI EP09]. The guide to bias at decision limits shows how.
- Report the patient count as a description, not a verdict. Give the results that crossed each limit, what repeat testing alone would move, and the interval. On a typical verification set a quiet count is not a finding that nothing happened.
- For a high-volume test, plan the first weeks after go-live, comparing routine results against the weeks before. The analyser change checklist puts the steps in order.
None of this replaces the laboratory’s own verification or its director’s judgement, and nothing on this page is regulatory advice.
Run it on your own pairs
The change check reads the paired results from your comparison in the browser. For one analyte, free, it gives the bias at each decision line you set and the results that crossed each line against those repeat testing alone would move. The public example runs the full report free. Your file is not uploaded.
Sources
- Clinical and Laboratory Standards Institute (2014). EP15: User Verification of Precision and Estimation of Bias. CLSI guideline, third edition, reaffirmed 2019. clsi.org/shop/standards/ep15/
- Clinical and Laboratory Standards Institute (2018). EP09: Measurement Procedure Comparison and Bias Estimation Using Patient Samples. CLSI guideline, third edition. clsi.org/standards/products/method-evaluation/documents/ep09/
- Linnet K (1999). Necessary sample size for method comparison studies based on regression analysis. Clinical Chemistry. doi.org/10.1093/clinchem/45.6.882
- Schmidt RL, Straseski JA, Raphael KL, Adams AH, Lehman CM (2015). A Risk Assessment of the Jaffe vs Enzymatic Method for Creatinine Measurement in an Outpatient Population. PLoS ONE. doi.org/10.1371/journal.pone.0143205
- Schmidt RL, Kordy MA, Genzen JR, Straseski JA, Greene DN, Lehman CM (2015). A mathematical procedure to estimate the impact of a change in method on discordance or misclassification at a decision limit in laboratory method comparison studies. Clinica Chimica Acta. doi.org/10.1016/j.cca.2014.10.043