Page 1

Page 1 of 9 Is there a QC Gap? Quality assurance Is there a QC Gap? Rate your practice against theory Jul 2007 Zoe C. Brooks & Carol Wambolt This essay is a follow-up to the 2006 series expressing concern that there is a widespread gap between QC theory and QC practice. Now we are giving you, the reader, the opportunity to rate your own QC practice and to help prove or disprove that gap. If the QC theories generally recommended by recognized QC experts are indeed the right way to perform laboratory quality control, the consequences of a significant gap between their theories and the typical laboratory practice pose potentially life-threatening consequences for patients. PROCESS We have prepared a survey that will help you see if you are performing QC the way the experts recommend – or if there is a gap between expert theory and your practice. Take the questionnaire below, then transfer your results into the attached Excel spreadsheet. Please e-mail your spreadsheet to Zoe Brooks so the data can be analyzed, anonymously of course. Please answer the questions truthfully – as reflected by your examination of the facts at this time (not what is planned or written or what you think should be done). Results will be accepted up to 30 days from the date of publication of this article. We will e-mail you a consolidated report of the study findings, so you can benchmark your performance against other participants. We hope to complete the data analysis, send reports back to you and have a discussion forum in place with survey results within 45 days of the closing date. QUALITY CONTROL – THEORY VS. PRACTICE The first essay in this series (Quality control in theory and practice – a gap analysis) compared performance-driven quality control (Fig. 1) (theoretical) to “typical” QC practice (Fig. 2). Read on; take the quiz to see where you rate! FIGURE 1: Theoretical QC practice 7/13/07 http://www.bloodgas.org/9D5EE2CE-0E8A-4B81-8548-CE39F181862F.W5doc?frames=no…

Page 2

Page 2 of 9 Is there a QC Gap? FIGURE 2: Typical QC practice The five steps of performance-driven quality control shown in Fig. 1 reflect the theory recommended by QC experts, as evidenced by the quotations from many well-known authors shown below. PD-QC theory step 1: Define performance standards; select candidate method(s); perform validation studies. Research and select an analytical process that will produce results at a rate of x to y per day, at a cost of x to y each, with accuracy of x % and precision of y % Define performance standards for each analyte: if a sample has a true value of x units, reported results must be within y units or z % of that value Expert theories generally agree on requirements for method selection and validation, including the application of performance standards. Other critical method factors or characteristics, such as cost/test, specimen types, specimen volumes, time required for analysis, rate of analysis, equipment required, personnel required, efficiency, safety, etc., must be considered during the selection of the analytical method. [Westgard ] [1] For about 40 years, there has been a steady stream of publications concerned with the generation and application of quality specifications. There appeared to be a real conflict about how to set quality specifications, but a decisive recent advance was that a consensus was reached in 1999 on global strategies to set quality specifications in laboratory medicine. The hierarchy is shown below: 1. Assessment of the effect of analytical performance on specific clinical decision-making quality specifications in specific 7/13/07 http://www.bloodgas.org/9D5EE2CE-0E8A-4B81-8548-CE39F181862F.W5doc?frames=no…

Page 3

Page 3 of 9 Is there a QC Gap? TABLE 1: PD-QC theory step 1 PD-QC theory step 2: Method performance (e.g. accuracy, precision, total error) meets performance standards? Ensure performance standards are met before reporting any patient results TABLE 2: PD-QC theory step 2 PD-QC theory step 3: Design/modify and implement daily QC processes to detect significant change Select QC samples. Select QC samples with analyte levels that monitor clinical decision points Verify that QC samples react the same way as patient samples (i.e. changes in accuracy or precision of patient results are reflected by proportional changes in accuracy or precision in QC sample results) Select a QC strategy clinical situations 2. Assessment of the effect of analytical performance on general clinical decision-making 3. Professional recommendations 4. Quality specifications laid down by regulation or by external quality assessment scheme (EQAS) organizers 5. Published data on the state of the art [Fraser] [2] The experts have recommended application of performance standards (TEa limits) to method validation and quality control since the 1960s. One of the first recommendations for establishing quality standards was published by Tonks in 1963; these standards were presented in the form of allowable total errors. [3] In 1970, Cotlove et al utilized within- subject biological variation to derive standards for allowable SDs. [4] A few years later in 1974, Westgard et al formulated criteria, based on the medical usefulness of test results, which could be used to decide whether an analytical method has acceptable precision and accuracy. [5] Expert theories have recommended choosing QC rules by comparing method accuracy and precision to defined performance standards for decades. The laboratory must establish the number, type and frequency of testing control materials, using the performance specifications established by the laboratory as specified in §493.1253(b)(3). [CLIA regulations] [6] We rely on controls to behave the same way as patient samples and detect errors in the analytical system. The ideal control material will: 1. Mimic the matrix and viscosity of patient specimens 2. Be both physically and chemically 7/13/07 http://www.bloodgas.org/9D5EE2CE-0E8A-4B81-8548-CE39F181862F.W5doc?frames=no…

Page 4

Page 4 of 9 Is there a QC Gap? consisting of frequency of testing, QC rules and processes to create and examine QC charts. Choose a QC strategy that will always detect changes that would cause any results to fail to meet the defined performance standard. Calculate the mean as an indicator of accuracy Calculate the SD or CV as an indicator of imprecision Assign the calculated mean and SD on the QC chart Calculate the critical systematic error (SEc) Select appropriate control strategy based on SEc TABLE 3: PD-QC theory step 3 PD-QC theory step 4: QC flags or chart examination indicate change in accuracy or precision of daily QC? Test QC samples. Plot all results on QC charts. Apply rules. Examine charts. If there are no QC flags, report patient results TABLE 4: PD-QC theory step 4 PD-QC theory step 5: Compare to performance standards; validate data; as sensitive to changes in the analytical system as patient samples 3. Contain all the analytes the laboratory measures in a given department 4. Contain concentrations of analytes at or near medical decision limits [Plaut] [7] “The appropriateness of an optimal QC strategy seems very much to depend on the quality required, as well as the expected instability of the analytical method (e.g., type, magnitude and frequency of errors).” [Wandrup ] [8] “The act of establishing your own means and SDs brings in the performance specifications observed in your own laboratory.” [Carey] [9] “A laboratory metric that has been used to select and design QC procedures is the critical systematic error, SEc, which can be calculated from the tolerance or quality requirement defined for the test and the imprecision and inaccuracy observed for the method […] The implication is that the error detection capability of the QC procedure should complement the performance capability of the process. With high process capability, the errors that cause defective results will be large and more easily detected by statistical quality control. As process capability decreases, the errors that must be detected get smaller, which requires better detection capabilities for the statistical QC procedure.” [Westgard] [10] The experts agree that a well- designed QC chart and strategy will identify change in an analytical system. (Such change must be evaluated before patient results are reported.) If none of the rules […] are violated, accept the run and report patient results. If any one of the rules […] is violated, the run is out-of-control; do not report patient test results. [Westgard] [11] The very premise of quality control states that if change in method performance is 7/13/07 http://www.bloodgas.org/9D5EE2CE-0E8A-4B81-8548-CE39F181862F.W5doc?frames=no…

Page 5

Page 5 of 9 Is there a QC Gap? improve or replace method if necessary If QC flags indicate that the accuracy or precision of the method has changed, compare the mean and SD of the current data population to your performance standards If the changed analytical process still produces results within allowable limits of the correct/true value, adjust the QC process and carry on If your method no longer meets performance standards, then stop reporting results while you make sure the numbers are correct and take corrective action if indicated TABLE 5: PD-QC theory step 5 unacceptable (no longer meets performance standards) you should stop reporting patient results. Results obtained for the performance characteristics (mean and SD) should be compared objectively to well- documented quality specifications, e.g., published data on the state of the art, performance required by regulatory bodies such as CLIA `88, or recommendations documented by expert professional groups. In addition, quality specifications can be derived from analysis of performance on clinical decision-making. [Fraser] [12] “To evaluate laboratory performance, the analytical imprecision and bias (obtained from the internal quality control protocol) is compared against the quality specifications (standards) for these two components of analytical error [accuracy and precision]. The analytical procedures that deviate from the standards have to be reviewed by laboratory professionals, and processes for improving performance implemented.” [Ricos] [13] READY TO RATE YOUR QC PRACTICE VS. EXPERT THEORY? DO YOU HAVE A GAP? The following quiz will let you compare your QC practice to the theories presented above. Enter a number to indicate where your practice rates on a scale from the example described for a value of 0.0 to the example for a value of 5.0. Mark an “X” if this question does not apply or you do not know. Mark an “E” if you feel that your processes are equally effective but not the same as our example of a rating of 5. If you complete this quiz in the attached Excel file, the spreadsheet will add your score and provide an interpretation. These are the 15 steps involved in performance- driven quality control: When selecting an analytical process, compare claims of accuracy and precision. These claims form an important component of the selection process. Define performance standards for each analyte that state “if a sample has a true value of x units, reported Score each step, where a value of “0.0” is similar to this example… We do not include accuracy and precision on our criteria list. Never. We do not have performance standards or TEa limits defined for any analyte. “This is a good lab. We are good people. Score a value of “5.0” if your practice is similar to this example… Accuracy and precision are the most important items on our criteria list. We set performance standards for all tests. Enter your ra- ting here 7/13/07 http://www.bloodgas.org/9D5EE2CE-0E8A-4B81-8548-CE39F181862F.W5doc?frames=no…

Page 6

Page 6 of 9 Is there a QC Gap? results must be within y units or z % of that value.” Ensure performance standards are met before reporting any results. Select QC samples with analyte levels that monitor clinical decision points. Verify that QC samples react the same way as patient samples (i.e. changes in accuracy or precision of patient results are reflected by proportional changes in accuracy or precision in QC sample results). Periodically calculate the mean of a defined set of QC points as an indicator of accuracy. Periodically calculate the SD or CV of a defined set of QC points as an indicator of imprecision. Assign the current calculated mean and SD on the QC chart. Periodically calculate the margin for error or critical systematic error (SEc) for each QC sample. Select appropriate No. We use Yes. We Our methods are all good.” Never. Always. No. We just accept what the manufacturer provides. No. We never verify changes in QC with patient samples, or monitor patient values. We don’t check for this when purchasing new controls. No. We use a running mean. We don’t examine or assess mean values on a regular basis. No. We use a running SD. We don’t examine or assess SDs or CVs on a regular basis. No. We assign a mean from history or the package insert or whatever. and/or… “The SD assigned on the chart is not the actual method SD; it comes from PT limits, or package inserts or we just multiply it a few times so we don’t get false QC flags.” Never. Regularly. Yes. If necessary, we purchase separate controls or make samples. Yes. We always verify changes in QC with patient samples, and monitor patient values or moving averages. We check new QC samples for this before purchasing. Yes. We calculate, examine and assess mean values on a regular basis. Yes. We calculate and examine and assess SDs or CVs on a regular basis. Yes. We always assign the current calculated mean and SD on the QC chart.We rely on our QC strategy to alert us to change. When a shift in the mean occurs, we update the mean on the QC chart (after making sure the system still meets performance standards). 7/13/07 http://www.bloodgas.org/9D5EE2CE-0E8A-4B81-8548-CE39F181862F.W5doc?frames=no…

Page 7

Page 7 of 9 Is there a QC Gap? control strategies (frequency of testing, QC rules, and processes to create and examine QC charts) for each QC sample on each analyte based on margin for error (SEc). Choose a QC strategy that will detect changes that would cause results to fail to meet the performance standards defined for each QC sample. Plot all results on QC charts. Apply rules, examine charts, and report patient results only if there are no QC flags. If QC flags indicate that the accuracy or precision of the method has changed, compare the mean and SD of the current data population to performance standards. If the changed analytical process still produces results within allowable limits of the correct/true value, adjust the QC process and carry on. whatever QC software comes with our instrument or LIS or QC samples. We never compare QC results to performance standards. Or … “Someone (who cannot be questioned) decided to use a 1- 2s or 1-3s rule for all controls on all tests.” Never. We don’t plot results. No. We report results and then someone examines the QC later. Or, we report all results – QC flags don’t make us stop reporting. The doctors need the results. If we start getting a new mean, then that must be what the control should be now. Or ..“Change is OK if the supervisor says so.”Or ..“If you can explain the change, it’s OK.”Or ..“If the change is “not too big”, it’s OK.” Sort of. We change the values on the chart whenever we start getting a new mean or SD. Then the chart looks better. Or .. No. Whenever there is a change we call for technical support. “All change is bad. It must be proactively select QC strategies and implement performance- driven quality control. We regularly compare QC results to performance standards and adjust the QC process if the method performance changes (as noted by a change in mean or SD of a new data set.) Always. All values. Yes. We always check QC before reporting patients.We never report results on runs with QC rejects until the cause of the flag has been determined and we are sure the method still meets performance standards. If we start getting a new mean or SD, then we calculate Total Error and SEc to make sure the method is within allowable error. If the method is within TEa, we change the mean or SD on the QC chart and, if advisable, adjust the QC rules and process. We realize change can be for the better – change is not always bad. 7/13/07 http://www.bloodgas.org/9D5EE2CE-0E8A-4B81-8548-CE39F181862F.W5doc?frames=no…

Page 8

Page 8 of 9 Is there a QC Gap? If the method no longer meets performance standards, then stop reporting results while you make sure the numbers are correct and take corrective action if indicated. TABLE 6: 15 steps involved in performance-driven quality control eliminated.” No. We never stop reporting results. The doctors need the results. Yes. We never release results that may be wrong, and therefore lead the clinician to the wrong decision and subsequent action, thus harming the patient. If you scored a perfect 5 on each question, your total score would be 75. Did you find a gap? Please enter your rating in the attached Excel file and e-mail the completed file with the quiz and your comments so we can assess the overall existence and magnitude of the gap. Your privacy is 100% assured. CONCLUSION We hope that you will help prove or disprove and take the first steps toward closing this gap – by taking the time to submit this survey. We have talked with many people who agree that this gap between QC theory and practice exists. The four prior essays ended with the words “Once again, these are my observations, and I truly hope that many of you will stand up and prove me wrong.” Not one person has stood up. We realize that we may be opening a Pandora’s box. But if this gap does indeed exist, then it poses a danger to patient care, and it should be closed. REFERENCES 1. Westgard JO. Method validation – real world applications. http://www.westgard.com/qcapp14.htm 2. Fraser CG. Biological variation and quality for POCT. www.bloodgas.org, Quality assurance, Jun 2001 3. Tonks, DA. A study of the accuracy and imprecision of clinical chemistry determinations in 170 Canadian laboratories. Clin. Chem. 1963;9:217-33. 4. Cotlove E, Marris E, Williams G. Biological and analytical components of variation in long-term studies of serum constituents in normal subjects: III. Physiological and medical implications. Clin. Chem. 1970;16:1028- 32. 5. Westgard JO, Carey NR, Wold S. Criteria for judging precision and accuracy in method development and evaluation. Clin. Chem. 1974;20:825-33. 6. CLIA regulations §493.1245 Standard: Control procedures 7. Brooks, Z. Quality control – from data to decisions. Basic concepts, troubleshooting, designing QC systems. Educational Courses. Zoe Brooks Quality Consulting. 2003, (Chapter 2 Course 1 “What should we look for in QC material” contributed by David Plaut) 8. Wandrup JH. Quality control of multi-profile blood gas testing: Key notes regarding simple quality assurance planning. www.bloodgas.org, June 2001. 9. Carey NR. Tips on managing the quality of immunoassays. http://www.westgard.com/guest4.htm#bio 10. Westgard JO. Six sigma quality management and requisite laboratory QC. http://www.westgard.com/essay36.htm 11. Westgard JO. QC - the multirule interpretation. http://www.westgard.com/lesson18.htm 12. Fraser CG, Hyltoft-Petersen P. Analytical performance characteristics should be judged against objective 7/13/07 http://www.bloodgas.org/9D5EE2CE-0E8A-4B81-8548-CE39F181862F.W5doc?frames=no…

Page 9

Page 9 of 9 Is there a QC Gap? quality specifications .Clin. Chem. 1999;45:321-23. 13. Ricos C, Alvarez V, Cava F, Garcia-Lario JV, Hernandez A, Jimenez CV, Minchinela J, Perich C, Simon M. Analytical Quality Commission of the Spanish Society of Clinical Chemistry and Molecular Pathology (SEQC). Biological variation database, http://www.westgard.com/guest17.htm Other references: 1. Brooks Z, Wambolt C. Use some horse sense with QC. MLO March 2007 http://www.mlo- online.com/articles/0307/ 0307clinical_issues.pdf 2. Online quizzes and discussions at www.zoebrooksquality.com/harmonize 3. Westgard JO. Quality planning and control strategies. www.bloodgas.org, 2001. 4. Westgard JO. A six sigma primer. www.bloodgas.org, 2002 5. Westgard JO, Quam EF, Barry PL. Selection grids for planning QC procedures. Clin Lab Sci 1990; 3: 271-78 6. Fraser CG, Kallner A, Kenny D, Hyltoft-Petersen P. Introduction: strategies to set global quality specifications in laboratory medicine. Scand J Clin Lab Invest 1999; 59: 477-78. 7. Brooks, Z. Performance-Driven Quality Control. AACC Press, Washington DC, 2001 ISBN 1-899883-54-9 AUTHORS Zoe C. Brooks Zoe Brooks Quality Consulting 8070 Highway 17 West Worthington Ontario Canada P0M 3H0 zoe@zoebrooksquality.com Carol Wambolt 1540 Maple Avenue Twin Falls, Idaho 83301 cwambolt@cableone.net 7/13/07 http://www.bloodgas.org/9D5EE2CE-0E8A-4B81-8548-CE39F181862F.W5doc?frames=no…