Understanding the Laboratory Proficiency Testing Cycle - Part 3

Understanding the Laboratory Proficiency Testing Cycle (External Quality Assessment – EQA) 

This presentation is part 3 of a series highlighting Proficiency Testing as a Laboratory Quality Partner. 

 This series is put together by Dr. Michael A Noble BA MD FRCPC,  the material founder and now retired chair of CMPT – Canadian (formerly Clinical) Microbiology Proficiency Testing Program. It is based on his national and international experience and expertise in Proficiency Testing (PT/EQA). Dr Noble continues as a member on the advisory committee of CMPT and collaborates with other EQA programs including Oneworld Accuracy

 In 1946 F. W. Sunderman returned to the United States after serving in the military to find that community confidence in medical laboratories was so low that it became common practice to send samples to multiple laboratories and mean results rather than trust the results of any single laboratory. 

He was so concerned that he set up the first Quality Assessment study for common chemistry analysis in medical laboratories. He learned, with concern, the community was right in their concern on laboratory reliability 

(see: Belk WP, Sunderman FW. Am J Clin Pathol. 1947 Nov;17(11):853-61). 

Shortly after, the College of American Pathologists began its national Proficiency Testing program. Much has changed in medical laboratories over the intervening 75 years, but as long as laboratorians test and report on samples, there is a continuing need for monitoring performance using Proficiency Testing, also referred to as External Quality Assessment (EQA) laboratories regardless of industry. It is specified in both ISO/IEC 17025:2017 (General requirements for the competence of testing and calibration laboratories) and in ISO15189:2012* (Medical laboratories — Requirements for quality and competence).

*Note: ISO Standards 15189 and 17043 are about to be republished as :2023  The new standards are not currently available

PT/EQA process of quality assessment is also a subject to accreditation published as ISO/IEC17043:2010* (Conformity assessment — General requirements for proficiency testing).   I suspect that the process of assessment has not fundamentally changed substantially since Sunderman’s first early works, but it is now much better defined. 

To better visualize the process I have created a cycle graphic 

From my perspective PT/EQA must keep in mind the Eight Positions of a performance cycle. I have named these positions by the key person or persons holding that position, starting with the Organizer and ending with the Quality Team.  The graphic indicates this is a cycle that repeats with each test challenge and event.

These positions do not work in isolation silos; they work in teams, depending on the task.  I see four teams:

Team 1: The Quality Team
Team 2: The Preparation Team
Team 3: The Distribution Team
Team 4: The Assessment-Education Team, and then back to Team 1.

Depending on the size and complexity of the organization, each position may represent a unique person or group. In a smaller organization, especially if they are just getting started, they may well find that number of people involved is fewer, although I doubt that any group would be able to succeed with fewer than four.   In the olden days, it was possible to start with little specific knowledge about the process of EQA and learn as you go. I doubt that is the case any longer. 

 PT/EQA cycle. 

Team 1- The Quality Team.  Organizer and Quality
The Organizer

As with all things laboratory, the PT/EQA Cycle begins and ends with Quality. (Indeed Quality is integral throughout). Among many activities, the Organizer is the person responsible for setting the mission and vision and quality policies of the program. The organizer sets the annual goals and objectives, and reviews quality performance and has the sign-off responsibility for the program Quality Manual.

The Organizer is the one responsible for the program's Annual Report, including the review of non-conformances and opportunities for improvement.  They report on if the organization met its expected goal and objectives, and presents its expectations for coming year.   The Organizer is, in ISO terms, the lead of Top Management. If the Organizer is not Quality engaged, the program will ultimately fail. 

Quality

At the end of every test event the Quality Team needs to review not only the results of laboratory performance, but even more importantly, the performance of the EQA scheme.

Did the samples meet the expectations of scheme? Did they meeting homogeneity and stability requirements? Did the get sent at the right time and did they go to the right places? Did they make clear the contents with respect to instructions and safety? Was the expected date for reporting clear? Were the results reviewed prior to sending out. Did the scheme receive any complaints, concerns, or comments or compliments on the results? Were any-and-all addressed as required? Did the results of this test event lead to a concern about any suppliers, or about any staff members? Were there any opportunities for improvement identified, and if so was an investigation pursued? Did this initiate an internal audit, and if so was it completed and reviewed?

Does the Quality Team and the Organizer, have their eye on their own Accreditation status? Are they in-step with their own requirements? Are they ready to consider whether or not they want, need or require to taking their first important step towards their own Accreditation? 

PT/EQA has a position of incredible power and influence that can impact on laboratories reputation and accreditation. It holds that position because of its close attention to its own quality status. 

The next post will highlight
Team 2: Organizer- Planner- Research&Development- Production

 

 

The “Guiding Principles” of External Quality Assessment – PT/EQA

 

                               

This presentation is part 2 of a series highlighting 
Proficiency Testing as a Laboratory Quality Partner.

This series is put together by Dr. Michael A Noble BA MD FRCP the material founder and now retired chair of CMPT – Canadian (formerly Clinical) Microbiology Proficiency Testing Program.  The content of the series iis based on national and international experience and expertise in both providing and teaching about Proficiency Testing (EQA). 
Dr Noble continues to serve on the advisory committee of CMPT and collaborates with Oneworld Accuracy.

The International Organization for Standardization (ISO) includes in one of its essential standards ISO9001:2015 (Quality Management) a list of what is calls the Principles of Quality Management that includes: customer focus; leadership;  engagement of people; process approach; improvement; evidence-based decision making; relationship management.

In my opinion citing of Principles should be an essential part of every standard.  With disappointment, ISO does not include similar preambles to either ISO15189: 2012 (Medical Laboratories – Requirements for quality and competence) or ISO17043:2010 (Conformity Assessment – General Requirements for Proficiency Testing). 

The later document, with particular interest in Proficiency Testing (PT/EQA) does however contain a preamble that speaks about a series of what it calls ten “typical purposes for interlaboratory comparisons”.  I have not included the full list here, but I strongly encourage readers to check the Introduction section of the standard.

I agree that in the widely diverse universe of laboratory testing, PT/EQA can have a wide number of “purposes”.  But within that list of purposes there are some that go beyond purpose and are core to what we do.  I argue they could and indeed should be separated out as more than “purposes” and would be better defined as “guiding principles”, which I think are qualities based on key values that help businesses and customers make beneficial decisions and create a distinct culture.

I don’t know if this list is going to be seen in the next version of ISO17043 (2023), but in my opinion I think they are worth consideration, even if they are not included going forward.  The list includes:.

A.   Evaluation of the performance

·       The evaluation of processes including stability and reliability of reagents, supplies, equipment and the environment is readily covered in the laboratory by Quality Control and internal audits. 
The external evaluation of performance can be done by direct observation by accreditation bodies but can only be done at scale through PT/EQA. 

B.   Identification of problems and initiation of actions for improvement

·       Highlighting problems  (non-conformances or opportunities-for-improvement) is done, not in search of penalties, but to light errors that potentially can harm customers.  
PT/EQA clients and providers alike should understand that laboratories that have a perfect track record with EQA/PT may still have errors, but at a frequency below the detection limits of PT/EQA.  More importantly, laboratories having even a few deficiencies in their PT/EQA challenges is making many more errors that are not being identified.  Laboratories that investigate their EQA/PT including in clinical samples, often find the same errors have occurred before.

C.   Provision of additional confidence to laboratory customers

·       Let me tell you a story.  Increasingly laboratories are putting their accreditation certificates out in the front lobby of their institution where the public can see it, but their PT/EQA certificates hang on the walls in the laboratory.  We in the laboratory know that we are doing well; what we need to do is share that information with the clients that use the laboratories that we assess.  It is the customers who benefit from knowing that the laboratories are performing well.

D.   Education of participating laboratories based on the outcomes of such comparisons

·       Education is more than a purpose; without an educational foundation PT/EQA is only serves as an error detection system, which can prompt some facilities to finding opportunities to cheat and collude. 

·       Another story… in our program we include an educational narrative in all our reports.  It is usually not more that 400 words but provides perspective and learning principles.  In a number of surveys of clients over the years, the smaller laboratories without access to journals and conferences  identified our critiques as their most available and most important educational information for continuing education.

E.   Establishment of the effectiveness of test or measurement methods

·       I include this within my list of PT/EQA principles, but I can concede that not all would agree.  In fields where there are a variety of choices on measurement methods, PT/EQA can offer some unique opportunities for study.  When samples are both homogeneous and stable, and there is a strong informatics foundation for analyzing performance, and there is a certain homogeneity in the testing laboratory pool, there can be value in looking at test methodology on side-by-side comparisons.  Only PT/EQA studies can create information on studies done by multiple laboratories looking for a common answer.  This can be an important part of the educational value-added through PT/EQA.

 

In summary I argue there are some guiding principles that should be central to every PT/EQA program.  I applaud the original crafters of ISO17043 in identifying these. I only wish they had identified them as more than just “purposes”.

But while I am on the topic let me offer a sixth Principle which probably should not only be included, but should be first on the list…   Quality First.  I think this speaks for itself.

Understanding the "Why" of Laboratory Proficiency Testing

 

Understanding the Why of Laboratory Proficiency Testing
(External Quality Assessment – EQA)

This presentation is part 1 of a series highlighting  
Proficiency Testing as a Laboratory Quality Partner.

When a doctor or patient or family members is told that a laboratory has processed a sample and come to an answer or conclusion, the people involved usually assume that the answer the have been given is correct, and usually they are correct.

Unfortunately that is not always the case.   While most errors are of little consequence, sometimes the error can lead to harm to people and public health and can lead to complications and sometimes worse.   What studies indicate is that most errors are either not detected or are detected but not reported. 

What the world needs is a program that can monitor laboratories, detect errors help them learn and avoid bad outcomes. Fortunately that program exists and is known as Proficiency Testing or External Quality Assessment (EQA)

Proficiency Testing (EQA) is single most important method to ensuring that laboratories are capable to performing tests in a manner that will provide accurate, relevant, understandable information, in a timely manner. 

Other techniques such as Quality Control (QC) can ensure that the materials and reagents and equipment that are being used are working as expected, and QC can also ensure that the environment in which the tests are performed is safe and meets requirements, and the staff have the knowledge and understanding necessary to perform the testing. 

Measuring test performance is not easily addressed by QC.  

Test performance measurement can be done in two ways.  Accreditation bodies often perform direct observation which means an auditor watches a person perform a certain test.  If they do it consistent with the written procedure, the auditor records the test was done in conformance. 

EQA measures differently by looking at the result.  If the result is consistent with the expected result, it is deemed as acceptable; if it is not consistent then it can be rated as “essentially consistent” or “not consistent” or “not consistent and dangerous”.

EQA can determine if test performance results in deriving correct results at high frequency every year.  Accreditation body observers have few opportunities to watch people performing tests, maybe 2 or 3 once every year or two.   EQA monitors laboratories 3 or 4 times a year with a spectrum of tests resulting often in 80 to 100 observations every year. 

Both approaches can detect non-conformances or errors (or opportunities for improvement), but the more observations there are the higher the probability of detecting problems. 

Studies on laboratory errors indicate erors fall into two main clusters; one group that are "one-offs", the other that are more repetitive.  When EQA picks up an error, it may be difficult to say what type has been detected.  When one considers that laboratories can do thousands if not millions of tests every year, the EQA-to-Performed Tests ratio is always going to be very small. But when a laboratory has even just a few errors,  the statistics of probability strongly suggest that this is a laboratory having repetitive problems.     

So if a laboratory can participate in EQA and not have errors detected,  that does not mean the laboratory is  error-free; it means that the error frequency is sufficiently low and they were not picked up.  On the other hand, when EQA does pick up even a few errors what it assuredly means is that the laboratory is making LOTS of mistakes… the vast majority of which are remaining undetected.

So, when the quality focused laboratory is notified of an EQA error, the staff and management are not only obliged to  investigate that error to figure out why it happened and put in corrective actions, they ALSO must do a look back and see if they have made a similar errors on patient samples  in the past.  

Bottom line…

Laboratories are busy active places generating lots of important information, but all laboratories can make errors, some of which can cause harm.  Programs like External Quality Assessment (or EQA or Proficiency Testing or PT) exist to help laboratories become aware of errors so that they can focus on cause and correction.   

Consider your EQA program as one of your QUALITY PARTNERS.