Understanding the Laboratory Proficiency Testing Cycle - Part 3

Understanding the Laboratory Proficiency Testing Cycle (External Quality Assessment – EQA) 

This presentation is part 3 of a series highlighting Proficiency Testing as a Laboratory Quality Partner. 

 This series is put together by Dr. Michael A Noble BA MD FRCPC,  the material founder and now retired chair of CMPT – Canadian (formerly Clinical) Microbiology Proficiency Testing Program. It is based on his national and international experience and expertise in Proficiency Testing (PT/EQA). Dr Noble continues as a member on the advisory committee of CMPT and collaborates with other EQA programs including Oneworld Accuracy

 In 1946 F. W. Sunderman returned to the United States after serving in the military to find that community confidence in medical laboratories was so low that it became common practice to send samples to multiple laboratories and mean results rather than trust the results of any single laboratory. 

He was so concerned that he set up the first Quality Assessment study for common chemistry analysis in medical laboratories. He learned, with concern, the community was right in their concern on laboratory reliability 

(see: Belk WP, Sunderman FW. Am J Clin Pathol. 1947 Nov;17(11):853-61). 

Shortly after, the College of American Pathologists began its national Proficiency Testing program. Much has changed in medical laboratories over the intervening 75 years, but as long as laboratorians test and report on samples, there is a continuing need for monitoring performance using Proficiency Testing, also referred to as External Quality Assessment (EQA) laboratories regardless of industry. It is specified in both ISO/IEC 17025:2017 (General requirements for the competence of testing and calibration laboratories) and in ISO15189:2012* (Medical laboratories — Requirements for quality and competence).

*Note: ISO Standards 15189 and 17043 are about to be republished as :2023  The new standards are not currently available

PT/EQA process of quality assessment is also a subject to accreditation published as ISO/IEC17043:2010* (Conformity assessment — General requirements for proficiency testing).   I suspect that the process of assessment has not fundamentally changed substantially since Sunderman’s first early works, but it is now much better defined. 

To better visualize the process I have created a cycle graphic 

From my perspective PT/EQA must keep in mind the Eight Positions of a performance cycle. I have named these positions by the key person or persons holding that position, starting with the Organizer and ending with the Quality Team.  The graphic indicates this is a cycle that repeats with each test challenge and event.

These positions do not work in isolation silos; they work in teams, depending on the task.  I see four teams:

Team 1: The Quality Team
Team 2: The Preparation Team
Team 3: The Distribution Team
Team 4: The Assessment-Education Team, and then back to Team 1.

Depending on the size and complexity of the organization, each position may represent a unique person or group. In a smaller organization, especially if they are just getting started, they may well find that number of people involved is fewer, although I doubt that any group would be able to succeed with fewer than four.   In the olden days, it was possible to start with little specific knowledge about the process of EQA and learn as you go. I doubt that is the case any longer. 

 PT/EQA cycle. 

Team 1- The Quality Team.  Organizer and Quality
The Organizer

As with all things laboratory, the PT/EQA Cycle begins and ends with Quality. (Indeed Quality is integral throughout). Among many activities, the Organizer is the person responsible for setting the mission and vision and quality policies of the program. The organizer sets the annual goals and objectives, and reviews quality performance and has the sign-off responsibility for the program Quality Manual.

The Organizer is the one responsible for the program's Annual Report, including the review of non-conformances and opportunities for improvement.  They report on if the organization met its expected goal and objectives, and presents its expectations for coming year.   The Organizer is, in ISO terms, the lead of Top Management. If the Organizer is not Quality engaged, the program will ultimately fail. 

Quality

At the end of every test event the Quality Team needs to review not only the results of laboratory performance, but even more importantly, the performance of the EQA scheme.

Did the samples meet the expectations of scheme? Did they meeting homogeneity and stability requirements? Did the get sent at the right time and did they go to the right places? Did they make clear the contents with respect to instructions and safety? Was the expected date for reporting clear? Were the results reviewed prior to sending out. Did the scheme receive any complaints, concerns, or comments or compliments on the results? Were any-and-all addressed as required? Did the results of this test event lead to a concern about any suppliers, or about any staff members? Were there any opportunities for improvement identified, and if so was an investigation pursued? Did this initiate an internal audit, and if so was it completed and reviewed?

Does the Quality Team and the Organizer, have their eye on their own Accreditation status? Are they in-step with their own requirements? Are they ready to consider whether or not they want, need or require to taking their first important step towards their own Accreditation? 

PT/EQA has a position of incredible power and influence that can impact on laboratories reputation and accreditation. It holds that position because of its close attention to its own quality status. 

The next post will highlight
Team 2: Organizer- Planner- Research&Development- Production

 

 

Celebrating World Accreditation Day – 2022 (Plus…..!)

 

Celebrating World Accreditation Day – 2022 (Plus…..!)

When I first started this blog in June 2012, one of the topics that we said that we would discuss was Quality Partners.  I made a rookie mistake by using the phrase but didn’t define it.  We had been using the term in our Certificate Course as far back as 2008, but that was a closed system that few readers would have an opportunity to read.

What we described in the course and later here was that Quality Partners were organizations that worked with and around medical laboratories with the sole intent of helping laboratories develop and maintain their Quality Management Systems with the goals being error reduction and quality improvement. 

As best as I can tell we were the first and only to use this phrase in this context, and maybe still am.  The point is that the laboratory is surrounded by organizations intent of keeping Quality front of mind.  We include in this list:

• Standards Development Bodies
• Accreditation Bodies
• Proficiency Testing Bodies
• Educators
• Professional Organizations
• Equipment and Reagent Suppliers.

 

I return to this concept because every year at this time the International Laboratory Accreditation Cooperation (ILAC) recognizes and celebrates the importance of Accreditation in the process to improve laboratories and protect the public.

Having spent most of my career being involved in Quality Partnership (Standards Development, Accreditation, Proficiency Testing, Educator) I am aware that most laboratorians, even today, still think of Quality Partners more as adversaries.  They tend to see us and generally appreciate us as grudgingly helpful but more often as picky and intrusive.  The over-riding thought is “we don’t make mistakes”. 

And for the most part, they are right… most laboratories do a good job most of the time.  Until they don’t. 

What we have seen over the last few years has been all sorts of false positive and false negative COVID-19 testing errors.  (see:  https://cmpt.ca/covid-19-proficiency-testing-one-year-later/)  We see concerning errors in Proficiency Testing in Gram Staining and in Pre-examination knowledge (see: https://cmpt.ca/publications-newsletter/annual-report/).   The problem is that  wrong information goes out and can impact on patients and their healthcare workers when faulty decisions are made.

Without going into details, last year we saw an organization with little experience in proficiency sample put staff at considerable risk and harm from inappropriate use of the samples.  Had they thought to contact us rather than go directly to harm, we may have reduced anguish and concern.  We are there to prevent bad things from happening.

So from my perspective, I congratulate ALL our Quality Partner colleagues with particular reference to our Accreditation colleagues for remaining committed and helping laboratorians help themselves and protect the public. 

And ILAC, many thanks for shining the light.

But here is a thought.  Maybe the time has come to acknowledge not only the standards development organizations, and the laboratory accreditation organizations, but to formally recognize ALL the laboratory Quality Partners for their contributions for making healthcare safer.

 

Innovation and the Modern Medical Laboratory 2016

A number of years ago I was invited to give a presentation on Quality practices for the research laboratory.  I recommended at good practices would include (1) adherence to established standards (2) Calibrate your equipment regularly (2) strict adherence to quality control (3) ensure that graduate students are properly trained and competency assessed before performing assays (4) where feasible work with a partner laboratory or participate in existing applicable proficiency testing programs to ensure that your procedures are working properly and (5) on a regular basis arrange for some form of external assessment.  

A senior person at the presentation said that he was very successful in funding awards, and never did any of those steps.  

Is it any wonder that only about 10-30 percent of published research is reproducible (see: www.jove.com August 2013).

I raise this story because it relates to a recent threat on LinkedIn by the ASQ “What Do you think? Is Quality shifting to be the Driver of Innovation? .  I have previously commented that Quality is not so much the driver of Innovation, but may be making Innovation better.

Medical laboratories are largely populated by folks who are creative by nature, and always ready to try out something new.  Sometimes it resembles innovation by distraction by the shiny bauble.  But most of the time it is about a self-driven desire to tweak existing processes, as in “if I do “this”, I bet I can make this assay easier, or faster, or better.  

 

The reality is that usually, the assay is not improved, and sometimes it is made worse.  At the very least what happens all too often is that even critical basics of accountable change do NOT occur.  There is little or no verification that the changes result in consistent values, or validation that changes don’t have unsuspected impacts due to age, gender, medications, or geography.  A few studies get done, some basic statistics are generated, and off we go.  We have another new Laboratory Developed Test (LDT) that gets presented as an abstract at a meeting, or gets published in a journal, and everyone is happy, except maybe the clinician who gets a wrong test interpretation, or the patient that gets put on a wrong treatment track.  

 

Stories come to mind.  The first is the tragic story of misinterpreted test results for her2 (breast tumor responsiveness to the drug Herceptin) that resulted in patient mistreatment and death and led to the hugely expensive and public Cameron Commission in Newfoundland.  

 

Over recent years, medical laboratories have been galloping forward creating new tests to find new information, largely based on new molecular (i.e. DNA based) assays in the disciplines of genetics, and detection of microbial pathogens of all sorts.  

 

New laboratory ideas are being generated on a daily basis and new procedures are being created almost as quickly, most commonly using DNA strands that are created in-house.  Laboratories often do a side-by-side comparison of the information generated by the “standard” test, and if the new assay is pretty much in line with the standard assay, then the transition is made.  And if there was not previous assay, but the test worked for other targets, and it seems to work well here, then that is good enough evidence to go forward. The result of this is more tests, more results, new knowledge, and better and faster patient information. And some of the information is very good and very helpful. 

The problem of course is that sometimes results don’t quite go as planned. 

 

In our Quality course we talk about Quality Partners helping laboratories advance their Quality improvement.  But we point out that the impact of the partners on laboratory performance pales in comparison to the power of an unhappy public who gets fatigued with new ideas bad outcomes and demands something with a lot more rigor.  Today it is the US Food and Drug Administration (FDA) that has made it really clear that the day of free-reign LDTs is over, at least in the US.  The sheriff is going to be laying down some new rules.  If laboratories are not going to discipline themselves, the gov’ment is going to do it for them. 

Needless to say lots of laboratories are unhappy.  Definitions are being made that may or may not work well.  Restrictions are being imposed, and delays are inevitable. 

Gee whiz.  What did they think was going to happen?  When you upset the public enough, you have little control over what happens next.  

But just to be clear, medical laboratories are not the only organizations that have fallen into the rush to innovation trap.  Think Theranos. 

Without going too far into the story (you can read it everywhere and anywhere) this was a new-tech company that figured they could revolutionize all blood sample testing; less blood, easy collection, faster results, simpler testing, better than the best sliced bread.  Going to revolutionize and take over the laboratory industry.

If only it had worked, which it turns out it did not.  And now there are enough bad outcomes to bring in the gov’ment and the lawyers.  It is going to be a very ugly summer for Theranos.

 

There are tons of opportunity and need for research and improvement and innovation in the medical laboratory arena. But there are some basics (PDSA and basic quality management come to mind) that would increase the odds for success.  

There is always a need for new ideas and improvement, and some basic application of Quality Management would go a long way to move the needle in the right direction.

 

The State of North America Laboratory Quality 2015

The State of North America Laboratory Quality 2015

Earlier this week I was invited to participate in the 2015 edition of Robert Michel’s and Dark Daily’s annual Laboratory Quality Confab held in New Orleans.  I was the third speaker in a symposium on the challenges facing US laboratories.  The lead speaker was Ellen Gabler, a journalist with the Milwaukee Journal Sentinel who had written a number of articles on the questionable quality of what the Commission on Medicaid and Medicare Services (CMS) and the Clinical Laboratory Improvement Act (CLIA) refer to as unwaived tests, i.e. tests that are so easy to perform correctly they require no quality assessment, and on the state of laboratory inspection (accreditation).  [in a nutshell… big glaring problems abound with both].   

The next speaker was Elissa Passiment, the Executive Vice President for the American Society for Clinical Laboratory Science and importantly the head (former) of the Clinical Laboratory Improvement Advisory Committee (CLIAC), the body that reviews activities associated with CLIA and reports to Congress.  Elissa talked about the challenges that CLIA has posed, and continues to pose to US laboratories.  And as she put it CLIA has become a driver to the bottom rather than an inspiration to excellence (Those aren’t quite Elissa’s words, but certainly the sentiment!).

I talked about Quality Partners (a favorite subject of mine) [see: http://www.medicallaboratoryquality.com/2014/12/competition-and-quality-partner-dynamic.html ] , emphasising both their strengths and weaknesses and how laboratories need to progress through the development of Quality Progress Plans, basically a page out of the newly published crown document of organizational quality, ISO9001:2015. 

I thought the symposium went well, although my guess is that it was way over the heads of most of the audience.  While I give them all points for wanting to be at a laboratory quality conference, for 80 percent of the audience this was their first time attending.  More importantly, and without putting too much importance on show of hand displays during conference presentations,  when I asked how many worked in laboratories where they regularly performed internal audits or if they maintained an Opportunities For Improvement list, for both questions I got about a 10-15 percent response.  So their notions about quality are not really yet on the foundations of quality improvement.

Of interest to me, following my presentation, Robert Michel asked me in front of the audience with microphone in hand, how I felt about Canada being a country with provincial control over laboratory quality and would we do better if we had CLIA. 

My response was that it is a tragedy that in Canada our federal government has never engaged in a process, voluntary or mandatory to insist on an across-the-country approach to laboratory Quality, leaving a true hodgepodge mess.  But on the other hand I thank my lucky stars that we do not have to put up with the tired and obsolete mess that the US Congress has made with CLIA.  

 If there is a true and singular tragedy it is that in 1967 the US Congress create a ground breaking approach through the creation of a quality concept, but over time, with the introduction of lobby power over quality power, and political expedience over any interest in patient care, CLIA has fallen from being a force for quality to one of the exact and very opposite.  CLIA today almost guarantees Quality-by-luck rather than Quality-by-design.

Even a quick look at CLIA rules today points to the absence of quality assessment for the vast majority of tests, the absence of any semblance of clinical appropriateness requirements, test acceptability tolerances that one could drive a Mack truck through, and a series of silly regulations about who you can and cannot talk too, or how many times you can or cannot test a proficiency testing sample as if any of that is monitorable or has anything to do with Quality.  What it speaks to is how poor regulations lead to gaming rather than improvement.  

Tired, broken, lack of innovation.  Not at ringing endorsement.

Given the choice, between Canadian hodgepodge and American tired-obsolete, I think if I really needed to have a laboratory test performed today, I would have far greater confidence on the accuracy, quality, and interpretability of the test performed in a Canadian laboratory.

Today we have a new government in Canada with a 4-5 year majority mandate (in other countries we would call that a benevolent dictatorship), and perhaps a new opportunity for a new beginning.  Perhaps this is the right new time to start communicating with my federal government contacts.