Fishing and Proficiency Testing

Fishing and Proficiency Testing

In our UBC Certificate Course in Laboratory Quality Management we introduced a concept which we call Quality Partners.  These are agencies and organizations whose primary purpose is to enhance the quality of medical laboratories.  Without these partners, the likelihood of a medical laboratory developing any semblance of quality enhancement and error reduction for all intents and purposes, negligible.  The primary Quality Partners are Accreditation, Proficiency Testing providers, Standards Development organizations, Education providers, Professional Organizations, and Suppliers.  While I am biased, of this group, the only  partner that has ever demonstrated the ability to actually reduce errors is the Proficiency Testing provider group.

When PT exists with good and competent programs, laboratories receive samples that closely resemble actual samples, with the singular but essential difference being that the sample has been pre-tested and stabilized so that the correct result is known by the provider.  The challenge to the laboratory is to test the sample and come to the same value.  If the laboratory can do that they are told they are proficient; if they cannot they are told they have opportunities for improvement to learn how and why they made mistakes, and to determine how they can avoid such errors going forward.  

PT providers are a committed group of people, dedicated to the premise that helping laboratories overcome errors directly contributes to patient care and patient safety. 

While my own personal focus in on water testing and patient care microbiology sample testing, most developed countries have PT providers not only for medical laboratory testing and water, they also test for food and engineering, and many many other types of laboratory tests. 

Unfortunately, developing countries see the value of developed medical laboratory testing, and see the value of quality partners, but they do not have local access to PT.  And that is a problem.

For many, the solution is to purchase PT samples from a larger country, who is always more than happy to provide them.  That the samples were never designed to address the issues of developing countries and that to get the samples to the laboratories at considerable distance they have to compromise their quality by freeze-drying the samples is of little concern.  When they write the informative sheets, they are written from the perspective of the large parent country and without any perspective for the developing region.  

So the product they receive is costly, not designed for their purpose, not optimised for their purpose, and not assessed or graded for their purpose.   That does not seem to be much of a deal, and from what I have seen, the success rates on the samples is rarely barely close to acceptable. 

 

We have taken a different approach.

Rather than just shipping samples, we bring one (or preferably two) able persons to our centre and give them intensive training in producing samples and setting up a basic program that will allow them to start a program, select, produce, transport samples relevant to their setting, and show them how to grade them and produce a critique.   As mentioned it is an intensive program.  Some get the information they want in one session, others come back for a second session.  

Over the duration of our program we have usually had one country participate each year.  This year we will have had representatives from 3 countries.  And we are getting contacts and expressions of interest for next year.  So by an awareness scale we seem to be having success.  

This summer we will be doing a survey to see if the countries that have visited us  have implemented programs and if those programs are still in operation.  (Satisfaction and Loyalty).

Without getting too sophomoric, the whole experience reminds me the proverb “Give a man a fish, and you feed him for a day; show him how to catch fish, and you feed him for a lifetime.”  Contrary to popular myth, this is not biblical nor Greek or by Confucius in origin, although I can understand the foundation for the various myths. 

The original version was “…give a man a fish he is hungry again in an hour; if you teach him to catch a fish you do him a good turn."  This was written originally by a British author, Anne Isabella Ritchie and published within a story entitled Mrs. Dymond, in an American magazine Littell's Living Age printed in September 1885.

Personally I prefer the original version.

PS:  If you want to know more contact us through www.cmpt.ca

The Value of the AGM

This is a busy time for the Medical Laboratory Quality arm of University of British Columbia; we had the Annual General Meeting for Clinical Microbiology Proficiency Testing (CMPT) last week and this week we have the Quality Management Conference for Medical Laboratories by the Program Office for Laboratory Quality Management (POLQM).  That is a ton of work, and if it weren’t for all the people that work with me, and for W Edwards Deming, it would have been a disaster.  

CMPT and POLQM staff are all strongly committed to Quality and that and a lot of hard work make the two programs work; but it was our collective commitment to ISO9001:2008 and Deming’s PDSA that created the structure that allowed us to be as organized as we are.  

The AGM went very well.  While we don’t unfortunately have records readily available, to my recognition we have been hosting an AGM since at least 2000; perhaps longer.   It is a convenient time for all the committee members and our stakeholders, especially the provincial accreditation bodies with whom we work, to get together in one room and sort out what is working with CMPT and what needs improvement.  The following day, the clinical bacteriology committee gets together in camera, and selected the next set of challenge samples that will be sent out between May 2014 and April 2015.  

Perhaps the most important part of the AGM from my perspective is when I share with the group how we have fared with last year’s Goals and Objectives, and more importantly the AGM is when I announce our new set of  G&Os for next year.

Last year we had a total of 6 G&Os, 4 were program oriented and 2 were Quality oriented.  All 6 were met.  

 This year we have a new set of 9 that address fiscal issues including personnel and hardware requirements, and importantly a plan to move forward on a new recognition pathway.  (As we progress forward, I will write more.)  It should be an exciting year.

One of the G&Os not listed at last year’s meeting, but added in as an extra in mid-year was the intent to allow medical laboratories that use CMPT products and services to send representatives to subsequent meetings with the view that being  present and more importantly being actively involved  during Question and Answer sessions could help create stronger relationships with those laboratories.  

It was a good idea, and well planned and promoted, and had some success.  We had people who planned to come, although one had to withdraw.  What I was hoping to have, questions arising during Q&A, didn’t occur.  Perhaps these new AGM attenders felt intimidated by the questions from the regular attenders, or perhaps they were content to just sit and listen; I can’t say.  But, I think the old adage of “you can’t win the lottery if you don’t buy a ticket” applies.  If you are not present during the Question and Answer period, then you are less likely to have your questions asked or answered.  

So I count this as a partial victory; a bunch read the announcements, some responded and a few attended.  I consider that as a good start.  Maybe next year a few more will attend and we will get a few questions from the floor.

As a final comment, within ISO 17043, the standard for competency assessment for proficiency testing programs, one finds requirement for management review and review meetings.  That is standard good quality, whether it is derived from 17025 or 9001.  What is not mentioned is any requirement or recommendation or a note suggesting that programs consider hosting a stakeholder meeting to bring together interested parties, including the customers to talk about product and service.  

I think that is a flaw in the standard that could and indeed should be addressed in future editions.

Competence Assessment: how regular is regular?

By now most readers of MMLQR know that I buy into the concepts of Quality and that I follow Quality Management practices in my laboratory and academic practices.  But I also see it as my obligation to understand and interpret Quality and apply it to the extent possible and to the extent reasonable.  To my way of thinking, embracing the limits of possible and reasonable creates an atmosphere of achievability.  Going beyond that is like starting on an exercise program by running a marathon every day.  You can do that once, and maybe even do that twice, but at a certain point it ceases to be sustainable and falling off track becomes unavoidable.  The challenge is to find the level that works for you and your organization from which you can build going forward.

It is with that in mind that I reference ISO 15189:2012 on the topic of competence.  Here is what the document says:
 “Competence assessment.  Following appropriate training, the laboratory shall assess the competence of each person to perform assigned managerial or technical tasks according to established criteria.  Reassessment shall take place at regular intervals. Retraining shall occur when necessary.”

This is not a bad idea.  One needs to be confident that people that are working in your laboratory know what they are doing.  And the concept of reassessment at “regular” intervals as opposed to “annual” intervals makes competency assessment more likely to be a program that can be sustained.

Recently I did a routine on-line survey of the laboratories that participate in our Clinical Microbiology Proficiency Testing program, which is a pan-Canadian proficiency testing program that is committed to clinically relevant proficiency testing. 

One of the questions I asked was about competency testing because I wanted to know if laboratories used CMPT samples as part of their competency assessment process.

The first question that I asked was “Does your laboratory perform competency assessments for laboratory personnel?”  to which I gave 5 choices for response.  You can see the results in Figure 1.

I will tell you that in my opinion, while most responders (67%) said they did so on a regular basis, I personally was hoping that more would have selected the second choice (new trainees, new hires, and return from absentee).  Surprising to me, not a single responder selected this choice.  While it is true that choice 1 was much closer to the letter of the regulation, the second choice is much more practical and pragmatic; in my opinion, it is still an acceptable option.

In my laboratory all my staff have been with me for a long time, well in excess of 12 years.  In Canada that is pretty typical of most laboratories.  All staff were trained well at the beginning and have grown into the positions; they are experts in what they do.  By all the measures that we follow (contamination rates, late rates, complaints and complements, sustained contracts versus lost contracts) they do very well. We have had people take prolonged maternity leave (in Canada employees can take up to one year off for parental leave), after which they underwent retraining and reassessment. 

In CMPT I do not bother with routine or “regular” competency assessment any more, except for special cases.  Rather I focus on output and performance.

In my world, it is true that some people can have life challenges, with drugs (including alcohol), or illness, or stress/anxiety all of which can impact on performance.  For some, the onset may be insidious, and may impact on work may be gradual or subtle.  But the reality is that most people don’t go through crisis and even when they do, competency assessment is far to blunt an instrument to rely upon for picking up subtleties.  In other words, in my world, once we have gone past the point of recent hires and retrains and those who take extended leave, having active competency assessment, even on a “regular” basis can be excessive.

All activities are Time, Effort, Energy, and Money (TEEM) consumers, all of which are finite in the medical laboratory.  Running competency program for all personnel takes a lot of time and effort and energy.  And I find that that those of assets that I can better consume in other meaningful Quality oriented activity.  Having an intact program is important, but so is picking your battles, and maintaining your options.  It is as much about balance as it is about requirements. 

So when the discussion comes with the accreditation auditor, I suggest that laboratories understand that the crafters of the standard signalled the need for flexibility in Competency Assessment by using the term “regular”, and that it is your prerogative to explain and justify how you have interpreted and used that flexibility to your laboratory’s advantage. 

It is up to you to sort out how you measure performance and how  regular  “regular” has to be.

PS: For more discussion on Competency Assessment, consider attending the POLQM Quality Management Conference for Medical Laboratories.

http://polqm.ca/conference_2013/conference_2013/conference_home.html

Laboratory Size, Complexity and Performance

Size Matters.

Tomorrow is our Annual General Meeting for CMPT, our proficiency testing program.  I have updated my meeting presentation to include all our review materials for the year that ended May 2012 (2011-2012). 

One subject that gets addressed in laboratory performance, not in the sense of finger pointing, but as a measure of our performance as a Quality Assessment program.

I have talked about this subject before, but obviously without particular effect.  To me it is a big deal.  Others may not see it that way, but I think they would be wrong.

We run CMPT which does proficiency testing in clinical and water bacteriology across Canada.  Proficiency testing programs structure themselves in one of two ways, either they give all laboratories the same challenges all the time, or they do not.  We do not.

When we started we stratified the laboratories into two groups.  The largest was category A are large complex laboratories that are expected to handle samples from all sorts of situation.  Often in addition to being large facilities with their own sample load, they also receive referral samples from smaller laboratories. The second group, category B, was large laboratories that tended to work at a lower complexity level.  They were less likely to receive cerebral spinal fluid samples or samples from severely immune compromised patients.  [Note: with the increase in homecare, and political will to ensure more services outside the core, the lines between categories A and B have blurred over time].  

As our program expanded we developed two additional categories; category C which are generally small rural laboratories, and category C1 are facilities whose primary function is to do sample set-up and then pack and ship.  The may be asked to so some basic tests which are defined to include preparing and examining Gram stained slides.  In my mind I would not call microscopic examination a basic procedure.

With 4 categories we decided from the get-go that large complex [A labs] would get a large volume of PT samples that include basic samples, complex samples, and very complex samples.  category B would get many of the same set, but would not get the very complex, largely because they would likely not see those samples often.  Similarly category C got fewer samples, and even less complexity, and category C1 received samples one-notch less.  If laboratories wanted to compare themselves [I hate that!], at least they could see themselves in a pool of comparable organizations.  The point was that we expect to see all laboratories performing well, if not perfect, as compared to others in their group.  

Unfortunately it doesn't work out that way.

Every year, the smaller the laboratory, the less well it performs, and over the decade the trend has never corrected and if anything continues to deteriorate.  This is not a good thing.  

Our best guess is that what this reflects is the relationship between size and complexity, and money, and opportunities for Quality.  The larger the laboratory, the more younger staff they can hire, the more continuing education they can support.  Money buys success.  The smaller laboratories have fewer staff, fewer fresh staff, less money for continuing education and fewer opportunities for Quality.  And the smaller the laboratory, the worse the situation.  

Proficiency testing should not be about inter-laboratory comparison in the sense of a competition.  That is an obsolete old notion retained mainly in the minds of bureaucrats.  But in this sense group to group comparison has value and merit.  

People living in small towns deserve to know that if they get a test done at a local medical laboratory they should have the assurance that the test should be received, set-up, performed and read, and interpreted accurately.  At least at a level higher than 70 percent.  I think that is part of the social contract.  It certainly is part of Patient Safety.

Maybe what is happening is they process all their regular sample work just fine; they only make their errors on the PT samples.  

Maybe. 

But here is a scary thought.  Imagine what the graph would look like if we gave all the laboratories regardless of size or complexity the same challenge samples.

Follow us

@Qualitology

Laboratory Performance 2012 - a warning

I have been tracking Proficiency Testing performance in laboratories for a long time, but most objectively over the last 10 years.  I presented this information a year ago, and I am presenting it again, updated by another year.  

What the information says is that as the size and complexity of the laboratory gets smaller (category A is the most complex and category C1 is the least) the laboratories performance on PT challenges gets poorer.  This is true year over year and is magnified because the larger laboratories get more samples with more diversity and more complexity and the smaller laboratories get samples that are less diverse and less complex.  


Now I understand that performance on proficiency testing is not the sole, nor necessarily the best indicator for clinical laboratory performance, but I will argue that it is a very good proxy.  While good PT performance does not necessarily ensure good clinical performance, less good performance reflects problems, often of a systematic or structural nature.  If a laboratory does not perform well on samples that closely simulate typical clinical samples (and across Canada, that is the norm) it is a cautionary flag that clinical performance may be suffering as well.  


And that is why having a low percent achieved score on PT challenges is a problem that should be monitored and addressed.


I have argued in the past, over and over and over, that the system is stacked against the smaller laboratory because they have fewer staff, less direct supervision, and less time and resources for continuing education, and little support at key levels.  


But this year starts to see a new trend in a number of provinces where this problem takes on a new urgency.  Health care across Canada is both broken and broke.


As the developed world struggles to find the elusive financial recovery, all jurisdictions are cutting back.  We use all sorts of positive euphemisms such as “doing better with less” and “innovative restructuring” but the public healthcare system is losing staff at every level, and much more emphasis is being put on home care. 


Frankly I don’t personally see that as all problematic.  Hospital care was never that special anyways and we have all become really aware of how much damage we do in hospital care with unwashed hands, deteriorating cleanliness, and less than optimal nutrition.  So if you have someone at home, or can arrange to have someone at home, to provide recuperative and longer term homecare service, that is probably a good thing.  


Back in the eighties (now some twenty-five years ago!) the mantra was “closer-to-home”.  Now it is becoming “in-the-home”.


But as folks go home it means that they live further away from the big facilities with the big laboratories with all their resources.  Folks are at home and much closer to the smaller laboratories.  Smaller laboratories are finding their work load is going up, and their complexity is going up.  Now they are seeing samples from patients that they have not seen for a long time.  But the problem is that if they were under-equipped for small volume low complexity work, they are in a worse position now.  


A colleague of mine has recently taken on the challenge of providing the additional support and supervision for the smaller laboratories in his province.  And that is a good thing.  If that does not become the norm then you can predict that the support system for home care will flounder and that will only make a weak situation much worse. 

So here is the message:  Healthcare collapsing; homecare arising.  Get help for the small “closer-to-home” laboratories or suffer the consequences.  




Addendum:
The full CMPT annual report will be out and available in early October.  You can read last year’s report at www.CMPT.ca