Learning about Risk and Prevention through Continual Improvement

Learning about Risk and Prevention through Continual Improvement

Yesterday we had our 12^th external audit by our ISO certification body in 12 years.  You would think that by this time we would have the process down pat.  It should be a walk in the park.   Well this is sort of true.  In the early days we would look at audit day with a certain degree of anxiety and distress.  These days those negative concerns are gone, and in fact we look forward to the exercise because we still find opportunities for learning and opportunities for improvement.  

Some would say that now that ISO 17043 (Conformity assessment — General requirements for proficiency testing) exists there is no longer a place for assessment of proficiency testing programs to ISO 9001.  I can see the argument since ’43 is more specific, but to my mind there is no document more specific and exacting to quality management than 9001.  So while we will likely seek an external auditor that we have confidence can do a competent assessment to ’43, we will continue to stick with 9001.  Indeed, to my way of thinking, I suspect that when the smoke clears, we will likely find a way to do both.  

 

One thing that I appreciate about our current ISO 9001 assessment body (SAI Global) is that they provide us with a knowledgeable and experienced auditor with a lot of knowledge and skill in the laboratory area.  Several of his clients are in the laboratory arena.  He is knowledgeable about calibration and equipment and quality control, and our assessment covers these areas thoroughly.  In addition to our regular close look at our quality system and management review and customer satisfaction plus plus, we also get a solid review of critical laboratory expectations.  So in a sense we feel we currently are getting the best of both worlds right now.

 

So how did we do.  We were found to have no non-conformities.  We continue to meet all the requirements.  But we are not perfect.  Although we do two major internal audits each year, (one in February-March and one in July-August) and had the documentation available, when I wrote up my Management Review report, I forgot to add in an “Internal Audit” section.  We probably could have gotten away with that except we publish the MR review in our Annual Report (available on-line at www.CMPT.ca) and so the absence of an IA section was pretty obvious.  This was written as an “OFI-Opportunity For Improvement” rather than as a non-conformity.  I could go back and edit the MR review, but we agreed that the more appropriate approach would be to include the section in our next MR in our next Annual Report.  

 

The other thing that came up is a common problem, but I think we have found a good solution.  Like most organizations we are weak when it comes to having a documented list of Preventive Actions in our report.  It looks like we continue to be only re-active to problems rather than pro-active through prevention.  But in fact that is not true.  Our problem is one of recognition and value.  

 

We make changes and improvements all the time.  We changed our glass slide labels to a better material so that they would not fall off and not get covered with stain.  We changed our packaging to reduce the risks of leakage to a level beyond requirements.  We implemented our earthquake preparedness safety kit, and revised the labelling on some of our reagent bottles.  There have been probably 15-20 of these types of changes going on over the last year or two, and while we have thought of them as products improvements and safety requirements (which they are) they are also true Preventive Actions because each of them reduces a Risk of inconvenience to ourselves or our customers.  

 

 So now as we continue along our path of innovation and continual improvement we can also look at ourselves from a Risk and Preventive Action perspective.  And if that was all we learned from this year’s audit, it was well worth the cost.

 

As a PS note, some explanation may be necessary.  If we are so satisfied with our current assessment process, why would be even be thinking about including additional assessments?  The answer is that for some of our customers (clients) seeing the words “Accredited to ISO 17043” is becoming more important, and so I suspect our move to accommodate their concerns  will come to fruition fairly quickly. 

 

 

June is Quality Month

There is so much (maybe too much?) happening on my Quality front in the few weeks.  I don’t think that I could have avoided this log jam through better planning or calendar management.   Sometimes, and the expression goes, things happen.


It is final examination time for our on-line Certification Course for Laboratory Quality Management.   I am marking answers on standards development, quality partners, quality associated definitions, and the strengths and weaknesses of proficiency testing.  The other faculty are all doing their fare share as well.  Fortunately Maggie and Doug, our coordinator and IT specialist have figured out a way to automate and improve our ease of electronic access to the responses.  It is probably saving me at least 2 hours on the grading process which likely will mean less stress and a more enjoyment in marking.  So far, I am really impressed with the answers that I have graded.  It is interesting and encouraging to see how much Quality information the participants have accumulated over the last 6 months.


Next week is all back-to-back-to-back meetings.  First is the Annual meeting of the Canadian Standards Association and there are going to be some very interesting meetings for leaders of Strategic Steering Committees to look at ways that we can benefit from each others experiences and perhaps find ways to harmonize activities across some pretty diverse industry boarders.


At the same meeting CSA is holding a first Canadian Academic Challenge, which will in some way I suspect connect with the World Standards Academic Day that was just held in Bali.   There is no doubt that CSA needs to find ways to better and more strongly connect with academia and the student and young faculty pool.  The reality of the currently active  standards community is that it is getting older, and the gomers (me included) need to start looking to the next younger group if we want to sustain a Quality presence.


The day after the CSA meeting there will be two days of meetings as the Canadians involved with ISO Technical Committee 176, the committee responsible for ISO9001:2008 start to prepare proposals to revise and amend that document.  This will be a very interesting exercise because the document is very mature.  But if the document does not undergo its own continual improvement process, then what will that say about its role as the very cornerstone of Quality.  Change and Improve, or Die.


And the day after the ISO TC 176 meeting is the joint meeting of the Canadian Society for Clinical Chemistry (CSCC) and its Quebec sister society (QSBC).  This meeting is being focused on the changes that are happening in Quality for medical laboratories across Canada.   There will be a substantial national and international presence and I am looking forward to being one of the keynote speakers, hoping that I remember to not burn myself out at all the other meetings being held throughout the week.


And then when I get back we start with the new cycles for our CMPT proficiency testing Annual Meeting (October) and our celebration of our 30th anniversary of provision of high quality proficiency testing services and 11th year of the Program Office for Laboratory Quality Management, and hopefully receive some positive news about our current grant proposals.  


In the next short while I will be announcing some pretty exciting changes that will start next year in our Certificate Course.


And we need to get started preparing the ISO Technical Committee 212 meeting in Berlin, and for a new PhD student intending to focus his research and study on medical laboratory Quality management.  He will be arriving in Vancouver in August.  I am pretty sure that he will be the first person in Canada to take on Quality management as a focus of primary interest.   I suspect I will hear more when I attend the Canadian Academic Challenge meeting.


My personal opinion is that this sort of pace of activities is about to become our new normal.  After a brief break in July we will be working at this sort of pace for the next 6 months, probably longer.  It may indeed go on for years.   


It seems the light has turned on in Canada and we are new starting to see all the huge benefits of being engaged in laboratory Quality at every level; a Quality explosion perhaps.


I call that a good thing.

A Happy and Successful re-Certification

Congratulations to us.  Our proficiency testing program as been internationally assessed and again our quality management program was recertified as in compliance with ISO 9001:2008.  This makes for 11 consecutive years.  For 10 of these 11 assessments (including this year) we were found to have no non-conformances.  There were two minor opportunities for improvement, both of which we were able to address on the spot.  We are pretty pleased with our accomplishment.  And when we last checked with our laboratory customers.  As mentioned previously near 90 percent of survey responders said that our meeting the standard demonstrated a commitment to quality and increased our credibility.  So that makes a win-win for us and our customers. 

In conversation, some suggest that the problem with our standard is that while it makes comment on our paperwork, it doesn’t prove that we are actually competent.  They say that a 9001 assessment does not speak to competency assessment.  They say that only an accreditation body can do make that judgement, to which I say “horse feathers!”.  Actually I usually say a lot more than that, but for purposes of keeping this web journal at an appropriate level of professionalism and decorum, I will leave it as “horse feathers”. 

We as comfortable with our recognition as are our customers. 

I know there is another standard for proficiency testing programs entitled ISO 17043:2010 Conformity assessment -- General requirements for proficiency testing.  It is an excellent  standard, one for which I participated in the development.  And one day we will probably look at it.   The document shares with 9001 the intent to help assure other organizations that the assessed program is competent with Quality being the prime objective.  That is a laudable goal.   My concern is not in either document, but is in the people doing the assessment.  I don’t have a lot of confidence that many assessment bodies have either the knowledge or experience to do a proper accreditation audit.  

The body that audits us every year for the last 11 years knows us, knows what we do, and knows our strengths and areas in which we can improve.   We have gone through this experience together and have mutually learned during the past decade.  We have worked with three different assessors from the company, and while each was a different person with individual personality and traits, over the years and we have found a consistency in their approach to our assessment.  We understand that we are not their only client and that they are not our coach or mentor.  But we understand them and they seem to understand our operation. 

Certification bodies must have  hundreds if not thousands of organizations wanting to be assessed to ISO 9001:2008, and they all get tons of opportunity and experience.  They know and understand the conventions of standard interpretation.   They understand and appreciate the subtleties and nuances.  They also know how to put context to the document.  They know it, they eat and breathe it.  Put simply, they know what they are doing, and they do it well.

On the other hand, there are very few proficiency testing programs in the world, and there are likely far more accreditation bodies than there are proficiency testing providers.  It is doubtful that any single accreditation body has done  more than one or two (maybe 3) assessments to ISO17043.  It is doubtful that many of them have had the opportunity to learn and understand the nuances of the requirements of the standard, and for certain they are not in the position to appreciate the differences or subtleties of different organizations.  Getting assessed in that sort of situation increases the risk of individual interpretation, and that in my mind is a lot at risk, indeed too much risk for too little reward.  Frankly I would rather let them learn on other people and hone some understanding before letting then come to us.

Its kind of like having a choice between a surgeon who done a procedure a hundred times, one that read about it.

In a few years, when the standard has been around for a while, and more people know what to look for, then we will consider the additional recognition. 

But not now.

Today we are very happy and very proud to be a part of a very small minority of programs that have gone through 11 successive external quality assessments and come out of it with honour and recognition. 

And as best as I can tell, so are our customers.

History of Quality - part 2

I was recently invited to participate in a meeting focused on the process of renewal and continual improvement for an organization responsible for Quality standards.  I assume that this is standard practice for all organizations committed to Quality, but there is that old adage about what happens when you ASSUME.  I won’t go there, but I will say full credit to the Canadian Standards Association for being prepared to both talk-the-talk and walk-the-walk. 

Anyways, during the introductory part of the meeting, a comment was made about CSA and Canada and its role and prominence in international Quality.  Not that Canada takes the responsibility as Secretariat for the ISO Technical Committee 176 (which it does) and not that Canada is a leader in Environmental Quality (which it is) but rather that ISO 9000, the cornerstone of all modern Quality Management was based upon a Canadian document specifically Can3 – Z299: 1985 Quality Assurance Program. 

That’s interesting, I thought that the source was another document, specifically BS5750.  Not only had I read that elsewhere but it was supported by information from the modern purveyor-of-all-knowledge (Wikipedia).  Wikipedia clearly and definitively states that the seed document to ISO9000 (first published in 1985) was the standard BSI 5750 published in 1979.  

The provenance of that document was the (US) Mil-Q-standard 9858 which was provided first to the pre-NATO structure (ABCA – America, Britain, Canada, Australia) and then to NATO where it became part of the Allied Quality Assurance Publications (AQAP) quality series.  

British Standards Institute (BSI) borrowed it from NATO to create its own quality document (BS 5179) which was later changed to (BS 5750).  [note: According to Joseph Juran: A History of Managing for Quality – 1995, 5750 was essentially a direct copy from NATO].  Britain then offered BS5750 as the  seed document for ISO9000.  Canada doesn’t come up anywhere, with the exception of being a part of ABCA and NATO.

What is missing here is that if you spend the time going to other sources, especially deep inside www.iso.org, you find the following: " When ISO/TC 176 embarked on the development of generic quality management standards for worldwide application it was able to take advantage of a substantial base of national experience notably in the United Kingdom and in Canada.

In the United Kingdom the BS 5750 standards were well on their way to broad acceptance and in Canada a series of national standards known as CSA Z299 were also widely used.”  

As I pursued further, it is clear that Z299 was a document developed from Mil-Q-9858 through NATO AQAP although, in contrast to BS5750 it was substantially changed to accommodate the new audience.  

I suspect what happened was that the ISO technical committee went on a search for seed documents Britain offered BS5750 and Canada offered Z299.1:1985.  Japan apparently also offered a document.  Based on these seeds, ISO 9000:1985 was created as a collective amalgam.  

All this basically makes sense.  The Quality seed document of note  was the US Mil-Q-9858, created by the US Military to improvement the Quality of companies selling products for military use.  By the language and structure, one can say almost definitively it was written either totally by W. Edwards Deming, or persons strongly influenced by Deming.  NATO used the document for similar purposes, but its potential value for civilian commercial guidance was acknowledged around the world.  Many countries, probably not limited to Britain and Canada created national documents.  I suspect (but can’t say with certainty) that Japan got to the same point not through Mil-Q- and NATO, but more likely from Deming directly.

CAN3 Z299.85 was first designed and written in the 1970’s but was published for the first time (?) in 1985, two years before ISO9000 series.  It is still available and still in circulation, and indeed is still being used as a certification standard today.  (Note that the US Military has dropped Mil-Q-9858 and replaced it with ISO9000).  Z299:1985 is not found in the current CSA catalogue.

So while perhaps the introductory comment was a bit of an overstatement, Canada’s role here has been under-recognized, probably by our own actions. 

It is not surprising that Canada’s role in this process has been missed in history.  It is a common Canadian trait to be at the forefront, but to hide in the background and let others take credit.  It is not a healthy trait.  

There are a few lessons here for me:  First off, the historical forefront and record of Quality does go through Canada and in particular through the Canadian Standards Association.  And second, don’t trust Wikipedia as an absolute source.  It is only as good as the author without a fact checker.  

As a final comment, someone at CSA or Standards Council of Canada could and should correct the Wikipedia record.  Credit when and where credit is due. 



Originally posted Nov 23, 2011.  Edited for grammar and spelling Feb 6, 2015

Monitoring Satisfaction through Noble's Rules

In the laboratory business we have always thought it was all about the science and not about the business.

But we were wrong.
ISO as well as WHO and CLSI (and before them, Deming and Crosby) all acknowledge the importance of “Customer satisfaction”.

It is not so much that the customer is always right, but that the customer should always have a voice and should be heard. There is an expectation to have some form of customer input on a regular basis, perhaps as often as once a year.

The reason that the standards development bodies have included this as a requirement and the basis for policy is because it doesn’t matter if you are an academic providing a course, or a laboratory providing documented information, or a manufacturer providing umbrellas, or a proficiency testing provider, or an equipment and reagents supplier, if your customers are not happy, then bad things start to happen.

In the private product or service sector that probably means customers stop coming. And that becomes the business killer.
 
In the public sector laboratory, the customer may not have a choice of which laboratory they have to use, but that won’t stop complaints, reputation slurs, increased threat of litigation. (Incidentally, this applies to accreditation bodies as well.)
Sooner or later you risk becoming the interest of the public and the media.  
Or even worse, think about the embarrassment and humiliation of a public inquiry.

All of those are major career killers.

So what to do. In the business world, the godsend solution for customer satisfaction has become the on-line survey. It is so easy to create an on-line survey and send it out to all your important customers. So easy, in fact, that it has become too easy. 
Anyone foolish enough to give your email address to a hotel or car-rental or restaurant gets inundated with surveys. We have become a world of survey send-outers and survey send-inners, and most of it is a waste of time.

Most surveys are poorly designed; are way too long, too complex, and far too diffusely focused. If a survey takes more than 2 -3 minutes to complete, you can guarantee that either it will not be completed, or will be completed with junk information. 

Also, you have to remember that responders  always have their own bias one way or another,and probably have interpreted the questions in ways that you never dreamed of. Creating most surveys has become high risk of being counter productive for addressing customer satisfaction. As they say “Fast, easy, slick and wrong”.

If you still feel compelled to resort to surveys, spent some time at setting them up so that you might get some information that you can consider. (We call that PDSA) . 

After years of learning the hard way, I figured out a set of simple rules  that anyone interested in developing a Satisfaction Survey can follow.  I arrogantly coined them as Noble's Rules for Successful Satisfaction Surveys.  

They don't guarantee success, but not keeping them in mind will pretty much guarantee failure.

(1) Focus them to a single issue.
The more you try to pack into a survey, the worse it gets.  Pick a topic and get out.


(2) Ask the question that needs to be asked, even if you may not like the answer.  
It’s very easy to create surveys that will always give you positive feedback by simply avoiding any potentially controversial or challenging issues, but how can you study or learn what people think if you don’t open up the discussion.

(3) limit the survey to only a few questions , best is to keep it to 5-6 and NEVER more than 10, and make them as uncomplicated as possible . 
Get in, ask a few questions, and get out.  Don't give them a chance to get bored.

(4) make sure that it can always be completed in 3 minutes or less. Boredom is a guarantee for incomplete surveys loaded with random nonsense answers.  It would be better if they didn't send the response in, because the nonsense becomes pollution and the pollution leads to terrible interpretation. 

(5) Pre-test the questions to reduce (you can never avoid) ambiguity. 
Make your questions VERY simple.  Confusing questions get confusing answers.

(6) Avoid requiring an answer. That is the other  guaranteed invitation to bogus information. 
Making people answer questions, makes people angry.  Sometimes you can't avoid them, but keep them to an absolute minimum.

(7) Pick your audience and stick with it.  
General send-outs are a total waste of time.

(8) Where you can, avoid satisfaction surveys. 
More effective solutions for monitoring satisfaction is looking at objective measures.  For example, count how many complaints come in and how many are resolved within a specific time. 
Set up a system to catalogue every complaint, something that most laboratories never do. All those telephone and hall-way gripes are complaints and they need to be included. 

You may not think they were important, but the person who mentioned them did.

Oops, My mistake!!

We had our recertification audit for ISO9001:2008 last week, and for the first time since 2004 have been found wanting. Darn. For the first time we have been cited for a non-conformance. When asked to see our internal audit record, we were unable to provide one.


That is not to say we have not done a lot of the checking. We routinely check and update our Quality Manual, and update and revise our SOPs as required. We have updated our job descriptions as required. We checked our Document Control Log and Records. We have our Management Review of incidents and opportunities for improvement, and have monitored customer satisfaction through survey and complaints. But we did not complete an internal audit document to demonstrate that this was all done.


Some years we have asked an external Quality consultant to do an audit for us (an external-internal audit), but we did not get around to that, because we figured we could do it ourselves. We were wrong.


You might think that I consider this a waste of time. After-all we had done all the critical stuff; we just didn’t get around to writing it down.

 You would be wrong. We take our certification seriously. The assessor was right. We were wrong.

Having been involved in the assessment process for 8 years, we should have known better. We did get busy with a number of issues, some of them regular events, if not planned, at least they were not unanticipated. We also had a number of exceptional issues that needed addressing. None of them were issues that would could have controlled though implementation of preventive action. That is not said as excuse, but as explanation.


CMPT has benefited from its active Quality Management program. As we have become more organized, have taken on more projects, and been more effective and efficient in working through them. We have more programs, we have extended our teaching programs and our research and development. We have gained control over our finances.


From my perspective, we have strongly benefited from adopting ISO 9001:2008 as the basis of our Quality Management system, and further benefited by taking our commitment the extra step forward by working with a qualified and competent Certification Body to do regular assessments, and recognize our quality system.

Our problem is one that in that effectiveness and efficiency follow a normal curve pattern, or more likely a normal curve with negative skew, which means that effectiveness and efficiency fall off faster than they develop, which means that challenges don’t just creap in; the pace is more of a gallop.

Small organizations need to be very careful because becoming more effective and efficient can result in improved balance sheet,but may not be sufficient to generate enough revenue to hire additional hands-on personnel, so that at a certain point they need to back off on expansion of activities, even though they may feel they can keep going.
In our situation, our neglecting to complete our paper work may be an indicator that we have reached that point.

 So, now we have two things to do. First we need to do a audit now to detect how many other gaps we had created, and second we may have to do an assessment of what we are doing and what are our options.
  All this happens at an interesting time. I have been at a meeting recently where all one hears about how valuable a service Accreditation Bodies provide, and how poor a service Certification Bodies provide (They are just in it more the money). Accreditation Bodies do provide a valuable Quality Partner service, but over the last while I have seen a number of accredited laboratories doing a terrible job; and at the same time we have been assisted by a certification review that found a critical deficiency.

 So the rhetoric is not consistent with reality.



At least not with my reality.