The Challenges of Quality oriented research

A few years ago I wrote a paper "Does external evaluation of laboratories improve patient safety" (Clinical Chemistry 2007).  The premise of the article was that literature and studies link positive correlations between awareness of standards and accreditation and performance on proficiency testing and on opinion/knowledge about quality issues, but there are none that demonstrate any of these important issues are definitively associated with improved or better laboratory performance.  

The answer about why this is the case is clear.  In the first place there are a huge numbers of variables that are known to have an impact on laboratory performance.  We have talked about many of them in MMLQR.  Factors such as busyness, noise, distraction, temperature, staff levels, stress levels, personnel personal issues, supervision, laboratory financial health, patient factors all play a role in generation of error and would not likely be evenly distributed.  Errors are usually not reported, and even at their worst happen uncommonly and irregularly, and that would mean that a study would have to occur for an extended period of time to generate enough information to draw a conclusion.  Put succinctly, to perform study on impact of accreditation, proficiency testing, and knowledge on laboratory performance would be near impossible.

That being said, it is important that I stress that in my paper I opined that while it may NOT be possible to prove that being accredited or doing well on proficiency testing was likely to result in improved performance, it would nonetheless be inappropriate and completely wrong to end Quality Assurance programs

On the other hand, it seems to me that there are certain sectors where a study could be developed, at least theoretically.  Take the construction industry.  In theory one could identify two groups of construction organizations, one group known to be certified to ISO 9001 and another group known to NOT be certified to ISO 9001.  Find them in a common geographic region, with relatively same types and numbers of employees, generally being involved in the same activities such as building homes or roads or bridges.  Put together a survey instrument to ask about opinions and knowledge and then talk to their customers about turnaround time, error, complaints, etc.  The study might take a couple years and a million dollars or so, but it is theoretically possible.  It would be reasonable to suggest that if Quality participation was seen to definitively improve construction performance, it would support the argument to maintain it in heath.

So I was really excited when I saw an article title in the January 2012 edition of ASQ Quality Management Journal entitled “The Effects of ISO Certification on Organization Workmanship Performance” by Jospeh Iwaro and Abrahams Mwasha, both from the University of West Indies in Trinidad and Tobago.  I was especially pleased to find in the abstract “It was found that ISO9001 certified organizations performed better in workmanship performance..."

Unfortunately, at least the way I read their article, as much as it has a lot of good information, I could not in the manuscript as published and printed the direct evidence to support better performance.  My interpretation of the study is that a survey was undertaken with a spectrum of people in the construction industry about their opinions on a number of factors that would be linked to better performance.  People working in ISO certified companies were significantly more likely to say as compared to people working in companies not ISO certified that they had seen or believed that workmanship performance was enhanced by improved communication, better documentation, better supervision, better work performance, and better management, but that is different from actually looking at the product of the performance, such as error free homes or bridges or roads.  I can understand why people in certified organizations would believe that ISO certification improves performance, even though there is good evidence that most certifications are ineffective (see Boiral and Amara, QMJ July 2009).

Assuming that I have read and understood this paper by Iwaro and Mwasha properly, it provides interesting information that is strongly supportive of Quality participation, even if it is not the definitive study.  But is is still much appreciated and is very much worth the read.

Here is what I believe:

1.    Research in the impacts of Quality participation is very difficult but is nonetheless very important.

2.    Doing direct impact studies may be impossible, but at a certain point a preponderance of support from indirect studies will “prove” the value of Quality participation.

3.    In the mean time, there is already enough information and evidence to support that standards development, quality management procedures, accreditation and/or certification oversight, and participation in external quality assessment help organizations improve.

4.    It is a bad and dangerous decision to consider it acceptable to avoid Quality participation as an approach to reducing expenses. 

Smart organizations understand the value of Quality participation, even if they can’t definitively prove its value.