Researcher beware - change is coming

The problems with research

It’s always interesting when things come together.  

Over the last few days I was at an international meeting where the discussion about the potential value of a Best Practice Guideline document for Researchers was raised because of a general experience that too many research dollars are wasted on non-reproducible and irrelevant studies.  An interesting proposal I thought, but perhaps a tad overstated.  

Then I get to the airport and pick up the Economist (March 15-21 2014) and turn to the Science and Technology section and there is an article called Metaphysicians which mentions John Ionnidis ( Why Most Published Research Findings are False ). I have discussed this author and this paper before [see: http://www.medicallaboratoryquality.com/2012/12/quality-and-medical-research.html ] because of his interest in doing “research on research”.  According to the article, a scourge of modern science is the “waste of effort”.  In 2010 “$200 BILLION (85% of total expenditure on medical research) was squandered on studies that were flawed in their design, redundant, never published, or poorly reported”.  Assuming those numbers are true, that would certain confirm there is a clear need for help.

According to Ionnidis too many researchers staggeringly over interpret statistical significance in studies that are far too small.  Further they have a lack of insight in proper study design and manifest “publication bias” where positive data gets written up for presentation and negative data gets ignored or worse.

Over the years we have observed that graduate students working in research laboratories seeming lack knowledge and interest and respect in running essential common equipment such as autoclaves.  Commonly the problem is that they are required to use the equipment, but did not receive appropriate training.  In essence they were just pushing buttons.  ( I suspect in the minds of their Principle Investigators, autoclaves are mundane compared to DNA analysis.)  

Furthermore common procedures of quality control and quality assurance are often incomplete or inadequate and generally not understood.   

What they do not seem to understand is that in the absence of basic Quality principles NO research can or should be trusted.

Apparently Ionnidis has doing something about it.  He has opened up a Center for meta research with the name Meta-Research Innovation Center at Stanford and the even more appealing acronym METRICS.  That will help to define the problems and perhaps develop some answers.  

Here are some questions that I would like to see addressed:

   A.               Would increased training in laboratory Quality result in reduced non-reproducible research and increased value for dollar spent?

   B.               Could accreditation of research laboratories result in reduced non-reproducible research and increased value for dollar spent?

   C.               Would introduction of proficiency testing into research laboratories result in reduced non-reproducible research and increased value for dollar spent?

I was intrigued by the notion of a Best Practices in Research Guideline however I also understand that in the current environment such a document would be pursued by those few laboratories that recognize the concepts of standardization and continual improvement.  The vast majority would, I suspect, never become aware of its existence, or never purchase it, or never read it, or read it but never consider it as relevant or appropriate to their laboratory.

I know this sounds a tad cynical, but for how long have we taken that approach in the health sciences. “It is not our problem, we are too smart, if it is a guideline then we can ignore it and if it is a regulation we can obscure it.”

This may well be the time when far more aggressive research oversight becomes a reality.  If a highly qualified author writes something in Accreditation and Quality Assurance (an excellent journal) its impact is strictly limited.  But if articles end up in the Economist, that is a different story.  Business folks read the Economist, as do lawyers, and politicians, and University presidents, and many of the general public.  

Sooner or later, the right (wrong?) person is going to start putting 2 and 2 together; wasting $200,000,000,000 is an insult to the public purse.   And then the regulations start happening, and some people lose their jobs, and some people end up in jail.

Even if we spend $2B on setting up a stringent oversight we are way ahead of the game.  And the impact on jobs would be negligible because for every research laboratory that is shut down, some would likely move into consultancy or oversight. 

Seems like a win.        

The Challenges of Quality oriented research

A few years ago I wrote a paper "Does external evaluation of laboratories improve patient safety" (Clinical Chemistry 2007).  The premise of the article was that literature and studies link positive correlations between awareness of standards and accreditation and performance on proficiency testing and on opinion/knowledge about quality issues, but there are none that demonstrate any of these important issues are definitively associated with improved or better laboratory performance.  

The answer about why this is the case is clear.  In the first place there are a huge numbers of variables that are known to have an impact on laboratory performance.  We have talked about many of them in MMLQR.  Factors such as busyness, noise, distraction, temperature, staff levels, stress levels, personnel personal issues, supervision, laboratory financial health, patient factors all play a role in generation of error and would not likely be evenly distributed.  Errors are usually not reported, and even at their worst happen uncommonly and irregularly, and that would mean that a study would have to occur for an extended period of time to generate enough information to draw a conclusion.  Put succinctly, to perform study on impact of accreditation, proficiency testing, and knowledge on laboratory performance would be near impossible.

That being said, it is important that I stress that in my paper I opined that while it may NOT be possible to prove that being accredited or doing well on proficiency testing was likely to result in improved performance, it would nonetheless be inappropriate and completely wrong to end Quality Assurance programs

On the other hand, it seems to me that there are certain sectors where a study could be developed, at least theoretically.  Take the construction industry.  In theory one could identify two groups of construction organizations, one group known to be certified to ISO 9001 and another group known to NOT be certified to ISO 9001.  Find them in a common geographic region, with relatively same types and numbers of employees, generally being involved in the same activities such as building homes or roads or bridges.  Put together a survey instrument to ask about opinions and knowledge and then talk to their customers about turnaround time, error, complaints, etc.  The study might take a couple years and a million dollars or so, but it is theoretically possible.  It would be reasonable to suggest that if Quality participation was seen to definitively improve construction performance, it would support the argument to maintain it in heath.

So I was really excited when I saw an article title in the January 2012 edition of ASQ Quality Management Journal entitled “The Effects of ISO Certification on Organization Workmanship Performance” by Jospeh Iwaro and Abrahams Mwasha, both from the University of West Indies in Trinidad and Tobago.  I was especially pleased to find in the abstract “It was found that ISO9001 certified organizations performed better in workmanship performance..."

Unfortunately, at least the way I read their article, as much as it has a lot of good information, I could not in the manuscript as published and printed the direct evidence to support better performance.  My interpretation of the study is that a survey was undertaken with a spectrum of people in the construction industry about their opinions on a number of factors that would be linked to better performance.  People working in ISO certified companies were significantly more likely to say as compared to people working in companies not ISO certified that they had seen or believed that workmanship performance was enhanced by improved communication, better documentation, better supervision, better work performance, and better management, but that is different from actually looking at the product of the performance, such as error free homes or bridges or roads.  I can understand why people in certified organizations would believe that ISO certification improves performance, even though there is good evidence that most certifications are ineffective (see Boiral and Amara, QMJ July 2009).

Assuming that I have read and understood this paper by Iwaro and Mwasha properly, it provides interesting information that is strongly supportive of Quality participation, even if it is not the definitive study.  But is is still much appreciated and is very much worth the read.

Here is what I believe:

1.    Research in the impacts of Quality participation is very difficult but is nonetheless very important.

2.    Doing direct impact studies may be impossible, but at a certain point a preponderance of support from indirect studies will “prove” the value of Quality participation.

3.    In the mean time, there is already enough information and evidence to support that standards development, quality management procedures, accreditation and/or certification oversight, and participation in external quality assessment help organizations improve.

4.    It is a bad and dangerous decision to consider it acceptable to avoid Quality participation as an approach to reducing expenses. 

Smart organizations understand the value of Quality participation, even if they can’t definitively prove its value.

Progress in Quality Research

I have been immersed in the arena of European laboratory quality over the last few days.  It has been a very interesting and positive experience.  I have had the opportunity to participate as a jury member for a candidate’s dissertation and defence of her PhD thesis on laboratory Quality.  To my knowledge this was the first (maybe second) ever PhD based on research on medical laboratory Quality and Proficiency Testing (External Quality Assessment –EQA).  Incredibly interesting and exciting.  What I saw was a level of creative and innovative enquiry and a growing awareness in laboratory Quality as a valid subject of study.  To my mind this is directly attributable to the development and dissemination of ISO 15189.  

As we continue to see progress with new laboratory technologies and techniques that will expand the technical information we will also see growth in the tools that ensure that this new technology has a Quality foundation.  Having gone through an era of too many tests that were fast, slickly packaged but complicated with poor sensitivity and even poorer specificity, the progress in studies on better quality monitoring will shine light on assay inadequacies and do much for continual diagnostic improvement.  

I have invited the candidate to appear in this web-journal site as a guest and I look forward to her presentation and insights soon.

 

During the PhD dissertation and defense some key observations were presented and recommendations for change were offered.  One recommendation suggested a role for a more uniform approach to quality for European laboratories.  Over the last 5-7 years European countries have been changing their approach to accreditation by requiring that the number of accreditation bodies reduce to only one official one per country.  That will be similar to the Canadian situation where there is only one provincial accreditor.  For some countries this may be seen as a restriction because in the past some countries have had multiple accreditation bodies, but this will result in increased intra-country consistency and uniformity in quality assessment.  There was also thoughts on the development of a over-arching role for EU policies for a more unified inter-country approach to Quality - something akin to the US CLIA.

Before that happens, I see an opportunity for some fundamental comparative study that could shed light on the modern value for supervening policies including mandatory accreditation.

In the United States there is a quality system that is driven centrally through the CLIA requirements under the authority of Health and Human Services (HHS) and the Center for Medicare and Medicaid Services (CMS).  While states are allowed to have their own accreditation program, they must as a minimum meet the requirements of CLIA.  In each state there is a 100 percent requirement for accreditation.  

There is another model in Canada where we have a  decentralized approach.   Each province has the authority to decide on the role and necessity of  laboratory accreditation.  Most provinces have their own single accreditation program or use the accreditation program from another Canadian province.  (Some provinces allow the intrusion of a foreign accreditation program, although the benefits or reasoning for this escapes me.)  In almost every province where there is accreditation, the requirement is mandatory and there is 100 percent participation. 

In Europe they have a system similar to Canada but different.  If one considers each of the EU countries as an independent but integrated unit, then each country has a status similar to a US state or a Canadian province.  In the EU, like Canada, there is no overarching “CLIA-like” structure.  But in the EU countries, where there is accreditation, it is voluntary, so that the participation in most countries is only around 20 percent.

So we have 3 jurisdictions US, Canada, Europe; each with comparable development of medical laboratories, each with over 1000 medical laboratories, but with three different stable structures for Quality and Quality Assessment.  I can’t imagine a model better suited for study and analysis.