Researcher beware - change is coming

The problems with research

It’s always interesting when things come together.  

Over the last few days I was at an international meeting where the discussion about the potential value of a Best Practice Guideline document for Researchers was raised because of a general experience that too many research dollars are wasted on non-reproducible and irrelevant studies.  An interesting proposal I thought, but perhaps a tad overstated.  

Then I get to the airport and pick up the Economist (March 15-21 2014) and turn to the Science and Technology section and there is an article called Metaphysicians which mentions John Ionnidis ( Why Most Published Research Findings are False ). I have discussed this author and this paper before [see: http://www.medicallaboratoryquality.com/2012/12/quality-and-medical-research.html ] because of his interest in doing “research on research”.  According to the article, a scourge of modern science is the “waste of effort”.  In 2010 “$200 BILLION (85% of total expenditure on medical research) was squandered on studies that were flawed in their design, redundant, never published, or poorly reported”.  Assuming those numbers are true, that would certain confirm there is a clear need for help.

According to Ionnidis too many researchers staggeringly over interpret statistical significance in studies that are far too small.  Further they have a lack of insight in proper study design and manifest “publication bias” where positive data gets written up for presentation and negative data gets ignored or worse.

Over the years we have observed that graduate students working in research laboratories seeming lack knowledge and interest and respect in running essential common equipment such as autoclaves.  Commonly the problem is that they are required to use the equipment, but did not receive appropriate training.  In essence they were just pushing buttons.  ( I suspect in the minds of their Principle Investigators, autoclaves are mundane compared to DNA analysis.)  

Furthermore common procedures of quality control and quality assurance are often incomplete or inadequate and generally not understood.   

What they do not seem to understand is that in the absence of basic Quality principles NO research can or should be trusted.

Apparently Ionnidis has doing something about it.  He has opened up a Center for meta research with the name Meta-Research Innovation Center at Stanford and the even more appealing acronym METRICS.  That will help to define the problems and perhaps develop some answers.  

Here are some questions that I would like to see addressed:

   A.               Would increased training in laboratory Quality result in reduced non-reproducible research and increased value for dollar spent?

   B.               Could accreditation of research laboratories result in reduced non-reproducible research and increased value for dollar spent?

   C.               Would introduction of proficiency testing into research laboratories result in reduced non-reproducible research and increased value for dollar spent?

I was intrigued by the notion of a Best Practices in Research Guideline however I also understand that in the current environment such a document would be pursued by those few laboratories that recognize the concepts of standardization and continual improvement.  The vast majority would, I suspect, never become aware of its existence, or never purchase it, or never read it, or read it but never consider it as relevant or appropriate to their laboratory.

I know this sounds a tad cynical, but for how long have we taken that approach in the health sciences. “It is not our problem, we are too smart, if it is a guideline then we can ignore it and if it is a regulation we can obscure it.”

This may well be the time when far more aggressive research oversight becomes a reality.  If a highly qualified author writes something in Accreditation and Quality Assurance (an excellent journal) its impact is strictly limited.  But if articles end up in the Economist, that is a different story.  Business folks read the Economist, as do lawyers, and politicians, and University presidents, and many of the general public.  

Sooner or later, the right (wrong?) person is going to start putting 2 and 2 together; wasting $200,000,000,000 is an insult to the public purse.   And then the regulations start happening, and some people lose their jobs, and some people end up in jail.

Even if we spend $2B on setting up a stringent oversight we are way ahead of the game.  And the impact on jobs would be negligible because for every research laboratory that is shut down, some would likely move into consultancy or oversight. 

Seems like a win.        

Quality and the Research Laboratory -one more time with vigour

Last week I had the opportunity to make a presentation to a group of laboratory sciences graduate students.  My topic was on Quality, a topic that I suspect was pretty marginal in their sphere of knowledge or interest. 

I had two agendas; first that they should be aware that Quality can be viewed both only as a subjective characteristic based on market influenced notions of value and craftsmanship, and specialness, and at the same time as an objective measurable based on specifications, and requirements, and commitment.  The second was to introduce to them not only that Quality in the objective sense has a role to play in every research laboratory, but to go further, the absence of Quality awareness makes everything that they think and do, null and void.

I started with the notion that there are tiers of Quality, starting from the base of Quality Control, then Quality Assessment, and finally Quality Management. 

To be fair, I acknowledge that not all laboratories will attain a level of achievement that includes a full Quality Management System.  Many clinical laboratories have at best a perfunctory Quality Manual and a pretty iffy Document Control system or Process Control and unfortunately most clinical laboratory directors do little Management Review.  But I am also aware that these are by-and-large completely absent in research laboratories unless their funding agency demands that they demonstrate they follow Good Manufacturing Practices (GMP).   

Many researchers scoff at the notion that they should participate in any form of Quality Assessment, thinking that exclusively means some form of proficiency testing or inter-laboratory comparison, and they forget about the simple basics of internal audit, and competency assessment, that let them know whether anything is being done the way that they think it is supposed to be done. 

But the sad reality is that even the most basic Quality Control is by-and-large absent.  All too often basic assessment of equipment and reagents and supplies through use of control materials barely occurs, and when it is done, it rarely is used critically.  There is little use of control charts (sometimes referred to as Levey-Jennings) and a near complete absence on its interpretation. 

I was not alone in the room being knowledgeable on the subject.  One of the seminar leaders lobbed be a really good softball question.  Is there not value in reproducibility as a reflection of accuracy?  If a value is tested multiple times and the same value is achieved, doesn't that give credibility to the value?   Tempting, but sadly, not.  What it did was open up the conversation on accuracy versus precision and bias.  It reminds me of an age-old description of surgeons, confident, quick, adept and wrong. 

Research is a critical part of health progress, and to be fair there has been great progress of the last couple hundred years.  But investigation is slow and erratic, and expensive beyond expensive.  For every step forward, there are dozens of steps back.  That is the nature of unraveling new knowledge.  But we make the situation worse and not better when graduate students don’t know of understand about the roles of Quality Control and Competency.  I would argue that for every laboratory that uses a standard piece of analytic equipment, there should be availability of programs that provide materials to ensure that the equipment is being used properly.  You could call that a variation of proficiency testing or competency assessment.  I would also call it common sense.  I would further argue that an aware funding agency would require the regular use of such challenges, perhaps even tying funding continuity to performance.

Our universities and training centres have an obligation to teach and guide graduate students to mold them into being excellent investigators.  When we leave the very basics of Quality out of the experience and equation, we are just perpetuating our folly, not fulfilling our obligations for investigation improvement. 

We have another session with the group next week.