Researcher beware - change is coming

The problems with research

It’s always interesting when things come together.  

Over the last few days I was at an international meeting where the discussion about the potential value of a Best Practice Guideline document for Researchers was raised because of a general experience that too many research dollars are wasted on non-reproducible and irrelevant studies.  An interesting proposal I thought, but perhaps a tad overstated.  

Then I get to the airport and pick up the Economist (March 15-21 2014) and turn to the Science and Technology section and there is an article called Metaphysicians which mentions John Ionnidis ( Why Most Published Research Findings are False ). I have discussed this author and this paper before [see: http://www.medicallaboratoryquality.com/2012/12/quality-and-medical-research.html ] because of his interest in doing “research on research”.  According to the article, a scourge of modern science is the “waste of effort”.  In 2010 “$200 BILLION (85% of total expenditure on medical research) was squandered on studies that were flawed in their design, redundant, never published, or poorly reported”.  Assuming those numbers are true, that would certain confirm there is a clear need for help.

According to Ionnidis too many researchers staggeringly over interpret statistical significance in studies that are far too small.  Further they have a lack of insight in proper study design and manifest “publication bias” where positive data gets written up for presentation and negative data gets ignored or worse.

Over the years we have observed that graduate students working in research laboratories seeming lack knowledge and interest and respect in running essential common equipment such as autoclaves.  Commonly the problem is that they are required to use the equipment, but did not receive appropriate training.  In essence they were just pushing buttons.  ( I suspect in the minds of their Principle Investigators, autoclaves are mundane compared to DNA analysis.)  

Furthermore common procedures of quality control and quality assurance are often incomplete or inadequate and generally not understood.   

What they do not seem to understand is that in the absence of basic Quality principles NO research can or should be trusted.

Apparently Ionnidis has doing something about it.  He has opened up a Center for meta research with the name Meta-Research Innovation Center at Stanford and the even more appealing acronym METRICS.  That will help to define the problems and perhaps develop some answers.  

Here are some questions that I would like to see addressed:

   A.               Would increased training in laboratory Quality result in reduced non-reproducible research and increased value for dollar spent?

   B.               Could accreditation of research laboratories result in reduced non-reproducible research and increased value for dollar spent?

   C.               Would introduction of proficiency testing into research laboratories result in reduced non-reproducible research and increased value for dollar spent?

I was intrigued by the notion of a Best Practices in Research Guideline however I also understand that in the current environment such a document would be pursued by those few laboratories that recognize the concepts of standardization and continual improvement.  The vast majority would, I suspect, never become aware of its existence, or never purchase it, or never read it, or read it but never consider it as relevant or appropriate to their laboratory.

I know this sounds a tad cynical, but for how long have we taken that approach in the health sciences. “It is not our problem, we are too smart, if it is a guideline then we can ignore it and if it is a regulation we can obscure it.”

This may well be the time when far more aggressive research oversight becomes a reality.  If a highly qualified author writes something in Accreditation and Quality Assurance (an excellent journal) its impact is strictly limited.  But if articles end up in the Economist, that is a different story.  Business folks read the Economist, as do lawyers, and politicians, and University presidents, and many of the general public.  

Sooner or later, the right (wrong?) person is going to start putting 2 and 2 together; wasting $200,000,000,000 is an insult to the public purse.   And then the regulations start happening, and some people lose their jobs, and some people end up in jail.

Even if we spend $2B on setting up a stringent oversight we are way ahead of the game.  And the impact on jobs would be negligible because for every research laboratory that is shut down, some would likely move into consultancy or oversight. 

Seems like a win.        

Saving Science: The Argument for Quality Assurance

About a year ago, I wrote on the subject of challenges in the medical research laboratory community, with particular reference to an article in a newspaper article and a published article by John Ioannidis.  [ See: Quanlity and the Research Laboratory  - http://www.medicallaboratoryquality.com/2012/12/quality-and-medical-research.html ]  The discussion continues with the October 19, 2013  edition of The Economist: Unreliable research - Trouble at the lab.

The article presents to a number of studies that point out that the process of peer review is badly flawed (not really a surprise!!) but it points to an actual viable solution. 

The story references John Bohannon, a biologist at Harvard, recently submitted a paper on cancer research 304 journals describing themselves as using peer review. What made this article different was that (a) the paper was sent under a pseudo name and (b) it was intentionally flawed with errors in study design, analysis and interpretation of results. What was problematic was that 157 of the journals accepted it for publication.

Consistent with that observation, Fiona Godlee, editor (1998) of British Medical Journal, did a similar study by sending a similarly flawed article to more than 200 of the BMJ’s regular reviewers. On average, only 25 percent of flaws were reported.  No one picked up 100% and some reported none.

BMJ then did a follow-up study in which they told reviewers it was a test; despite this performance did not improve.  And a subsequent study from UCSF indicated that over time reviewer performance did not get better, it got worse.

This is pretty gruesome.

When credible journals like the Economist get on a subject, you may want to dispute some or all of their facts and ideas, but the reality is that their international impact on public and government and funding organization opinions is substantial.  Articles like this can make 1% or 5% or more impact on research funding, without causing a single eye to be bat.  In a time of economic fragility, saving a billion or two from research budget can seem like an easy way to deal with budget crunch.  We can ignore it, or slur it, but we do this at our collective peril.

Wearing my “Gee, what a surprise” hat, let me say that all this supports my argument that institutions need to clean up their act considerably.  More importantly it reinforces my argument that traditional published knowledge is no more reliable than information well presented by a trusted person in a standardized structure, but submitted electronically through instruments including blogs.  

But wearing my other “let’s get on and fix this” had, it seems to me there is a viable solution.  Medical and non-medical testing and calibration laboratories can be improved through quality assessment.  In our own recent examination, 75 percent of laboratories that were correcting proficiency testing error found system errors that impacted on their routine testing.  [See: http://www.medicallaboratoryquality.com/2013/08/proficiency-testing-does-improve-quality.html ]

I propose the following:  all credible journals need to develop a quality assessment strategy for all manuscript reviewers. 

1.    Reviewers are selected by virtual of their knowledge and skills in particular disciplines.

2.    Selected reviewers are retained for a finite period of time.

3.    Selected reviewers are expected to participate in proficiency testing of their reviewing skills on a regular basis, arbitrarily twice a year. 

4.    Reviewers who are found proficient, can continue on with reviewing.

5.    Reviewers who are found to have opportunities for improvement, either demonstrate their improvement, or move on.

It seems to me this is a viable solution, and is consistent with a variety of proficiency challenges that we do on a regular basis.  Acquiring manuscripts that are deemed flawed or not flawed would not take a long time.  Editing for freshness would not be particularly challenging.  

The review process would improve.  The quality of published literature would improve.  The confidence in published literature would improvement.