No Uncertainty about Uncertainty

In the previous entry I made a comment about the reference to Measurement Uncertainty in the CRC Press book Quality Assurance in the Pathology Laboratory edited by Maciej Bogusz. I noted that the author's sole argument for laboratories to calculate their (more appropriately) uncertainty of measurement was because it was expressed as a requirement in the standard ISO17025. (It is also included in 15189).

I mentioned that in my opinion, there is NO reason for doing something worse than doing it solely because it is a requirement in a standard or that an accreditation body said to do it.

Here are some GOOD reasons why laboratories should perform certain processes and procedures:
1: We created a policy to which we are committed.
2: It is a legal requirement in the places that we work. Adherence reduces the risk of liability
3: It is a customer requirement and expectation.
4: It creates a better, safer work environment.
5: It amends an error and reduces the risk of repeat.
6: Adherence enhances our financial health.

The concept of uncertainty of measurement is an extensive of technical philosophy that no measurement is absolute. Factors including the precision and stability of equipment, consistency and quality of reagents, technical competency and reproducibility of operator skill, knowledge and talent can all influence the result of a measurement. All measurements should have some form of error bars around them.
I have absolutely no problem with that concept. This concept grew out of studies in the physical sciences where the most minor of minor error can result in a rocket being fired at the moon but hitting Mars instead, or where scatter within test results may get interpreted as cold fusion, or where the tiniest of deviations can result in huge alterations in interpretation of collisions in an atomic accelerator.
In all these situations the study and analysis can be under complete control it is appropriate to define the UM and take it into account during interpretation. I got it, I understand it, I believe it.

But here's a news flash. medical laboratories are not closed system research centers. Most of the life of a clinical sample is far beyond our control, and for all intents and purposes is likely to remain that way. The patient and the collection are almost always at a distance from the laboratory, and there are too many variables that can impact on the sample in a way that we can not control There is the technique of collection, the stability of the container and its contents (including specific additives). There is the temperature in the collection site, the duration of time for transport, the temperature at transport, the amount of agitation during transport. There is the amount of time the sample sits on a workbench before it is accessioned.

And those are the ones that we know. How about all the factors we don't know.

Generally have an idea or impression about the uncertainty and impacts of variables, but we have no way to calculate their impact. Metrologists understand this but their answer is that is OK we will just develop a list of all the variables that we can think of and make an ESTIMATE or QUESTIMATE on their value and on their impact. This is called the Uncertainty Budget. So now what we have done is taken a tool that was designed to calculate precise error bar values, but instead ended up with a best guess that may be close or not, may be valid or not, may be reproducible or not. But we have a Number and we can can now tell the accreditation team that we have a number (good), but then we go an tell our customer either as a patient or a surgeon or a cardiologist that we have a number (bad). What we don't tell them is that we have no idea of how much confidence we have in the value. We just give then the value.
That is what I can both misleading and DUMB.

So what should we do. Well we can do some things with confidence. We can test our analyzer repeatedly and we can plot the range of values that we get when we test a Certified Reference Material and we can calculate the analyzers range. Laboratories have done that for years and reported on test trueness, precision and bias. If we interpret quality control tests against that range we can say with confidence that the equipment has a certain allowable error range, and we can say with confidence that if we measured a sample concentration as 6.2 umol/L that the true value is somewhere likely between 6,12 and 6.28.
But that is not generating a value for uncertainty of measurement.

Here is what is so annoying about this. Anyone who has worked in a laboratory understands that calculating precision and bias is an important aspect of being a laboratorian. Anyone who has worked in a laboratory understands that making estimates or guesstimates for variables for which we have not basis for the estimate or guess is a fool's game. So why do we end up with standards with requirements that make no sense to the laboratorian.

It's because dumb things happen around the standards development, crafting, and negotiation table. and once something gets into the standard regardless of how inappropriate it is, and how wrong it is, it is almost impossible to get folks to acknowledge the error and actually fix it.

Resident Quality Seminars as Adult Education

A few weeks ago I wrote (See “Communicating Quality and the Principles of Adult Education - January 30th 2011) that adults learn what they want, and  how they want and accept the knowledge if it makes sense and is consistent with what we already know.  The educator needed to be seen to be organized and pragmatic and relevant and interesting.  If adults are motivated they learn better.  

So with this in mind, can I make any objective comments about my own teaching capabilities?

I got back the results from my Resident Seminar Series.  
 
Background 
To summarize, I put on a seminar series for anatomic and general pathology residents.  The series begins the week after New Years and my sessions start at 08:00 AM.  A draw to the academic half day program is that they get a free breakfast when they attend.  
Before I start the seminar series  I create an anonymous on-line survey that includes a series of 10 questions on quality facts, along with some demographic information.  I then present 7 hours of seminars over 4 sessions and give 1 hour to a guest presenter, and after that asked the group to respond to a second on-line survey which asked the same questions as before and added some additional comments on the series based on a 7 point Likert ranked scale with a maximum value of 6.0

Attendance
The pre-seminar survey had 25 responders.  
On average there were 29-31 attenders to the seminar series.  Nearly everyone attended all 4 sessions.
The post-seminar survey had 14 responders.

Findings
So my first point.  Adult learners do what they want and respond to surveys if there is something on the table.  After the series we were able to get 14 responders (despite repeated requests).  Not getting a better response rate in the post-survey was annoying, but it is what is is.  

My second point:  There was improvement in the responses to all the fact based questions.  There was significant improvement (yates corrected p<0.05) in 2 questions.

My third point:  The responders found the information relevant to their career (5.5/6.0)  informative (5.36/6.0), relevant to their residency (5.29/6/0), and relevant to preparation for examinations (5.07/6.0)  The weakest response was when the group found the information interesting (5.0/6.0).  

Interpretation.  
Overall I can this a success.  Not an overwhelming success, but a success none the less.  

What was encouraging was that after the series, one resident  approached the hospital quality manager to get engaged in a project, and one resident approached me on a similar topic.

The increase in knowledge performance was ojectively positive and in part significantly so.  And there is a powerful opinion and recognition that knowing something about quality is important in a laboratory career.

Discussion and Conclusion
The objective measures point to the seminar series as a measurable success with opportunities for improvement.

Teaching my on;line course is the ideal adult education situation.  The participants seek us out because they want the information.  There is money on the table.  They feel free to participate actively.
With residents, they are present because they feel required to be there, even if they don’t care.  At worst they still get a free breakfast.  There is no money on the table.  They feel free to drop out if they are not being entertained.   

Note to myself:  
Look forward to doing the seminar series again, maybe next year, more likely in 2 years.  
Make some changes to beef up interest.  
And importantly don’t take the weak response rate too personally.  If they really thought the series stunk they probably would have taken full advantage to let me know.  

PS:
The information will be presented more completely and formally during the POLQM Quality Weekend Workshop in June.  There will be an education breakout session.

The Science of Qualitology

I like the ASQ’s Quality Management Journal because it publishes articles in a science and experimental structure that I understand and expect to see in a journal.  The article that I was looking at was analyzing factors associated with Quality in hospital settings. (seeR.E Carter, S.C. Lonial, and P.S. Raju.  2010.  Impact of Quality Management on Hospital Performance: An Empirical Investigation.  QMJ.  17(4): 8-24).

The study design was based on a survey sent to hospital executives in 175 organizations in mid-US (Kentucky, Ohio, Tennessee, Minnesota, and Mississippi)). The surveys were sent to Hospital CEOs who were in turn supposed to pass them on to senior folks like the VP administration, Quality manager, Support services manager, Director of nursing.  This was very ambitious.

The conclusions they came to were what I would expect; when it comes to quality size and stress matter.  The more uncertainty in the institution, the larger the institution, the less likely they were to have “measurable” evidence of Quality. 

The “measure” of Quality in this study looked at 5 markers for financial performance, 4 markers of market/service development and 4 markers of quality outcomes.  That, in my opinion was a set unlikely to give a clear picture of hospital quality.

And that brings me to my point. 
What are the objective measures that one can monitor as an indicator for success or failure for introduction of Quality activities in medical laboratories?
Not success in accreditation or proficiency testing scores. They are too readily manipulated  (see   M.A. Noble.  2007.  Does External Evaluation of Laboratories Improve Patient Safety?    Clinical Chemistry and Laboratory Medicine.  Clin Chem Lab Med.  45(6):753-756). 

Not numbers of reported incidents or OFI’s.  They are too open to flexible interpretation.  OFI reports, if anything are like unemployment rates.  A downward movement in rates may mean more people are being employed, or it may mean that fewer people are bothering to look.  And a rise may mean more people are unemployed, or it may mean more people are hopeful and are again more actively looking.  In the same way  a rise in the OFI’s rate may mean more problems are being identified and reported meaning poorer Quality, or it may mean more engagement leading to more reporting meaning better Quality.

How about client or staff satisfaction?  Maybe, but again, very manipulatable and too vague.
And in Canada, financial stability or instability are completely inappropriate since 99 percent (or more) of resources come from the government purse.

So we have a dilemma.  For good studies we need measurable and interpretable and  monitorable outcomes on both a micro- and macro-  basis. We do this on a micro- scale all the time (call that Quality Indicators).  But to move from interesting to convincing and compelling, we will need to define our macro- outcomes as well. 

For Quality to create a lasting imprint in medical laboratories, we are going to have to speak the language of laboratory personnel, pathologists and technologists.  We will need the language of science and experimentation. outcome and conclusion. 

Any and all ideas are most certainly welcome.
m

PS: Absence of strong interpretable measures makes grant funding difficult, maybe impossible.  I have learned this the hard way.