An Honest Mess

Evidence-Based Medicine is an attempt to simplify and streamline a complex reality into a more manageable structure. Reflective EBM proponents know that matters are very complicated. But they understand that practitioners have very limited time (and abilities) to deal with these complexities. They must use heuristics and simplifications. Sometimes this will lead to sub-optimality in patient care. Overlooking or undervaluing certain evidence can produce oversights—effective treatments are not used, treatments are used when they are predictably ineffectual, avoidable side-effects are incurred. But the truth is: in practice, sub-optimality is presently unavoidable. For now, we cannot take everything into consideration. We must always overlook or undervalue some information, some factors, and some possibilities. We suffer from the frame problem: we have to pick the options which we’re going to consider and the factors which we’ll take into account somehow—and that somehow cannot include appraising all the others.[1]

In short, we should not expect to take everything in consideration. That is not pragmatic. Michael Lewis asked whether “an honest mess” was preferable to “a tidy lie”. An honest mess is not practical: leaving practitioners to wade through huge amounts of information without guidance cedes optimality to the abilities and perceptions of the individual clinician. There will be a great deal of variation in optimality of treatment, and in comprehensiveness and coherence of approaches. Clinicians will face the frame problem individually, and their individual framing assumptions will determine predictable optimality. Sub-optimality is unavoidable. So EBM will be a success if it decreases sub-optimality. We cannot ask to remove it altogether. We should not expect even to minimize it. If EBM practitioners treat patients less sub-optimally than non-EBM practitioners, then EBM is a success.

But there is an important problem to be acknowledged in the tidy lie. Tidy lies are often so tidy and aesthetically-pleasing that they are widely apprehended as truths. The classical notion that truth is simple and elegant, that a simpler theory is a better one, confers the ring of truth upon a tidy lie.[2] Suppose we forget that the tenets and processes espoused by EBM proponents are heuristics—ways to achieve better results by selectively ignoring certain relevant information. Suppose we instead come to view them as basic principles, truths; suppose we take them without the pinch of salt. Then we might come to the mistaken view that the information (and indeed the entire kinds of information) which is sidelined or overlooked by EBM is inherently not valuable. We might come to believe that considering this information would not increase treatment optimality. The biggest and clearest sin is: we might come to believe that we cannot do better than apply an EBM-style method. We might stop viewing EBM approaches as a pragmatic shortcut for time-pressured and resource-limited clinicians. We might use the principles of EBM to justify ignoring the information which EBM sidelines (due only to a lack of time and resources) in principle and always.

Is an honest mess preferable to a tidy lie? I’d like to skirt around the question, and defend two oxymorons: a tidy mess, or an honest lie. The honest lie is perhaps the clearest way to retain EBM as a heuristic without mistaking EBM’s principles for fundamental rules and methodological truths. If we are going to use the tools developed by EBM proponents, then we should always be clear that they are a stark oversimplification, and that many of the forms of evidence and information relegated to indifference by EBM approaches are important and relevant. The information which EBM proponents sideline is information which can improve patient outcomes and policy decisions, and ultimately save lives. It is not information which can be universally replaced by EBM’s preferred sources. We should be honest about the lie. It is uncomfortable to honestly expose our lies, but ultimately less dangerous than mistaking them for truths. In my work, I’ve been arguing that hierarchies of evidence have reinforced the lie and been consistently interpreted as a deeper truth, which makes them dangerous.

Finally, I might suggest that a tidy mess — messy territory that is well-mapped and charted, with some small honest lies to simplify the mapping process — might be a workable pragmatic approach to clinical practice. Whether a sufficiently tidy mess can be achieved is an independent matter from whether EBM can be harmful through the misunderstanding of a tidy lie as a tidy truth, or whether this can be corrected by rearranging EBM as an honest lie. But if such a map could be drawn, and could retain the practicality of EBM approaches while mitigating some of the downsides, then we could move closer to the pragmatic ideal in clinical practice.

What I call the ‘pragmatic ideal’ is that approach which is both pragmatic, in the sense of practical and practicable, and which has the least predictable sub-optimality of any similarly pragmatic view. There is predictable sub-optimality when we are aware that ignoring certain kinds of information can lead to sub-optimal treatment. That is, we can predict that (sometimes) ignoring or undervaluing this information leads to sub-optimal treatment, given the surrounding framework. The pragmatic ideal does not necessarily lead to the least actual sub-optimality in practice. There may be unpredicted or unpredcitable sub-optimality from several sources. A pragmatic ideal only minimizes predictable sub-optimality compared to other pragmatic approaches.

 

[1] In the classic version of the frame problem, a robot is unable to solve a problem (a bomb about to explode) because it must consider all possible solutions—which takes so long that the bomb explodes before it can act. Here, we have the problem that we cannot consider all possible treatments and all patient-features which could potentially interfere with the effectiveness of those treatments in time. Sometimes this will be due to similar urgencies. Usually, it is simply due to the overwhelming size of the task and the finite resources of clinicians and data analysts.

[2] The question of simplicity as an epistemic virtue is an interesting one. David Lewis’s famous optimism that “nature might be kind” and obey a simple, clear best theory (should we be smart enough to find it) fits badly with the experience of social and pragmatic scientists in fields like medicine. A better view might be that simpler theories are more likely to be wrong in medical science—more akin to John Green’s slogan: “The truth resists simplicity”.