Do not centralize evidence. If you believe that all the evidence you need to make your decisions is of a single kind, you will be so much easier to mislead. Those who say ‘I’ll believe it when I see this kind of study’, once their singular preference is declared, are so much easier to mislead, dupe or otherwise fool. A smart person believes only on the basis of a diverse base of evidence.
In my work, I have been defending a single thesis, albeit in various forms: if you believe that a single form of evidence can answer all of your questions, prove all of your claims, then you leave yourself susceptible to misconception and deception at an industrial scale. Here, I will claim this only with respect to medicine – that the idea that a single form of clinical trial can provide all the answers that medical practitioners need will first embitter and then endanger patients. However, I believe it applies far more broadly across the gamut of evidence.
Hierarchies of evidence have placed Randomised Controlled Trials on a pedestal as the ideal study, producing the ideal form of evidence. Some proponents of Evidence-Based Medicine have argued for increasingly basing treatment and policy decisions on the results of such studies, by virtue of them being the best form of evidence. On the back of this, we have a proliferation of approaches to generating misleading RCT results, alongside mismatched attempts to force a fit between a very restrictive and particular methodological approach on the one hand, and the broad-ranging and varied concerns of clinical medicine on the other.
There is no such thing as ideal evidence. Evidence generation involves trade-offs. You trade precision against generality. Study size conflicts with consistency of interventions. Studying a diverse population makes controlling for variation harder, while studying a homogeneous one jeopardises generalisation. Every decision in designing a study has harms and benefits for the kind of evidence produced. There is no study design which avoids all ills. We should stop pretending that any methodology can fix this problem and acknowledge and agree that no individual study will do. Nor is it necessary to try to force the RCT methodology to fill each point on these spectra – it is possible that approaches designed and refined to answer a distinctive question will do it better than an adaptation of the RCT design, and without sacrificing the original’s benefits.
There is a market for evidence. Rival approaches can fill distinct niches. There is a huge range of information to be acquired. There is no need for every scientist, every practitioner-researcher and every pharmaceutical company to retrace the same paths. But we have rushed towards what is prized as the sweetest low-hanging fruit: an average effect. The mantra is: determine the average effect of some treatment as precisely as possible and in the broadest possible population. In reality, this is not the most important information to clinicians or patients. What is important is for clinicians and scientific medical researchers to have at their disposal a range of information: about averages, sure, but also about variation, about range, and about predictors of variation.
Unfortunately, we have missed the competitive aspect of the marketplace of evidence. Where competition exists in medical evidence, the stakes are: who can produce the best trial to provide the most precise estimate of the average treatment effect in some population. This is false competition. A true competition in evidence involves attempts to innovate and deviate, producing new things which are appropriate to different concerns and questions. A detailed and thorough evidence-base is not one which has the most trials of the single ‘strongest’ kind, but rather the one that is most varied, contested and elaborated. The hallmark of scientific medicine will be an array of approaches applied in conflicting ways to different interpretations of a problem. It is only when we can draw upon studies which employ diverse approaches and produce distinctive contributions that we can truly claim to have an evidence base from which to make claims about the effects which treatments have.