Five years ago today I submitted my PhD thesis, Hierarchies of Evidence in Evidence-Based Medicine, at the London School of Economics. In those five years, the work remains my most well-read piece. Like most PhD students, I never expected it to be noticed by many more people than my supervisors and examiners. But the thesis has been downloaded more than 7,500 times on LSE’s website alone, surpassing Arhat Singh Verdi’s 2010 thesis What is Truth? as the most read Philosophy thesis in the institution’s repository, where it continues to collect hundreds of new reads per month. It started to pick up dozens of citations from a weird and wonderful cross-section of academic literature – everything from recommendations on thromboprophylaxis in major joint arthroscopy, to self-management behaviours in dementia care-givers, to data-driven approaches to sustainability. I was fascinated to see the critique of evidence hierarchies diffuse out from its home in medicine to other fields which find themselves under pressure to conform to the influence of an evidence ranking which was originally designed to express views about the state of clinical epidemiology. I was proud to see the thesis featured by Tom Siegfried in Science News. Touring medical conferences of all stripes, and later in other fields struggling to articulate their clash with the ideals of evidence-based policy, I found that there was certainly an appetite amongst practitioners and policy-makers to subject hierarchies to critical scrutiny and to the same demand for evidence which EBM made of practitioners the world over.
Since the piece appears to have continued traction, and as I have continually worked on issues in evidence and ethics in medicine in the years since, the five year anniversary seemed a good time to collect some thoughts on what has changed – and what has not – since September 2015.
- 1) Practitioners are increasingly uncertain about how to understand hierarchies
The most impactful experiences of the last five years, for me, are in seeing the disconnect between different practitioners in how they understand evidence hierarchies, and how they expect others to. There are two polarised responses to work in the philosophy of EBM when I address medical conferences. From the same room, I regularly hear both that I have overblown what everyone knows to be a loose-held heuristic which is not taken seriously by well-informed practitioners, and that I am mounting a radical and anti-scientific challenge against a fundamental pillar of what it means to do medicine. Thankfully, the two camps of critics usually answer each other. My approach is to let them fight it out over the buffet table, as the most important thing which adherents of either extreme can learn from me is that their own view is far from universal. To be sure, some practitioners definitely think that the evidence hierarchy is a fixed, immutable fact. Others think it is a useful rule of thumb with rare exceptions. Still others find it reductive and restrictive, a device used to knock serious scientific research out of contention if it does not fit the statistically orthodox mould. When the test of evidence is trial by twitter, there may be limited space for the nuance of non-standard methods.
- 2) Evidence hierarchies are far from dead or declining
Some proponents of EBM seem increasingly embarrassed by evidence hierarchies. To them, hierarchies are a relic of a less sophisticated period in the movement’s history. They have moved on. I often hear that evidence hierarchies – while a firm fixture of the early Evidence-Based Medicine canon – are on their way out or already six feet under. But if EBM has moved on, both practice and practitioners have not. Hierarchies remain widespread in medical school curricula, editorial policies, funding body criteria and policy doctrine. There is a proliferation of new hierarchies, revisions and reorderings independently of the more nuanced (but still ultimately RCT-beholden) GRADE approach.
In particular, evidence hierarchies are expanding their reach into domains far beyond healthcare. As evidence-based policy offers a convenient bulwark amidst anti-scientific rhetoric from populist politicians, hierarchies slip inside more and more social scientific research in areas such as education, criminal justice and social work. Unfortunately, there seems to be a tendency to import the most stripped-back and simplified of hierarchies. Sophisticated pseudo-hierarchical approaches like GRADE rarely make the jump, perhaps because the nuance and improvement over the simple lists of old is so intimately connected to authors’ understanding of the specific evidential needs of healthcare. Hierarchies remain influential within and beyond medicine. Whether the view of hierarchies as inviolable laws of science or as weak heuristics is expanding, though, is less clear.
- 3) Foregrounding heterogeneity of treatment effects is radical until it’s really not.
The core argument of my thesis is that hierarchies of evidence misidentify what evidence medical practitioners need in order to deliver the best outcomes for patients. Evidence hierarchies generally assess how accurately and reliably a study design can measure the differential average treatment effect – that is, the difference between the average outcome in one treatment group and the average outcome in another. But what practitioners really want evidence about is the range, consistency and causes of variation in the effects of treatments. That is the crux of a great many debates in medical science. By forcing the debates to be filtered through the lens of the RCT, though, we deform and distort the scientific problems which practitioners care about.
This view was received as distinctly unorthodox when I presented it in the run-up to publication of my thesis. But there was already a coalescing movement around this idea – which is very far from novel – under names like ‘personalized medicine’ and ‘precision medicine’. While each of these movements have brought different concerns to the fore (particularly focusing in overly reductive and restrictive terms on genomic medicine and genetic markers of disease and treatment response), there remains a long way to go if medicine is going to be able to reverse decades of statistical and philosophical degradation and assert that a truly scientific medicine must look to provide the evidence practitioners need, not the evidence which is easiest to produce through a monolithic method or defend on statistical grounds.
Of course, this sort of approach has had some high-profile adherents in recent years, perhaps none more so than President Barack Obama, whose genomic dream and funding glut for precision medicine put a lot of eyes on the challenges of identifying, understanding and predicting variation in treatment effects. The sprint towards machine learning in healthcare has done much to amplify clamour for the use of big data to tailor and target treatment. But despite the investment and promise, there is a long way to go before any of the bold bets made by adherents of these movements begin to pay out. We should be wary of swinging too far back the other way. The RCT has always had a role to play in medical science. We can reject the reductive equation of medical science with RCT evidence and promote the use of scientific method – theorising, prediction, severe tests and progressive revision – without repudiating a valuable method in the process. If we succeed, we’ll likely have to do it all again (and support those who already are) in other fields reeling to cope with their own evidence-based movements.