Hierarchies of evidence in Evidence-Based Medicine (EBM) come in many varieties and have been very influential in medical practice and policy since the late 1990s. However, two fundamental problematic assumptions underpin the use of hierarchies of any kind in clinical practice: (1) that evidence can and should be appraised in isolation, and (2) that clinical treatment decisions made on the basis of information for which there is strong evidence are the most likely to be good decisions. Both of these assumptions are mistaken, and therefore that hierarchical methods of appraising evidence in clinical practice are suboptimal.
In order to have the best chances of making a good decision in the clinical setting, we need evidence for each stage in an inference to a claim about the likely effects of potential treatments on the individual patient. Crucial to this process is information about the heterogeneity or homogeneity of the treatment effects. Different patients react differently to the same treatment regimen. However, some treatments produce more consistent effects than others. Some treatments are effective for some patients, but ineffective or harmful for others. In many cases, it may be possible to identify features of patients which modify the effectiveness of treatments and can serve as predictors of high or low responsiveness to the treatment, or high or low susceptibility to adverse side-effects, and thereby predict which patients are likely to receive more or less benefit than average from the therapy. Knowing the average treatment effect in a population, no matter how confidently and accurately, is not sufficient to make a strong evidence-based prediction about the likely effects of a treatment on an individual. This information must be accompanied by information about the heterogeneity and predictors of variation in the treatment’s effects. Basing inferences on averages alone smuggles in an implicit assumption of homogeneous treatment effects. This step in the inference should be based on evidence.
Hierarchies of evidence enshrine the idea that if there is a strong evidence for information used in making a decision, then the clinician has found all of the relevant evidence. But this ignores the possibility that the information given is only part of that relevant to the decision. Hierarchies prioritise methodologies like RCTs which are adept at providing evidence about the differential average treatment effect between two treatment regimens in some population. But clearly evidence about variation in effects and causes of variation and about side-effect profiles are important and relevant to most patients’ decision-making procedures. Giving information which is evidence-based is not the fullest service for the patient where the information given is incomplete.