Hierarchies of evidence can be classified as exhaustive or inexhaustive. A hierarchy is exhaustive if and only if it gives some ranking to any given piece of medical evidence – there’s nothing it leaves out. But there’s a dilemma which hierarchy authors face: either way, hierarchies of evidence present a serious issue.
There are several ways in which a hierarchy might be inexhaustive. It might just overlook a particular type of evidence (e.g. it doesn’t include subgroup analyses). It might be limited to a particular category of evidence – for instance, many EBM hierarchies and earlier clinical epidemiology hierarchies were limited only to epidemiological evidence and omit mechanistic evidence, experience, laboratory research, etc. It might only take account of single pieces of evidence, and not take account of ways of amalgamating evidence, such as meta-analyses, systematic reviews, databases and consensus conferences, which (contrary to some arguments) are distinct from the underlying evidence because they can provide distinct insights. It may just rank the currently existing methods; it has no catch-all-else category to take account of possible future developments in the methodologies of medical research.
The Exhaustiveness Problem is the following dilemma:
If a hierarchy is exhaustive, there is no real room left for developing future methods. The only way to fully cover new methods, it seems, is to have some catch-all-else category – but this category will probably be ranked low (lest I invent a stupid new method), meaning that if I come up with some innovative method better than ever RCTs, it will still be low-ranked. There are two ways to grasp this horn of the dilemma. One is to argue that it is not possible to innovate any new research method which is superior to RCTs. I’m yet to see any argument to this end, and it seems like something of a failure of imagination (as well as omitting consideration of new techniques like machine learning which might prove fertile methodological ground). The other option is to put forward an adaptive model of hierarchies, in which each iteration of a hierarchy is presented with a conscious awareness that it must be revised regularly when new methods are developed. In this case, we shouldn’t act as if the hierarchy is final in any sense, and the hierarchy will not license inferences about whether we have the best possible evidence.
If a hierarchy is inexhaustive, it no longer follows from the hierarchy that RCT evidence is the best evidence. This undermines arguments and positions which attempt to base practice on it. It might be that the best evidence isn’t even on the hierarchy. Guiding practice on the basis of an inexhaustive hierarchy can at best form a part of a fuller approach to evidence, at worst seems irresponsible.
First written 28/01/2012, revised 08/07/2015.