195 Hierarchies: A Systematic Database Update

Hierarchies of evidence have long been, and remain, a fundamental part of Evidence-Based Medicine. Hierarchies are a way to rank or assess the quality, strength or level of evidence based primarily on the methodology by which a study was conducted. My 2015 Thesis analyses and criticises these structures in detail. As part of that project, I assembled the first comprehensive database of evidence hierarchies, which was first released online on 20/12/2011. Back then, the database contained around 60 hierarchies. It was updated to include around 85 hierarchies in 2016 after the publication of the thesis.

But since then, many more hierarchies have been published and some which passed undetected before have come to light. The work of Joseph Vere in particular has been important in approaching hierarchies of evidence and putting new light on the range and variation of hierarchies available to practitioners. Vere also rightly noted that the database of hierarchies which I had assembled was limited by the inexhaustive search criteria used.

I therefore resolved to conduct a systematic review of the medical literature to identify any hierarchies which were missing from the database. I completed this review on 13/11/2017, but due to other commitments was unable to update the database at that time. Returning to this project after reading Vere’s latest paper, I have now added all of the hierarchies which were identified in that review to the database, as well as conducting a further review to find any hierarchies published since 13/11/2017, up to 08/10/2020. In all, this takes the total number of distinct hierarchies in the database to 195. A full list of the additions can be found at the foot of the Database page.

In the process, I found that whilst the GRADE ranking system is quite widespread, the most cited and used hierarchy in the medical literature appears to be the Australian NHMRC ranking. I also saw that there remain many new hierarchies being published, and older designs being adapted and reformulated for new applications. This is true both within EBM’s traditional domain of clinical medicine and beyond: in dentistry, nursing, public health, medical education and more. It is also very clear that pyramid-style presentations of evidence rankings are increasingly common and widespread in the medical literature – a concerning development given that these ways of presenting evidence hierarchies tend to be considerably less nuanced than the likes of the Australian NHMRC or GRADE rankings.

Search protocol:

The following search protocol was used to identify unique hierarchies of evidence. PubMed/MEDLINE and CENTRAL were searched, using the search terms “hierarchy of evidence”, “hierarchies of evidence”, and “evidence hierarchy”. A supplementary search of Google Scholar for the same search terms was performed to locate other sources of hierarchies, following the same procedure. All identified accessible English-language books, articles and reports were screened to determine whether any object (whether textual or graphical) matching the definition of an evidence hierarchy was present. Any hierarchies found were added to the database. A hierarchy of evidence was defined as per Blunt 2015, as :

“A procedure for making an inference about some epistemic property of the evidence provided by studies, or some epistemic relation which holds between the evidence provided by two or more studies, primarily based on a ranking of the design or methodology used in the studies.”

If a hierarchy was specifically cited as originating in another source by the authors, the prior source was followed up and checked. Where the hierarchies were identical, the prior source was added to the database, and the replication was removed. It was not uncommon for the cited sources to contain no hierarchy or a substantively different hierarchy. In these cases, both were included. As such, there are many cases in which a text included in this database explicitly states that its hierarchy is not novel, but has been included.

If the new entry was found to be identical to any previous uncited hierarchy (either already or subsequently detected by the search protocols), it was removed from the database. For the purposes of this review, two hierarchies are “identical” where they have the same number of levels and sub-levels, the same interpretation property, rank only the same methodologies or evidence bases, and have identical presentation in terms of conditions and modifiers. The earliest published source is always retained, except where priority cannot be conclusively established, in which case both papers are cited under a single record.

In addition to sources found through this search, or cited in sources found through this search, any papers not identified in this search, but cited in previous systematic reviews of evidence grading systems, were also included, as were all papers previously cited in Hierarchies of Evidence in Evidence-Based Medicine (2015). These papers were reviewed according to the same procedure as those found via the searches.

You can find the full database here.