This post relates to a small but significant database update to the database of Hierarchies of Evidence.
I have previously discussed some of the controversy about the origins of evidence hierarchies. Some cite Campbell and Stanley’s (1963) “Experimental and Quasi-experimental Designs for Research” as the inspiration, while others (I think, correctly) identify a Canadian Task Force on the Periodic Health Examination report (1979) as the origin. But less attention has been paid to the more recent development of reformatting evidence hierarchies as pyramids. My thesis (Blunt, 2015) sidelined the pyramidal turn as an uninteresting distinction of format. But in the years since, pyramids have more or less taken over as the main way to depict an evidence hierarchy, largely replacing the standard list or table style, in competition only with more sophisticated presentations like GRADE (2004).
Following some recent correspondence with a colleague in epidemiology, I was prompted to revisit this lacuna, and ask where the evidence pyramid originates. The database I have compiled based on my thesis and more recent systematic review contains several different pyramidal formulations. The earliest amongst these which would be recognisable as such to a modern aficionado of evidence pyramids was Wagoner et al.’s 2004 pyramid, which was included in SUNY Downstate Medical Center’s EBM Tutorial course. This rather brash pyramid looks as so:
Already, we should note that this is a unique evidence hierarchy in itself, independently of any claim to be the original evidence pyramid. For one, it has a run for “Randomized Controlled Double Blind Studies”, which is an unusual conditional framing of RCTs. There is no rung for unblinded RCTs, which is interesting in itself. Furthermore, the framing of expert opinion as “Ideas, Editorials, Opinions” is unique to this pyramid, and the inclusion of “Animal research” and “In vitro (‘test tube’) research” are also both unusual amongst the 196 evidence hierarchies I have studied. What’s more, the choice to place ideas and opinions above laboratory research is a startling and unusual one – while they are often condemned to the same low level, relegating this bench research to a station below even “opinions” is a strong claim indeed.
A great many sources in the EBM literature, including my own thesis and database, cite this SUNY pyramid to 2004. The reasons for this may vary – in particular, there may be a case of citogenesis here, in which earlier citations are replicated into newer works. For me, there were two reasons for citing this to 2004. First, this is how it was cited in several works which directly referenced or even included the pyramid, from which I discovered its existence. Second, in 2011 when I first accessed the SUNY EBM Tutorial which contained the above image, the page included a footnote stating “Last updated January 6, 2004”. This was the best option to pass for a date for an online resource of this kind. Citing grey literature such as this tutorial page for a SUNY course has rarely been well-standardised, and that shows in the way this literature has been cited across EBM texts. Since 2004, this is the citation that the majority of papers that I have seen, in the EBM literature, use for this pyramid.
Almost all papers, mine excluded, simply cite SUNY as the authors here. However, further information about authorship was available elsewhere on the EBM Tutorial website, which credits Betty Wagoner, Reference Librarian at the Medical Research Library of Brooklyn, for preparing the original material, alongside co-author Martin Mellish, and a range of consultants and editors. Correspondence is asked to be directed to Dr. Andrea Markinson. As such, I attribute the evidence pyramid to Wagoner et al., and it is my assumption that either Betty Wagoner or potentially Andrea Markinson are the likely originators of this pyramid.
However, in my systematic review process, one article which was detected undermined this simple timeline. Although not containing a novel hierarchy of its own, a paper by Evidence-Based Dentistry proponents Jane Forrest and Syrene Miller of the University of Southern California contained a black-and-white version of the SUNY evidence pyramid. This paper was published in 2001, implying that the SUNY pyramid must already have been available online at that point (see Forrest & Miller, 2001a).
The Internet Archive’s Wayback Machine allows us to view archived versions of old sites, periodically taking snapshots of what is available online. But searching using the Wayback Machine is not simple – really, a URL is needed to explore their records. The commonly cited URL for the SUNY pyramid is: http://library.downstate.edu/ebm2/2100.htm. Nothing remains at that site, but the Wayback Machine has 31 captures of the page dating between 2004-2021. The earliest of these is on 23rd March 2004. This page lists “Last updated: January 6, 2004”, so does not help in dating this artefact.
However, there have been two distinct versions of the SUNY Downstate EBM Tutorial, which the library team list as version 1.3 and version 2. The commonly cited hierarchy which dates from 2004 is part of the version 2 tutorial. The version 1.3 tutorial can be found by removing the ‘2’ from the URL, using http://library.downstate.edu/ebm/2100.htm. Again, this page is no longer available online. But importantly, the first time it was archived on Wayback Machine was on 29th August 2002. At that point, we find a page for the evidence pyramid. However, the image itself is not available, casting a little additional doubt over whether the pyramid was available through this site at that time. Unfortunately, neither do these archived pages contain a “last updated” statement to indicate when prior to August 2002 the page was most recently altered. This, then, still would not definitively account for the presence of the pyramid in Forrest & Miller’s 2001 paper.
Using Wayback Machine, we can navigate to the Table of Contents for the SUNY EBM Tutorial page. The earliest archive for this page is on 5th January 2001. That archive of the contents page lists: “Last Updated: August 15, 2000”. That contents page, at that stage, lists “A Guide to Research Methods”, of which the “Evidence Pyramid” page was a part. So, potentially, we could infer the existence of the pyramid in at the latest August 2000, which would, at least, be consistent with Forrest & Miller’s 2001 publication in the Journal of Evidence-Based Dental Practice. We cannot find documented proof that the SUNY pyramid was online at this site prior to 2002, but the journal article from 2001 and contents page apparently dating to 2000 does suggest that the pyramid could date to 2000 at the latest. This interpretation is reinforced by the identical (albeit grayscale) pyramid offered in Forrest & Miller’s 2001 paper to the one included in the ‘Version 2’ of the SUNY course in 2004 and beyond.
A final twist finds some resolution, however. I searched the literature for “evidence pyramid” and “SUNY” for papers prior to 2004 to try to identify any further citations of the original version of the pyramid which might offer further detail or identify the pyramid prior to 2001. Very few results were returned, with all but one of these papers being written or cowritten by Jane Forrest. However, two papers did provide some interest. First, another 2001 paper in the British Journal of Social Work by Stephen A. Webb. Webb’s paper is a substantial critique of the application of the ideas of evidence-based policy to social work. Webb is listed, at the time, as teaching at the University of Bradford in the UK. In his paper, Webb writes:
Indeed, the SUNY-based health sciences ‘Evidence-based medical course’ provides an ‘evidence-based pyramid’ which places methodologies such as randomized controlled trials, cohort and case control studies at the top, with ideas and opinions at the bottom of the pyramid.Webb (2001), pp.65-66.
Although not a complete description of the pyramid, and without any citation to help us determine its origin, Webb does corroborate the presence of such a tool in 2001, and moreover shows that the SUNY pyramid had already exhibited a fairly wide reach. Not only had evidence-based dentistry scholars from Southern California picked it up, so had an scholar of social work from Bradford, England.
Secondly, and most promisingly for dating the origin of the pyramid, another paper by Jane Forrest and Syrene Miller, of almost identical title to the first and so nearly overlooked in my search, provided an important clue. In Forrest & Miller (2001b), the authors list some resources which would be of interest to dental practitioners wishing to learn more about evidence-based practice, and crucially, on page 55, provide the URL for the SUNY evidence pyramid. The URL is http://www.servers.medlib.hscbklyn.edu/ebm/2100.htm, notably a different location to the commonly-cited SUNY source, hosted instead on the Medical Research Library of Brooklyn’s servers. Their citation for this source lists access dates of 4/4/2001, but cites the source as from 1997.
Plugging this new source for the Wagoner et al. pyramid into the Wayback Machine, we find that it was first archived on 11th May 2000, considerably early than the SUNY website sources. Moreover, this version has the full-colour image of the evidence pyramid, identical to that in the “Version 2” SUNY offering, in full display. We can now definitively state that the Wagoner et al. evidence pyramid dates at the latest to May 2000.
Finally, the contents pages for the Medical Research Library of Brooklyn version of the course provides authorial information and an original date of publication. Here, they write: “Copyright © SUNY (State University of New York), 1997.” This explains Forrest & Miller’s attribution of the resource to 1997 in (2001b). We cannot definitively say that the evidence pyramid was part of the original version of these course materials as published in 1997, particularly as later revision dates have been provided up to 2000, and because this authorship page is clear that revisions were repeatedly made:
Revisions to original course material by Dr. Andrea Markinson, Assistant Director for Educational Services.
First Design revisions by Anita Ondrusek, Mary Doherty, and Ki-tae Mok. Second Design revisions by Dr. Andrea Markinson.Medical Research Library of Brooklyn, “Author and Copyright Information” as of 17/04/2000, provided by Internet Archive.
So, while no firm date prior to 2000 can be established for the Wagoner et al. evidence pyramid, given the spread of usage of this evidence pyramid beyond its initial circle, I feel justified in amending the evidence hierarchy database to reflect 1997 for this evidence pyramid, with the proviso that the earliest version of the image still accessible dates to May 2000. This would definitively make the Wagoner et al. evidence pyramid the first known pyramidal evidence hierarchy in the Evidence-Based Medicine orbit.
- Blunt, C. (2015). Hierarchies of evidence in evidence-based medicine (Doctoral thesis, London School of Economics and Political Science).
- Campbell, D.T. & Stanley, J.C. (1963) Experimental and Quasi-Experimental Designs for Research. Boston: Hougton Miffling Co.
- Canadian Task Force on the Periodic Health Examination. (1979) “The Periodic Health Examination”. CMAJ 121 1193-1252″
- Forrest, J.L. & Miller, S.A. (2001a) “Enhancing your practice through evidence-based decision making: Finding the best clinical evidence”, Journal of Evidence-Based Dental Practice, 1(3): 227-36.
- Forrest, J.L. & Miller, S.A. (2001b) “Enhancing your practice through evidence-based decision making”, Journal of Evidence-Based Dental Practice, 1(1): 51-7.
- GRADE Working Group. (2004) “Grading quality of evidence and strength of recommendations”. British Medical Journal, 328, 1490
- Wagoner, B. et al. (1997) “Guide to Research Methods: The Evidence Pyramid”. SUNY Downstate Medical Center: Medical Research Library of Brooklyn, EBM Tutorial, version 1.3, first known publication at latest 11 May 2000, archived at https://web.archive.org/web/20000511202739/http://www.servers.medlib.hscbklyn.edu/ebm/2100.htm
- Wagoner, B. et al. (2004) “Guide to research methods: the Evidence Pyramid”, SUNY Downstate Medical Center: Medical Research Library of Brooklyn, EBM Tutorial, version 2, archived at https://web.archive.org/web/20041101013240/http://library.downstate.edu/EBM2/2100.htm (23/03/2004)
- Webb, S.A. (2001) “Some considerations on the validity of evidence-based practice in social work”, British Journal of Social Work, 31: 57-79.
Last updated: 15/02/2022