It's always been hard to keep up with the medical literature, especially to figure out what original articles are of high enough quality to warrant a change in your clinical practice. It's not enough to just read the abstract, or to be reassured because the authors are from a reputable institution or the article is published in a reputable journal. I've been teaching Evidence Based Medicine (EBM) in various formats for over 20 years, including a full graduate school course for a while. I've learned a lot, both from reading but also from my students and colleagues, about how to sort through the jungle of words and diagrams in medical articles to pick up those rare pearls of good information.
EBM officially came into being in the early 1990s and, like most things, it has evolved. What hasn't changed much, however, are the forces that result in low quality evidence being published and advertised:
- Pressure on researchers to "publish or perish." This not only involves job security and academic promotion but also a natural desire to make a name for oneself.
- Pressure from academic institutions to ask their researchers to "hype" their studies in the hopes of increasing organizational rankings in national publications and also increase charitable donations.
- Complicity from the lay press, anxious to describe in breathtaking fashion a new study, even if it has no direct relevance to clinical practice or improving lives of their viewers/readers.
- Efforts from commercial organizations, such as pharmaceutical companies, test developers, and device manufacturers, to sell their products.
- Predatory journals who will publish anything for a price. (One "gotcha" study showed how one of these journals published a report taken straight from the pages of a "Seinfeld" script - clearly totally bogus and obviously published in such a journal without any editorial review.)
- Failure of the medical community as a whole to convey the inherent uncertainty in medical science - very few things are absolute "facts."
All of this just got worse in the pandemic era. Individual clinicians, researchers, and organizations seem bent on being the first to report the newest covid finding, and publishers and the lay press are anxious to help them. Unfortunately, things have moved too fast. Just recently, 3 major journals (New England Journal of Medicine, Lancet, and Annals of Internal Medicine) retracted publications due to, in my opinion, sloppy editing - plain rookie mistakes likely due to being in too much of a rush. (Actually as I'm writing this I heard about a potential new retraction with Proceedings of the National Academy of Sciences regarding mode of transmission of SARS-CoV-2). It is even harder now for those of us at the point of care to digest the onslaught of poor science looking for the truly helpful articles. However, there is still hope, and here are some quick guides to survival in the Pandemic Era of Medical Practice (PEMP, I just made up that acronym).
The image above is one I've used many times, most recently at a talk I gave at the AAP NCE meeting last fall. It is my version of the "evidence pyramid," a hierarchy of studies much misunderstood by the general medical public. Simply explained, results utilizing the study design types at the lower end of the pyramid are more likely to be shown to be wrong when subsequent studies, usually from a higher design type in the pyramid, are performed. Also, note that pure bench studies and animal studies aren't even part of the pyramid; those studies would not immediately impact clinical (human) medical practice. Also be aware that a poorly-designed randomized controlled trial (RCT) wouldn't be near the top of the pyramid; bad science can occur at all levels and trumps the pyramid ranking.
The vast majority of design types we are seeing related to COVID-19 are case series, i.e. just a report of what was tried and what happened, usually of a retrospective nature. It's not that these studies are bad, but compared to a randomized, placebo-controlled double blind trial of a new therapy, it just doesn't stand up. The gap between the lower and upper ends of the pyramid are magnified when we are dealing with a completely new disease like COVID-19.
(BTW, if you are wondering about GOBSAT, I wish I had invented the acronym but I didn't. It stands for Good Ol' Boys Sittin' Around a Table, another word for expert opinion. Again, if that is all we have to go on, I'm certainly interested in what experts think, but it's astonishing over the years how often GOBSAT opinions are reversed when better studies are performed.)
So, here's a quick and dirty approach of how I keep up with the flood of medical studies. First, I look at the abstract. If it sounds like something worth reading more, I then go immediately to the Methods section of the article. Yes, I know that section is the most painful of all, but that's where I figure out study design and whether the study may have critical flaws that would affect study results. Also, in spite of modern-day editing, even the best journals still allow conclusions to appear in the abstract that aren't supported by the study itself; usually they just represent the authors' conjectures but aren't labelled as such. If the Methods section doesn't pass muster, I don't read the rest of the article. If, however, the Methods look reasonably sound (remember, this is biology, we can't expect perfection in any study) I look through the results and discussion to see if this is something that would apply to my patients.
One more point that has just surfaced during PEMP. I'm starting to see increased alerts about manuscripts submitted to pre-publication web sites. Prior to the pandemic, these were sites where authors submitted data to be looked at by other scientists. They were not necessarily even submitted to a journal, just a way to increase transparency and actually a good thing. One key important fact is that the documents have not undergone any peer review at all. Unfortunately, now many authors are submitting results of case series and the like to these sites, and the lay press and even otherwise sound academicians are referring to these as "publications" when in fact they are nothing of the sort. As a reviewer for many medical journals and author of a few scientific articles, I can tell you that most articles submitted for publication undergo many, many significant changes before publication. I wouldn't advise clinicians to even look at these postings, they are useful only if someone is trying to design a research study on a similar topic. Some of the web sites include medrxiv.org and biorxiv.org. Again, nothing wrong with these sites other than how they are currently being misused by a few individuals.
So, I would advise you all not to be too discouraged by the confusion and flood of information. Listen to the lay press so you know what your patients and families are hearing, read the key articles, and be prepared to answer questions in your practice.