Well, definitely no gospel with this study design type, though sometimes it seems as if some medical researchers place MA on a very high pedestal. In the days when I was teaching a graduate school course in evidence-based medicine, I asked learners to approach MA as the most dangerous form of medical study, because so few people were aware of the nuances of study design and how that could impact interpretation and translation into clinical practice.
This past week I saw 2 new MA studies published that offer an opportunity to look into this more deeply.
Time to Buy Ocean Spray Stock?
Definitely not, but the granddaddy/momma of all MA and EBM, the Cochrane Library, has a significant update to its MA of use of cranberry juice and related products to prevent urinary tract infections. Cochrane is the ultimate in high-quality MA with a very standardized process to conduct such reviews and a rigorous editorial process. Access to full Cochrane reviews requires a subscription, but the summaries and often helpful "Plain language summaries" are open to all.
However, these gold standard MAs don't always translate into immediately useful clinical practice changes. I have found them to be more helpful in pointing the way for future studies to answer lingering questions.
The current cranberry/UTI review was published last month and updates the last review published in 2012. The new update now has twice the number of studies (50) as in 2012, so gives a better estimate of effects. The entire review runs to 150 pages, just to give you an idea of the level of detail.
Below I have taken the section of the results that shows the summary analysis for studies in children; that dark diamond shows a significantly lower relative risk for children given a cranberry product as opposed to placebo or control in preventing symptomatic UTI. (With apologies for the column alignment in the table, it defeated my limited web design skills.)
As most pediatric healthcare providers know, authorities recommend limiting the use of antibiotics as prophylaxis for recurrent UTI in children, with the main use drawback being development of antibiotic resistance. Should we now recommend cranberry products for children at risk for recurrent UTI?
To answer this, we need to look closely at the methodology of the Cochrane MA, in particular the characteristics of the studies chosen for the review. This requires a close look at the dreaded Methods section of the study. Of the studies included above, Afshar, Salo, and Wan compared cranberry juice to placebo; Ferrara was a cranberry/lingonberry syrup combination compared to Lactobacillus treatment; and the Dotis study looked at cranberry capsules compared to no treatment. Most of the studies looked at children with more than 1 prior UTI, but Salo enrolled children after their first UTI. Only one study enrolled more than 100 children. So, you can see from the start that we have some highly variable study characteristics.
What is more of a problem with most MAs is the fact that, in order to include only studies of high quality where data are very clearly documented, the study milieu bears little resemblance to real-life situations. In our clinical practices we don't have access to research nurses who are hovering over all the subjects to better ensure treatment compliance and data collection. This is really the difference between reporting efficacy (how an intervention behaves under ideal circumstances) versus effectiveness (how it works in real-world situations). Also, we don't know how the cranberry intervention compares to other non-antibiotic strategies such as promoting better voiding habits in children.
This is an excellent study, and the data clearly show a benefit (efficacy) for the cranberry interventions. From a practical standpoint, I don't think I would start recommending this for children who have experienced UTI, but if a parent wanted to try it I wouldn't be opposed. Just remember, any intervention to prevent recurrent UTI must be maintained long-term; it's not something to try for a month and then forget about.
Is Covid Vaccination Associated with Bell Palsy?
The short answer: probably yes, but that's not the correct question. What you, I, and our patients should want to know is how Bell palsy (BP) rates vary between the vaccinated and unvaccinated; i.e. is the risk of peripheral facial nerve paralysis likely to be more or less with vaccination versus covid disease? A new MA study attempted to shine some light on this question. Both randomized controlled trials and observations studies were eligible for inclusion, and the studies were heavy on adults with only a few adolescent studies. The methodology was generally of high quality.
The researchers reported several outcome comparisons. Looking just at the observational studies, there were fewer "events" (BP cases) in the vaccinated versus unvaccinated participants, and many fewer events comparing vaccine recipients to people experiencing SARS-CoV 2 infection. They found no difference comparing Pfizer to Oxford/AstraZeneca vaccine recipients in observational studies. So, from these comparisons, it's very clear that if you are worried about Bell palsy, you want to be vaccinated.
What's really interesting, though, is that if you look just at randomized controlled trials comparing mRNA vaccination with placebo, the reverse was seen: vaccine recipients actually had more events.
Why are we seeing such a contradiction of associations? Well, it's not really a contradiction, it's a great example of the difficulties in both using study results and explaining them to the general public. One really needs to have an understanding of overall BP rates as well as how data are collected under a variety of circumstances.
The baseline rate of BP in the general population, without covid considerations, is around 15-30 per hundred thousand people per year. The BP rate in the vaccine recipients in the RCTs in this report was 18/100,000/year, similar to previous reports. However, the rate of BP following actual infection with SARS-CoV-2 in the general population has ranged from 32.3 to 82 per 100,000, significantly higher. So, this report still says it is better to be vaccinated against this virus than to be infected with it, which is really the question we care about most. In an RCT, study subjects might have a lower covid infection rate anyway (people enrolling in a vaccine study probably lean towards more infection-cautious behaviors than in the general population) and tracking is limited to a specified time following study entry. These are nuances between RCTs and observational studies that can make it appear that conclusions contradict one another.
All the data from this MA were from the published studies themselves; some MA are able to use patient-level data (data on individual patients, rather than the aggregate report from the publication), but his was not the case here. So, the investigators were unable to evaluate multiple factors that might have clarified our understanding. I certainly hope investigators (and pharmaceutical companies!) will move to better data sharing in the future to allow for more thorough analyses and comparisons of studies via patient-level data.
I suspect some in the anti-vax movement could take data from this study out of context to persuade people to avoid covid vaccination. If SARS-CoV-2 goes away (not gonna happen any time soon, sadly), then purely from a BP standpoint you could argue against receiving the vaccine, but we have many more examples of vaccination benefits. And, we have all now seen what happens with infections that "went away," only to flare again when vaccination rates fall. It is still better to be vaccinated against covid than to experience infection.
One more caveat - remember that these types of studies only show an association between events and cannot prove or refute cause and effect. However, with knowledge of the mechanisms of BP with other infections and vaccinations, most authorities would conclude that BP can be caused by covid vaccination, just at a lower rate than caused by the infection itself.
Meta-Analysis Fatigue?
You might well be experiencing this by now, but if you want a little more explanation I did attempt, early in the pandemic, to explain Cochrane reviews and to provide some tips about keeping up with and interpreting the medical literature. Always tough, but manageable.