Seems like Shark Week has been everywhere the past few days. I can't recall ever watching anything on the Discovery Channel's Shark Week offerings, but I was very aware of it because I live a 20-minute walk from their (now former) corporate headquarters in Silver Spring, MD. Every year around Shark Week they transformed their several story building into a shark.
Perhaps this all influenced me to provide you with an in-depth look at how I approach evaluating an article that might change medical practice. It's a routine I've refined over several years and taught in my prior EBM classes as well as in other settings.
A Deep (Nose)dive
In early June I saw the advance publication of a randomized controlled trial of antibiotic versus placebo for acute bacterial sinusitis. I made the conscious decision at that time not to mention it in this blog because I decided it did not have direct applicability to clinical practice. I might also have been swayed by the fact that I wanted to mention my new washing machine in that blog, and, well, some things just gotta make room for the important stuff.
However, the article appeared in last week's JAMA and was accompanied by an editorial in JAMA as well as an editorial in JAMA Otolaryngology, plus appeared in several medical feeds. Given all the hoopla, I acknowledge I made a mistake in early June and should revisit the article, this time in depth. I'll lead you through the process and my conclusions.
Step 1 - Is it worth my time to evaluate an article in depth?
This step only takes a minute, usually from reading the title and the abstract. If it sounds intriguing and has a study design likely to yield accurate results (e.g. a double-blinded randomized controlled trial for therapeutic interventions) I keep going.
This report is a randomized controlled trial aiming to determine "if antibiotic therapy [for acute bacterial sinusitis] could be appropriately withheld in prespecified subgroups." The abstract sounded interesting with a good study design, so on to the next step.
Step 2 - Immediately go to the Methods section; do not read the Introduction, Results, or Discussion sections.
Whenever I think about this step I'm reminded of the sea (intentional shark pun) of horrified faces in my student/trainee audiences when they first hear this. There's nothing more mundane than the Methods section of a scientific article, but if the methods aren't sound I don't waste my time reading the rest of the paper.
This was a study taking place over 6 years, extended due to pandemic interference with enrollment, at 6 different study sites. In a multi-center study, especially one interrupted by a pandemic, there is a risk that lower-enrolling centers will make mistakes in following the protocol that could cause loss of subjects or missing data. Looking at the enrollment and follow-up diagram, fortunately this did not appear to be a big problem for this study. Also, looking at their Table 1, most of the subjects were enrolled at just 2 centers: Children's Hospital of Pittsburgh (about 3/4 of enrollees) and Kentucky Pediatric and Adult Research (about 15%). The latter organization is a for-profit company, a feature that always raises my eyebrows with medical research, but they've been in business in rural Kentucky since the 1980s, and I don't have any particular concerns about them.
Eligible subjects were children ages 2-11 years who fulfilled criteria for persistent or worsening acute sinusitis in the AAP's 2013 clinical practice guideline. They also needed to have an initial score of at least 9 on the validated Pediatric Rhinosinusitis Symptom Scale (PRSS) which was used as the primary means of evaluating treatment results. Here's a quick look at the scale from the article's online supplemental information.
Points are assigned as 0 for No, then 1-5 from Almost None to An Extreme Amount. As you can see, the minimum score of 9 to be eligible for the study can be achieved with fairly mild symptoms. More on that later.
I won't mention this in depth, but I felt the exclusion criteria were reasonable. In particular, those with more severe symptoms such as temperature > 39 C for 3 or more consecutive days were excluded.
The enrolled children were stratified according to whether parents reported presence or absence of colored nasal discharge and then randomized randomly to receive amoxillin/clavulanate 90 mg/kg/day/6.4 mg/kg/day or placebo divided in 2 doses for 10 days, in a double-blinded fashion. Anticipating later problems with side effects that did occur, the authors mention that they chose amoxicillin/clavulanate over amoxicillin alone to provide broader coverage for all pathogens, presumably those producing beta-lactamase. So, I did note this wasn't quite a real-world situation where amoxicillin might be a better choice based on less GI side effects. Children had nasopharyngeal swabs for Streptoocccus pneumoniae, Haemophilus influenzae, and Moraxella catarrhalis obtained at enrollment, and outcomes were compared based on these results.
I immediately wondered about the placebo which was said to be "matching." Most pediatric healthcare providers, as well as parents of young children, know that it is pretty easy to determine if a child is receiving an antibiotic containing amoxicillin due to the odor of the medication and also sometimes urine odor. I wanted to find out if the placebo was truly a good match for the study drug, but the article itself had no further details. Fortunately, the online supplemental materials included the 73-page study protocol. On page 31 beginning on line 790, the placebo is well-described and included reassurance that it was identical in appearance, odor, and texture to the amox/clav.
I'll spare you the statistical details; although I'm not a statistician, I didn't notice anything nonstandard in the analytic methods.
Step 3 - Results
Finally we get to some of the meat of the article! In total, 254 children were evaluable in the treatment group and 256 in the placebo group. Almost 2/3 of enrollees were 2-5 years of age, so a bit skewed towards the younger part of the age range. I was interested in the PRSS scores of those evaluated. In general there was good distribution along the PRSS range, and the 2 treatment groups were similar:
Now, what about the outcomes. The investigators predetermined that a difference in PRSS scores of 1.0 was clinically meaningful. I would question this, given that it doesn't take much to drop a point from a PRSS score - just rating stuffy nose as dropping from "a little" to "almost none" would do it. Here is what was seen for the group as a whole:
Here's what was found when broken down by culture results:
I did notice in all the analyses that children experienced the biggest 1-day drop in PRSS scores in that first day, regardless of treatment group. This represents either natural history or perhaps a placebo effect of just being enrolled in the study and parents being required to pay close attention to their child's symptoms.
With further analysis, it turned out that children with cultures showing pneumococcus or Hi showed statistically significant improvement with antibiotic compared to placebo. Presence of Moraxella, absence of any pathogens in culture, and snot color had no predictive value of response. So, the implication is that culture showing pneumococcus or Hi in culture, along with the clinical AAP guidelines, could be used to further narrow the number of children receiving antibiotic treatment unnecessarily.
In terms of the down side, the main problem was clinically significant diarrhea seen in 11% of the antibiotic group and 5% of the placebo group. If amoxicillin had been used the rate in the antibiotic group likely would have been much lower.
Step 4 - Now read the rest of the article
I look particularly at the Discussion section to see how the authors propose using the results and also how they describe limitations of their study. They describe their study findings as being used in 1 of 4 ways:
- Treat everyone fulfilling AAP guidelines, no testing for pathogens
- Start presumptive treatment for everyone fulfilling AAP guidelines, test for pathogens, and stop treatment if no pneumococcus or Hi found
- Test for pathogens, wait for results, and treat those who have positive pathogen test
- Watchful waiting for everyone
All of those options seem possible to me, given that diagnosis is so difficult and the vast majority of these children do fine as long as they are followed closely. Multiple practical problems arise with options 2 and 3 because they require sending a culture and presumably making a special request to the lab to look for and report presence of any pneumococcus or Hi. Depending on the office practice, this takes time and has additional problems/delays if a weekend or holiday is involved, plus making sure the patient isn't lost to follow up. If your practice is situated in a hospital like Children's Hospital of Pittsburgh this might be more feasible than if this is a practice site that sends cultures to a commercial lab.
The authors listed 5 study limitations, some of which seemed trivial to me. First, their original target sample size wasn't reached due to the pandemic. However, their results still reached statistical significance, so this isn't a real limitation. Second, changes in gut microbiome weren't examined. As we find out more about how the microbiome interacts with other aspects of health, it becomes even more important to limit unnecessary antibiotic use. Third, as stated previously children with certain severe symptoms were excluded use of cultures in that setting can't be assessed. Similarly, the fourth limitation was that children not meeting study entry criteria, or even those with parent characteristics less likely to enroll in research studies, might have introduced bias. This is true of all prospective clinical trials so hardly needs to be mentioned here. The fifth and final limitation mentioned was that the gold standard for diagnosis of bacterial sinusitis, quantitative culture of a sinus aspirate, wasn't used. This would have been ideal, but it would be extremely difficult to enroll a child with a relatively minor illness if they would be required to undergo an invasive procedure with little direct benefit to the child.
The main study limitation I appreciated is that the differences in PRSS scores between antibiotic and placebo groups weren't terribly large, an average of 1.69 points. I wonder whether parents would find this an important difference, particularly since after a couple days we are talking about the milder end of the scoring system. There isn't enough detail provided, even in the supplemental data, to know how many individual patients experienced a decrease in symptoms that might be important to parents.
Step 5 - Is there evidence of bias in the study?
The study was funded by NIAID, probably the cleanest funding source there is in terms of bias. I did note that a couple of the authors disclosed financial arrangements with a company called Kaizen Bioscience. I had to research the company a little, it is a small for-profit pharmaceutical company. One of the arrangements involved a proprietary formulation of amoxicillin/clavulanate, so this is a very real potential conflict of interest with the study. On the other hand, I am acquainted with a few of the study's authors, and I don't for a minute suspect they would purposely alter their report based on this connection. The problem, however, is really with implicit bias, the bias we all have but aren't aware of. Did the authors subconsciously "spin" their report to appear more favorable? I don't view this article any differently based on the potential for implicit bias, it's just something to keep in mind particularly if future studies show different results.
After all of this, how do I put everything together?
I wish I had been keeping track of time, but I estimate it took me at least 3 hours of working through the article, not counting time thinking about it while in the shower or writing this post. Ultimately I came back to my original thought - this study isn't ready for adoption into clinical practice.
But, after picking through this with a very fine-(shark)tooth comb, this study's design and reporting are almost impeccable, really a model for others who are beginning clinical treatment research careers.
In the interests of not boring you completely, I left out a lot of other details in the study (see metatranscriptomics for example). Read the article if you have more interest in pediatric sinusitis.
'Demic Doldrums
We continue to see some uptrending in SARS-CoV-2 in wastewater, but still at low levels, according to Biobot.
Similarly, CDC reports slight upticks in covid hospitalizations and ED visits.
Dr. Rivers at Force of Infection notes that the current upticks resemble what we saw in the summer of 2021, just before the delta wave. I think it's a little early to be saying that, plus so far nothing as virulent as the delta variant is prominent. I wouldn't be surprised by at least a mini-wave this fall.
Shark Week
I decided to explore more about Shark Week; one of my (surfer) brothers-in-law did survive a shark attack many years ago, requiring several surgical procedures. I was sorry to learn that the Shark Week shows have become a form of docufiction, whereby true documentary filming is overlaid with fictional information or staged events to increase viewers or readers. In our current age of fake news and artificial intelligence doctoring of reality, we don't need this.