When Thai and US Army scientists announced last month that their experimental HIV vaccine reduced the risk of contracting the disease by 31%, it caused quite a stir. After all, every one of the 100 or so previous HIV vaccine trials over the last 20 years had failed completely.
Alas, a second analysis of the $105 million study that was released weeks after the announcement suggests the apparent, moderate benefit may have been caused by a statistical fluke.
Worse yet, it turns out that the results of the second analysis were available to the scientists when they announced their original findings.
“We thought very hard about how to provide the clearest, most honest message,” said Jerome Kim, an Army scientist involved with the study. “We stand by the fact that this is a vaccine with a modest protective effect.”
Kim’s team based its initial announcement on a “modified intent to treat analysis,” which includes all volunteers that enrolled in the study, whether they received the full course of the vaccine or not. Such analyses reflect real world situations in which some people don’t show up for all shots in the vaccine sequence.
Using this analysis, the scientists determined that there was only a 3.9% chance that the observed 31% reduction in HIV among the vaccine-treated group was caused by a simple statistical fluke…in scientific parlance, this is considered to be a borderline significant result.
The second, so-called “per protocol” analysis included just the study participants that received all vaccines in the regimen at the right time.
Such analyses normally corroborate intent to treat findings, but for unknown reasons, it did just the opposite in this study. It suggested there was a 16% chance the results could have been due to chance alone—far too high to support claims that the vaccine was efficacious.
Some AIDS activists and scientists believe the vaccine merits further study, but worry that the botched announcement might undermine support for the vaccine and HIV vaccine trials generally.
“I would have preferred to have seen both results straight up. It might spring back on (the scientists), and that would be unfortunate,” Mitchell Warren, director of the AIDS Vaccine Advocacy Coalition said.