The Use of Multiple Primary Outcomes in Randomized Controlled Trials of Chinese Herbal Medicine

Background. Multiple primary outcomes are commonly used in randomized controlled trials (RCTs) of Chinese herbal medicine (CHM). Analysis and interpretation of the results of CHM RCTs with many outcomes are not clear. No previous studies have systematically assessed the use of multiple primary outcomes in this area. This study aimed to assess the reporting of multiple primary outcomes and the statistical methods used to adjust multiplicity in RCTs of CHM. Methods. Search for RCTs of CHM published in English between January 2010 and December 2019 in MEDLINE, EMBASE, and the Cochrane Central Register of Controlled Trials (CENTRAL) was undertaken. We randomly selected 20% of the included RCTs as the analyzing sample of this study. The number of multiple primary outcomes, the methods used to adjust the multiplicity in statistical analysis and sample size estimate, and the trial information were collected. For RCTs that adopted multiple primary outcomes without the multiplicity adjustment, we used Bonferroni correction to adjust. Results. 227 CHM RCTs were included in our study. 92 (40.5%) failed to report what their primary outcome was. Of 135 (59.5%) RCTs that reported primary outcome, 93 (68.9%) reported one and 42 (31.1%) reported more than one primary outcome (range 2-5). Of 42 RCTs that reported multiple primary outcomes, only 5 adjusted for multiple outcomes. If multiplicity had been accounted for using Bonferroni correction, 10 (37.0%) RCTs that reported a significant result had demonstrated a nonsignificant result, giving the adjusted P value. Only one of the 42 RCTs calculated sample size based on multiple primary outcomes. Adopting multiple primary outcomes showed a slow growth trend with the publication year. The proportion of primary outcome reported explicitly in RCTs was different in terms of the nationality of the first author (P=0.004), in which mainland China has the lowest proportion (55.8%). The highest percentage of the studies with primary outcome reporting explicitation was mental and behavioural disorders (83.3%), and the most frequently adopting multiple primary outcomes were studies on the disease of the nervous system (66.7%). The percentage of reporting primary outcome explicitly was associated with sample size (P<0.001); for the percentage of RCTs adopting multiple primary outcomes, there was no statistically significant difference (P=0.739). Conclusions. Multiple primary outcomes are prevalent in CHM RCTs. However, appropriate methods are not usually taken in most of the analyses to safeguard the inferences against multiplicity. Sample size estimation based on multiple primary outcomes is still lacking. These issues complicate the interpretability of trial results and can lead to spurious conclusions. Guidelines to improve analyzing and reporting for multiple primary outcomes in CHM RCTs are warranted.

Automated summary

Multiple primary outcomes are commonly used in Chinese herbal medicine RCTs, but appropriate methods for adjusting multiplicity are often not utilized, and sample size estimation based on multiple primary outcomes is lacking. The explicit reporting of primary outcomes is associated with nationality, research area, and sample size.