A comparison of systematic reviews and guideline-based systematic reviews in medical studies

The question of how citation impact relates to academic quality accompanies every decade in bibliometric research. Although experts have employed more complex conceptions of research quality for responsible evaluation, detailed analyses of how impact relates to dimensions such as methodological rigor are lacking. But the increasing number of formal guidelines for biomedical research offer not only the potential to understand the social dynamics of standardization, but also their relations to scientific rewards. By using data from Web of Science and PubMed, this study focuses on systematic reviews from biomedicine and compares this genre with those systematic reviews that applied the PRISMA reporting standard. Besides providing an overview about growth and location, it was found that the latter, more standardized type of systematic review accumulates more citations. It is argued that instead of reinforcing the traditional conception that higher impact represents higher quality, highly prolific authors could be more inclined to develop and apply new standards than more average researchers. In addition, research evaluation would benefit from a more nuanced conception of scientific output which respects the intellectual role of various document types.

Automated summary

The question of how citation impact relates to academic quality is a recurring theme in bibliometric research. While experts have used more complex conceptions of research quality for evaluation, detailed analyses of how impact relates to dimensions like methodological rigor are lacking. Increasing formal guidelines for biomedical research not only provide insight into the social dynamics of standardization, but also their relationships to scientific rewards.