Challenges of measuring software impact through citations: An examination of the lme4 R package

The rise of software as a research object is mirrored by increasing interests in quantitative studies of scientific software. However, inconsistent citation practices have led most existing studies of this type to base their analysis of software impact on software name mentions, as identified in full-text publications. Despite its limitations, citation data exists in much greater quantities and covers a broader array of scientific fields than full-text data, and thus can support investigations with much wider scope. This paper aims to analyze the extent to which citation data can be used to reconstruct the impact of software. Specifically, we identify the variety of citable objects related to the Ime4 R package and examine how the package's impact is dispersed across these objects. Our results shed light on a little-discussed challenge of using citation data to measure software impact: even within the category of formal citation, the same software object might be cited in different forms. We consider the implications of this challenge and propose a method to reconstruct the impact of Ime4 through its citations nonetheless. (C) 2019 Elsevier Ltd. All rights reserved.