4.6 Article

Metamorphic Robustness Testing: Exposing Hidden Defects in Citation Statistics and Journal Impact Factors

Journal

IEEE TRANSACTIONS ON SOFTWARE ENGINEERING
Volume 47, Issue 6, Pages 1164-1183

Publisher

IEEE COMPUTER SOC
DOI: 10.1109/TSE.2019.2915065

Keywords

Robustness; Database systems; Software; Fuzzing; Google; Metamorphic robustness testing; metamorphic testing; fault-based testing; software robustness; oracle problem; citation count; journal impact factor; Scopus; Web of Science; Google Scholar; verification and validation

Funding

  1. Australian Research Council [LP160101691]
  2. Australian Government Research Training Program scholarship
  3. Morphick Solutions Pty Ltd
  4. Australian Research Council [LP160101691] Funding Source: Australian Research Council

Ask authors/readers for more resources

The presence of hyphens in paper titles can distort citation counts and impact factor evaluations, challenging the common belief that they are reliable measures. This study reveals a strong negative correlation between journal impact factor and the percentage of papers with hyphenated titles, impacting the assessment of paper and journal impact.
We propose a robustness testing approach for software systems that process large amounts of data. Our method uses metamorphic relations to check software output for erroneous input in the absence of a tangible test oracle. We use this technique to test two major citation database systems: Scopus and the Web of Science. We report a surprising finding that the inclusion of hyphens in paper titles impedes citation counts, and that this is a result of the lack of robustness of the citation database systems in handling hyphenated paper titles. Our results are valid for the entire literature as well as for individual fields such as chemistry. We further find a strong and significant negative correlation between the journal impact factor (JIF) of IEEE Transactions on Software Engineering (TSE) and the percentage of hyphenated paper titles published in TSE. Similar results are found for ACM Transactions on Software Engineering and Methodology. A software engineering field-wide study reveals that the higher JIF-ranked journals are publishing a lower percentage of papers with hyphenated titles. Our results challenge the common belief that citation counts and JIFs are reliable measures of the impact of papers and journals, as they can be distorted simply by the presence of hyphens in paper titles.

Authors

I am an author on this paper
Click your name to claim this paper and add it to your profile.

Reviews

Primary Rating

4.6
Not enough ratings

Secondary Ratings

Novelty
-
Significance
-
Scientific rigor
-
Rate this paper

Recommended

No Data Available
No Data Available