期刊
SYSTEMATIC REVIEWS
卷 10, 期 1, 页码 -出版社
BMC
DOI: 10.1186/s13643-021-01700-x
关键词
Systematic review; Scoping review; Text mining; Natural language processing; Machine learning; Classification model; Abstract screening; Automation
资金
- Tier 1 Canada Research Chair in Knowledge Translation
- Squires-Chalmers Chair for the Physician-in-Chief of Department of Medicine, St. Michael's Hospital
- University of Toronto
- Tier 2 Canada Research Chair in Knowledge Synthesis
- Ontario Ministry of Research, Innovation, and Science
- Natural Sciences and Engineering Research Council (NSERC) Discovery Grants Program
- Canada Research Chairs Program
- NSERC Industrial Research Chairs Program
A semi-automated workflow was developed for abstract screening in systematic reviews using text mining and machine learning methods, achieving high efficiency and accuracy. The workflow demonstrated good performance in practice, effectively reducing workload and improving review efficiency.
Background: Current text mining tools supporting abstract screening in systematic reviews are not widely used, in part because they lack sensitivity and precision. We set out to develop an accessible, semi-automated workflow to conduct abstract screening for systematic reviews and other knowledge synthesis methods. Methods: We adopt widely recommended text-mining and machine-learning methods to (1) process title-abstracts into numerical training data; and (2) train a classification model to predict eligible abstracts. The predicted abstracts are screened by human reviewers for (true) eligibility, and the newly eligible abstracts are used to identify similar abstracts, using near-neighbor methods, which are also screened. These abstracts, as well as their eligibility results, are used to update the classification model, and the above steps are iterated until no new eligible abstracts are identified. The workflow was implemented in R and evaluated using a systematic review of insulin formulations for type-1 diabetes (14,314 abstracts) and a scoping review of knowledge-synthesis methods (17,200 abstracts). Workflow performance was evaluated against the recommended practice of screening abstracts by 2 reviewers, independently. Standard measures were examined: sensitivity (inclusion of all truly eligible abstracts), specificity (exclusion of all truly ineligible abstracts), precision (inclusion of all truly eligible abstracts among all abstracts screened as eligible), F1-score (harmonic average of sensitivity and precision), and accuracy (correctly predicted eligible or ineligible abstracts). Workload reduction was measured as the hours the workflow saved, given only a subset of abstracts needed human screening. Results: With respect to the systematic and scoping reviews respectively, the workflow attained 88%/89% sensitivity, 99%/99% specificity, 71%/72% precision, an F1-score of 7996/79%, 98%/97% accuracy, 63%/55% workload reduction, with 12%/11% fewer abstracts for full-text retrieval and screening, and 0%/1.5% missed studies in the completed reviews. Conclusion: The workflow was a sensitive, precise, and efficient alternative to the recommended practice of screening abstracts with 2 reviewers. All eligible studies were identified in the first case, while 6 studies (1.5%) were missed in the second that would likely not impact the review's conclusions. We have described the workflow in language accessible to reviewers with limited exposure to natural language processing and machine learning, and have made the code available to reviewers.
作者
我是这篇论文的作者
点击您的名字以认领此论文并将其添加到您的个人资料中。
推荐
暂无数据