Definition of Cancer Research: Journals, Titles, Abstracts or Keywords?

Three versions of a 'filter' used to identify papers on cancer research, as defined by Cancer Research, UK, and interpreted by four experts, have been compared. The first filter was based only on specialist journals and had unacceptably low recall. The second filter was based both on journals and on title words, and had both precision and recall above 0.9. The third filter was based additionally on words in the abstract and/or keywords provided in the paper: it improved the recall to almost unity but the precision was severely degraded, with many false positives. The three filter versions were compared in terms of the outputs of 15 countries in the Web of Science in recent years, and in some instances, gave differing indicators of their performance (numbers of papers and citations) which could give conflicting messages for science policy.