Cats or CAT scans: Transfer learning from natural or medical image source data sets?

Transfer learning is a widely used strategy in medical image analysis. Instead of only training a network with a limited amount of data from the target task of interest, we can first train the network with other, potentially larger source data sets, creating a more robust model. The source data sets do not have to be related to the target task. For a classification task in lung computed tomography (CT) images, we could use both head CT images and images of cats as the source. While head CT images appear more similar to lung CT images, the number and diversity of cat images might lead to a better model overall. In this survey, we review a number of articles that have studied similar comparisons. Although the answer to which strategy is best seems to be 'it depends', we discuss a number of research directions we need to take as a community to gain more understanding of this topic.