Transfer learning for histopathology images: an empirical study

Histopathology imaging is one of the key methods used to determine the presence of cancerous cells. However, determining the results from such medical images is a tedious task because of their size, which may cause a delay in results for days. Even though CNNs are widely used to analyze medical images, they can only learn short-term dependency and ignore long-term dependency, which could be crucial in processing higher dimensional histology images. Transformers, however, make use of a self-attention mechanism, which might be helpful to learn dependencies across an entire set of features. To process histology images, deep learning models require a large number of images, which is usually not available. Transfer learning, which is often used to deal with this issue, involves fine-tuning a trained model for use with medical images by adding features. In context, it is essential to analyze which CNNs or transformers are more conducive to transfer learning. In this study, we performed an empirical study to evaluate the performance of different pre-trained deep learning models for the classification of lung and colon cancer on histology images. Vision transformer and CNN models pre-trained on image-net are analyzed for the classification of histopathology images. We performed an experiment on the LC25000 dataset for the evaluation of models. The dataset consists of five classes, two belong to colon and three belong to lung cancer. The insights and observations obtained from an ablation study performed on different pre-trained models show vision transformers perform better than CNN based models for histopathology image classification using transfer learning. Moreover, the vision transformer with more layers of ViT-L32 performs better than ViTB32 with fewer layers.

Automated summary

This study evaluates the performance of different pre-trained deep learning models for the classification of histopathology images. The results suggest that Vision Transformers are more suitable for histopathology image classification using transfer learning compared to CNN models.