Vision Transformers-Based Transfer Learning for Breast Mass Classification From Multiple Diagnostic Modalities

Abstract

Breast mass evaluation is crucial for early breast cancer diagnosis via imaging. While Convolutional Neural Network (CNN)-based deep learning (DL) has enhanced this process, it suffers from computational complexity and limited spatial encoding. Vision Transformer (ViT)-based DL, more adept at encoding spatial information, presents a promising alternative. This study introduces a ViT-based transfer learning (TL) method for breast mass classification. Three ViT-based TL architectures pretrained on ImageNet were proposed and evaluated using ultrasound and mammogram datasets. Comparative analysis against ViT trained from scratch and CNN-based TL was conducted. Results showed the ViT-based TL method achieving the highest area under curve (AUC) of 1 +/- 0 for both datasets, outperforming ViT from scratch and yielding similar or better performance compared to CNN-based TL. Despite its computational cost, ViT-based TL demonstrates superior classification capabilities for breast mass images. This research provides a foundational framework for future studies exploring ViT-based TL in breast cancer diagnosis.