Self-Supervised Pre-training of Swin Transformers for Label-Efficient Classification of Medical Images

Abstract

An effective deep learning approach that can build discriminative representations with little annotation cost is urgently needed to keep up with the exponential rise of medical imaging data. However, medical domain large-scale labeled datasets are limited owing to annotation complexity and expert dependency, which makes supervised training of vision transformers often a challenge. Our focus here is on medical images label-efficient categorization via self-supervised pre-training of Swin Transformers. The Swin Transformer can now capture both local and global contextual dependencies in medical imaging modalities including X-ray, CT, and MRI using the suggested method, which makes use of masked image modeling techniques and contrastive learning. The model is fine-tuned using few labeled samples after pre-training on large-scale unlabeled medical datasets, drastically lowering the need for annotation. Based on experimental assessments conducted on benchmark datasets, it has been found that self-supervised Swin Transformers achieve better classification accuracy, resilience to sparse data, and cross-modal generalizability than traditional CNNs and supervised ViT models. Based on these results, self-supervised transformer-based pre-training could be a good option for medical images categorization that is both scalable and efficient with labels.