From Semantic Segmentation of Natural Images to Medical Image Segmentation Using ViT-Based Architectures

Abstract

We address the problem of medical image segmentation in the context of limited training data. Our approach builds on the capabilities of the Vision Transformer (ViT) and the recent Segmenter model, adapting them for the task of medical image segmentation. By leveraging Segmenter models pre-trained on moderately-sized datasets like ADE20K, we demonstrate their effectiveness when fine-tuned on smaller and scarce medical imaging datasets, specifically those for skin lesions and polyps. Employing our proposed training strategy, the adapted Segmenter model both matches and surpasses the current state-of-the-art on three key medical image datasets: ISIC2018 for skin lesions, and CVCClinicDB and ETIS-LaribPolypDB for polyps, while maintaining competitive performance on Kvasir-SEG and CVC ColonDB.

Tasks

Reproductions