Exploring the Limits of Deep Image Clustering using Pretrained Models
Nikolas Adaloglou, Felix Michels, Hamza Kalisch, Markus Kollmann
Code Available — Be the first to reproduce this paper.
ReproduceCode
- github.com/HHU-MMBS/TEMI-official-BMVC2023OfficialIn paperpytorch★ 29
Abstract
We present a general methodology that learns to classify images without labels by leveraging pretrained feature extractors. Our approach involves self-distillation training of clustering heads based on the fact that nearest neighbours in the pretrained feature space are likely to share the same label. We propose a novel objective that learns associations between image features by introducing a variant of pointwise mutual information together with instance weighting. We demonstrate that the proposed objective is able to attenuate the effect of false positive pairs while efficiently exploiting the structure in the pretrained feature space. As a result, we improve the clustering accuracy over k-means on 17 different pretrained models by 6.1\% and 12.2\% on ImageNet and CIFAR100, respectively. Finally, using self-supervised vision transformers, we achieve a clustering accuracy of 61.6\% on ImageNet. The code is available at https://github.com/HHU-MMBS/TEMI-official-BMVC2023.
Tasks
Benchmark Results
| Dataset | Model | Metric | Claimed | Verified | Status |
|---|---|---|---|---|---|
| CIFAR-10 | TEMI CLIP ViT-L (openai) | Accuracy | 0.97 | — | Unverified |
| CIFAR-10 | TEMI DINO ViT-B | NMI | 0.89 | — | Unverified |
| CIFAR-100 | TEMI DINO ViT-B | Accuracy | 0.67 | — | Unverified |
| CIFAR-100 | TEMI CLIP ViT-L (openai) | Accuracy | 0.74 | — | Unverified |
| ImageNet | TEMI DINO (ViT-B) | Accuracy | 58 | — | Unverified |
| ImageNet | TEMI MSN (ViT-L) | Accuracy | 61.6 | — | Unverified |
| ImageNet-100 (TEMI Split) | TEMI CLIP ViT-L (openai) | NMI | 0.9 | — | Unverified |
| ImageNet-100 (TEMI Split) | TEMI MSN ViT-L | NMI | 0.89 | — | Unverified |
| ImageNet-100 (TEMI Split) | TEMI DINO ViT-B | NMI | 0.86 | — | Unverified |
| ImageNet-200 | TEMI DINO ViT-B | NMI | 0.85 | — | Unverified |
| ImageNet-200 | TEMI CLIP ViT-L (openai) | NMI | 0.88 | — | Unverified |
| ImageNet-200 | TEMI MSN ViT-L | NMI | 0.87 | — | Unverified |
| ImageNet-50 (TEMI Split) | TEMI DINO ViT-B | NMI | 0.86 | — | Unverified |
| ImageNet-50 (TEMI Split) | TEMI CLIP ViT-L (openai) | NMI | 0.92 | — | Unverified |
| ImageNet-50 (TEMI Split) | TEMI MSN ViT-L | NMI | 0.88 | — | Unverified |
| STL-10 | TEMI DINO ViT-B | Accuracy | 0.99 | — | Unverified |