Self-Supervision Can Be a Good Few-Shot Learner

Code

• github.com/bbbdylan/unisiam
  OfficialIn paperpytorch★ 37
• github.com/MindSpore-paper-code-2/code3/tree/main/unisiam
  mindspore★ 0
• github.com/2023-MindSpore-1/ms-code-217/tree/main/unisiam
  mindspore★ 0
• github.com/yangyucheng000/MSpaper/tree/main/unisiam
  mindspore★ 0

Abstract

Existing few-shot learning (FSL) methods rely on training with a large labeled dataset, which prevents them from leveraging abundant unlabeled data. From an information-theoretic perspective, we propose an effective unsupervised FSL method, learning representations with self-supervision. Following the InfoMax principle, our method learns comprehensive representations by capturing the intrinsic structure of the data. Specifically, we maximize the mutual information (MI) of instances and their representations with a low-bias MI estimator to perform self-supervised pre-training. Rather than supervised pre-training focusing on the discriminable features of the seen classes, our self-supervised model has less bias toward the seen classes, resulting in better generalization for unseen classes. We explain that supervised pre-training and self-supervised pre-training are actually maximizing different MI objectives. Extensive experiments are further conducted to analyze their FSL performance with various training settings. Surprisingly, the results show that self-supervised pre-training can outperform supervised pre-training under the appropriate conditions. Compared with state-of-the-art FSL methods, our approach achieves comparable performance on widely used FSL benchmarks without any labels of the base classes.

Tasks

Benchmark Results

Reproductions