SOTAVerified

Big Transfer (BiT): General Visual Representation Learning

2019-12-24ECCV 2020Code Available2· sign in to hype

Alexander Kolesnikov, Lucas Beyer, Xiaohua Zhai, Joan Puigcerver, Jessica Yung, Sylvain Gelly, Neil Houlsby

arXivPDF

Code Available — Be the first to reproduce this paper.

Reproduce

Code

Abstract

Transfer of pre-trained representations improves sample efficiency and simplifies hyperparameter tuning when training deep neural networks for vision. We revisit the paradigm of pre-training on large supervised datasets and fine-tuning the model on a target task. We scale up pre-training, and propose a simple recipe that we call Big Transfer (BiT). By combining a few carefully selected components, and transferring using a simple heuristic, we achieve strong performance on over 20 datasets. BiT performs well across a surprisingly wide range of data regimes -- from 1 example per class to 1M total examples. BiT achieves 87.5% top-1 accuracy on ILSVRC-2012, 99.4% on CIFAR-10, and 76.3% on the 19 task Visual Task Adaptation Benchmark (VTAB). On small datasets, BiT attains 76.8% on ILSVRC-2012 with 10 examples per class, and 97.0% on CIFAR-10 with 10 examples per class. We conduct detailed analysis of the main components that lead to high transfer performance.

Tasks

Few-Shot LearningFine-Grained Image ClassificationImage ClassificationOut-of-Distribution GeneralizationRepresentation Learning

Benchmark Results

DatasetModelMetricClaimedVerifiedStatus
Oxford 102 FlowersBiT-M (ResNet)Accuracy99.3—Unverified
Oxford 102 FlowersBiT-L (ResNet)Accuracy99.63—Unverified
Oxford-IIIT PetsBiT-L (ResNet)Accuracy96.62—Unverified
Oxford-IIIT PetsBiT-M (ResNet)Accuracy94.47—Unverified

Reproductions