Better plain ViT baselines for ImageNet-1k
2022-05-03Code Available1· sign in to hype
Lucas Beyer, Xiaohua Zhai, Alexander Kolesnikov
Code Available — Be the first to reproduce this paper.
ReproduceCode
- github.com/google-research/big_visionOfficialIn paperjax★ 3,388
- github.com/jeongsoop/rgb-no-morepytorch★ 57
- github.com/osiriszjq/impulse_initpytorch★ 15
- github.com/conceptofmind/Simple-ViT-flaxjax★ 5
- github.com/yuyangshu/retinavitjax★ 1
- github.com/pwc-1/Paper-9/tree/main/5/vitmindspore★ 0
- gitlab.com/birder/birderpytorch★ 0
Abstract
It is commonly accepted that the Vision Transformer model requires sophisticated regularization techniques to excel at ImageNet-1k scale data. Surprisingly, we find this is not the case and standard data augmentation is sufficient. This note presents a few minor modifications to the original Vision Transformer (ViT) vanilla training setting that dramatically improve the performance of plain ViT models. Notably, 90 epochs of training surpass 76% top-1 accuracy in under seven hours on a TPUv3-8, similar to the classic ResNet50 baseline, and 300 epochs of training reach 80% in less than one day.