Co-Scale Conv-Attentional Image Transformers
Weijian Xu, Yifan Xu, Tyler Chang, Zhuowen Tu
Code Available — Be the first to reproduce this paper.
ReproduceCode
- github.com/mlpc-ucsd/CoaTOfficialIn paperpytorch★ 235
- github.com/rwightman/pytorch-image-modelspytorch★ 36,538
- github.com/naver-ai/vidtpytorch★ 318
- github.com/rishikksh20/CoaT-pytorchpytorch★ 15
- github.com/mindspore-courses/External-Attention-MindSpore/blob/main/model/backbone/CoaT.pymindspore★ 0
- github.com/Mind23-2/MindCode-170mindspore★ 0
- github.com/BR-IDL/PaddleViT/tree/develop/image_classification/CoaTpaddle★ 0
- github.com/leondgarse/keras_cv_attention_models/tree/main/keras_cv_attention_models/coattf★ 0
- gitlab.com/birder/birderpytorch★ 0
Abstract
In this paper, we present Co-scale conv-attentional image Transformers (CoaT), a Transformer-based image classifier equipped with co-scale and conv-attentional mechanisms. First, the co-scale mechanism maintains the integrity of Transformers' encoder branches at individual scales, while allowing representations learned at different scales to effectively communicate with each other; we design a series of serial and parallel blocks to realize the co-scale mechanism. Second, we devise a conv-attentional mechanism by realizing a relative position embedding formulation in the factorized attention module with an efficient convolution-like implementation. CoaT empowers image Transformers with enriched multi-scale and contextual modeling capabilities. On ImageNet, relatively small CoaT models attain superior classification results compared with similar-sized convolutional neural networks and image/vision Transformers. The effectiveness of CoaT's backbone is also illustrated on object detection and instance segmentation, demonstrating its applicability to downstream computer vision tasks.