Vision Transformers for Dense Prediction

Code

• github.com/huggingface/transformers
  pytorch★ 158,292
• github.com/isl-org/MiDaS
  pytorch★ 5,348
• github.com/intel-isl/MiDaS
  pytorch★ 5,348
• github.com/kritiksoman/GIMP-ML
  pytorch★ 1,542
• github.com/SforAiDl/vformer
  pytorch★ 164
• github.com/antocad/FocusOnDepth
  pytorch★ 155
• github.com/EPFL-VILAB/3DCommonCorruptions
  pytorch★ 87
• github.com/Expedit-LargeScale-Vision-Transformer/Expedit-DPT
  pytorch★ 10
• github.com/alexeyab/midas
  pytorch★ 8
• github.com/danielzgsilva/MonoDepthAttacks
  pytorch★ 3

Abstract

We introduce dense vision transformers, an architecture that leverages vision transformers in place of convolutional networks as a backbone for dense prediction tasks. We assemble tokens from various stages of the vision transformer into image-like representations at various resolutions and progressively combine them into full-resolution predictions using a convolutional decoder. The transformer backbone processes representations at a constant and relatively high resolution and has a global receptive field at every stage. These properties allow the dense vision transformer to provide finer-grained and more globally coherent predictions when compared to fully-convolutional networks. Our experiments show that this architecture yields substantial improvements on dense prediction tasks, especially when a large amount of training data is available. For monocular depth estimation, we observe an improvement of up to 28% in relative performance when compared to a state-of-the-art fully-convolutional network. When applied to semantic segmentation, dense vision transformers set a new state of the art on ADE20K with 49.02% mIoU. We further show that the architecture can be fine-tuned on smaller datasets such as NYUv2, KITTI, and Pascal Context where it also sets the new state of the art. Our models are available at https://github.com/intel-isl/DPT.

Tasks

Benchmark Results

Reproductions