ZoeDepth: Zero-shot Transfer by Combining Relative and Metric Depth

Code

• github.com/isl-org/ZoeDepth
  OfficialIn paperpytorch★ 2,807
• github.com/isl-org/MiDaS
  pytorch★ 5,348
• github.com/intel-isl/MiDaS
  pytorch★ 5,348
• github.com/thygate/stable-diffusion-webui-depthmap-script
  pytorch★ 1,850
• github.com/cake-lab/Mobile-AR-Depth-Estimation
  pytorch★ 24
• github.com/GuidoManni/BodySLAM
  pytorch★ 23

Abstract

This paper tackles the problem of depth estimation from a single image. Existing work either focuses on generalization performance disregarding metric scale, i.e. relative depth estimation, or state-of-the-art results on specific datasets, i.e. metric depth estimation. We propose the first approach that combines both worlds, leading to a model with excellent generalization performance while maintaining metric scale. Our flagship model, ZoeD-M12-NK, is pre-trained on 12 datasets using relative depth and fine-tuned on two datasets using metric depth. We use a lightweight head with a novel bin adjustment design called metric bins module for each domain. During inference, each input image is automatically routed to the appropriate head using a latent classifier. Our framework admits multiple configurations depending on the datasets used for relative depth pre-training and metric fine-tuning. Without pre-training, we can already significantly improve the state of the art (SOTA) on the NYU Depth v2 indoor dataset. Pre-training on twelve datasets and fine-tuning on the NYU Depth v2 indoor dataset, we can further improve SOTA for a total of 21% in terms of relative absolute error (REL). Finally, ZoeD-M12-NK is the first model that can jointly train on multiple datasets (NYU Depth v2 and KITTI) without a significant drop in performance and achieve unprecedented zero-shot generalization performance to eight unseen datasets from both indoor and outdoor domains. The code and pre-trained models are publicly available at https://github.com/isl-org/ZoeDepth .

Tasks

Reproductions