FVO: Fast Visual Odometry with Transformers

Abstract

Hybrid pipelines that combine deep learning with classical optimization have established themselves as the dominant approach to visual odometry (VO). By integrating neural network predictions with bundle adjustment, these models estimate camera trajectories with high accuracy. Still, hybrid VO methods fall short of the speed and capabilities of pure end-to-end approaches. Current hybrid frameworks rely on massive, pre-trained 3D networks to predict geometry. Because these backends are trained to be scale-ambiguous and frozen rather than retrained, the pipelines essentially inherit this limitation and, by design, fails to estimate absolute scale. Furthermore, their slow optimization and post-processing steps bottleneck the pipeline's inference speed. We propose to replace post-processing entirely by formulating monocular visual odometry as a direct relative pose regression problem. This formulation enables us to train a fast, high-capacity transformer to predict relative camera poses and corresponding confidences using only camera poses as supervision. More importantly, it allows us to employ a confidence-aware inference scheme that aggregates overlapping pose predictions for robust trajectory estimation. We demonstrate on multiple visual odometry benchmarks that our method, Fast Visual Odometry (FVO), successfully leverages diverse data to achieve competitive or superior performance while being nearly 2 times faster than the fastest baselines.

Reproductions