Vision-and-Language Navigation: Interpreting visually-grounded navigation instructions in real environments

Code

• github.com/peteanderson80/Matterport3DSimulator
  Officialpytorch★ 0
• github.com/YicongHong/Recurrent-VLN-BERT
  pytorch★ 202
• github.com/google-research-datasets/RxR
  tf★ 176
• github.com/MarSaKi/NvEM
  pytorch★ 78
• github.com/YicongHong/Entity-Graph-VLN
  pytorch★ 46
• github.com/batra-mlp-lab/vln-sim2real
  pytorch★ 45
• github.com/hlr/vln-trans
  pytorch★ 14
• github.com/batra-mlp-lab/vln-chasing-ghosts
  pytorch★ 9

Abstract

A robot that can carry out a natural-language instruction has been a dream since before the Jetsons cartoon series imagined a life of leisure mediated by a fleet of attentive robot helpers. It is a dream that remains stubbornly distant. However, recent advances in vision and language methods have made incredible progress in closely related areas. This is significant because a robot interpreting a natural-language navigation instruction on the basis of what it sees is carrying out a vision and language process that is similar to Visual Question Answering. Both tasks can be interpreted as visually grounded sequence-to-sequence translation problems, and many of the same methods are applicable. To enable and encourage the application of vision and language methods to the problem of interpreting visually-grounded navigation instructions, we present the Matterport3D Simulator -- a large-scale reinforcement learning environment based on real imagery. Using this simulator, which can in future support a range of embodied vision and language tasks, we provide the first benchmark dataset for visually-grounded natural language navigation in real buildings -- the Room-to-Room (R2R) dataset.

Tasks

Benchmark Results

Reproductions