  Navigation guided by natural language instructions presents a challenging
reasoning problem for instruction followers. Natural language instructions
typically identify only a few high-level decisions and landmarks rather than
complete low-level motor behaviors; much of the missing information must be
inferred based on perceptual context. In machine learning settings, this is
doubly challenging: it is difficult to collect enough annotated data to enable
learning of this reasoning process from scratch, and also difficult to
implement the reasoning process using generic sequence models. Here we describe
an approach to vision-and-language navigation that addresses both these issues
with an embedded speaker model. We use this speaker model to (1) synthesize new
instructions for data augmentation and to (2) implement pragmatic reasoning,
which evaluates how well candidate action sequences explain an instruction.
Both steps are supported by a panoramic action space that reflects the
granularity of human-generated instructions. Experiments show that all three
components of this approach---speaker-driven data augmentation, pragmatic
reasoning and panoramic action space---dramatically improve the performance of
a baseline instruction follower, more than doubling the success rate over the
best existing approach on a standard benchmark.

Computer Vision and Pattern Recognition;Computation and Language