We introduce "Talk The Walk", the first large-scale dialogue dataset grounded in action and perception. The task involves two agents (a "guide" and a "tourist") that communicate via natural language in order to achieve a common goal: having the tourist navigate to a given target location. The task and dataset, which are described in detail, are challenging and their full solution is an open problem that we pose to the community. We (i) focus on the task of tourist localization and develop the novel Masked Attention for Spatial Convolutions (MASC) mechanism that allows for grounding tourist utterances into the guide's map, (ii) show it yields significant improvements for both emergent and natural language communication, and (iii) using this method, we establish non-trivial baselines on the full task.
The paper introduces a new dataset "Talk the Walk" that are dialogs
between a guide and a tourist, where the guide is to help the tourist
navigate to a target location.  The guide has access to a map and the
target location, but he relies on the tourist to communicate her
state (location) by natural language.
Pros.:
The task represented in the dataset can be highly challenging, and
respectable effort went into creating the dataset based on real city
neighborhoods.   The description and analysis of the dataset are
detailed.  The paper is well written up to the end of page 3.
The dataset by itself is a good contribution to the scientific
community when it is shared.  There could be many topics open
for studying within the same data.
The list of references and related work is exceptionally thorough
and useful for researchers interested in the topic.
Cons.:
The description of the experiments done with the dataset, however,
suffers from being overly cryptic.  The methods are not sufficiently
motivated, very few alternatives are presented and argued against,
and the several sections give a dry report of the sequences of things
the authors did.  It is not clear how others may find value in the
results and conclusions.
While the paper opens with the emphasis of a real-world setting,
after a series of simplifications (e.g. landmark typing, perfect perception)
it seems that much of the full complexity of the natural task is taken out,
and the main goal of the study is no longer clear.
For example, since there are only 9 types of landmarks,
in a small neighborhood there are not many combinations to draw reference to.
Simple observation sequences of such can easily narrow down the location
uncertainty.  It is important to highlight what the remaining open
issues are that make the task still challenging.
Misc.:
Examples are missing in the discussion of the experimental tasks.
e.g. in the study of emergent language, what could be a message that a
tourist may generate to describe his location?  what makes it hard for
the guide to decode it?  Likewise, what could be an example state of
the tourist and the description of that state in natural language?
Without the examples, it is difficult for the reader to have a sense
of the challenges in each task.
Table 8 is the first place where (finally) some utterances are presented.
However the description in the table or in the text is not sufficient
to convey the point that is supposed to be explained by the example.
The descriptions of the landmarks are restricted to the type of business
at the location with 9 possibilities.  Is the list of 9 exhaustive?
Are there any exceptions  (e.g. schools)?  How are such exceptions
represented and treated in the dialogs?
To what extent the difficulty of the tasks depends on the variability
of the combination of landmarks visible at each location?
What could be a simplest way to do this without neural-modeling?
e.g. with the many limitations that are built into the task and its
representations, will a simple decision tree based instruction method suffice?
Or a traditional algorithm that relies on repeated exploration and evaluation?
It is surprising that such possibilities are not even mentioned.
A complex neural architecture does not seem to be well justified unless
it is motivated by the need to overcome limitations of a classical method.
Irrelevant to the research effort, a thought about the dataset is that,
in these days with popular uses of GPS, the reliance on such dialogs for
navigation feels a bit backwards.