In this work, we propose a goal-driven collaborative task that contains language, vision, and action in a virtual environment as its core components. Specifically, we develop a Collaborative image-Drawing game between two agents, called CoDraw. Our game is grounded in a virtual world that contains movable clip art objects. The game involves two players: a Teller and a Drawer. The Teller sees an abstract scene containing multiple clip art pieces in a semantically meaningful configuration, while the Drawer tries to reconstruct the scene on an empty canvas using available clip art pieces. The two players communicate via two-way communication using natural language. We collect the CoDraw dataset of ∼10K dialogs consisting of ∼138K messages exchanged between human agents. We define protocols and metrics to evaluate the effectiveness of learned agents on this testbed, highlighting the need for a novel crosstalk condition which pairs agents trained independently on disjoint subsets of the training data for evaluation. We present models for our task, including simple but effective nearest-neighbor techniques and neural network approaches trained using a combination of imitation learning and goal-driven training. All models are benchmarked using both fully automated evaluation and by playing the game with live human agents.
The paper proposes a game of collaborative drawing where a teller is
to communicate a picture to a drawer via natural language.  The picture
allows only a small number of components and a fixed and limited set
of detailed variations of such components.
Pros:
The work contributed a dataset where the task has relatively objective
criteria for success.  The dataset itself is a valuable contribution
to the community interested in the subject.   It may be useful for
purposes beyond those it was designed for.
The task is interesting and its visual nature allows for easy inspection
of the reasons for successes or failures.  It provides reasonable grounding
for the dialog.  By restricting the scope of variations through the options
and parameters, some detailed aspects of the conversation could be explored
with carefully controlled experiments.
The authors identified the need for and proposed a "crosstalk" protocol
that they believe can prevent leakage via common training data and
the development of non-language, shared codebooks that defeat the purpose
of focusing on the natural language dialog.
The set up allows for pairing of human and human, machine and machine,
and human and machine for the two roles, which enables comparison to
human performance baselines in several perspectives.
The figures give useful examples that are of great help to the readers.
Cons.:
Despite the restriction of the task context to creating a picture with
severely limited components, the scenario of the dialogs still has many
details to keep track of, and many important facets are missing in the
descriptions, especially on the data.
There is no analysis of the errors.  The presentation of
experimental results stops at the summary metrics, leaving many
doubts on why they are as such.
The work feels somewhat pre-mature in its exploration of the models
and the conclusions to warrant publication.  At times it feels like the
authors do not understand enough why the algorithms behave as they do.
However if this is considered as a dataset description paper and
the right expectation is set in the openings, it may still be acceptable.
The completed work warrants a longer report when more solid conclusions
can be drawn about the model behavior.
The writing is not organized enough and it takes many back-and-forth rounds
of checking during reading to find out about certain details that are given
long after their first references in other contexts.  Some examples are
included in the followings.
Misc.
Section 3.2, datasets of 9993 dialogs:
Are they done by humans?   Later it is understood from further descriptions.
It is useful to be more explicit at the first mention of this data collection effort.
The way they relate to the 10020 scenes is mentioned as "one per scene", with a footnote on some being removed.
Does it mean that no scene is described by two different people?  Does this
limit the usefulness of the data in understanding inter-personal differences?
Later in the descriptions (e.g. 4.1 on baseline methods) the notion of
training set is mentioned, but up to then there is no mentioning of how
training and testing (novel scenes) data are created.
It is also not clear what training data include: scenes only?
Dialogs associated with specific scenes?  Drawer actions?
Section 4.1, what is a drawer action?  How many possibilities are there?
From the description of "rule-based nearest-neighbor drawer" they seem to be
corresponding to "teller utterance".
However it is not clear where they come from.  What is an example of a drawer action?
Are the draw actions represented using the feature vectors discussed in the later sections?
Section 5.1, the need for the crosstalk protocol is an interesting observation,
however based on the description here, a reader may not be able to understand
the problem.  What do you mean by "only limited generalization has taken place"?  Any examples?
Section 5, near the end: the description of the dataset splits is too cryptic.
What are being split?  How is val used in this context?
All in all the data preparation and partitioning descriptions need substantial clarification.
Section 6:  Besides reporting averaged similarity scores, it will be useful to report some error analysis.
What are the very good or very bad cases?  Why did that happen?
Are the bad scenes constructed by humans the same as those bad scenes
constructed by machines?  Do humans and machines tend to make different errors?