  For embodied agents, navigation is an important ability but not an isolated
goal. Agents are also expected to perform specific tasks after reaching the
target location, such as picking up objects and assembling them into a
particular arrangement. We combine Vision-and-Language Navigation, assembling
of collected objects, and object referring expression comprehension, to create
a novel joint navigation-and-assembly task, named ArraMon. During this task,
the agent (similar to a PokeMON GO player) is asked to find and collect
different target objects one-by-one by navigating based on natural language
instructions in a complex, realistic outdoor environment, but then also ARRAnge
the collected objects part-by-part in an egocentric grid-layout environment. To
support this task, we implement a 3D dynamic environment simulator and collect
a dataset (in English; and also extended to Hindi) with human-written
navigation and assembling instructions, and the corresponding ground truth
trajectories. We also filter the collected instructions via a verification
stage, leading to a total of 7.7K task instances (30.8K instructions and
paths). We present results for several baseline models (integrated and biased)
and metrics (nDTW, CTC, rPOD, and PTC), and the large model-human performance
gap demonstrates that our task is challenging and presents a wide scope for
future work. Our dataset, simulator, and code are publicly available at:
https://arramonunc.github.io

Computation and Language;Artificial Intelligence;Computer Vision and Pattern Recognition;Robotics