  In the real world, autonomous driving agents navigate in highly dynamic
environments full of unexpected situations where pre-trained models are
unreliable. In these situations, what is immediately available to vehicles is
often only human operators. Empowering autonomous driving agents with the
ability to navigate in a continuous and dynamic environment and to communicate
with humans through sensorimotor-grounded dialogue becomes critical. To this
end, we introduce Dialogue On the ROad To Handle Irregular Events (DOROTHIE), a
novel interactive simulation platform that enables the creation of unexpected
situations on the fly to support empirical studies on situated communication
with autonomous driving agents. Based on this platform, we created the Situated
Dialogue Navigation (SDN), a navigation benchmark of 183 trials with a total of
8415 utterances, around 18.7 hours of control streams, and 2.9 hours of trimmed
audio. SDN is developed to evaluate the agent's ability to predict dialogue
moves from humans as well as generate its own dialogue moves and physical
navigation actions. We further developed a transformer-based baseline model for
these SDN tasks. Our empirical results indicate that language guided-navigation
in a highly dynamic environment is an extremely difficult task for end-to-end
models. These results will provide insight towards future work on robust
autonomous driving agents. The DOROTHIE platform, SDN benchmark, and code for
the baseline model are available at https://github.com/sled-group/DOROTHIE.

Artificial Intelligence;Computation and Language;Computer Vision and Pattern Recognition;Robotics