  Adversarial attacks are valuable for providing insights into the blind-spots
of deep learning models and help improve their robustness. Existing work on
adversarial attacks have mainly focused on static scenes; however, it remains
unclear whether such attacks are effective against embodied agents, which could
navigate and interact with a dynamic environment. In this work, we take the
first step to study adversarial attacks for embodied agents. In particular, we
generate spatiotemporal perturbations to form 3D adversarial examples, which
exploit the interaction history in both the temporal and spatial dimensions.
Regarding the temporal dimension, since agents make predictions based on
historical observations, we develop a trajectory attention module to explore
scene view contributions, which further help localize 3D objects appeared with
the highest stimuli. By conciliating with clues from the temporal dimension,
along the spatial dimension, we adversarially perturb the physical properties
(e.g., texture and 3D shape) of the contextual objects that appeared in the
most important scene views. Extensive experiments on the EQA-v1 dataset for
several embodied tasks in both the white-box and black-box settings have been
conducted, which demonstrate that our perturbations have strong attack and
generalization abilities.

Computer Vision and Pattern Recognition