Recent work in adversarial machine learning started to focus on the visual perception in autonomous driving and studied Adversarial Examples (AEs) for object detection models. However, in such visual perception pipeline the detected objects must also be tracked, in a process called Multiple Object Tracking (MOT), to build the moving trajectories of surrounding obstacles. Since MOT is designed to be robust against errors in object detection, it poses a general challenge to existing attack techniques that blindly target objection detection: we find that a success rate of over 98% is needed for them to actually affect the tracking results, a requirement that no existing attack technique can satisfy. In this paper, we are the first to study adversarial machine learning attacks against the complete visual perception pipeline in autonomous driving, and discover a novel attack technique, tracker hijacking, that can effectively fool MOT using AEs on object detection. Using our technique, successful AEs on as few as one single frame can move an existing object in to or out of the headway of an autonomous vehicle to cause potential safety hazards. We perform evaluation using the Berkeley Deep Drive dataset and find that on average when 3 frames are attacked, our attack can have a nearly 100% success rate while attacks that blindly target object detection only have up to 25%.
This paper is about conducting evasion attacks against Multiple Object Tracking (MOT) techniques. Compared to existing work on adversarial examples against object detection, to attack MOT techniques, the adversary needs to successfully fool multiple frames, and the authors show that by naively using existing attack approaches, the adversary needs to achieve 98% single-frame attack success rate to fool the tracking system, which is too hard for existing attack algorithms.  Therefore, this paper proposes a smart way of attacking MOT techniques by leveraging the properties of the tracking algorithm. In particular, they generate adversarial perturbations to remove the original bounding box while adding a fake bounding box that has some overlap with the original bounding box, so that the system will compute the movement of the object in a wrong way. They evaluate on videos in Berkeley Deep Drive dataset, and show that by attacking 2~3 frames, they can achieve nearly 100% attack success rate, while the attack success rate is only 25% if the tracking algorithm is not considered when crafting the attacks.
There has been a long line of work studying adversarial attacks against autonomous driving systems, which aims at revealing the security threat of such attacks in the real world. While the community has made promising progress, to my best knowledge, this paper is the first work considering attacking MOT techniques, and they suggest the importance of this attack scenario by demonstrating that attacking objection detection alone is not sufficient in terms of fooling the tracking technique. Meanwhile, the proposed attack method is interesting and effective, even if the perturbation is only bounded in a patch located at a reasonable place in the frame, e.g., the target car to attack.  Therefore, I lean towards accepting this paper, as it contributes to a new perspective of adversarial attacks.
However, I have some questions about evaluation.
1. To generate adversarial frames for the trajectory of the same car, is the adversarial patch inserted since the first frame and always stays the same, or different patches are needed to attack different frames of the same trajectory? I feel that if the adversarial patch needs to change a lot in the entire trajectory, then the practicality of the attack would be compromised.
2. How large does the adversarial patch need to be in order to successfully launch the attacks? And does the position of the patch affect the attack performance?
3. I wonder if the same adversarial patch may work beyond a single context, i.e., with the same patch, the same car can fool the tracking technique with different background scenes. I am not familiar with the details of the dataset in their evaluation, but it would be helpful to show relevant results if it is easy to simulate using their dataset.
------------
Post-rebuttal comments
I thank the authors for their response, and I keep my original assessment.
------------