CAMOU: LEARNING A VEHICLE CAMOUFLAGE FOR PHYSICAL ADVERSARIAL ATTACK ON OBJECT DETECTORS IN THE WILD
In this paper, we conduct an intriguing experimental study about the physical adversarial attack on object detectors in the wild. In particular, we learn a camouflage pattern to hide vehicles from being detected by state-of-the-art convolutional neural network based detectors. Our approach alternates between two threads. In the first, we train a neural approximation function to imitate how a simulator applies a camouflage to vehicles and how a vehicle detector performs given images of the camouflaged vehicles. In the second, we minimize the approximated detection score by searching for the optimal camouflage. Experiments show that the learned camouflage can not only hide a vehicle from the image-based detectors under many test cases but also generalizes to different environments, vehicles, and object detectors.
The authors investigate the problem of learning a camouflage pattern which, when applied to a simulated vehicle, will prevent an object detector from detecting it. The problem is frames as finding the camouflage pattern which minimises the expected decision scores from the detector over a distribution of possible views of this pattern applied to a vehicle (e.g. vantage point, background, etc). This expectation is approximated by sampling detector scores when applying the detector considered to images synthesised using a number of vantage points and scene contexts. In order to generate a gradient signal with respect to the camouflage applied (the simulated image rendering is non-differentiable) the approach considers learning a clone network which takes as input the camouflage pattern, the vehicle model and a given environment and outputs the vehicle detector’s devision values. The clone network and the optimal camouflage are learned alternately in order to obtain a relatively faithful approximation. The approach is evaluated in simulation using two standard object detectors (Mask R-CNN and YOLOv3) on two vehicle models over a range of transformations.
Pros:
———
- interesting challenge
- the clone network provides an interesting solution to the challenge of having a (non-differentiable) simulator in the loop.
Cons:
———
- the fact that this is done in simulation is understandable but to some degree negates the authors’ point that physicality matters because it is harder than images. How effective is a learned camouflages in reality? It would be great to get at least some evidence of this.
- if the sim-2-real gap is too big it is not clear how this approach could ever be feasibly employed in the real world. Some intuition here would add value.
- the approach presented critically hinges on the quality of the clone network. Some analysis of robustness of the approach with respect to network performance would add value.
- little is offered in terms of analysis of the camouflages generated. The three failure modes triggered and discussed are intuitive. But are there any insights available as to what aspect of the camouflages triggers these behaviours in the object detectors? This could also add significant value.
In summary, this work addresses an interesting problem but it is not clear how impactful the approach will be in real-world settings and hence how significant it it. Some of the more technically interesting aspects of the submission (e.g. the quality of the clone network, any learning derived from the camouflages generated) are not explored. 
Misc comments: 
- equs 1, 2 & 3: min_* should be argmin_*