The complexity of large-scale neural networks can lead to poor understanding of their internal details. We show that this opaqueness provides an opportunity for adversaries to embed unintended functionalities into the network in the form of Trojan horse attacks. Our novel framework hides the existence of a malicious network within a benign transport network. Our attack is flexible, easy to execute, and difficult to detect. We prove theoretically that the malicious network's detection is computationally infeasible and demonstrate empirically that the transport network does not compromise its disguise. Our attack exposes an important, previously unknown loophole that unveils a new direction in machine learning security.
This paper proposed a novel an very interesting attacking scenario (the authors called it the Trojan horse attack) that aims to embed a secret model for solving a secret task into a public model for solving a different public task, through the use of weight permutations, where the permutations can be considered as a key in the crypto setting. The authors prove the computational complexity (NP-completeness) of detecting such as a secret model. Experimental results show that it is possible to secretly embed multiple and different secret models into one publish model via joint training with permutation, while the performance of each model is similar to the individually trained models.
Overall, the trojan horse attacking scenario considered in this paper is novel and provides new insights to the research in adversarial machine learning. The finding that permutation along is able to embed multiple models is highly non-trivial. While I agree with the authors' explanations on the difference between trojan horse attack versus multi-task learning (shared data or not), my main concern is the lack of comparison and discussion to another secrecy-based attack scheme, the "Adversarial Reprogramming of Neural Networks" published in 2019. In their adversarial reprogramming attack, the model weights also remain unchanged (and un-permuted). To train the secret "model", Elsayed et al. used a trainable input perturbation to learn how to solve the secret task. Although Elsayed et al. did not consider the case of reprogramming for multiple secret tasks, I believe the proposed method and adversarial reprogramming share common goals and their attacks are both stealthy in the sense final model weights are unchanged. I would like to know the authors' thoughts on the proposed attack v.s. adversarial reprogramming to better motivate the importance of the considered attack scenario. In my perspective, they have the same threat model but adversarial reprogramming seems to be even stealthier as it does not use the secret data to jointly train the final model. Some discussion and numerical comparisons will be very useful for clarifying the advantage of the proposed method over adversarial reprogramming in terms of "attacks". I am happy to increase my rating if the authors address this main concern.
*** Post-rebuttal comments
I thank the authors for the clarification. I actually quite like the idea and believe the threat model is valid if addressed fin a clearer manner. I look forward to a future version of this work.
***