Due to the lack of enough training data and high computational cost to train a deep neural network from scratch, transfer learning has been extensively used in many deep-neural-network-based applications. A commonly-used transfer learning approach involves taking a part of a pre-trained model, adding a few layers at the end, and re-training the new layers with a small dataset. This approach, while efficient and widely used, imposes a security vulnerability because the pre-trained model used in transfer learning are usually available publicly to everyone, including potential attackers. In this paper, we show that without any additional knowledge other than the pre-trained model, an attacker can launch an effective and efficient brute force attack that can craft instances of input to trigger each target class with high confidence. We assume that the attacker does not have access to any targetspecific information, including samples from target classes, re-trained model, and probabilities assigned by Softmax to each class, and thus called target-agnostic attack. These assumptions render all previous attacks impractical, to the best of our knowledge. To evaluate the proposed attack, we perform a set of experiments on face recognition and speech recognition tasks and show the effectiveness of the attack. Our work sheds light on a fundamental security challenge of the Softmax layer when used in transfer learning settings.
The paper proposed a method of attacking deep neural networks that were trained using transfer learning. The primary claim is that the proposed technique only requires knowledge of the base model (i.e., if the frozen parameters taken from pretrained VGG, ResNET models are known, then that is sufficient) and doesn’t require any samples of target classes of the post-transfer application to successfully attack the DNN.
Core idea is that Softmax layers used in classification can be exploited to craft a perturbation that will fool the network. The algorithm uses a brute force approach to iterate through each neuron on the final layer before the softmax. For each activated neuron, all other neurons are zeroed out and a regularized loss is computed between the activated feature and the neuron. The gradient of the loss function is then used to craft a perturbation using typical k-step gradient based method (as used in FGSM, PGD attacks). In other words, they are trying to craft a perturbation that will zero out all other class probability and put maximum weight on the target adversary class when applied to the network.
Experiments were conducted on VGG-Face and Pannous Speech dataset.
1. Paper is easy to follow and different empirical results show a few intuitive observations for a few different settings. 
2. The main issue is that the algorithmic is rather simplistic and seems impractical. Most security applications that motivate the paper can easily defend themselves by simply limiting the number of attempts. Even if this core motivation issue is discarded, the algorithmic contribution is very minimal, no analytical understanding either. 
3. Some comparison with Black Box models should be added. Although authors claim that Black Box models would require many queries, a specific comparison and contrasting with number of attempts and queries would help understand the critical advantages of the proposed technique. 
4. Table 1 provides a good overview of the results but it should include standard deviations for experiments that were averaged.
5. Is there a notion of imperceptibility/attack budget here? Some of the attacks are clearly not imperceptible. This should be discussed. 
6. Section 6 seems a little brief and can use more details. Authors claims that using EVM, “the model successfully defeat all our crafted images” but “there is 7.38% chance that the EVM-base model classifies the input as one of the target classes”. This statement is a little confusing.