TOWARDS REVERSE-ENGINEERING BLACK-BOX NEURAL NETWORKS
Many deployed learned models are black boxes: given input, returns output. Internal information about the model, such as the architecture, optimisation procedure, or training data, is not disclosed explicitly as it might contain proprietary information or make the system more vulnerable. This work shows that such attributes of neural networks can be exposed from a sequence of queries. This has multiple implications. On the one hand, our work exposes the vulnerability of black-box neural networks to different types of attacks -we show that the revealed internal information helps generate more effective adversarial examples against the black box model. On the other hand, this technique can be used for better protection of private content from automatic recognition models using adversarial examples. Our paper suggests that it is actually hard to draw a line between white box and black box models. The code is available at goo.gl/MbYfsv.
-----UPDATE------
Having read the responses from the authors, and the other reviews, I am happy with my rating and maintain that this paper should be accepted.
----------------------
In this paper, the authors trains a large number of MNIST classifier networks with differing attributes (batch-size, activation function, no. layers etc.) and then utilises the inputs and outputs of these networks to predict said attributes successfully. They then show that they are able to use the methods developed to predict the family of Imagenet-trained networks and use this information to improve adversarial attack.
I enjoyed reading this paper. It is a very interesting set up, and a novel idea.
A few comments:
The paper is easy to read, and largely written well. The article is missing from the nouns quite often though so this is something that should be amended. There are a few spelling slip ups ("to a certain extend" --> "to a certain extent", "as will see" --> "as we will see")
It appears that the output for kennen-o is a discrete probability vector for each attribute, where each entry corresponds to a possibility (for example, for "batch-size" it is a length 3 vector where the first entry corresponds to 64, the second 128, and the third 256). What happens if you instead treat it as a regression task, would it then be able to hint at intermediates (a batch size of 96) or extremes (say, 512).
A flaw of this paper is that kennen-i and io appear to require gradients from the network being probed (you do mention this in passing), which realistically you would never have access to. (Please do correct me if I have misunderstood this)
It would be helpful if Section 4 had a paragraph as to your thoughts regarding why certain attributes are easier/harder to predict. Also, the caption for Table 2 could contain more information regarding the network outputs.
You have jumped from predicting 12 attributes on MNIST to 1 attribute on Imagenet. It could be beneficial to do an intermediate experiment (a handful of attributes on a middling task).
I think this paper should be accepted as it is interesting and novel.
Pros
------
- Interesting idea
- Reads well
- Fairly good experimental results
Cons
------
- kennen-i seems like it couldn't be realistically deployed
- lack of an intermediate difficulty task