The output of a neural network depends on its parameters in a highly nonlinear way, and it is widely assumed that a network's parameters cannot be identified from its outputs. Here, we show that in many cases it is possible to reconstruct the architecture, weights, and biases of a deep ReLU network given the ability to query the network. ReLU networks are piecewise linear and the boundaries between pieces correspond to inputs for which one of the ReLUs switches between inactive and active states. Thus, first-layer ReLUs can be identified (up to sign and scaling) based on the orientation of their associated hyperplanes. Later-layer ReLU boundaries bend when they cross earlier-layer boundaries and the extent of bending reveals the weights between them. Our algorithm uses this to identify the units in the network and weights connecting them (up to isomorphism). The fact that considerable parts of deep networks can be identified from their outputs has implications for security, neuroscience, and our understanding of neural networks.
This paper introduces an approach to recover weights of ReLU neural networks by querying the network with specifically constructed inputs. The authors notice that the decision regions of such networks are piece-wise linear corresponding to activations of individual neurons. This allows to identify hyperplanes that constitute the decision boundary and find intersection points of the decision boundaries corresponding to neurons at different layers of the network. However, weights can be recovered only up to permutations of neurons in each layer and up to a constant scaling factor for each layer.
The algorithm consists of two parts: Identifying parameters of the first layer and subsequent layers. First they sample a lot of line segments and find their intersections with the decision boundary, i.e. where pre-activations equal 0. Then they sample some points around the intersections and estimate the hyperplanes up to a scaling factor and a sign. For the first layer they check whether some of the hyperplanes belong to it. For the consecutive layers they proceed by moving from the intersection points along the hyperplanes until the decision boundary bends. Again, by identifying which of the bends correspond to the intersection of the current layer's hyperplanes with the previous layers' ones, they are able to recover parameters of the current layer by computing the angles of these intersections.
This paper tackles a very interesting and important problem that might have huge implications for security and many other aspects. However, I'm leaning towards Reject for the following reasons:
1. The algorithm's description is either incomplete or unclear. There are such core functions as PointsOnLine and TestHyperplane, whose pseudo-code would be very helpful for understanding. For example, the authors say that PointsOnLine performs a "binary search," but then this function can find only one (arbitrary) intersection of a line segment with a decision boundary, while each sampled line can intersect multiple ones. If it is not binary search, then the asymptotic analysis given in the end of Sec. 4.2 is incorrect. Even more mysterious is TestHyperplane, from the provided intuition I do not understand how it is possible to distinguish hyperplanes corresponding to the first layer vs. the other layers. In Sec. 4.3, second paragraph, the choice of R is unclear. How to chose it to make sure that the closest boundary intersects it?
The authors consider a very limited setting of only fully-connected (linear) layers with ReLU activations. In this case it is easy to see that the resulting decision boundary is indeed piece-wise linear with a lot of bending. Authors themselves notice, that "the algorithm does not account for weight sharing." For CNN this will lead to each virtual neuron in each channel to have its own kernel weights, although there must be one kernel per channel. Also the authors admit, that pooling layers affect partitioning of the activation regions, making the proposed approach inapplicable. The authors did not discuss whether the proposed approach can handle batchnorm layers. Such non-linear transformations could pose serious problems. All this rules out applications to, for example, all CNN-based architectures, that prevail in computer vision. The authors mention, that their "algorithm holds with slight modification" for ResNets, but as mentioned earlier convolutional, pooling and batchnorm layers make it not so trivial (if at all possible).
2. Experimental evaluation is extremely limited: It is all contained in just one paragraph. Although it is mentioned that "it is often possible to recover parameters of deep ReLU networks," they evaluated their approach on very shallow and narrow networks (only 2 layers, 10 to 50 neurons in each). The immediate question here is why this algorithm is not applied to sufficiently deep NN? At least a network that could classify MNIST reasonably well. Actually, this would be a better proof-of-concept: Given a pre-trained MNIST classifier, apply the proposed method, recover the weights and check if you get the same output as from the original network. Whereas here the evaluation is given as a normalized relative error of the estimated vs. the true weights. Which raises the question of how the scaling factor was chosen? Recall, that the proposed method estimates network's parameters only up to an arbitrary scaling factor. My guess, is that for the Figures 3 and 4 (both right) the estimated weights were re-scaled optimally to minimize the relative error. But in the end, one is interested in recovering the original weights of the network, not relative ones.
I am very confused by Fig. 3 left: Why is the number of queries going down as the number of neurons increases? Should it not be that with more neurons the ambiguity also increases, requiring more queries? Again, this analysis is very limited, it would be very interesting to see, how many more queries one needs for deeper layers of the network. But for this experiments with deeper than 2 layers networks are necessary.
3. The choice of parameters is unclear and not discussed. How long should the line segments be, how many of them. How many points are sampled and within which radius to identify hyperplanes, how to choose 'R'. And how all these choices affect accuracy and performance.
Overall, the paper looks rather incomplete to me and requires a major revision. It will definitely benefit if the "slight modification" for the case of ResNets is included. Also, experimental evaluation should be completely re-done and extended.