Understanding theoretical properties of deep and locally connected nonlinear network, such as deep convolutional neural network (DCNN), is still a hard problem despite its empirical success. In this paper, we propose a novel theoretical framework for such networks with ReLU nonlinearity. The framework bridges data distribution with gradient descent rules, favors disentangled representations and is compatible with common regularization techniques such as Batch Norm, after a novel discovery of its projection nature. The framework is built upon teacher-student setting, by projecting the student's forward/backward pass onto the teacher's computational graph. We do not impose unrealistic assumptions (e.g., Gaussian inputs, independence of activation, etc). Our framework could help facilitate theoretical analysis of many practical issues, e.g. disentangled representations in deep networks.
This paper gives a model for understanding locally connected neural networks. The main idea seems to be that the network is sparsely connected, so each neuron is not going to have access to the entire input. One can then think about the gradient of this neuron locally while average out over all the randomness in the input locations that are not relevant to this neuron. Using this framework the paper tried to explain several phenomena in neural networks, including batch normalization, overfitting, disentangling, etc.
I feel the paper is poorly written which made it very hard to understand. For example, as the paper states, the model gives a generative model for input (x,y) pairs. However, I could not find a self-contained description of how this generative model works. Some things are described in Section 3.1 about the discrete summarization variables, but the short paragraph did not describe: (a) What is the "multi-layer" deterministic function? (b) How are these z_\alpha's chosen? (c) Given z's how do we generate x? (d) What happens if we have z_\alpha and z_\beta and the regions \alpha and \beta are not disjoint? What x do we use in the intersection?
In trying to understand the paper, I was thinking that (a)(b) The multilayer deterministic function is a function which gives a tree structure over the z_\alpha's, where y is the root. (I have no idea why this should be a deterministic function, intuitively shouldn't y be chosen randomly, and each z_\alpha chosen randomly conditioned on its parent?)  (c) there is a fixed conditional distribution of P(x_\alpha|z_\alpha), and I really could not figure out (d). The paper definitely seems to allow two receptive fields to intersect as in Figure 1(b).
Without understanding the generative model, it is impossible for me to evaluate the later results. My general comments there is that there are no clear Theorems that summarizes the results (the Theorems in the paper are all just Lemmas that are trying to work towards the final goal of giving some explanations, but the explanations and assumptions are not formally written down). Looking at things separately (as again I couldn't understand the single paragraph describing the generative model), the Assumption in Theorem 3 seems extremely limiting as it is saying that x_j is a discrete distribution (which is probably never true in practice). I wouldn't say "the model does not impose unrealistic assumptions" in abstract if you are going to assume this, rather the model just makes a different kind of unrealistic assumptions (Assumptions in Theorem 2 might be much weaker, but it's hard to judge that).
==== After reading the revision
The revised version is indeed more clear about how the teacher network works, and I have tried to understand the later parts of the paper again. The result of the paper really relies on the two assumptions in Theorem 2. Of the two assumptions, the first one seems to be intuitive (and it is OK although exact conditional independence might be slightly strong). The second assumption is very unclear though as it is not an assumption that is purely about the model/teacher network (which are the x and z variables), it also has to do with the learning algorithm/student network (f's and g's). It is much harder to reason about the behavior of an algorithm on a particular model and directly making an assumption about that in some sense hides the problem. The paper mentioned that the condition is true if z is fine-grained, but this is very vague - it is definitely true if z is super fine-grained to satisfy the assumption in Theorem 3, but that is too extreme.
Overall I still feel the paper is a bit confusing and it would benefit from having a more concrete example. I like the direction of the work but I can't recommend for recommendation at this stage.