DEEP LAYERS AS STOCHASTIC SOLVERS
We provide a novel perspective on the forward pass through a block of layers in a deep network. In particular, we show that a forward pass through a standard dropout layer followed by a linear layer and a non-linear activation is equivalent to optimizing a convex objective with a single iteration of a τ -nice Proximal Stochastic Gradient method. We further show that replacing standard Bernoulli dropout with additive dropout is equivalent to optimizing the same convex objective with a variance-reduced proximal method. By expressing both fully-connected and convolutional layers as special cases of a high-order tensor product, we unify the underlying convex optimization problem in the tensor setting and derive a formula for the Lipschitz constant L used to determine the optimal step size of the above proximal methods. We conduct experiments with standard convolutional networks applied to the CIFAR-10 and CIFAR-100 datasets and show that replacing a block of layers with multiple iterations of the corresponding solver, with step size set via L, consistently improves classification accuracy.
This paper theoretically verifies an equivalence between stochastic solvers on a particular class of convex optimization problems and a forward pass through a dropout layer followed by a linear layer and a non-linear activation. Experiments show that replacing a block of layers with multiple iterations of the corresponding solver improves classification accuracy. My detailed comments are as follows. 
*Positive points: 
1. The perspective is novel and interesting, i.e., training a forward pass through a dropout layer followed by a linear layer and a non-linear activation is equivalent to optimizing a convex problem by a Proximal Stochastic Gradient method. More importantly, this perspective has been theoretically verified. 
2. In the experiments, training networks with solvers replacing deep layers is able to improve accuracy significantly. 
*Negative points:
1. Some technical details are not clear and many notations are used without clear explanations. Specifically, many notations based on (Bibi & Ghanem, 2017) make the paper hard to follow. Moreover, there are many mistakes in proofs. Please revise the paper according to the following comments.
2. There are many limitations for the proposed method. Specifically, the theoretical results are hard to be extended to more general neural networks (e.g., ResNet) with Batch Normalization which are widely used.
3. The experiment section should be significantly improved. There are only two datasets (i.e., CIFAR-10, CIFAR-100). It would be convincing that more baselines are compared on other datasets, such as ImageNet.
*Detailed comments:
**Comments on technical issues.
1. In Problem (1), the definition of $g(x)$ and $f¬_i()$ should be provided for clarity.
2. The motivation and some details of Function (2) should be provided since $F(x^l)$ is important for proving the equivalence between stochastic solvers and a forward network. In addition, $x$ should be corrected as $x^l$.
3. Is Equation (3) wrong? Based on the definition of Prox-GD in (Xiao & Zhang, 2014), it should be $x^l=Prox(x^{l-1} – 1/L \nabla F(x^l)) = Prox((I-1/L A)x^{l-1} + 1/L (AA^T x^l + b))$ which is different from Equation (3). Moreover, the Lipschitz constant w.r.t. maximal eigenvalue should be proved.
4. In Definitions D.1 and D.2, what is the definition of $fold_{H0}$? Is the dimensionality of $bdiag(D)$ wrong? Why is $bdiag(D)$ an identity mapping when $n_3=n_4$?
5. There are some issues on Equation (7) and its proofs. Is $A(:, i, :, :)$ and $\vec{X}(i, :, :, :) $ wrong? It affects the results of Equation (8). Does Equation (25) miss the operator $fold_{HO}$ in Appendix G? Please check the proofs of Proposition 1.
6. There are some issues on proofs of Lemma 2. Why are $F_H \otimes F_W \otimes I_{n_1}$ and $F_H \otimes F_W \otimes I_{n_2}$ orthogonal? Is the third and fourth equality in (24) wrong? For the fourth equality in (24), Eigen decomposition seems to be for a matrix, not a tensor.
**Comments on Experiments
1. Training Networks is equivalent to optimizing proximal solvers. Why can training networks with solvers replacing blocks of layers improve accuracy? Reasonable explanations should be provided.
2. Optimizing a convex optimization problem can easily obtain the optimal solution. What happens if solvers are used to replace more blocks of layers? Complexity analysis for these should be provided.
3. The experiments are only conducted on two datasets (i.e., CIFAR-10, CIFAR-100). It would be better to compared more baselines on other datasets, such as ImageNet.