ON RANDOM DEEP WEIGHT-TIED AUTOENCODERS: EXACT ASYMPTOTIC ANALYSIS, PHASE TRANSI- TIONS, AND IMPLICATIONS TO TRAINING
We study the behavior of weight-tied multilayer vanilla autoencoders under the assumption of random weights. Via an exact characterization in the limit of large dimensions, our analysis reveals interesting phase transition phenomena when the depth becomes large. This, in particular, provides quantitative answers and insights to three questions that were yet fully understood in the literature. Firstly, we provide a precise answer on how the random deep weight-tied autoencoder model performs "approximate inference" as posed by Scellier et al. (2018), and its connection to reversibility considered by several theoretical studies. Secondly, we show that deep autoencoders display a higher degree of sensitivity to perturbations in the parameters, distinct from the shallow counterparts. Thirdly, we obtain insights on pitfalls in training initialization practice, and demonstrate experimentally that it is possible to train a deep autoencoder, even with the tanh activation and a depth as large as 200 layers, without resorting to techniques such as layer-wise pre-training or batch normalization. Our analysis is not specific to any depths or any Lipschitz activations, and our analytical techniques may have broader applicability.
This paper studies auto-encoders under several assumptions: (a) the auto-encoder's layers are fully connected, with random weights, (b) the auto-encoder is weight-tied, (c) the dimensions of the layers go to infinity with fixed ratios. The main contribution of the paper is to point out that this model of random autoencoder can be elegantly and rigorously analysed with one-dimensional equations. The idea is original and will probably lead to new directions of research. Already the first applications that the paper suggests are exciting.
The paper does a good job in justifying assumptions (a), (b) and (c) in the introduction. It is convincing in the fact that this point of view may bring practical insights on training initialization for real-world autoencoders. Thus my opinion is that this paper brings original and significant ideas in the field.
One flaw of this paper is that the writing might be clearer. For instance when presenting the technical theorem (Theorem 1), it would be useful to have an intuitive explanation for the theorem and the state-evolution-like equations. However, I believe that there are some easy fixes that would greatly improve the clarity of the exposition. Here is a list of suggestions: 
- In Section 2.1, a large number of notations are introduced. It would help a lot if the authors made a graphical representation of these. For instance, a diagram where every linearity / non-linearity is a box, and the different variables $x_l$, $\hat{x}_l$ appear would help a lot. 
- Section 2.2 is rather technical. The authors could try to give some more intuition of what's happening. For instance, they could spend more time after the theorem explaining what $\tau_l, \gamma_l$ and $\rho_l$ mean. They could also introduce the notation S_sig and S_var early and this section (and not in Section 3), because it helps interpreting the parameters. It would also help if they could write a heuristic derivation of the state-evolution-like equations. From the paper, the only way the reader can understand the intuition behind those complicated equations is to look at the proof of Theorem 1 (which is rather technical). 
- In Section 3.1, I did not understand the difference between interpretations 1 and 2. Could the authors clarify? 
- In Section 3.4, I did not understand the sentence: "In particular, near the phase transition of \gamma, S_sig/S_var = \Omega(\beta^{1.5}". If one uses the \Omega notation, it means that some parameter is converging to something. What is the parameter? As a consequence, I did not understand this paragraph. 
- In Section 3.5, the authors should make clear from the beginning why they are running those specific simulations. What hypothesis are they trying to check? I finally concluded that they are running simulations to check if the hypothesis they make in the first paragraph are true. They also want to compare with some other criteria in the literature, named EOC, that also gives insights about the trainability of the network. However, they could explicitly say in the beginning of the second paragraph that this is the goal.
- In a similar spirit, the authors should end Section 3.5 with a clear conclusion on whether or not the framework enables us to predict the trainability of the autoencoder. 
Minor edits / remarks: 
- Typo: last but one paragraph of the introduction: "whose analysis is typically more straighforwards" -> "straightforward".
- At the end of Section 3.2: what can be proved about the behavior of \gamma / \sqrt{\rho}? It is obviously a central quantity and the authors do not say what happens in the phases where \gamma and \rho go to infinity for instance. Is it because it is hard to analyse?