Hierarchically embedding smaller networks in larger networks, e.g. by increasing the number of hidden units, has been studied since the 1990s. The main interest was in understanding possible redundancies in the parameterization, as well as in studying how such embeddings affect critical points. We take these results as a point of departure to devise a novel strategy for escaping from flat regions of the error surface and to address the slow-down of gradient-based methods experienced in plateaus of saddle points. The idea is to expand the dimensionality of a network in a way that guarantees the existence of new escape directions. We call this operation the opening of a tunnel. One may then continue with the larger network either temporarily, i.e. closing the tunnel later, or permanently, i.e. iteratively growing the network, whenever needed. We develop our method for fully-connected as well as convolutional layers. Moreover, we present a practical version of our algorithm that requires no network structure modification and can be deployed as plug-and-play into any current deep learning framework. Experimentally, our method shows significant speed-ups.
• We develop a practical version of our algorithm that requires no network structure modification and can be deployed as plug-and-play into any current deep learning framework.
• We show experimentally that our method escapes effectively from flat areas during training.
In Section 2, we provide background, motivation, and explain in detail how to open the tunnel by solving an interposed optimization problem. We discuss the fully connected case as well as the practically relevant case of convolutional layers. In Section 3, we describe how to close the tunnel, i.e. how to project back the expanded network onto one of the original dimensionality. We also propose a practical version of our idea that does not require an explicit modification of the network structure, based on a well-chosen reorganization of the weights. Finally, experiments and related work are presented in Sections 4 and 5 respectively.
This is an interesting submission and I appreciate especially connecting to a body of literature which is not normally well known in our community (e.g. Fukumizu&Amari). I think the perspective is definitely new and probably quite relevant not only for practical approaches to escape saddles but also to understand learning in deep learning.  I have a few notes and suggestions:
1) Name of the paper: 
I think is not descriptive of the approach and actually the words “magic” makes it sound strange. I think this will reduce the amount of people reading the work. Please consider something more descriptive like: “Escaping saddle points by increasing capacity”. Or something else more inspired, but that also hints what the work is about. 
2) Notation: 
The notation used is not ML friendly (or generically) to the average reader. I strongly suggest to use b_v for bias, W_uv for weights, and not theta_v and theta_uv which is not typical notation. ‘u’ and ‘v’ are somewhat non-typical choices either, though I understand that they come from the graph notation. Transfer functions are usual sigma. In the text you explain the process by starting with a u’ and then add the clone which is u. Normally you should have started with u and add the clone that is u’. x_u for the value of unit u (assuming this is in the middle of a deep net) is also quite a strange notation. I can guess the authors might be from a slightly different community, but I’m worried about people from the target audience (ICLR) being turned away from the work or even worse confused because of notations.
3) Related work
There is the Net2Net work that is related to what is going on here that is not cited (https://arxiv.org/pdf/1511.05641.pdf). I think there was some follow-up work after this.
4) Symmetry breaking
I do not understand how symmetry is broken. If I clone a unit, and have a new variant of it u’ that now has the same outbound connections but multiplied by alpha (while the original unit by 1-alpha) then while the norm of the gradients differ, their direction does not. Wouldn’t this mean that the units will track each other and hence no tunnel is open? In Net2Net dropout was used to break symmetry (i.e. a source of noise that would pick one path over the other). There is no source of noise here to break symmetry. 
5) Diagrams and analysis
Connected to this, I feel like this could have been represented clearly with a diagram showing the net before and after. There could be some analysis, a more extended discussion of where the symmetry breaking comes from, empirical evidence that it does. I’m not necessarily worried that experiments are not scaled up, I’m more concerned that the hypothesis and solution is only tested by means of change in performance. What is this tunnel doing? How does it change the Hessian at the saddle? Any visualization to reinforce the intuition of what the approach is doing? 
6) Closing tunnels & re-organizing
I don’t understand the mechanism for reorganizing weights and closing tunnels. It seems first of all to confirm my intuition that there is no symmetry breaking since we can “close” the tunnel by simple algebraic manipulation. So if those two units always stay in sync how do you actually change the error surface? How do you take advantage of this extra capacity to solve anything. Regardless, when it comes to re-organizing, it seems you pick two of these units that are in sync (previous tunnel I guess) and collapse them to open a new tunnel, right? How does this change anything? Which unit needs to be cloned? Any? So then why is the previous tunnel not efficient anymore?