We present a Homotopy Training Algorithm (HTA) to solve optimization problems arising from neural networks. The HTA starts with several decoupled systems with low dimensional structure and tracks the solution to the high dimensional coupled system. The decoupled systems are easy to solve due to the low dimensionality but can be connected to the original system via a continuous homotopy path guided by the HTA. We have proved the convergence of HTA for the non-convex case and existence of the homotopy solution path for the convex case. The HTA has provided a better accuracy on several examples including VGG models on CIFAR-10. Moreover, the HTA would be combined with the dropout technique to provide an alternative way to train the neural networks.
Summary
This paper proposes an algorithm to address the issue of nonlinear optimization
in high dimensions and applies it to convolutional neural networks (VGG models) on CIFAR 10. 
They show 11% relative reduction in error for this particular task with this
particular network. In addition, they prove additional theoretical results on
the convergence of SGD using their method in the convex case as well as
convergence of SGD to a stationary point in the nonconvex case when the homotopy
parameter is fixed which is not done in practice.
Given an optimization problem, their method first solves multiple independent
lower-dimensional optimization problems each with a subset of the parameters and
then optimizes a new objective function controlled by a monotonically decreasing
parameter L that interpolates the original objective function and the
previously-solved lower dimensional problems. L can be seen as a regularization
parameter that is gradually decreased as we optimize the new optimization
function. When L = 0, we recover the original optimization problem.
The authors prove that (1) SGD with their procedure will find a stationary point
under the Robbins-Monro conditions for a fixed L and (2) SGD with their
procedure will converge for convex problems as L is decreased to 0.
Decision and reasoning
This paper should be rejected because (1) the proposed algorithm attempts to address 
the original issue of high dimensional nonlinear optimization of neural networks but
violates the algorithm's assumption in practice, (2) the
empirical evaluations are lacking - having only evaluated their method on a toy
problem with up to only 6 dimensions and a relatively simple image classification task,
and (3) the assumption of fixing the homotopy parameter in the theorem on the
non-convex case directly violates the intention of the algorithm. 
Regarding (1): The proposed procedure requires initializing L at a large value
and reducing L towards 0 in order to recover the original optimization problem.
However, in practice for CIFAR 10, the authors initialize L to be 0.01 and
gradually reduces it to 0.005 which is hardly the original intent of the
algorithm. There is also no demonstration whether or not this gradual reduction in
L actually has an effect on the optimization of the new objective function. For
example, since the start and end values of L are similar, will we get similar
results if we simply fix L to be 0.005 or 0.01? The authors also show that their
method outperforms a quasi-newton method by combining the optimization with
their procedure on a non-convex example by Chow et al. 2013. However, this example
only goes up to n=6 dimensions, which is hardly comparable to the original
problem of high dimensional non-convex optimization that this paper sought to
address.
Regarding (2): The authors evaluated their procedure on CIFAR10, a relatively
simple image classification task that modern neural networks can solve easily
and is not representative of the types of nonlinear optimization problems
prevalent in deep learning. There's also an issue of using only VGG networks for
their evaluations while VGGs are typically eschewed in favor of ResNets today.
Given that the optimization is easier with residual connections, it may be the case that
their procedure does not significantly improve the accuracy of ResNets.
Regarding (3): By fixing L in Theorem 3.1, the authors essentially show that SGD
converges to a stationary point for their new objective function which can be
seen as a regularized version of the original objective function, which is not a
strong result. Furthermore, fixing L goes against the original procedure's
motivation of recovering the original optimization function as L decreases to 0.
Additional comments and questions
There are passages that are difficult to understand because not enough context
is given. For example in the "remark" passage, it is not clear where the
"necessary condition" comes from. In addition it seems like it doesn't even
type-check since the first term is 2n dimensional while the second term is 4n
dimensional. 
There are also many errors in the writing that hinder the presentation. A subset of
them includes:
- "nerual netowrks on roboticsKonda et al." -> "neural networks on robotics Konda et al."
- "based on homotopy continuation method" -> "based on the homotopy continuation methods"
- "random chosen point" -> "randomly chosen point"
- "we choose \tilde{\theta} = 0 in the dropout" -> reword
- Fourth term in Equation 3 should be \theta_2 - \tilde{\theta_2}
- "By gradually increasing parameter L" -> "By gradually decreasing parameter L"
- "where \xi is a random variable due to random algorithms" -> reword and possibly say the randomness is from SGD
- After equation 6, should have b_i instead of \beta_i
- In equation 20, should be g(\theta_*^0) instead of g(\theta_*^1)
- In theorem 3.2 you never explained what \theta_*^{L_k} is
- "We compared the traditional optimization method (the quasi-Newton method)" -> which quasi-Newton method?
- Figures 2 and 3 label the x-axis with "epochs". However only 4 epochs were run, so I believe the x-axis should be "iterations""
Besides improving the quality of writing in the paper, I would strongly suggest that the
authors improve their empirical evaluation.  Possibilities include evaluating on
CIFAR 100 or ImageNet, using a wider variety of networks including ResNets,
evaluating on tasks other than image classification.