In recent years, stochastic gradient descent (SGD) and its variants have been the dominant optimization methods for training deep neural networks. However, SGD suffers from limitations such as the lack of theoretical guarantees, vanishing gradients, excessive sensitivity to input, and difficulties solving highly non-smooth constraints and functions. To overcome these drawbacks, alternating minimizationbased methods for deep neural network optimization have attracted fast-increasing attention recently. As an emerging and open domain, however, several new challenges need to be addressed, including 1) Convergence depending on the choice of hyperparameters, and 2) Lack of unified theoretical frameworks with general conditions. We, therefore, propose a novel Deep Learning Alternating Minimization (DLAM) algorithm to deal with these two challenges. Our innovative inequality-constrained formulation infinitely approximates the original problem with non-convex equality constraints, enabling our proof of global convergence of the DLAM algorithm under mild, practical conditions, regardless of the choice of hyperparameters and wide range of various activation functions. Experiments on benchmark datasets demonstrate the effectiveness of DLAM.
The motivation for this paper is as follows: 
Why SGD:
1. Easy
2. applied in online settings 
However:
1. Convergence proof has been done by others, but the assumptions of their proofs cannot be applied to problems involving deep neural networks, which are highly nonsmooth and nonconvex
2. Suffer from gradient vanish
3. sensitive to the input
The paper is based on new ideas  on alternative minimization method. In particular the  loss function is reformulated as nested function associated with multiple linear and nonlinear transformations. This nested structure is then decomposed into a series of linear and nonlinear equality constraints by introducing auxiliary variables and penalty hyperparameters. Then  multiple subproblems are generated which can be minimized alternatively. (using ADMM, BCD)
However, these methods suffer from problems:
1. Convergence properties are sensitive to penalty parameters.
2. Lack of unified theoretical frameworks with general conditions. (Not fully proved the
convergence. Most of the work is based on some assumptions)
Main Assumption: Activation functions are quasilinear functions. 
pros (+):
1. Generic, easy to extend it to convolutional layers and recurrent layers.
2. Transform the Neural network optimization problem into inequality constrained
problem (new point of view)
3. Ensure convexity of subproblems. (quadractic approximation of activation function).
4. Ensure global minima, and converges to the critical point (where 0 is in the set of
partial differential of F(F(W,b,,z,a) is the augment loss function) with respect to W*,
and set of partial differential of F with respect to b*).
5. The convergence rate is O(1/k), k is the number of iterations of updating (W,b,z,a).
6. inequality-constraint prevents the output of a nonlinear function from changing much
and reduces sensitivity to the input.
7. Matrix inversion is avoided by quadratic approximation
8. No need strict and complex condition, such as KL properties to prove convergence.
Instead, this algorithm needs simple and mild conditions to guarantee convergence,
and cover most of the loss function and activation functions.
9. The choice of hyperparameters has no effect on convergence.
10. Much better performance on MNIST
cons(-):
1. penalty parameters need to be tuned as ADMM does.
2. The regularizations $\omega$ is l1 and l2, otherwise, it will not be closed-form
solutions. The regularization like dropout, batch-norm can not be added into this algorithm.
3. The dataset is relatively naive. Only MNIST and Fashion MNIST. It would be better to include little more sophisticate dataset like ImageNet or Cifar10. Since the computation complexity is O(d^2) where d is the dimension of the features, large scale of image like ImageNet could cause problem. (Comparison: features of MNIST: 196, features of Fashion MNIST:784, features of ImageNet: 65536).
4. The Adagrad has better performance than DLAM on 500*500 on Fashion MNIST and close performance on MNIST. As the authors suggest: DLAM performed competitively for the Fashion MNIST dataset. This can be interpreted as follows:  the proposed algorithm performs well on easy datasets, however the performance is not much better than other algorithms  on Fashion MNIST, which is little more challenging. What would be your comment about this criticism? Can you provide any theoretical insight?  Certainly  it would strengthen the paper if there were more comparisons of  DLAM on more complex dataset.