We propose a novel technique for improving the stochastic gradient descent (SGD) method to train deep networks, which we term PoweredSGD. The proposed PoweredSGD method simply raises the stochastic gradient to a certain power γ ∈ [0, 1] elementwise during iterations and introduces only one additional parameter, namely, the power exponent γ (when γ = 1, PoweredSGD reduces to SGD). We further propose PoweredSGD with momentum, which we term PoweredSGDM, and provide convergence rate analysis on both PoweredSGD and PoweredSGDM methods. Experiments are conducted on popular deep learning models and benchmark datasets. Empirical results show that the proposed PoweredSGD and PoweredSGDM obtain faster initial training speed than adaptive gradient methods, comparable generalization ability with SGD, and improved robustness to hyper-parameter selection and vanishing gradients. PoweredSGD is essentially a gradient modifier via a nonlinear transformation. As such, it is orthogonal and complementary to other techniques for accelerating gradient-based optimization.
This paper investigates an SGD variant (PowerSGD) where the stochastic gradient is raised to a power of $\gamma \in [0,1]$.  The authors introduce PowerSGD and PowerSGD with momentum (PowerSGDM). The theoretical proof of  the convergence is given and experimental results show that the proposed algorithm converges faster than some of the existing popular adaptive SGD techniques.  Intuitively, the proposed PowerSGD can boost the gradient (since $\gamma \in [0,1]$) so it may be helpful for the gradient of the lower layers of a deep network which may be hit by the vanishing gradient issue. This may give rise to a faster convergence.   So overall the idea makes sense but I have the following concerns. 
1. The major issue I have with this paper is Theorem 3.1 on the ergodic convergence rate of the proposed PowerSGD.  At the first glance, it is $O(\frac{1}{T})$ which is faster than the conventional SGD convergence rate $O(\frac{1}{\sqrt{T}})$.  But after a closer look, this rate is obtained by a very strong assumption on the batch size $B_{t}=T$.  In other words, when the number of iterations is large, the batch size will be large too.  I consider this assumption unrealistic.  Given that $T$ is typically very large (it is iterations, not epochs),  it will require a huge batch size, probably close to the whole training set. In this case, it is basically a GD, not SGD any more. That's why the rate is $O(\frac{1}{T})$, which is the convergence rate of GD.   I would like to see a convergence proof where the batch size $B_{t}$ is treated as a small constant like other SGD proofs assume.  Actually in the experiments the authors never use an increasing batch size. Instead, a constant batch size 128 is used. Therefore,  the faster convergence demonstrated in the experiments can not be explained by Theorem 3.1 or Theorem 3.2. 
2. There are numerous inaccuracies in the proof given the supplementary material.  For instance, in Eq.7,  $\nabla f(x) \sigma(\nabla f(x))$  should be $\nabla f(x)^{T} \sigma(\nabla f(x))$   The random variable $\xi_{t}$ should be a scalar on training samples, not a vector, etc..  The authors should clean it up. 
3. It would be helpful to show the $\gamma$ value on each experiment with different tasks. It would be good to know how $\gamma$ varies across tasks. 
4. I think in the comparative experiments, the plain SGD should be added as another reference algorithm. 
5. The term "PowerSGD" seems to have been used by other papers.