Designing energy-efficient networks is of critical importance for enabling stateof-the-art deep learning in mobile and edge settings where the computation and energy budgets are highly limited. Recently,  framed the search of efficient neural architectures into a continuous splitting process: it iteratively splits existing neurons into multiple off-springs to achieve progressive loss minimization, thus finding novel architectures by gradually growing the neural network. However, this method was not specifically tailored for designing energy-efficient networks, and is computationally expensive on large-scale benchmarks. In this work, we substantially improve  in two significant ways: 1) we incorporate the energy cost of splitting different neurons to better guide the splitting process, thereby discovering more energy-efficient network architectures; 2) we substantially speed up the splitting process of Wu et al. (2019), which requires expensive eigen-decomposition, by proposing a highly scalable Rayleigh-quotient stochastic gradient algorithm. Our fast algorithm allows us to reduce the computational cost of splitting to the same level of typical back-propagation updates and enables efficient implementation on GPU. Extensive empirical results show that our method can train highly accurate and energy-efficient networks on challenging datasets such as ImageNet, improving a variety of baselines, including the pruning-based methods and expert-designed architectures.
The paper is well written and flows very well. The idea is straightforward and easy to understand. Here are my feedbacks:
--Method--
Methodology-wise, the novelty is somewhat limited. The main technical contributions are: 1) formulate linear programming to reduce energy costs, and the formulation of linear programming is straightforward. 2) the claimed main contribution is an application of Rayleigh-Quotient gradient descent to approximate the eigenvalue calculations in the model proposed by Wu et al (2019). 
I believe computing the eigenvalue shall not be the only way to solve Eq.(3), e.g. using gradient-based method or Lagrange. My main question is whether computing the global optimum really matters? Since your method is still essentially an approximation algorithm, which may break the optimality condition here. From the experimental results, it seems that your approximation is quite close to the analytical solution. Therefore, it is unclear computing a global optimum really matters here.
I tried to buy the idea of escaping the saddle point with splitting (it indeed sounds straightforward and reasonable). It will be great if the author can add some empirical experiments to verify it.
--Content--
The entire section2 is at reviewing prior works, and I believe you should cut down the content here.
--Experiments--
1. Diversity of your tests: the author main uses MobileNet in testing their ideas. It seems that their method starts at MobileNet, then tries to improve it. I suggest authors adding other recent works, e.g. FBNet, MobileNetV3, into their tests to diversify the types of networks. 
2. Results variations: all figures, e.g. fig.3, fig.4, in the paper lack plotting their results variations. Since the optimization may converge to different local optimum, it is more persuasive to show their performance variations.
3. The final results are not surprising: in table.1 and table.2, as far as I'm aware, the mainstream accuracy for ImageNet under the mobile setting should be 75% top-1. And the SOTA top-1 accuracy on ImageNet is around 85.5%. Table.1 and table.2 do somewhat show the effectiveness of your method, but its significance is limited especially considering the limited novelty of methodology.
Minors:
In Fig.3 k = 6, it seems vanilla splitting is better than accuracy and parameters, except for 0.2 log higher flops. I don't think this makes a compelling case here.
In Fig.4, from flops 7 ~ 9, your results are similar to Bn especially accuracy > 0.6. When accuracy < 0.6, it is less interesting, and I believe the improvement should be huge.
Fig.5, could you please compare the time using MAGMA from NVIDIA? I had some experiences in implementing the LAPACK and BLAS on GPUs, and it should not be that slow.
Overall, this paper has some interesting results, which shows the eigenvalue can be approximated by Rayleigh-Quotient gradient descent, and show positive improvement. However, the methodological and experimental results can definitely be strengthened. The author may consider proving that achieving the global optimum on Eq.(3) really matters, then motivate the methodology. Thank you.