Mini-batch stochastic gradient methods (SGD) are state of the art for distributed training of deep neural networks. Drastic increases in the mini-batch sizes have lead to key efficiency and scalability gains in recent years. However, progress faces a major roadblock, as models trained with large batches often do not generalize well, i.e. they do not show good accuracy on new data. As a remedy, we propose a post-local SGD and show that it significantly improves the generalization performance compared to large-batch training on standard benchmarks while enjoying the same efficiency (time-to-accuracy) and scalability. We further provide an extensive study of the communication efficiency vs. performance trade-offs associated with a host of local SGD variants.
This paper proposes a new distributed computation technique for SGD training of deep neural networks. The proposed method is a modification of the local SGD which updates models distributed to several workers in a parallel way and synchronize the model parameters at every few epochs. The local SGD shows a nice performance but it is not robust against large mini-batch size. The proposed method is called post-local SGD that starts the local SGD after some epochs of standard mini-batch SGD. This modification makes the local SGD robust against the large mini-batch size. The authors conducted thorough experiments to investigate the performance of the proposed method. The experiments reveal that the proposed method gives better performances than the mini-batch SGD and the vanilla local SGD.
Pros:
- As far as I checked, the numerical experiments are strong. They checked several points of view. Several settings of mini-batch sizes and number of workers are compared. For more detailed points, they compared the proposed method for not only the vanilla SGD but also other optimizers such as momentum SGD. Moreover, different choices of timing of starting local SGD (t') are compared. 
- The proposed method is simple and easy to implement.
Cons:
- The local SGD itself is not new and has been already proposed (indeed, the authors are also explaining this point in the paper), and theoretical investigation of local SGD has also been well exploited. This method is just a combination of normal mini-batch SGD and local SGD. In that sense, there are not a so much eye-opening idea in the proposed method.
- SGLD interpretation is instructive, but it does not explain why the "post"-local SGD outperforms the local SGD. We can intuitively guess why post-local SGD is better but it is not a rigorous theory. 
- The post local SGD algorithm has more freedom than existing method, which results in more tuning parameters. There are parameters such as the batch-size before and after switching, the number of workers, the timing of switching and so on. It could be disadvantage of the proposed method.
Minor comment:
- Please give the precise meaning of the dominant eigenvalue.
- I think the setting that the data are distributed to all workers is acceptable. It is also an interesting setting from the HPC point of view (on the other hand, it would be less interesting in terms of federated learning).