Pipe-SGD: A Decentralized Pipelined SGD Framework for Distributed Deep Net Training
Distributed training of deep nets is an important technique to address some of the present day computing challenges like memory consumption and computational demands. Classical distributed approaches, synchronous or asynchronous, are based on the parameter server architecture, i.e., worker nodes compute gradients which are communicated to the parameter server while updated parameters are returned. Recently, distributed training with AllReduce operations gained popularity as well. While many of those operations seem appealing, little is reported about wall-clock training time improvements. In this paper, we carefully analyze the AllReduce based setup, propose timing models which include network latency, bandwidth, cluster size and compute time, and demonstrate that a pipelined training with a width of two combines the best of both synchronous and asynchronous training. Specifically, for a setup consisting of a four-node GPU cluster we show wall-clock time training improvements of up to 5.4× compared to conventional approaches.
This paper proposed a pipelined training setup for neural nets. The pipeline structure allows computation and communication to be carried out concurrently within each worker.     Experiments show speed improvements over existing methods. Overall I think the method and the evaluations are convincing. However, I believe the NIPS conference is not the appropriate venue for this paper for the following reasons.    This paper deals with a high performance computing problem, namely, how to get more throughput (computation) in a distributed network. This is a low level computer systems problem that does not involve machine learning, statistics or neural science.    As much as it is designed for deep neural nets, the nature of the approach is not really about improving SGD itself or neural nets architecture or coming up with a new loss.     In summary, I do believe this work has the potential of changing the basic library for distributed neural nets training because of its improved performance. I just feel that this paper would fit better to other high performance computing conferences, like many of the cited related work.