Continuous Propagation: Layer-Parallel Training
Continuous propagation is a parallel technique for training deep neural networks with batch size one at full utilization of a multiprocessor system. It enables spatially distributed computations on emerging deep learning hardware accelerators that do not impose programming limitations of contemporary GPUs. The algorithm achieves model parallelism along the depth of a deep network. The method is based on the continuous representation of the optimization process and enables sustained gradient generation during all phases of computation. We demonstrate that in addition to its increased concurrency, continuous propagation improves the convergence rate of state of the art methods while matching their accuracy.
 Pros: * asynchronous model-parallel training of deep learning models would potentially help in further scaling of deep learning models * the paper is clearly written and easy to understand Cons: * weak experiments: performance of algorithms are not analyzed in terms of wall-clock, and important baselines are not compared against, making it difficult to judge the practical usefulness of the proposed algorithm * weak theory: although the algorithm is claimed to be motivated by continuous-time formulation of gradient descent, neither convergence proof nor algorithm design really use the continuous-time formulation and discrete-time formulation seems to suffice; the proof is straightforward corollary of Lin et al. Summary: This paper proposes to update parameters of each layer in deep neural network asynchronously, instead of updating all layers simultaneously and synchronously. Authors derive this algorithm by first formulating gradient descent in continuous-time form and then modifying time dependencies between layers. While asynchronous updates of parameters in stochastic gradient descent has been explored (dating back to [1] in 1986, and authors should also be referring to [2]), to my knowledge application of these ideas to layer-by-layer model parallelism for deep neural networks has not been studied. Since model-parallel training across machines has not been very successful, and model-parallelism has been only exploited within machines, asynchronous model-parallel optimization is an important topic of research which has the promise of scaling deep learning models beyond the memory capacity of a single machine. Unfortunately, the practical usefulness of the algorithm has not been demonstrated. It remains unanswered whether this algorithm can be implemented efficiently in modern hardware architectures, or in which situations this algorithm will be more useful than existing algorithms. Experiments are all reported in terms of the number of updates (epochs), but this is not useful in judging the practical advantage of the proposed algorithm. What matters in practice is how fast the algorithm is in improving the performance as a function of _wall-clock time_, and I would expect that synchronous algorithms would be much faster than the proposed algorithm in terms of wall-clock time, as they can better exploit optimized tensor arithmetic on CPUs and GPUs. Also, authors should compare against mini-batch gradient descent, because this is the most popular way of training deep neural networks; authors has the burden of proof that the proposed algorithm is practically more useful than the existing standard method. Authors argue their algorithm is motivated by continuous-time formulation of stochastic gradient descent, but it is unclear to me whether the continuous-time formulation was really necessary to derive the proposed algorithm. The algorithm operates in discrete time horizon, and continuous time is not used anywhere. Authors rely mostly on Lin et al for the convergence proof, which is also based on discrete time horizon. Authors argue in page 1 that Continuous Propagation is statistically superior to mini-batch gradient descent, but I cannot find statistical superiority of the method. Also, the upper bound of the time-delay T slows down the convergence rate (Proposition in the appendix), so it is unclear whether asynchronous update is theoretically faster than synchronous mini-batch gradient descent. I think which algorithm is faster depends on values of L, T and M. Authors do not provide enough citations. Continuous-time characterization of gradient descent has a long history, and authors should provide citation of it, for example when (5) is introduced. Authors should provide more discussion of the history of model-parallel asynchronous SGD (such as [1] and [2]), and when mentioning alternatives like Czarnecki et al (2017), authors should discuss what advantages and disadvantages the proposed algorithm has against these alternatives. [1] Distributed Asynchronous Deterministic and Stochastic Gradient Optimization Algorithms (Tsitsiklis, Bertsekas and Athans, 1986) [2] Hogwild!: A Lock-Free Approach to Parallelizing Stochastic Gradient Descent (Niu et al, 2011)
You are a member of the scientific community tasked with peer review. Review the following paper title and abstract.
### Title + Abstract
Continuous Propagation: Layer-Parallel Training
Continuous propagation is a parallel technique for training deep neural networks with batch size one at full utilization of a multiprocessor system. It enables spatially distributed computations on emerging deep learning hardware accelerators that do not impose programming limitations of contemporary GPUs. The algorithm achieves model parallelism along the depth of a deep network. The method is based on the continuous representation of the optimization process and enables sustained gradient generation during all phases of computation. We demonstrate that in addition to its increased concurrency, continuous propagation improves the convergence rate of state of the art methods while matching their accuracy.
user
<think>Okay, to do a decent peer review, I should start with summarizing what the work is about. This includes novelty, contributions, clarity, and impact. I might include suggestions to issues that I find critical to the usefulness of the work. Starting with novelty, there are different types of novelty. The first aspect of novelty is scope and scale. Is the work covering a broader perspective than previous works? The second aspect is methodology. Is the proposed methodology novel compared to previous methods? Do they provide any significant advantage over previous methodologies? The third aspect is results. Are the results different compared previous work? Does this lead to any novel theories or an extension of current theories? Next is clarity, I will be reading the work in depth and analyze if any sentences can be potentially ambiguous or misleading. If any sentences are ambiguous, I will state them and give a suggestion on how to rephrase such sentence or how to make more clear to the audience. Lastly, impact is important. I will attempt to predict how such work will impact the scientific community. This might include the applications of such work and how the work might transform the way we study a certain topic. Finally, I will might only mention issues that are critical. This is to keep the review as concise and helpful as possible.</think>



 Pros: * asynchronous model-parallel training of deep learning models would potentially help in further scaling of deep learning models * the paper is clearly written and easy to understand Cons: * weak experiments: performance of algorithms are not analyzed in terms of wall-clock, and important baselines are not compared against, making it difficult to judge the practical usefulness of the proposed algorithm * weak theory: although the algorithm is claimed to be motivated by continuous-time formulation of gradient descent, neither convergence proof nor algorithm design really use the continuous-time formulation and discrete-time formulation seems to suffice; the proof is straightforward corollary of Lin et al. Summary: This paper proposes to update parameters of each layer in deep neural network asynchronously, instead of updating all layers simultaneously and synchronously. Authors derive this algorithm by first formulating gradient descent in continuous-time form and then modifying time dependencies between layers. While asynchronous updates of parameters in stochastic gradient descent has been explored (dating back to [1] in 1986, and authors should also be referring to [2]), to my knowledge application of these ideas to layer-by-layer model parallelism for deep neural networks has not been studied. Since model-parallel training across machines has not been very successful, and model-parallelism has been only exploited within machines, asynchronous model-parallel optimization is an important topic of research which has the promise of scaling deep learning models beyond the memory capacity of a single machine. Unfortunately, the practical usefulness of the algorithm has not been demonstrated. It remains unanswered whether this algorithm can be implemented efficiently in modern hardware architectures, or in which situations this algorithm will be more useful than existing algorithms. Experiments are all reported in terms of the number of updates (epochs), but this is not useful in judging the practical advantage of the proposed algorithm. What matters in practice is how fast the algorithm is in improving the performance as a function of _wall-clock time_, and I would expect that synchronous algorithms would be much faster than the proposed algorithm in terms of wall-clock time, as they can better exploit optimized tensor arithmetic on CPUs and GPUs. Also, authors should compare against mini-batch gradient descent, because this is the most popular way of training deep neural networks; authors has the burden of proof that the proposed algorithm is practically more useful than the existing standard method. Authors argue their algorithm is motivated by continuous-time formulation of stochastic gradient descent, but it is unclear to me whether the continuous-time formulation was really necessary to derive the proposed algorithm. The algorithm operates in discrete time horizon, and continuous time is not used anywhere. Authors rely mostly on Lin et al for the convergence proof, which is also based on discrete time horizon. Authors argue in page 1 that Continuous Propagation is statistically superior to mini-batch gradient descent, but I cannot find statistical superiority of the method. Also, the upper bound of the time-delay T slows down the convergence rate (Proposition in the appendix), so it is unclear whether asynchronous update is theoretically faster than synchronous mini-batch gradient descent. I think which algorithm is faster depends on values of L, T and M. Authors do not provide enough citations. Continuous-time characterization of gradient descent has a long history, and authors should provide citation of it, for example when (5) is introduced. Authors should provide more discussion of the history of model-parallel asynchronous SGD (such as [1] and [2]), and when mentioning alternatives like Czarnecki et al (2017), authors should discuss what advantages and disadvantages the proposed algorithm has against these alternatives. [1] Distributed Asynchronous Deterministic and Stochastic Gradient Optimization Algorithms (Tsitsiklis, Bertsekas and Athans, 1986) [2] Hogwild!: A Lock-Free Approach to Parallelizing Stochastic Gradient Descent (Niu et al, 2011)
assistant
 Pros: * asynchronous model-parallel training of deep learning models would potentially help in further scaling of deep learning models * the paper is clearly written and easy to understand Cons: * weak experiments: performance of algorithms are not analyzed in terms of wall-clock, and important baselines are not compared against, making it difficult to judge the practical usefulness of the proposed algorithm * weak theory: although the algorithm is claimed to be motivated by continuous-time formulation of gradient descent, neither convergence proof nor algorithm design really use the continuous-time formulation and discrete-time formulation seems to suffice; the proof is straightforward corollary of Lin et al. Summary: This paper proposes to update parameters of each layer in deep neural network asynchronously, instead of updating all layers simultaneously and synchronously. Authors derive this algorithm by first formulating gradient descent in continuous-time form and then modifying time dependencies between layers. While asynchronous updates of parameters in stochastic gradient descent has been explored (dating back to [1] in 1986, and authors should also be referring to [2]), to my knowledge application of these ideas to layer-by-layer model parallelism for deep neural networks has not been studied. Since model-parallel training across machines has not been very successful, and model-parallelism has been only exploited within machines, asynchronous model-parallel optimization is an important topic of research which has the promise of scaling deep learning models beyond the memory capacity of a single machine. Unfortunately, the practical usefulness of the algorithm has not been demonstrated. It remains unanswered whether this algorithm can be implemented efficiently in modern hardware architectures, or in which situations this algorithm will be more useful than existing algorithms. Experiments are all reported in terms of the number of updates (epochs), but this is not useful in judging the practical advantage of the proposed algorithm. What matters in practice is how fast the algorithm is in improving the performance as a function of _wall-clock time_, and I would expect that synchronous algorithms would be much faster than the proposed algorithm in terms of wall-clock time, as they can better exploit optimized tensor arithmetic on CPUs and GPUs. Also, authors should compare against mini-batch gradient descent, because this is the most popular way of training deep neural networks; authors has the burden of proof that the proposed algorithm is practically more useful than the existing standard method. Authors argue their algorithm is motivated by continuous-time formulation of stochastic gradient descent, but it is unclear to me whether the continuous-time formulation was really necessary to derive the proposed algorithm. The algorithm operates in discrete time horizon, and continuous time is not used anywhere. Authors rely mostly on Lin et al for the convergence proof, which is also based on discrete time horizon. Authors argue in page 1 that Continuous Propagation is statistically superior to mini-batch gradient descent, but I cannot find statistical superiority of the method. Also, the upper bound of the time-delay T slows down the convergence rate (Proposition in the appendix), so it is unclear whether asynchronous update is theoretically faster than synchronous mini-batch gradient descent. I think which algorithm is faster depends on values of L, T and M. Authors do not provide enough citations. Continuous-time characterization of gradient descent has a long history, and authors should provide citation of it, for example when (5) is introduced. Authors should provide more discussion of the history of model-parallel asynchronous SGD (such as [1] and [2]), and when mentioning alternatives like Czarnecki et al (2017), authors should discuss what advantages and disadvantages the proposed algorithm has against these alternatives. [1] Distributed Asynchronous Deterministic and Stochastic Gradient Optimization Algorithms (Tsitsiklis, Bertsekas and Athans, 1986) [2] Hogwild! : A Lock-Free Approach to Parallelizing Stochastic Gradient Descent (Niu et al, 2011)