An Automatic Operation Batching Strategy for the Backward Propagation of Neural Networks Having Dynamic Computation Graphs
Organizing the same operations in the computation graph of a neural network into batches is one of the important methods to improve the speed of training deep learning models and applications since it helps to execute operations with the same type in parallel and to make full use of the available hardware resources. This batching task is usually done by the developers manually and it becomes more dif- ficult when the neural networks have dynamic computation graphs because of the input data with varying structures or the dynamic flow control. Several automatic batching strategies were proposed and integrated into some deep learning toolkits so that the programmers don’t have to be responsible for this task. These strategies, however, will miss some important opportunities to group the operations in the backward propagation of training neural networks. In this paper, we proposed a strategy which provides more efficient automatic batching and brings benefits to the memory access in the backward propagation. We also test our strategy on a variety of benchmarks with dynamic computation graphs. The result shows that it really brings further improvements in the training speed when our strategy is working with the existing automatic strategies.
 This paper proposed a just-in-time optimization method of neural network calculation on dynamic computation graphs. The method focused on batching summation of gradients on the backward calculation which was performed independently in conventional toolkits, and experiments on 3 LSTM tasks showed that in several settings the proposed method improved the speed of backward computation. The proposed method is straightforward and reasonable in terms of improving the speed of the backward computation. Authors discussed the proposed method on only the neural network toolkits with a dynamic computation strategy, but this kind of optimization can be applied to any existing toolkits even which has a non-dynamic strategy. This point looks a kind of misleading of the discussion on the paper. The paper provided a detailed analysis of time consumption on only a success-case (Table 4). Unfortunately, Table 2 and 3 showed that the proposed method does not have a global effectiveness and suggest a necessity for a further discussion about when to use the proposed method. Since this discussion can surely be conducted by comparing analyses of success and failure-cases, authors should provide analyses of all experiments. A conceivable weakness of the method may be the increase of memory consumption. If the toolkit plan to perform batch operations for summations of gradients, it needs to store all available gradients about each use of the corresponding variables. If the variable has a large shape and is used very frequently (e.g., variables in the softmax layer), the amount of total memory consumed by its gradient tends to be a serious problem. The non-batching strategy can mitigate this problem by discarding gradient information as soon as it is propagated back to all preceding nodes. The paper does not provide any information about memory consumption but it is important to discuss this kind of perspective. Others: In Table 2 and 3, please provide the ratio of speeds which are more reasonable to judge the real improvement rather than the one-zero decision (showed as up/down arrows). In Table 4, why the time of the forward propagation slightly increased? You should write a full list of authors of the DyNet paper that the official README provided: https://github.com/clab/dynet/blob/master/README.md
You are a member of the scientific community tasked with peer review. Review the following paper title and abstract.
### Title + Abstract
An Automatic Operation Batching Strategy for the Backward Propagation of Neural Networks Having Dynamic Computation Graphs
Organizing the same operations in the computation graph of a neural network into batches is one of the important methods to improve the speed of training deep learning models and applications since it helps to execute operations with the same type in parallel and to make full use of the available hardware resources. This batching task is usually done by the developers manually and it becomes more dif- ficult when the neural networks have dynamic computation graphs because of the input data with varying structures or the dynamic flow control. Several automatic batching strategies were proposed and integrated into some deep learning toolkits so that the programmers don’t have to be responsible for this task. These strategies, however, will miss some important opportunities to group the operations in the backward propagation of training neural networks. In this paper, we proposed a strategy which provides more efficient automatic batching and brings benefits to the memory access in the backward propagation. We also test our strategy on a variety of benchmarks with dynamic computation graphs. The result shows that it really brings further improvements in the training speed when our strategy is working with the existing automatic strategies.
user
<think>Okay, to do a decent peer review, I should start with summarizing what the work is about. This includes novelty, contributions, clarity, and impact. I might include suggestions to issues that I find critical to the usefulness of the work. Starting with novelty, there are different types of novelty. The first aspect of novelty is scope and scale. Is the work covering a broader perspective than previous works? The second aspect is methodology. Is the proposed methodology novel compared to previous methods? Do they provide any significant advantage over previous methodologies? The third aspect is results. Are the results different compared previous work? Does this lead to any novel theories or an extension of current theories? Next is clarity, I will be reading the work in depth and analyze if any sentences can be potentially ambiguous or misleading. If any sentences are ambiguous, I will state them and give a suggestion on how to rephrase such sentence or how to make more clear to the audience. Lastly, impact is important. I will attempt to predict how such work will impact the scientific community. This might include the applications of such work and how the work might transform the way we study a certain topic. Finally, I will might only mention issues that are critical. This is to keep the review as concise and helpful as possible.</think>



 This paper proposed a just-in-time optimization method of neural network calculation on dynamic computation graphs. The method focused on batching summation of gradients on the backward calculation which was performed independently in conventional toolkits, and experiments on 3 LSTM tasks showed that in several settings the proposed method improved the speed of backward computation. The proposed method is straightforward and reasonable in terms of improving the speed of the backward computation. Authors discussed the proposed method on only the neural network toolkits with a dynamic computation strategy, but this kind of optimization can be applied to any existing toolkits even which has a non-dynamic strategy. This point looks a kind of misleading of the discussion on the paper. The paper provided a detailed analysis of time consumption on only a success-case (Table 4). Unfortunately, Table 2 and 3 showed that the proposed method does not have a global effectiveness and suggest a necessity for a further discussion about when to use the proposed method. Since this discussion can surely be conducted by comparing analyses of success and failure-cases, authors should provide analyses of all experiments. A conceivable weakness of the method may be the increase of memory consumption. If the toolkit plan to perform batch operations for summations of gradients, it needs to store all available gradients about each use of the corresponding variables. If the variable has a large shape and is used very frequently (e.g., variables in the softmax layer), the amount of total memory consumed by its gradient tends to be a serious problem. The non-batching strategy can mitigate this problem by discarding gradient information as soon as it is propagated back to all preceding nodes. The paper does not provide any information about memory consumption but it is important to discuss this kind of perspective. Others: In Table 2 and 3, please provide the ratio of speeds which are more reasonable to judge the real improvement rather than the one-zero decision (showed as up/down arrows). In Table 4, why the time of the forward propagation slightly increased? You should write a full list of authors of the DyNet paper that the official README provided: https://github.com/clab/dynet/blob/master/README.md
assistant
 This paper proposed a just-in-time optimization method of neural network calculation on dynamic computation graphs. The method focused on batching summation of gradients on the backward calculation which was performed independently in conventional toolkits, and experiments on 3 LSTM tasks showed that in several settings the proposed method improved the speed of backward computation. The proposed method is straightforward and reasonable in terms of improving the speed of the backward computation. Authors discussed the proposed method on only the neural network toolkits with a dynamic computation strategy, but this kind of optimization can be applied to any existing toolkits even which has a non-dynamic strategy. This point looks a kind of misleading of the discussion on the paper. The paper provided a detailed analysis of time consumption on only a success-case (Table 4). Unfortunately, Table 2 and 3 showed that the proposed method does not have a global effectiveness and suggest a necessity for a further discussion about when to use the proposed method. Since this discussion can surely be conducted by comparing analyses of success and failure-cases, authors should provide analyses of all experiments. A conceivable weakness of the method may be the increase of memory consumption. If the toolkit plan to perform batch operations for summations of gradients, it needs to store all available gradients about each use of the corresponding variables. If the variable has a large shape and is used very frequently (e.g., variables in the softmax layer), the amount of total memory consumed by its gradient tends to be a serious problem. The non-batching strategy can mitigate this problem by discarding gradient information as soon as it is propagated back to all preceding nodes. The paper does not provide any information about memory consumption but it is important to discuss this kind of perspective. Others: In Table 2 and 3, please provide the ratio of speeds which are more reasonable to judge the real improvement rather than the one-zero decision (showed as up/down arrows). In Table 4, why the time of the forward propagation slightly increased? You should write a full list of authors of the DyNet paper that the official README provided: https://github.com/clab/dynet/blob/master/README.md