Training Long Short-Term Memory With Sparsified Stochastic Gradient Descent
Prior work has demonstrated that exploiting the sparsity can dramatically improve the energy efficiency and shrink the memory footprint of Convolutional Neural Networks (CNNs). However, these sparsity-centric optimization techniques might be less effective for Long Short-Term Memory (LSTM) based Recurrent Neural Networks (RNNs), especially for the training phase, because of the significant structural difference between the neurons. To investigate if there is possible sparsity-centric optimization for training LSTM-based RNNs, we studied several applications and observed that there is potential sparsity in the gradients generated in the backward propagation. In this paper, we illustrate why the sparsity exists and propose a simple yet effective thresholding technique to induce further more sparsity during training an LSTM-based RNN training. Experiment results show that the proposed technique can increase the sparsity of linear gate gradients to higher than 80\% without loss of performance, which makes more than 50\% multiply-accumulate (MAC) operations redundant in an entire LSTM training process. These redudant MAC operations can be eliminated by hardware techniques to improve the energy efficiency and training speed of LSTM-based RNNs.
CONTRIBUTIONS When training LSTMs, many of the intermediate gradients are close to zero due to the flat shape of the tanh and sigmoid nonlinearities far from the origin. This paper shows that rounding these small gradients to zero results in matrices with up to 80% sparsity during training, and that training character-level LSTM language models with this sparsification does not significantly change the final performance of the model. The authors argue that this sparsity could be exploited with specialized hardware to improve the energy efficiency and speed of recurrent network training. NOVELTY Thresholding gradients to induce sparsity and improve efficiency in RNN training is a novel result to my knowledge. MISSING CITATIONS Prior work has explored low-precision arithmetic for recurrent neural network language models: Hubara et al, “Quantized Neural Networks: Training Neural Networks with Low Precision Weights and Activations”, https://arxiv.org/abs/1609.07061 Ott et al, “Recurrent Neural Networks With Limited Numerical Precision”, https://arxiv.org/abs/1608.06902 Low-precision arithmetic for recurrent networks promises to improve both training and inference efficiency. How does the proposed sparsification method compare with low-precision arithmetic? Are the ideas complementary? EXPERIMENTS The main experimental result of the paper (Section 4.1) is that training LSTM language models with sparse gradients does not affect convergence or final performance (Figure 5). This result is promising, but I do not think that this single experiment is enough to prove the utility of the proposed method. I also have some problems with this experiment. Plotting validation loss for character-level language modeling is not a standard way to report results; it is much more typical to report bits-per-character or perplexity on a held-out test set. These experiments also lack sufficient details for replication. What optimization algorithm, learning rate, and regularization were used? How were these hyperparameters chosen? Is the “truncated Wikipedia dataset” used for training the standard text8 dataset? In addition, the experiments do not compare with existing published results on this dataset. In the OpenReview discussion, the authors remarked that the “The final validation loss for the sparsified model is [...] almost the same as the baseline.” Comparing validation loss at the end of training is not the proper way to compare models. From Figure 5, it is clear that all models achieve minimal validation loss after around 10k iterations, after which the validation losses increase, suggesting that the models have slightly overfit the training data by the end of training. In Section 4.2 the authors claim to obtain similar experimental results with other network architectures, on other datasets (tiny-shakespeare and War and Peace), and for other tasks (image captioning and machine translation). However, the details and results of these experiments are not included in the paper, making it difficult to assess the utility of the proposed method and the significance of the results. ENERGY EFFICIENCY AND TRAINING SPEED One of the main claims of the paper is that sparse gradients can be exploited in hardware to reduce the training speed and improve the energy efficiency of recurrent network training, but these benefits are neither quantified nor demonstrated experimentally. Even without actually implementing custom hardware, would it be possible to estimate the expected improvements in efficiency through simulation or other means? Such results would significantly strengthen the paper. GRADIENT TRUNCATION AS REGULARIZATION In Figure 5 all models appear to reach a minimum validation loss at around 10k iterations and then overfit; at this point the Low model achieves even lower loss than the baseline. This is an interesting experimental result, but it is not discussed in the paper. Perhaps a low truncation threshold acts as a weak regularizer to prevent overfitting? Is this a general phenomenon of training recurrent networks with sparse gradients, or is it just a quirk of this particular experiment? This idea deserves more investigation, and could strengthen the paper. SUMMARY The core idea of the paper (thresholding gradients to induce sparsity and improve efficiency of RNN training) is interesting and practically useful, if a bit incremental. Nevertheless with thorough and deliberate experiments quantifying the tradeoffs between task performance, training speed, and energy efficiency across a variety of tasks and datasets, this simple idea could be the core of a strong paper. Unfortunately, as written the paper provides neither theoretical arguments nor convincing experimental results to justify the proposed method, and as such I do not believe the paper is ready for publication in its current form. PROS - The proposed method is simple, and seems to be a promising direction for improving the speed of training recurrent networks. CONS - No discussion of prior work on low-precision recurrent networks - Experimental results are not sufficient to validate the method - Many experimental details are missing - Results of key experiments (image captioning and machine translation) are missing
You are a member of the scientific community tasked with peer review. Review the following paper title and abstract.
### Title + Abstract
Training Long Short-Term Memory With Sparsified Stochastic Gradient Descent
Prior work has demonstrated that exploiting the sparsity can dramatically improve the energy efficiency and shrink the memory footprint of Convolutional Neural Networks (CNNs). However, these sparsity-centric optimization techniques might be less effective for Long Short-Term Memory (LSTM) based Recurrent Neural Networks (RNNs), especially for the training phase, because of the significant structural difference between the neurons. To investigate if there is possible sparsity-centric optimization for training LSTM-based RNNs, we studied several applications and observed that there is potential sparsity in the gradients generated in the backward propagation. In this paper, we illustrate why the sparsity exists and propose a simple yet effective thresholding technique to induce further more sparsity during training an LSTM-based RNN training. Experiment results show that the proposed technique can increase the sparsity of linear gate gradients to higher than 80\% without loss of performance, which makes more than 50\% multiply-accumulate (MAC) operations redundant in an entire LSTM training process. These redudant MAC operations can be eliminated by hardware techniques to improve the energy efficiency and training speed of LSTM-based RNNs.
user
<think>Okay, to do a decent peer review, I should start with summarizing what the work is about. This includes novelty, contributions, clarity, and impact. I might include suggestions to issues that I find critical to the usefulness of the work. Starting with novelty, there are different types of novelty. The first aspect of novelty is scope and scale. Is the work covering a broader perspective than previous works? The second aspect is methodology. Is the proposed methodology novel compared to previous methods? Do they provide any significant advantage over previous methodologies? The third aspect is results. Are the results different compared previous work? Does this lead to any novel theories or an extension of current theories? Next is clarity, I will be reading the work in depth and analyze if any sentences can be potentially ambiguous or misleading. If any sentences are ambiguous, I will state them and give a suggestion on how to rephrase such sentence or how to make more clear to the audience. Lastly, impact is important. I will attempt to predict how such work will impact the scientific community. This might include the applications of such work and how the work might transform the way we study a certain topic. Finally, I will might only mention issues that are critical. This is to keep the review as concise and helpful as possible.</think>



CONTRIBUTIONS When training LSTMs, many of the intermediate gradients are close to zero due to the flat shape of the tanh and sigmoid nonlinearities far from the origin. This paper shows that rounding these small gradients to zero results in matrices with up to 80% sparsity during training, and that training character-level LSTM language models with this sparsification does not significantly change the final performance of the model. The authors argue that this sparsity could be exploited with specialized hardware to improve the energy efficiency and speed of recurrent network training. NOVELTY Thresholding gradients to induce sparsity and improve efficiency in RNN training is a novel result to my knowledge. MISSING CITATIONS Prior work has explored low-precision arithmetic for recurrent neural network language models: Hubara et al, “Quantized Neural Networks: Training Neural Networks with Low Precision Weights and Activations”, https://arxiv.org/abs/1609.07061 Ott et al, “Recurrent Neural Networks With Limited Numerical Precision”, https://arxiv.org/abs/1608.06902 Low-precision arithmetic for recurrent networks promises to improve both training and inference efficiency. How does the proposed sparsification method compare with low-precision arithmetic? Are the ideas complementary? EXPERIMENTS The main experimental result of the paper (Section 4.1) is that training LSTM language models with sparse gradients does not affect convergence or final performance (Figure 5). This result is promising, but I do not think that this single experiment is enough to prove the utility of the proposed method. I also have some problems with this experiment. Plotting validation loss for character-level language modeling is not a standard way to report results; it is much more typical to report bits-per-character or perplexity on a held-out test set. These experiments also lack sufficient details for replication. What optimization algorithm, learning rate, and regularization were used? How were these hyperparameters chosen? Is the “truncated Wikipedia dataset” used for training the standard text8 dataset? In addition, the experiments do not compare with existing published results on this dataset. In the OpenReview discussion, the authors remarked that the “The final validation loss for the sparsified model is [...] almost the same as the baseline.” Comparing validation loss at the end of training is not the proper way to compare models. From Figure 5, it is clear that all models achieve minimal validation loss after around 10k iterations, after which the validation losses increase, suggesting that the models have slightly overfit the training data by the end of training. In Section 4.2 the authors claim to obtain similar experimental results with other network architectures, on other datasets (tiny-shakespeare and War and Peace), and for other tasks (image captioning and machine translation). However, the details and results of these experiments are not included in the paper, making it difficult to assess the utility of the proposed method and the significance of the results. ENERGY EFFICIENCY AND TRAINING SPEED One of the main claims of the paper is that sparse gradients can be exploited in hardware to reduce the training speed and improve the energy efficiency of recurrent network training, but these benefits are neither quantified nor demonstrated experimentally. Even without actually implementing custom hardware, would it be possible to estimate the expected improvements in efficiency through simulation or other means? Such results would significantly strengthen the paper. GRADIENT TRUNCATION AS REGULARIZATION In Figure 5 all models appear to reach a minimum validation loss at around 10k iterations and then overfit; at this point the Low model achieves even lower loss than the baseline. This is an interesting experimental result, but it is not discussed in the paper. Perhaps a low truncation threshold acts as a weak regularizer to prevent overfitting? Is this a general phenomenon of training recurrent networks with sparse gradients, or is it just a quirk of this particular experiment? This idea deserves more investigation, and could strengthen the paper. SUMMARY The core idea of the paper (thresholding gradients to induce sparsity and improve efficiency of RNN training) is interesting and practically useful, if a bit incremental. Nevertheless with thorough and deliberate experiments quantifying the tradeoffs between task performance, training speed, and energy efficiency across a variety of tasks and datasets, this simple idea could be the core of a strong paper. Unfortunately, as written the paper provides neither theoretical arguments nor convincing experimental results to justify the proposed method, and as such I do not believe the paper is ready for publication in its current form. PROS - The proposed method is simple, and seems to be a promising direction for improving the speed of training recurrent networks. CONS - No discussion of prior work on low-precision recurrent networks - Experimental results are not sufficient to validate the method - Many experimental details are missing - Results of key experiments (image captioning and machine translation) are missing
assistant
CONTRIBUTIONS When training LSTMs, many of the intermediate gradients are close to zero due to the flat shape of the tanh and sigmoid nonlinearities far from the origin. This paper shows that rounding these small gradients to zero results in matrices with up to 80% sparsity during training, and that training character-level LSTM language models with this sparsification does not significantly change the final performance of the model. The authors argue that this sparsity could be exploited with specialized hardware to improve the energy efficiency and speed of recurrent network training. NOVELTY Thresholding gradients to induce sparsity and improve efficiency in RNN training is a novel result to my knowledge. MISSING CITATIONS Prior work has explored low-precision arithmetic for recurrent neural network language models: Hubara et al, “Quantized Neural Networks: Training Neural Networks with Low Precision Weights and Activations”, https://arxiv.org/abs/1609.07061 Ott et al, “Recurrent Neural Networks With Limited Numerical Precision”, https://arxiv.org/abs/1608.06902 Low-precision arithmetic for recurrent networks promises to improve both training and inference efficiency. How does the proposed sparsification method compare with low-precision arithmetic? Are the ideas complementary? EXPERIMENTS The main experimental result of the paper (Section 4.1) is that training LSTM language models with sparse gradients does not affect convergence or final performance (Figure 5). This result is promising, but I do not think that this single experiment is enough to prove the utility of the proposed method. I also have some problems with this experiment. Plotting validation loss for character-level language modeling is not a standard way to report results; it is much more typical to report bits-per-character or perplexity on a held-out test set. These experiments also lack sufficient details for replication. What optimization algorithm, learning rate, and regularization were used? How were these hyperparameters chosen? Is the “truncated Wikipedia dataset” used for training the standard text8 dataset? In addition, the experiments do not compare with existing published results on this dataset. In the OpenReview discussion, the authors remarked that the “The final validation loss for the sparsified model is [...] almost the same as the baseline.” Comparing validation loss at the end of training is not the proper way to compare models. From Figure 5, it is clear that all models achieve minimal validation loss after around 10k iterations, after which the validation losses increase, suggesting that the models have slightly overfit the training data by the end of training. In Section 4.2 the authors claim to obtain similar experimental results with other network architectures, on other datasets (tiny-shakespeare and War and Peace), and for other tasks (image captioning and machine translation). However, the details and results of these experiments are not included in the paper, making it difficult to assess the utility of the proposed method and the significance of the results. ENERGY EFFICIENCY AND TRAINING SPEED One of the main claims of the paper is that sparse gradients can be exploited in hardware to reduce the training speed and improve the energy efficiency of recurrent network training, but these benefits are neither quantified nor demonstrated experimentally. Even without actually implementing custom hardware, would it be possible to estimate the expected improvements in efficiency through simulation or other means? Such results would significantly strengthen the paper. GRADIENT TRUNCATION AS REGULARIZATION In Figure 5 all models appear to reach a minimum validation loss at around 10k iterations and then overfit; at this point the Low model achieves even lower loss than the baseline. This is an interesting experimental result, but it is not discussed in the paper. Perhaps a low truncation threshold acts as a weak regularizer to prevent overfitting? Is this a general phenomenon of training recurrent networks with sparse gradients, or is it just a quirk of this particular experiment? This idea deserves more investigation, and could strengthen the paper. SUMMARY The core idea of the paper (thresholding gradients to induce sparsity and improve efficiency of RNN training) is interesting and practically useful, if a bit incremental. Nevertheless with thorough and deliberate experiments quantifying the tradeoffs between task performance, training speed, and energy efficiency across a variety of tasks and datasets, this simple idea could be the core of a strong paper. Unfortunately, as written the paper provides neither theoretical arguments nor convincing experimental results to justify the proposed method, and as such I do not believe the paper is ready for publication in its current form. PROS - The proposed method is simple, and seems to be a promising direction for improving the speed of training recurrent networks. CONS - No discussion of prior work on low-precision recurrent networks - Experimental results are not sufficient to validate the method - Many experimental details are missing - Results of key experiments (image captioning and machine translation) are missing