Quasi-Recurrent Neural Networks
Recurrent neural networks are a powerful tool for modeling sequential data, but the dependence of each timestep’s computation on the previous timestep’s output limits parallelism and makes RNNs unwieldy for very long sequences. We introduce quasi-recurrent neural networks (QRNNs), an approach to neural sequence modeling that alternates convolutional layers, which apply in parallel across timesteps, and a minimalist recurrent pooling function that applies in parallel across channels. Despite lacking trainable recurrent layers, stacked QRNNs have better predictive accuracy than stacked LSTMs of the same hidden size. Due to their increased parallelism, they are up to 16 times faster at train and test time. Experiments on language modeling, sentiment classification, and character-level neural machine translation demonstrate these advantages and underline the viability of QRNNs as a basic building block for a variety of sequence tasks.
This paper points out that you can take an LSTM and make the gates only a function of the last few inputs - h_t = f(x_t, x_{t-1}, ...x_{t-T}) - instead of the standard - h_t = f(x_t, h_{t-1}) -, and that if you do so the networks can run faster and work better. You*re moving compute from a serial stream to a parallel stream and also making the serial stream more parallel. Unfortunately, this simple, effective and interesting concept is somewhat obscured by confusing language. - I would encourage the authors to improve the explanation of the model. - Another improvement might be to explicitly go over some of the big Oh calculations, or give an example of exactly where the speed improvements are coming from. - Otherwise the experiments seem adequate and I enjoyed this paper. This could be a high value contribution and become a standard neural network component if it can be replicated and if it turns out to work reliably in multiple settings.
You are a member of the scientific community tasked with peer review. Review the following paper title and abstract.
### Title + Abstract
Quasi-Recurrent Neural Networks
Recurrent neural networks are a powerful tool for modeling sequential data, but the dependence of each timestep’s computation on the previous timestep’s output limits parallelism and makes RNNs unwieldy for very long sequences. We introduce quasi-recurrent neural networks (QRNNs), an approach to neural sequence modeling that alternates convolutional layers, which apply in parallel across timesteps, and a minimalist recurrent pooling function that applies in parallel across channels. Despite lacking trainable recurrent layers, stacked QRNNs have better predictive accuracy than stacked LSTMs of the same hidden size. Due to their increased parallelism, they are up to 16 times faster at train and test time. Experiments on language modeling, sentiment classification, and character-level neural machine translation demonstrate these advantages and underline the viability of QRNNs as a basic building block for a variety of sequence tasks.
user
<think>Okay, to do a decent peer review, I should start with summarizing what the work is about. This includes novelty, contributions, clarity, and impact. I might include suggestions to issues that I find critical to the usefulness of the work. Starting with novelty, there are different types of novelty. The first aspect of novelty is scope and scale. Is the work covering a broader perspective than previous works? The second aspect is methodology. Is the proposed methodology novel compared to previous methods? Do they provide any significant advantage over previous methodologies? The third aspect is results. Are the results different compared previous work? Does this lead to any novel theories or an extension of current theories? Next is clarity, I will be reading the work in depth and analyze if any sentences can be potentially ambiguous or misleading. If any sentences are ambiguous, I will state them and give a suggestion on how to rephrase such sentence or how to make more clear to the audience. Lastly, impact is important. I will attempt to predict how such work will impact the scientific community. This might include the applications of such work and how the work might transform the way we study a certain topic. Finally, I will might only mention issues that are critical. This is to keep the review as concise and helpful as possible.</think>



This paper points out that you can take an LSTM and make the gates only a function of the last few inputs - h_t = f(x_t, x_{t-1}, ...x_{t-T}) - instead of the standard - h_t = f(x_t, h_{t-1}) -, and that if you do so the networks can run faster and work better. You*re moving compute from a serial stream to a parallel stream and also making the serial stream more parallel. Unfortunately, this simple, effective and interesting concept is somewhat obscured by confusing language. - I would encourage the authors to improve the explanation of the model. - Another improvement might be to explicitly go over some of the big Oh calculations, or give an example of exactly where the speed improvements are coming from. - Otherwise the experiments seem adequate and I enjoyed this paper. This could be a high value contribution and become a standard neural network component if it can be replicated and if it turns out to work reliably in multiple settings.
assistant
This paper points out that you can take an LSTM and make the gates only a function of the last few inputs - h_t = f(x_t, x_{t-1}, ...x_{t-T}) - instead of the standard - h_t = f(x_t, h_{t-1}) -, and that if you do so the networks can run faster and work better. You*re moving compute from a serial stream to a parallel stream and also making the serial stream more parallel. Unfortunately, this simple, effective and interesting concept is somewhat obscured by confusing language. - I would encourage the authors to improve the explanation of the model. - Another improvement might be to explicitly go over some of the big Oh calculations, or give an example of exactly where the speed improvements are coming from. - Otherwise the experiments seem adequate and I enjoyed this paper. This could be a high value contribution and become a standard neural network component if it can be replicated and if it turns out to work reliably in multiple settings.