Surprisal-Driven Feedback in Recurrent Networks
Recurrent neural nets are widely used for predicting temporal data. Their inherent deep feedforward structure allows learning complex sequential patterns. It is believed that top-down feedback might be an important missing ingredient which in theory could help disambiguate similar patterns depending on broader context. In this paper, we introduce surprisal-driven recurrent networks, which take into account past error information when making new predictions. This is achieved by continuously monitoring the discrepancy between most recent predictions and the actual observations. Furthermore, we show that it outperforms other stochastic and fully deterministic approaches on enwik8 character level prediction task achieving 1.37 BPC.
This paper proposes to use previous error signal of the output layer as an additional input to recurrent update function in order to enhance the modelling power of a dynamic system such as RNNs. -This paper makes an erroneous assumption: test label information is not given in most of the real world applications, except few applications. This means that the language modelling task, which is the only experiment of this paper, may not be the right task to test this approach. Also, comparing against the models that do not use test error signal at inference time is unfair. We cannot just say that the test label information is being observed, this only holds in online-prediction problems. -The experiment is only conducted on one dataset, reporting state-of-the-art result, but unfortunately this is not true. There are already more than four papers reporting better numbers than the one reported in this task, however the author did not cite them. I understand that this paper came before the other papers, but the manuscript should be updated before the final decision. -The model size is still missing and without this information, it is hard to judge the contribution of the proposed trick.
You are a member of the scientific community tasked with peer review. Review the following paper title and abstract.
### Title + Abstract
Surprisal-Driven Feedback in Recurrent Networks
Recurrent neural nets are widely used for predicting temporal data. Their inherent deep feedforward structure allows learning complex sequential patterns. It is believed that top-down feedback might be an important missing ingredient which in theory could help disambiguate similar patterns depending on broader context. In this paper, we introduce surprisal-driven recurrent networks, which take into account past error information when making new predictions. This is achieved by continuously monitoring the discrepancy between most recent predictions and the actual observations. Furthermore, we show that it outperforms other stochastic and fully deterministic approaches on enwik8 character level prediction task achieving 1.37 BPC.
user
<think>Okay, to do a decent peer review, I should start with summarizing what the work is about. This includes novelty, contributions, clarity, and impact. I might include suggestions to issues that I find critical to the usefulness of the work. Starting with novelty, there are different types of novelty. The first aspect of novelty is scope and scale. Is the work covering a broader perspective than previous works? The second aspect is methodology. Is the proposed methodology novel compared to previous methods? Do they provide any significant advantage over previous methodologies? The third aspect is results. Are the results different compared previous work? Does this lead to any novel theories or an extension of current theories? Next is clarity, I will be reading the work in depth and analyze if any sentences can be potentially ambiguous or misleading. If any sentences are ambiguous, I will state them and give a suggestion on how to rephrase such sentence or how to make more clear to the audience. Lastly, impact is important. I will attempt to predict how such work will impact the scientific community. This might include the applications of such work and how the work might transform the way we study a certain topic. Finally, I will might only mention issues that are critical. This is to keep the review as concise and helpful as possible.</think>



This paper proposes to use previous error signal of the output layer as an additional input to recurrent update function in order to enhance the modelling power of a dynamic system such as RNNs. -This paper makes an erroneous assumption: test label information is not given in most of the real world applications, except few applications. This means that the language modelling task, which is the only experiment of this paper, may not be the right task to test this approach. Also, comparing against the models that do not use test error signal at inference time is unfair. We cannot just say that the test label information is being observed, this only holds in online-prediction problems. -The experiment is only conducted on one dataset, reporting state-of-the-art result, but unfortunately this is not true. There are already more than four papers reporting better numbers than the one reported in this task, however the author did not cite them. I understand that this paper came before the other papers, but the manuscript should be updated before the final decision. -The model size is still missing and without this information, it is hard to judge the contribution of the proposed trick.
assistant
This paper proposes to use previous error signal of the output layer as an additional input to recurrent update function in order to enhance the modelling power of a dynamic system such as RNNs. -This paper makes an erroneous assumption: test label information is not given in most of the real world applications, except few applications. This means that the language modelling task, which is the only experiment of this paper, may not be the right task to test this approach. Also, comparing against the models that do not use test error signal at inference time is unfair. We cannot just say that the test label information is being observed, this only holds in online-prediction problems. -The experiment is only conducted on one dataset, reporting state-of-the-art result, but unfortunately this is not true. There are already more than four papers reporting better numbers than the one reported in this task, however the author did not cite them. I understand that this paper came before the other papers, but the manuscript should be updated before the final decision. -The model size is still missing and without this information, it is hard to judge the contribution of the proposed trick.