RNNs with Private and Shared Representations for Semi-Supervised Sequence Learning
Training recurrent neural networks (RNNs) on long sequences using backpropagation through time (BPTT) remains a fundamental challenge. It has been shown that adding a local unsupervised loss term into the optimization objective makes the training of RNNs on long sequences more effective. While the importance of an unsupervised task can in principle be controlled by a coefficient in the objective function, the gradients with respect to the unsupervised loss term still influence all the hidden state dimensions, which might cause important information about the supervised task to be degraded or erased. Compared to existing semi-supervised sequence learning methods, this paper focuses upon a traditionally overlooked mechanism -- an architecture with explicitly designed private and shared hidden units designed to mitigate the detrimental influence of the auxiliary unsupervised loss over the main supervised task. We achieve this by dividing RNN hidden space into a private space for the supervised task and a shared space for both the supervised and unsupervised tasks. We present extensive experiments with the proposed framework on several long sequence modeling benchmark datasets. Results indicate that the proposed framework can yield performance gains in RNN models where long term dependencies are notoriously challenging to deal with.
 This paper introduces a recurrent neural network architecture that minimizes both supervised and unsupervised losses for sequence learning. The intuition behind of this work makes sense that the mixing gradients problem would degrade the performance of RNNs on sequence learning tasks. However, the experimental results are not convincing although quite big improvements on CIFAR10 are observed. Particularly, I am not entirely convinced by the authors* argument that adding auxiliary loss is helpful for addressing long-term dependencies. What I can see from Table 2 is that the performance of the proposed method in terms of accuracy increases as the ratio of shared parameters increases. In my opinion, these results can be also interpreted as a regularization effect. Datasets like MNIST and CIFAR can be used to show that the proposed idea is working as a proof of concept, but it might be difficult to draw a generalized conclusion from the results in the paper like the use of unsupervised loss during training helps. In my opinion, neural machine translation datasets might be an option if the authors would like to claim the use of an unsupervised loss is helpful to capture better long-range dependencies.
You are a member of the scientific community tasked with peer review. Review the following paper title and abstract.
### Title + Abstract
RNNs with Private and Shared Representations for Semi-Supervised Sequence Learning
Training recurrent neural networks (RNNs) on long sequences using backpropagation through time (BPTT) remains a fundamental challenge. It has been shown that adding a local unsupervised loss term into the optimization objective makes the training of RNNs on long sequences more effective. While the importance of an unsupervised task can in principle be controlled by a coefficient in the objective function, the gradients with respect to the unsupervised loss term still influence all the hidden state dimensions, which might cause important information about the supervised task to be degraded or erased. Compared to existing semi-supervised sequence learning methods, this paper focuses upon a traditionally overlooked mechanism -- an architecture with explicitly designed private and shared hidden units designed to mitigate the detrimental influence of the auxiliary unsupervised loss over the main supervised task. We achieve this by dividing RNN hidden space into a private space for the supervised task and a shared space for both the supervised and unsupervised tasks. We present extensive experiments with the proposed framework on several long sequence modeling benchmark datasets. Results indicate that the proposed framework can yield performance gains in RNN models where long term dependencies are notoriously challenging to deal with.
user
<think>Okay, to do a decent peer review, I should start with summarizing what the work is about. This includes novelty, contributions, clarity, and impact. I might include suggestions to issues that I find critical to the usefulness of the work. Starting with novelty, there are different types of novelty. The first aspect of novelty is scope and scale. Is the work covering a broader perspective than previous works? The second aspect is methodology. Is the proposed methodology novel compared to previous methods? Do they provide any significant advantage over previous methodologies? The third aspect is results. Are the results different compared previous work? Does this lead to any novel theories or an extension of current theories? Next is clarity, I will be reading the work in depth and analyze if any sentences can be potentially ambiguous or misleading. If any sentences are ambiguous, I will state them and give a suggestion on how to rephrase such sentence or how to make more clear to the audience. Lastly, impact is important. I will attempt to predict how such work will impact the scientific community. This might include the applications of such work and how the work might transform the way we study a certain topic. Finally, I will might only mention issues that are critical. This is to keep the review as concise and helpful as possible.</think>



 This paper introduces a recurrent neural network architecture that minimizes both supervised and unsupervised losses for sequence learning. The intuition behind of this work makes sense that the mixing gradients problem would degrade the performance of RNNs on sequence learning tasks. However, the experimental results are not convincing although quite big improvements on CIFAR10 are observed. Particularly, I am not entirely convinced by the authors* argument that adding auxiliary loss is helpful for addressing long-term dependencies. What I can see from Table 2 is that the performance of the proposed method in terms of accuracy increases as the ratio of shared parameters increases. In my opinion, these results can be also interpreted as a regularization effect. Datasets like MNIST and CIFAR can be used to show that the proposed idea is working as a proof of concept, but it might be difficult to draw a generalized conclusion from the results in the paper like the use of unsupervised loss during training helps. In my opinion, neural machine translation datasets might be an option if the authors would like to claim the use of an unsupervised loss is helpful to capture better long-range dependencies.
assistant
 This paper introduces a recurrent neural network architecture that minimizes both supervised and unsupervised losses for sequence learning. The intuition behind of this work makes sense that the mixing gradients problem would degrade the performance of RNNs on sequence learning tasks. However, the experimental results are not convincing although quite big improvements on CIFAR10 are observed. Particularly, I am not entirely convinced by the authors* argument that adding auxiliary loss is helpful for addressing long-term dependencies. What I can see from Table 2 is that the performance of the proposed method in terms of accuracy increases as the ratio of shared parameters increases. In my opinion, these results can be also interpreted as a regularization effect. Datasets like MNIST and CIFAR can be used to show that the proposed idea is working as a proof of concept, but it might be difficult to draw a generalized conclusion from the results in the paper like the use of unsupervised loss during training helps. In my opinion, neural machine translation datasets might be an option if the authors would like to claim the use of an unsupervised loss is helpful to capture better long-range dependencies.