Fraternal Dropout
Recurrent neural networks (RNNs) are important class of architectures among neural networks useful for language modeling and sequential prediction. However, optimizing RNNs is known to be harder compared to feed-forward neural networks. A number of techniques have been proposed in literature to address this problem. In this paper we propose a simple technique called fraternal dropout that takes advantage of dropout to achieve this goal. Specifically, we propose to train two identical copies of an RNN (that share parameters) with different dropout masks while minimizing the difference between their (pre-softmax) predictions. In this way our regularization encourages the representations of RNNs to be invariant to dropout mask, thus being robust. We show that our regularization term is upper bounded by the expectation-linear dropout objective which has been shown to address the gap due to the difference between the train and inference phases of dropout. We evaluate our model and achieve state-of-the-art results in sequence modeling tasks on two benchmark datasets - Penn Treebank and Wikitext-2. We also show that our approach leads to performance improvement by a significant margin in image captioning (Microsoft COCO) and semi-supervised (CIFAR-10) tasks.
 The proposed method fraternal dropout is a stochastic alternative of the expectation-linear dropout method, where part of the objective is for the dropout mask to have low variance. The first order way to achieve lower variance is to have smaller weights. The second order is by having more evenly spread weights, so there is more concentration around the mean. As a result, it seems that at least part of the effect of explicitly reducing the variance is just stronger weight penalty. The effect of dropout in the first place is the opposite, where variance is introduced deliberately. So I would like to see some comparisons between this method and various dropout rates, and regular weight penalty combinations. This work is very closely related to expectation linear dropout, except that you are now actually minimizing the variance: 1/2E[ ||f(s) - f(s*)|| ] is used instead of E [ ||f(s) - f_bar|| ]. Eq 5 is very close to this, except the f_bar is not quite the mean, but the value with the mean dropout mask. So all the results should be compared with ELD. I do not think the method is theoretically well-motivated as presented, but the empirical results seem solid. It is somewhat alarming how the analysis has little to do with the neural networks and how dropout works, let along RNNs, while the strength of the empirical results are all on RNNs. I feel the ideas interesting and valuable especially in light of strong empirical results, but the authors should do more to clarify what is actually happening. Minor: why use s_i and s_j, when there is never any reference to i and j? As far as I can tell, i and j serve as constants, more like s_1 and s_2.
You are a member of the scientific community tasked with peer review. Review the following paper title and abstract.
### Title + Abstract
Fraternal Dropout
Recurrent neural networks (RNNs) are important class of architectures among neural networks useful for language modeling and sequential prediction. However, optimizing RNNs is known to be harder compared to feed-forward neural networks. A number of techniques have been proposed in literature to address this problem. In this paper we propose a simple technique called fraternal dropout that takes advantage of dropout to achieve this goal. Specifically, we propose to train two identical copies of an RNN (that share parameters) with different dropout masks while minimizing the difference between their (pre-softmax) predictions. In this way our regularization encourages the representations of RNNs to be invariant to dropout mask, thus being robust. We show that our regularization term is upper bounded by the expectation-linear dropout objective which has been shown to address the gap due to the difference between the train and inference phases of dropout. We evaluate our model and achieve state-of-the-art results in sequence modeling tasks on two benchmark datasets - Penn Treebank and Wikitext-2. We also show that our approach leads to performance improvement by a significant margin in image captioning (Microsoft COCO) and semi-supervised (CIFAR-10) tasks.
user
<think>Okay, to do a decent peer review, I should start with summarizing what the work is about. This includes novelty, contributions, clarity, and impact. I might include suggestions to issues that I find critical to the usefulness of the work. Starting with novelty, there are different types of novelty. The first aspect of novelty is scope and scale. Is the work covering a broader perspective than previous works? The second aspect is methodology. Is the proposed methodology novel compared to previous methods? Do they provide any significant advantage over previous methodologies? The third aspect is results. Are the results different compared previous work? Does this lead to any novel theories or an extension of current theories? Next is clarity, I will be reading the work in depth and analyze if any sentences can be potentially ambiguous or misleading. If any sentences are ambiguous, I will state them and give a suggestion on how to rephrase such sentence or how to make more clear to the audience. Lastly, impact is important. I will attempt to predict how such work will impact the scientific community. This might include the applications of such work and how the work might transform the way we study a certain topic. Finally, I will might only mention issues that are critical. This is to keep the review as concise and helpful as possible.</think>



 The proposed method fraternal dropout is a stochastic alternative of the expectation-linear dropout method, where part of the objective is for the dropout mask to have low variance. The first order way to achieve lower variance is to have smaller weights. The second order is by having more evenly spread weights, so there is more concentration around the mean. As a result, it seems that at least part of the effect of explicitly reducing the variance is just stronger weight penalty. The effect of dropout in the first place is the opposite, where variance is introduced deliberately. So I would like to see some comparisons between this method and various dropout rates, and regular weight penalty combinations. This work is very closely related to expectation linear dropout, except that you are now actually minimizing the variance: 1/2E[ ||f(s) - f(s*)|| ] is used instead of E [ ||f(s) - f_bar|| ]. Eq 5 is very close to this, except the f_bar is not quite the mean, but the value with the mean dropout mask. So all the results should be compared with ELD. I do not think the method is theoretically well-motivated as presented, but the empirical results seem solid. It is somewhat alarming how the analysis has little to do with the neural networks and how dropout works, let along RNNs, while the strength of the empirical results are all on RNNs. I feel the ideas interesting and valuable especially in light of strong empirical results, but the authors should do more to clarify what is actually happening. Minor: why use s_i and s_j, when there is never any reference to i and j? As far as I can tell, i and j serve as constants, more like s_1 and s_2.
assistant
 The proposed method fraternal dropout is a stochastic alternative of the expectation-linear dropout method, where part of the objective is for the dropout mask to have low variance. The first order way to achieve lower variance is to have smaller weights. The second order is by having more evenly spread weights, so there is more concentration around the mean. As a result, it seems that at least part of the effect of explicitly reducing the variance is just stronger weight penalty. The effect of dropout in the first place is the opposite, where variance is introduced deliberately. So I would like to see some comparisons between this method and various dropout rates, and regular weight penalty combinations. This work is very closely related to expectation linear dropout, except that you are now actually minimizing the variance: 1/2E[ ||f(s) - f(s*)|| ] is used instead of E [ ||f(s) - f_bar|| ]. Eq 5 is very close to this, except the f_bar is not quite the mean, but the value with the mean dropout mask. So all the results should be compared with ELD. I do not think the method is theoretically well-motivated as presented, but the empirical results seem solid. It is somewhat alarming how the analysis has little to do with the neural networks and how dropout works, let along RNNs, while the strength of the empirical results are all on RNNs. I feel the ideas interesting and valuable especially in light of strong empirical results, but the authors should do more to clarify what is actually happening. Minor: why use s_i and s_j, when there is never any reference to i and j? As far as I can tell, i and j serve as constants, more like s_1 and s_2.