Learning RNNs with Commutative State Transitions
Many machine learning tasks involve analysis of set valued inputs, and thus the learned functions are expected to be permutation invariant. Recent works (e.g., Deep Sets) have sought to characterize the neural architectures which result in permutation invariance. These typically correspond to applying the same pointwise function to all set components, followed by sum aggregation. Here we take a different approach to such architectures and focus on recursive architectures such as RNNs, which are not permutation invariant in general, but can implement permutation invariant functions in a very compact manner. We first show that commutativity and associativity of the state transition function result in permutation invariance. Next, we derive a regularizer that minimizes the degree of non-commutativity in the transitions. Finally, we demonstrate that the resulting method outperforms other methods for learning permutation invariant models, due to its use of recursive computation.
Summary: this paper proposes a new principled methodology for deriving and training RNN neural networks for prediction of permutation invariant functions. Authors show on simple tasks their method may outperform DeepSets, the state of the art. Although the idea is interesting and the paper reflects thorough work, I believe in its current form results are too weak to deserve publication. More specifically. 1)Mathematical results and statements are mostly trivial and may well be omitted or included as an appendix. They don*t seem to convey anything profound (with the exception of theorem 3.6, but this follows from results on deepsets paper). Some of these results are also mostly anectodal 2)The regularization idea seems interesting, but I am concerned it is showing that the final learned networks have a deepset-like architecture: more specifically, theorem 3.6 shows RNN can implement permutation invariant functions by making identifying the parameters with the ones of deepperm. Also, as the authors mentioned, when learning a permutation invariant function then for any degree of regularization the regularization loss can be made zero. So for me, results seem to indicate that the network might have learned a deepperm kind of representation, which equivalently can be expressed as a RNN. Authors should make clear there are fundamental differents between both frameworks 3)Overall, the experimental validation section is weak and an extensive description of network architectures is lacking. Without them it is hard to resolve my concerns on 2).
You are a member of the scientific community tasked with peer review. Review the following paper title and abstract.
### Title + Abstract
Learning RNNs with Commutative State Transitions
Many machine learning tasks involve analysis of set valued inputs, and thus the learned functions are expected to be permutation invariant. Recent works (e.g., Deep Sets) have sought to characterize the neural architectures which result in permutation invariance. These typically correspond to applying the same pointwise function to all set components, followed by sum aggregation. Here we take a different approach to such architectures and focus on recursive architectures such as RNNs, which are not permutation invariant in general, but can implement permutation invariant functions in a very compact manner. We first show that commutativity and associativity of the state transition function result in permutation invariance. Next, we derive a regularizer that minimizes the degree of non-commutativity in the transitions. Finally, we demonstrate that the resulting method outperforms other methods for learning permutation invariant models, due to its use of recursive computation.
user
<think>Okay, to do a decent peer review, I should start with summarizing what the work is about. This includes novelty, contributions, clarity, and impact. I might include suggestions to issues that I find critical to the usefulness of the work. Starting with novelty, there are different types of novelty. The first aspect of novelty is scope and scale. Is the work covering a broader perspective than previous works? The second aspect is methodology. Is the proposed methodology novel compared to previous methods? Do they provide any significant advantage over previous methodologies? The third aspect is results. Are the results different compared previous work? Does this lead to any novel theories or an extension of current theories? Next is clarity, I will be reading the work in depth and analyze if any sentences can be potentially ambiguous or misleading. If any sentences are ambiguous, I will state them and give a suggestion on how to rephrase such sentence or how to make more clear to the audience. Lastly, impact is important. I will attempt to predict how such work will impact the scientific community. This might include the applications of such work and how the work might transform the way we study a certain topic. Finally, I will might only mention issues that are critical. This is to keep the review as concise and helpful as possible.</think>



Summary: this paper proposes a new principled methodology for deriving and training RNN neural networks for prediction of permutation invariant functions. Authors show on simple tasks their method may outperform DeepSets, the state of the art. Although the idea is interesting and the paper reflects thorough work, I believe in its current form results are too weak to deserve publication. More specifically. 1)Mathematical results and statements are mostly trivial and may well be omitted or included as an appendix. They don*t seem to convey anything profound (with the exception of theorem 3.6, but this follows from results on deepsets paper). Some of these results are also mostly anectodal 2)The regularization idea seems interesting, but I am concerned it is showing that the final learned networks have a deepset-like architecture: more specifically, theorem 3.6 shows RNN can implement permutation invariant functions by making identifying the parameters with the ones of deepperm. Also, as the authors mentioned, when learning a permutation invariant function then for any degree of regularization the regularization loss can be made zero. So for me, results seem to indicate that the network might have learned a deepperm kind of representation, which equivalently can be expressed as a RNN. Authors should make clear there are fundamental differents between both frameworks 3)Overall, the experimental validation section is weak and an extensive description of network architectures is lacking. Without them it is hard to resolve my concerns on 2).
assistant
Summary: this paper proposes a new principled methodology for deriving and training RNN neural networks for prediction of permutation invariant functions. Authors show on simple tasks their method may outperform DeepSets, the state of the art. Although the idea is interesting and the paper reflects thorough work, I believe in its current form results are too weak to deserve publication. More specifically. 1)Mathematical results and statements are mostly trivial and may well be omitted or included as an appendix. They don*t seem to convey anything profound (with the exception of theorem 3.6, but this follows from results on deepsets paper). Some of these results are also mostly anectodal 2)The regularization idea seems interesting, but I am concerned it is showing that the final learned networks have a deepset-like architecture: more specifically, theorem 3.6 shows RNN can implement permutation invariant functions by making identifying the parameters with the ones of deepperm. Also, as the authors mentioned, when learning a permutation invariant function then for any degree of regularization the regularization loss can be made zero. So for me, results seem to indicate that the network might have learned a deepperm kind of representation, which equivalently can be expressed as a RNN. Authors should make clear there are fundamental differents between both frameworks 3)Overall, the experimental validation section is weak and an extensive description of network architectures is lacking. Without them it is hard to resolve my concerns on 2).