Learning to Reason with Third-Order Tensor Products
We combine Recurrent Neural Networks with Tensor Product Representations to learn combinatorial representations of sequential data. This improves symbolic interpretation and systematic generalisation. Our architecture is trained end-to-end through gradient descent on a variety of simple natural language reasoning tasks, significantly outperforming the latest state-of-the-art models in single-task and all-tasks settings. We also augment a subset of the data such that training and test data exhibit large systematic differences and show that our approach generalises better than the previous state-of-the-art.
Summary    This paper presents a question-answering system based on tensor   product representations. Given a latent sentence encoding, different   MLPs extract entity and relation representations which are then used   to update an tensor product representations of order-3 and trained   end-to-end from the downstream success of correctly answering the   question. Experiments are limited to bAbI question answering, which is   disappointing as this is a synthetic corpus with a simple known   underlying triples structure. While the proposed system outperforms   baselines like recurrent entity networks (RENs) by a small difference   in mean error, RENs have also been applied to more real-world tasks   such as the Children's Book Test (CBT).   Strengths    - I like that the authors do not just report the best performance of     their model, but also the mean and variance from five runs.   - I liked the ablation study, showing which operations are needed for     which bAbI tasks.   Weaknesses    - Results on bAbI should be taken with a huge grain of salt and only     serve as a unit-test. Specifically, since the bAbI corpus is     generated from a simple grammar and sentence follow a strict triplet     structure, it is not surprising to me that a model extracting three     distinct symbol representations from a learned sentence     representation (therefore reverse engineering the underlying     symbolic nature of the data) would solve bAbI tasks. However, it is     highly doubtful this method would perform well on actual natural     language sentences. Hence, statements such as "trained [...] on a     variety of natural language tasks" is misleading. The authors of the     baseline model "recurrent entity networks" [12] have not stopped at     bAbI, but also validated their models on more real-world data such     as the Children's Book Test (CBT). Given that RENs solve all bAbI     tasks and N2Ns solve all but one, it is not clear to me what the     proposed method adds to a table other than a small reduction in mean     error. Moreover, the N2N baseline in Table 2 is not introduced or     referenced in this paper, so I am not sure which system the authors     are referring to here.  Minor Comments    - L11: LSTMs have only achieved on some NLP tasks, whereas traditional     methods still prevail on others, so stating they have achieved SotA     in NLP is a bit too vague.   - L15: Again, too vague, certain RNNs work well for certain natural     language reasoning tasks. See for instance the literature on natural     language inference and the leaderboard at     https://nlp.stanford.edu/projects/snli/   - L16-18: The reinforcement learning / agent analogy seems a bit     out-of-place here. I think you generally point to generalization     capabilities which I believe are better illustrated by the examples     you give later in the paper (from lines 229 to 253).   - Eq. 1: This seems like a very specific choice of combining the     information from entity representations and their types. Why is this     a good choice? Why not keep the concatenation of the kitty/cat outer     product and the mary/person outer product? Why is instead the     superposition of all bindings a good design choice?   - I believe section four could benefit from a small overview figures     illustrating the computation graph that is constructed by the     method.   - Eq. 7: At first, I found it surprising why three distinct relation     representation are extracted from the sentence representation, but     it became clearer later with the write, move and backling     functions. Maybe already mention at this point why the three     relation representations are going to be used for.   - Eq. 15: s_question has not been introduced before. I imagine it is a     sentence encoding of the question and calculated similarly to Eq. 5?   - Eq. 20: A bit more details for readers unfamiliar with bAbI or     question answering would be good here. "valid words" here means     possible answer words for the given story and question, correct?   - L192: "glorot initalization" -> "Glorot initialization". Also, there     is a reference for that method: Glorot, X., & Bengio, Y. (2010,     March). Understanding the difficulty of training deep feedforward     neural networks. In Proceedings of the thirteenth international     conference on artificial intelligence and statistics (pp. 249-256).   - L195: α=0.008, β₁=0.6 and β₂=0.4 look like rather arbitrary     choices. Where does the configuration for these hyper-parameters     come from? Did you perform a grid search?   - L236-244: If I understand it correctly, at test time stories with     new entities (Alex etc.) are generated. How does your model support     a growing set of vocabulary words given that MLPs have parameters     dependent on the vocabulary size (L188-191) and are fixed at test     time?   - L265: If exploding gradients are a problem, why don't you perform     gradient clipping with a high value for the gradient norm to avoid     NaNs appearing? Simply reinitializing the model is quite hacky.   - p.9: Recurrent entity networks (RENs) [12] is not just an arXiv     paper but has been published at ICLR 2017.