The vast majority of neural models in Natural Language Processing adopt a form of structureless distributed representations. While these models are powerful at making predictions, the representational form is rather crude and does not provide insights into linguistic structures. In this paper we introduce novel language models with representations informed by the framework of Holographic Reduced Representation (HRR). This allows us to inject structures directly into our wordlevel and chunk-level representations. Our analyses show that by using HRR as a structured compositional representation, our models are able to discover crude linguistic roles, which roughly resembles a classic division between syntax and semantics.
Summary:
========
Theis paper proposes a method for learning decomposable representations in the context of a language modeling task. Using holographic reduced representations (HRR), a word embedding is composed of a role and a filler. The embedding is then fed to an LSTM language model. There is also an extension to chunk-level representations. Experimentally, the model achieves perplexity comparable to a (weak) baseline LSTM model. The analysis of the learned representations shows a separation into syntactic and semantic roles. 
The paper targets an important problem, that of learning decomposable representations. As far as I know, it introduces a novel perspective using HRR and does so in the context of language modeling, which is a core NLP task. The analysis of the learned representations is quite interesting. I do have some concerns with regards to the quality of the language model, the clarity of some of the model description, and the validity of using HRR in this scenario. Please see detailed comments below. 
Comments:
=========
1. Section 2 refers to Plate (1995) for the conditions when the approximate decoding via correlation holds. I think it's important to mention these conditions and discuss whether they apply to the language modeling case. In particular, Plate mentions that the elements of each vector need to be iid with mean zero and variance 1/n (where n is the length of the vector). Is this true for the present case? Typically, word embeddings and LSTM states are do not exhibit this distribution. Are there other conditions that are (not) met?
2. Learning separate bases for different role-filler bindings is said to encourage the model to learn a decomposition of word representation. On the other hand, if I understand correctly, this means that word embeddings are not shared between roles, because s^w_i is also a role-specific vector (not just a word-specific vector). Is that a cause of concern? 
3. It's not clear to me where in the overall model the next word is predicted. Figure 1b has an LSTM that predicts filler embeddings. Does this replace predicting the next word in a vanilla LSTM? Equation 5 still computes a word score. Is this used to compute the probability of the next word as in equation 2?  
4. Comparison to other methods for composing words. Since much of the paper is concerned with composing words, it seem natural to compare the methods (and maybe some of the results) to methods for composing words. Some examples include [2] and the line of work on recursive neural networks by Socher et al., but there are many others. 
5. Perplexity results:
- The baseline results (100.5 ppl on PTB) are very weak for an LSTM. There are multiple papers showing that a simple LSTM can do much better. The heavily tuned LSTM of [1] gets 59.6 but even less tuned LSTMs go under 80 or 80 ppl. See some results in [1]. This raises a concern that the improvements from the HRR model may not be significant. Would they hold in a more competitive model? 
- Can you speculate or analyze in more detail why the chunk-level model doesn't perform well, and why adding more fillers doesn't help in this case? 
6. Motivation: 
- The introduction claims that the dominant encoder-decoder paradigm learns "transformations from many smaller comprising units to one complex emedding, and vice versa". This claim should be qualified by the use of attention, where there is not a single complex embedding, rather a distribution over multiple embeddings. 
- Introduction, first paragraph, claims that "such crude way of representing the structure is unsatisfactory, due to a lack of transparency, interpretability or transferability" - what do you mean by these concepts and how exactly is the current approach limited with respect to them? Giving a bit more details about this point here or elsewhere in the paper would help motivate the work. 
7. Section 3.3 was not so clear to me:
- In step 1, what are these r_i^{chunk}? Should we assume that all chunks have the same role embeddings, despite them potentially being syntactically different? How do you determine where to split output vectors from the RNN to two parts? What is the motivation for doing this?
- In prediction, how do you predict the next chunk embedding? Is there a different loss function for this? 
- Please provide more details on decoding, such as the mentioned annealing and regularization. 
- Finally, the reliance on a chunker is quite limiting. These may not be always available or of high quality. 
8. The analysis in section 4.3 is very interesting and compelling. Figure 2 makes a good point. I would have liked to see more analysis along these lines. For example, more discussion of the word analogy results, including categories where HRR does not do better than the baseline. Also consider other analogy datasets that capture different aspects. 
9. While I agree that automatic evaluation at chunk-level is challenging, I think more can be done. For instance, annotations in PTB can be used to automatically assign roles such as those in table 4, or others (there are plenty of annotations on PTB), and then to evaluate clustering along different annotations at a larger scale. 
10. The introduction mentions a subset of the one billion word LM dataset (why a subset?), but then the rest of the papers evaluates only on PTB. Is this additional dataset used or not? 
11. Introduction, first paragraph, last sentence: "much previous work" - please cite such relevant work on inducing disentangled representations.
12. Please improve the visibility of Figure 1. Some symbols are hard to see when printed. 
13. More details on the regularization on basis embeddings (page 4) would be useful. 
14. Section 3.3 says that each unique word token is assigned a vectorial parameter. Should this be word type? 
15. Why not initialize the hidden state with the last state from the last batch? I understand that this is done to assure that the chunk-level models only consider intra-sentential information, but why is this desired? 
16. Have you considered using more than two roles? I wonder how figure 2 would look in this case. 
Writing, grammar, etc.:
====================== 
- End of section 1: Our papers -> Our paper
- Section 2: such approach -> such an approach; HRR use -> HRR uses; three operations -> three operations*:*
- Section 3.1: "the next token w_t" - should this be w_{t+1)? 
- Section 3.2, decoding: remain -> remains 
- Section 3.3: work token -> word token 
- Section 4.1: word analogy task -> a word analogy task; number basis -> numbers of basis
- Section 4.2: that the increasing -> that increasing 
- Section 4.3: no space before comma (first paragraph); on word analogy task -> on a word analogy task; belong -> belongs
- Section 4.4: performed similar -> performed a similar; luster -> cluster 
- Section 5: these work -> these works/papers/studies; share common goal -> share a common goal; we makes -> we make; has been -> have been  
References
==========
[1] Melis et al., On the State of the Art of Evaluation in Neural Language Models
[2] Mitchell and Lapata, Vector-based Models of Semantic Composition