Widely used recurrent units, including Long-short Term Memory (LSTM) and the Gated Recurrent Unit (GRU), perform well on natural language tasks, but their ability to learn structured representations is still questionable. Exploiting reduced Tensor Product Representations (TPRs) -distributed representations of symbolic structure in which vector-embedded symbols are bound to vector-embedded structural positions -we propose the TPRU, a simple recurrent unit that, at each time step, explicitly executes structural-role binding and unbinding operations to incorporate structural information into learning. A gradient analysis of our proposed TPRU is conducted to support our model design, and its performance on multiple datasets shows the effectiveness of our design choices. Furthermore, observations on a linguistically grounded study demonstrate the interpretability of our TPRU.
This paper proposes a novel model of recurrent unit for RNNs which is inspired from tensor product representation (TPR) introduced by Smolensky et al. in 1990. The authors claim that this allows one to better incorporate structural information into learning and easier interpretability for the learned representations. The proposed approach is motivated by a theoretical analysis showing that using TPR in this context acts as a sort of pre-conditioner and stabilizes learning. Experiments on entailment tasks (given two statement, decide whether the first implies the second) are provided to validate the approach. 
I find the paper not easy to follow, with a non-negligible amount of typos in the notations and results. The advantage in terms of accuracy of the proposed approach seems marginal in the experiment, and the analysis of the interpretability of the learned representations could be improved: loosely speaking, particular examples of interpretability are given but sometimes without contexts or baselines to compare to (see the two last comments below). I know "interpretability" is a difficult property to assess but I think there may be more principled ways to showcase the approach.
I think this paper is not yet ready for publication: the proposed model is interesting and relevant but its validity could be better assessed and the paper needs some thorough proof-reading. 
* Questions / Comments *
- page 1: the authors write U^TR = I, but I believe this is only possible if the TPR dimension d is bigger than the number of roles N. Is this always the case? This should be clarified.
- related to the previous point:  if U^TR = I then shouldn't U^Tb_{t-1} simply be f_{t-1} in Eq. 1?
- before Eq.1, f should be from R^d\times R^d' to R^d, not from R^d \times R^d
- In Eq. 1, b_{t-1} and x_t are not of the same dimension, so the cannot be multiplied by the same matrix U (this is why the matrices V_x and V_b are introduced later on).
- there seems to be a problem with Eq. (7): db_t/d_{b_{t-1}} appears on both sides of the equality...
- Modification 1: what is \tilde{vb_t}? I don't remember seeing this notation introduced before.
- Table 1: constants should not be included in big O notation! To compare constants, one should give the exact number of operations needed for inference.
- POS tagging: Aren't there many other reasons that could lead to this correlation (beside the informal argument that "TPR captures structured information")? Maybe the authors should compare with something else, for example the PMI between values of hidden neurons in a learned RNN and POS tags. Out of context, the numbers in Table 5 are not informative.
- Polysemy: Only a very specific cherry picked example is given here. A more principled or in depth analysis of this phenomenon is needed to make a stronger case.
* Typos *
- " The number of parameter matrices *is* the same as that of..."
- page 4 "stables" -> "stabilizes" (but rephrasing the sentence altogether would be better).
- page 5: BiDAF misses the capital letters ("bidaf").
- "dev set" -> "validation set" or "development set".
- page 8: "provides research*ers with* an intuitive..."?