We introduce HUBERT 1 which combines the structured-representational power of Tensor-Product Representations (TPRs) and BERT, a pre-trained bidirectional Transformer language model. We show that there is shared structure between different NLP datasets that HUBERT, but not BERT, is able to learn and leverage. We validate the effectiveness of our model on the GLUE benchmark and HANS dataset. Our experiment results show that untangling data-specific semantics from general language structure is key for better transfer among NLP tasks.
This paper proposes an alternative way of reusing pretrained BERT for downstream tasks rather than the traditional method of fine-tuning the embeddings equivalent to the CLS token. 
For each bert embedded token, the proposed method aims at disentangling semantic information of the word from its structural role. Authors provide two ways to provide this disentagling using LSTM or transformer blocks. with several design choices such as: *  a regularization term to encourages the roles matrix to be orthogonal and hence each role carry independent information *  design the roles and symbols matrices so that the number of symbols is greater than the number of roles
In evaluation authors design several experiments to show that: 
* Does transferring disentangled role & symbol embeddings improve transfer learning
* the effectiveness of the TPR layer on performance?
* Transfer beyond Glue tasks? 
While those experiments provide empirical gains of the design choices, authors don't show enough study to attribute those  empirical gains to the presented design choices: 
One large claim in the paper is that empirical gains in the ability of transfer between similar tasks MNLI and GLUE is because of disentangling the semantics from the role representations. We don't know if the TPR layer really manages to do that, this could have been easily verified using for example clustering word senses of the same word. 
The empirical gains in transfer learning can be simply attributed to: 
- More params it seems adding an LSTM over bert embeddings already does some improvement, I would have loved to see this more exploited but it wasn't. This aligns with some recent findings that BERT is undertrained (Liu et al. 2019) https://arxiv.org/abs/1907.11692  
- Variance in the results (authors report only results of one single run not mean and std of several runs).
- More budget given to hyper-parameter search for the models proposed in the paper.  Hyper param budget isn't also reported in the paper. 
- other factors, not the ones associated with the claims in the paper: for example what authors claim is an ablation study was comparing several different models together. It would have been more interesting to see for example the effect of making the # symbols = # roles or removing the orthogonality loss from the roles matrix.
Conclusion: The paper introduces large claims and empirical results that correlate with, however the provided experiments are not done with enough control to attribute gains to the design choices provided in the paper.