Neural networks are able to perform tasks that rely on compositional structure even though they lack obvious mechanisms for representing this structure. To analyze the internal representations that enable such success, we propose ROLE, a technique that detects whether these representations implicitly encode symbolic structure. ROLE learns to approximate the representations of a target encoder E by learning a symbolic constituent structure and an embedding of that structure into E's representational vector space. The constituents of the approximating symbol structure are defined by structural positions -roles -that can be filled by symbols. We show that when E is constructed to explicitly embed a particular type of structure (string or tree), ROLE successfully extracts the ground-truth roles defining that structure. We then analyze a GRU seq2seq network trained to perform a more complex compositional task (SCAN), where there is no ground truth role scheme available. For this model, ROLE successfully discovers an interpretable symbolic structure that the model implicitly uses to perform the SCAN task, providing a comprehensive account of the representations that drive the behavior of a frequently-used but hard-to-interpret type of model. We verify the causal importance of the discovered symbolic structure by showing that, when we systematically manipulate hidden embeddings based on this symbolic structure, the model's resulting output is changed in the way predicted by our analysis. Finally, we use ROLE to explore whether popular sentence embedding models are capturing compositional structure and find evidence that they are not; we conclude by suggesting how insights from ROLE can be used to impart new inductive biases to improve the compositional abilities of such models.
This work focuses on the understanding of Deep NNs and of their generalization ability for compositional tasks and especially for partially composition tasks such as one encounters in natural language processing.
The core idea is to design and to learn a neural model (the ROLE network) able to analyse a target neural network (e.g. an encoder in an seq2seq architecture) by identifying the symbolic structures the target network manipulates (the symbols and their roles) in its representations and the compositional rules it has learned on these. The main motivations are the understanding of what is learnt by a such neural network and getting ideas on which architecture to choose for improving generalization on compositional tasks.
 
The experimental study shows that ROLE ’s results are relevant when dealing with a fully compositional model by design. This is a kind of sanity check, that ROLE may uncover the ground truth from the data, while no prior information is provided on the nature of the roles. 
The experiments on the SCAN dataset concern a standard RNN model learned from data. I am not sure of the nature of RNN used here. How many layers, which activation function ? The experimental section also provides insights on what is learned by the ROLE model and on how its (compositional) representations match the ones learned by the RNN.     
As far as in understand, beyond the understanding of the learned representations in RNNs, the paper motivates the work with the expectation that the gained knowledge might be useful for designing better neural nets, with improved generalization ability. Yet i don’t  see clearly what could be done on this line.