Contextualized word representations, such as ELMo and BERT, were shown to perform well on various semantic and structural (syntactic) task. In this work, we tackle the task of unsupervised disentanglement between semantics and structure in neural language representations: we aim to learn a transformation of the contextualized vectors, that discards the lexical semantics, but keeps the structural information. To this end, we automatically generate groups of sentences which are structurally similar but semantically different, and use metric-learning approach to learn a transformation that emphasizes the structural component that is encoded in the vectors. We demonstrate that our transformation clusters vectors in space by structural properties, rather than by lexical semantics. Finally, we demonstrate the utility of our distilled representations by showing that they outperform the original contextualized representations in a few-shot parsing setting.
Summary:
=========
This paper aims to disentangle semantics and syntax in contextualized word representations. The main idea is to learn a transformation of the contexualized representations that will make two word representations to be more similar if they appear in the same syntactic context, but less similar if they appear in different syntactic contexts. The efficacy of this transformation is evaluated through cluster analysis, showing that words better organize in syntactic clusters after the transformation, and through low-resource dependency parsing. 
The paper presents a simple approach to transform word representations and expose their syntactic information. The experiments are mostly convincing. I would like to see better motivation, more engagement with a wider range of related work, and more thorough quantitative evaluations. Another important question to address is also what kind of semantic/syntactic types of information are targeted, and how to handle the tradeoff between them, for instance for different purposes. 
Main comments:
==============
1. Motivation: I found the motivation for the problem understudied a bit lacking. The main motivation seems to be to disentangle semantic and syntactic information. But why should we care about that? Beyond reference to disentangling in computer vision, some more motivation would be good. The few-shot parsing is a good such motivation, although the results are a bit disappointing (see more on this below). Another possible motivation is potential applications of disentanglement in language generation. There is a line of work on style transfer also in language generation, and it seems plausible that the methodology could be applied to such tasks. 
2. The present work is well-differentiated from work on extracting syntactic information from word representations via supervised ways, as the current work does so in an unsupervised way. I don't quite get the terminological differentiation between "mapping" and "extracting" in the introduction, but the idea is clear. 
3. Have you considered alternative representations of word pairs besides the different of their transformations f(x)-f(y)? 
4. I found it interesting that the word representation from BERT is the concatenation of layer 16 with the mean of all the other layers. This is motivated by Hewitt and Manning's findings, and [5] found similar results. However, the different between layer 16 and others is not that large as to warrant emphasizing it so much. Perhaps a scalar mix with fine-tuning may work better, as in [5], or another method. Have you tried other word representations? I also wonder whether it makes sense to use different layers for different parts of the triplet loss, depending on whether to emphasize syntactic vs. semantic similarity. 
5. The introduction lays out connections to some related work, but leaves several relevant pieces missing. See examples below. 
6. The results in 3.3 are limited but useful. The comparison with a PCA-ed and reduced representation is well thought of, because of the risk with low-resource and high dimensionality. That said, I found the gap between the proposed syntax model and the ELMo-reduced disappointingly small. Even in the LAS, it seems like the difference is very small, ~0.5, although it's hard to tell from the figure. Providing the actual numbers and a measure of statistical significance would be helpful here. 
7. Some care should be taken to define what kind of semantics is targeted here. In several cases this is "lexical semantics", but then we have "meaning" in parentheses sometimes (end of intro). Obviously, there's much more to semantics and meaning that the lexical semantics, so a short discussion of how the work views other, say compositional semantics, would be good. 
Other comments:
===============
1. The introduction seeks a representation that will ignore the similarity between "syrup" in (2) and (4). I wonder if "ignoring" is too strong. One may not want to lose all lexical semantic information. Moreover, the proposed triplet loss does not guarantee that information is ignored (and justly so, in my opinion). 
2. In the example, "maple" and "neural" are said to be syntactically similar, although "maple syrup" is a noun compound while "neural networks" is an adjective-noun. Shouldn't they be treated differently then? Unless the notion of syntax is more narrow and just looks at unlabeled dependency arcs. 
3. Some experimental choices are left unexplained, such as k=6 (section 2.1) or mapping to 27 dims (section 2.3); these two seem potentially important. 
4. Section 2.3: do you also back-prop back into the BERT/ELMo model weights? 
5. The dataset statistics in section 3 do not match those in section 2.2. Please clarify. 
6. The qualitative cluster analysis via t-SNE (3.1) is compelling. It could be made stronger by reporting quantitative clustering statistics such as cluster purity before and after transformation. 
7. In the examples showin in 3.1, it would be good to give also the nearest neighbor before the transformation for comparison. 
8. The quantitative results in 3.2 convey the point convincingly. It's good to see also the lexical match measure going down. The random baseline is also a good sanity check to have. It would be good to provide full results with BERT, at least in the appendix and at least for section 3.2, maybe also for 3.3.
9. More related work: 
+ Work that injects syntactic information into word representations in a supervised way, such as [1,2]
+ Work that shows that word embeddings contain different kinds of information (syntactic/semantic), and propose simle linear transformations to uncover them. 
+ Engaging with the literature on style transfer in language generation would be good, as mentioned above for motivation, but also to situate this work w.r.t to related style transfer work. 
+ Another line of work that may be mentioned is the variety of papers trying to extract syntactic information from contextualized word representations, such as constructing trees from attention weights. There were a few such papers in BlackboxNLP 2018 and 2019. 
Typos, phrasing, formatting, etc.:
============================
- Abstract: a various of semantic... task -> various semantic... tasks; use metric-learning approach -> use a metric-learning approach; in few-shot parsing setting -> in a few-shot parsing setting
- Wilcox et al. does not have a year
- Introduction: few-shots parsing -> few-shot parsing
- Method: extract vectors -> extracts vectors; Operativly -> Operatively 
- Section 3: should encourages -> should encourage; a few-shots settings -> a few-shot setting
- 3.2: -- was not rendered properly
- 3.3: matrix that reduce -> reduces 
References
==========
[1] Levy and Goldberg. 2014. Dependency-Based Word Embeddings
[2] Bansal et al. 2014. Tailoring Continuous Word Representations for Dependency Parsing
[3] Artetxe et al. 2018. Uncovering divergent linguistic information in word embeddings with lessons for intrinsic and extrinsic evaluation
[4] Tenney et al. 2019. BERT Rediscovers the Classical NLP Pipeline
[5] Liu et al. 2019. Linguistic Knowledge and Transferability of Contextual Representations