DIFFERENTIABLE PERTURB-AND-PARSE: SEMI-SUPERVISED PARSING WITH A STRUCTURED VARIATIONAL AUTOENCODER
Human annotation for syntactic parsing is expensive, and large resources are available only for a fraction of languages. A question we ask is whether one can leverage abundant unlabeled texts to improve syntactic parsers, beyond just using the texts to obtain more generalisable lexical features (i.e. beyond word embeddings). To this end, we propose a novel latent-variable generative model for semi-supervised syntactic dependency parsing. As exact inference is intractable, we introduce a differentiable relaxation to obtain approximate samples and compute gradients with respect to the parser parameters. Our method (Differentiable Perturb-and-Parse) relies on differentiable dynamic programming over stochastically perturbed arc weights. We demonstrate effectiveness of our approach with experiments on English, French and Swedish.
This paper proposed a variational autoencoder-based method for semi-supervised dependency parsing. Given an input sentence s, an LSTM-based encoder generates a sentence embedding z, and a NN of Kiperwasser & Goldberg (2016) generates a dependency structure T. Gradients over the tree encoder are approximated by (1) adding a perturbation matrix over the weight matrix and (2) relax dynamic programming-based parsing algorithm to a differentiable format. The decoder combines standard LSTM and Graph Convolutional Network to generate the input sentence from z and T. The authors evaluated the proposed method on three languages, using 10% of the original training data as labeled and the rest as unlabeled data.
Pros
1. I like the idea of this sentence->tree->sentence autoencoder for semi-supervised parsing. The authors proposed a novel and nice way to tackle key challenges in gradient computation. VAE involves marginalization over all possible dependency trees, which is computationally infeasible, and the proposed method used a Gumbel-Max trick to approximate it. The tree inference procedure involves non-differentiable structured prediction, and the authors used a peaked-softmax method to address the issue. The whole model is fully differentiable and can be thus trained end to end.
2. The direction of semi-supervised parsing is useful and promising, not only for resource-poor languages, but also for popular languages like English. A successful research on this direction could be potentially helpful for lots of future work.
Cons, and suggestions on experiments
My main concerns are around experiments. Overall I think they are not strong enough to demonstrate that this paper has sufficient contribution to semi-supervised parsing. Below are details.
1. The current version only used 10% of original training data as labeled and the rest as unlabeled data. This makes the reported numbers way below existing state-of-the-art performance. For example, the SOTA UAS on English PTB has been >95%. Ideally, the authors should be able to train a competitive supervised parser on full training data (English or other languages), and get huge amount of unlabeled data from other sources (e.g. News) to further push up the performance. The current setting makes it hard to justify how useful the proposed method could be in practice.
2. The best numbers from the proposed model is lower than baseline (Kipperwasser & Goldberg) on English, and only marginally better on Swedish. This probably means the supervised baseline is weak, and it's hard to tell if the gains from VAE will retain if applied to a stronger supervised.
3. A performance curve with different amount of labeled and unlabeled data would be useful to better understand the impact of semi-supervised learning.
4. What's the impact of perturbation? One could simply use T=Eisner(W) as approximation. Did you observe any significant benefits from sampling?
Other questions
1. What's the impact of keeping the tree constraint on dependencies during backpropagation?  Have you tried removing the tree constraint like previous work?
2. Are sentence embedding and trees generated from two separate LSTM encoders? Are there any parameter sharing between the two?