Many tasks in natural language processing involve comparing two sentences to compute some notion of relevance, entailment, or similarity. Typically this comparison is done either at the word level or at the sentence level, with no attempt to leverage the inherent structure of the sentence. When sentence structure is used for comparison, it is obtained during a non-differentiable pre-processing step, leading to propagation of errors. We introduce a model of structured alignments between sentences, showing how to compare two sentences by matching their latent structures. Using a structured attention mechanism, our model matches possible spans in the first sentence to possible spans in the second sentence, simultaneously discovering the tree structure of each sentence and performing a comparison, in a model that is fully differentiable and is trained only on the comparison objective. We evaluate this model on two sentence comparison tasks: the Stanford natural language inference dataset and the TREC-QA dataset. We find that comparing spans results in superior performance to comparing words individually, and that the learned trees are consistent with actual linguistic structures.
This paper describes the use of latent context-free derivations, using
a CRF-style neural model, as a latent level of representation in neural
attention models that consider pairs of sentences. The model implicitly
learns a distribution over derivations, and uses marginals under this
distribution to bias attention distributions over spans in one sentence
given a span in another sentence.
This is an intriguing idea. I had a couple of reservations however:
* The empirical improvements from the method seem pretty marginal, to the
point that it's difficult to know what is really helping the model. I would
liked to have seen more explanation of what the model has learned, and
more comparisons to other baselines that make use of attention over spans.
For example, what happens if every span is considered as an independent random
variable, with no use of a tree structure or the CKY chart?
* The use of the \alpha^0 vs. \alpha^1 variables is not entirely clear. Once they
have been calculated in Algorithm 1, how are they used? Do the \rho values
somewhere treat these two quantities differently?
* I'm skeptical of the type of qualitative analysis in section 4.3, unfortunately.
I think something much more extensive would be interesting here. As one
example, the PP attachment example with "at a large venue" is highly suspect;
there's a 50/50 chance that any attachment like this will be correct, there's
absolutely no way of knowing if the model is doing something interesting/correct
or performing at a chance level, given a single example.