A Regularized Framework for Sparse and Structured Neural Attention
Modern neural networks are often augmented with an attention mechanism, which tells the network where to focus within the input. We propose in this paper a new framework for sparse and structured attention, building upon a smoothed max operator. We show that the gradient of this operator defines a mapping from real values to probabilities, suitable as an attention mechanism. Our framework includes softmax and a slight generalization of the recently-proposed sparsemax as special cases. However, we also show how our framework can incorporate modern structured penalties, resulting in more interpretable attention mechanisms, that focus on entire segments or groups of an input. We derive efficient algorithms to compute the forward and backward passes of our attention mechanisms, enabling their use in a neural network trained with backpropagation. To showcase their potential as a drop-in replacement for existing ones, we evaluate our attention mechanisms on three large-scale tasks: textual entailment, machine translation, and sentence summarization. Our attention mechanisms improve interpretability without sacrificing performance; notably, on textual entailment and summarization, we outperform the standard attention mechanisms based on softmax and sparsemax.
Summary
=======
  This paper presents a framework for implementing different sparse
  attention mechanisms by regularizing the max operator using convex
  functions. As a result, softmax and sparsemax are derived as special
  cases of this framework. Furthermore, two new sparse attention
  mechanisms are introduced that allow the model to learn to pay the
  same attention to contiguous spans. My concerns are regarding to the
  motivation of interpretability, as well as the baseline attention
  models. However, the paper is very well presented and the framework is
  a notable contribution that I believe will be useful for researchers
  working with attention mechanisms.
Strengths
=========
  - The provided framework encompasses softmax and sparsemax as special
    cases.
  - It is a drop-in replacement for softmax, so it can be readily used
    for extending a variety of existing neural networks that employ a
    softmax attention mechanism.
  - There seems to only be a small computational overhead (~25% increase
    in runtime).
  - While the attention weights produced by the sparse attention models
    are slightly more interpretable, using such sparse attention didn't
    hurt performance on RTE, MT and sentence summarization tasks.
Weaknesses
==========
  - Looking at Fig 1 and playing Devil's advocate, I would argue that
    interpretability doesn't really seem to improve that much with the
    variants of sparse attention presented here -- I could just
    threshold the weights of a normal softmax attention mechanism (or
    increase the contrast in Fig 1) and I would get a similar outcome. I
    do see however that with span-based sparse attention things start to
    look more interpretable but I have the impression that the three
    tasks are not the best fit for it. For instance, Q&A on SQuAD might
    have been a more interesting task as it asks for a span in a
    paragraph as the answer for a given question. This might have
    alleviated another weakness of the evaluation, namely that we don't
    see strong improvements when using span-based sparse attention. So
    right now the main takeaway is that we can obtain similar
    performance with slightly improved interpretability -- where I am
    still unsure how practically relevant the latter is.
  - Related to the point above, I do believe it is important to point
    out that interpretability of attention weights is problematic as the
    attentive RTE and MT (and maybe the summarization model too?) are
    not solely relying on the context vector that is obtained from the
    attention mechanism. Hence, any conclusions drawn from looking at
    the attention weights should be taken with a big grain of salt.
  - Moreover, the RTE attention baseline seems weak (though I understand
    that model is inherited from sparsemax). [35] report a test set
    accuracy of 83.5% whereas the baseline attentive RTE model here only
    achieves 81.66%, and the best performance for fusedmax reported here
    is 82.41%. In fact, what I would have liked to see is a comparison
    of a state-of-the-art attentive RTE/MT model with a drop-in
    replacement of fusedmax/oscarmax and an improvement in
    accuracy/BLEU.
Minor Comments
==============
  - I think it would be great if the limitations of this framework could
    be highlighted a bit more. For instance, I am wondering whether your
    framework could also support more global structured penalties such
    as the prior in MT that most of the attention should be along the
    diagonal and my hunch is that this would be difficult. See Cohn,
    Trevor, et al. "Incorporating structural alignment biases into an
    attentional neural translation model." arXiv preprint
    arXiv:1601.01085 (2016).
  - I believe in section 2 it could be made clearer when gradients are
    calculated by a solver and when not.