Attention mechanisms have advanced the state of the art in several machine learning tasks. Despite significant empirical gains, there is a lack of theoretical analyses on understanding their effectiveness. In this paper, we address this problem by studying the landscape of population and empirical loss functions of attentionbased neural networks. Our results show that, under mild assumptions, every local minimum of a two-layer attention model has low prediction error, and attention models require lower sample complexity than models not employing attention. Additionally for the popular self-attention, our theoretical results provide several guidelines for designing attention mechanisms. Our findings are validated with satisfactory experimental results on MNIST and IMDB reviews dataset.
Summary: This paper endeavors to develop some theoretical understanding of why attention mechanisms improve the performance of neural networks. It includes some theory around the sample complexity of models which have a multiplicative mask in them, as well as a few related experiments.
Review: Having some theoretical analysis of the benefits of attention is definitely needed and I commend the authors for working towards this goal. The main issue that I have with the paper is that the model initially considered (introduced in section 2) is unrelated to attention mechanisms because the attention weights "a" do not depend on the input in any way. If I understand correctly, the ground-truth function y_i = f*(a* . x_i) assumes there is a single globally-correct attention mask a* for all examples. This is not attention. Every attention mechanism that I am aware of depends on the input in some way (in other words, instead of a*, you'd have a function a*(x_i) which changes according to the input). Some additional discussion is needed to explain why it is interesting to study this dramatically simplified (to the point of not being relevant) model. For example, if it was explicitly described as a preliminary for the a*(x_i) case (section 3.2 onwards) it would make more sense. Similarly, the assumption ||a_0|| ≤ s_0 is not true for most attention masks in pratice. There are some hard attention mechanisms which try to induce binary masks/hard sparsity but they are extremely rarely used in practice. The combination of these two assumptions makes for what appears to me (if I'm understanding correctly) a somewhat trivial result that the "attention mechanisms constrain the parameters in a smaller space". Note that this criticism does not apply to section 3.2 where the attention mask is specified as a function of the input. Similarly section 4.2 relies on the "single attention mask" idea, which is the sole result considering generalization in the paper. Finally 5.1.1 consists of an experiment which assumes a model of the form G*(x . a*), where a* is a fixed random binary mask. Training a model to learn G* and a* has very little to do with attention - it is a learning problem where you have removed a subset of the features and are trying to determine the parameters of the function and which features were removed. The L_1 penalty on "a" is certainly a good way to learn to only include a sparse subset of your features, but again has very little relevance to most attention mechanisms (i.e. I know of almost no work which regularizes the attention mask in this way).
Overall I think most of the results in this paper are not relevant to attention models and as such it has limited impact. I would suggest reframing the paper around a focus on what the authors call the "self-attention" case (which is really just normal attention), possibly using the fixed a* case as a preliminary/motivation. I also would suggest removing the discussion of regularization the attention mask as I am not aware of this being used in practice. Finally I would suggest some more direct discussion of how the results apply to models which are used in practice.