Attention based Transformer architecture has enabled significant advances in the field of natural language processing. In addition to new pre-training techniques, recent improvements crucially rely on working with a relatively larger embedding dimension for tokens. Unfortunately, this leads to models that are prohibitively large to be employed in the downstream tasks. In this paper we identify one of the important factors contributing to the large embedding size requirement. In particular, our analysis highlights that the scaling between the number of heads and the size of each head in the existing architectures gives rise to this limitation, which we further validate with our experiments. As a solution, we propose a new way to set the projection size in attention heads that allows us to train models with a relatively smaller embedding dimension, without sacrificing the performance.
This work studies the head size <--> head number tradeoff in multihead attention. It argues and formally establishes that (1) the expressivity of an attention head is determined by its dimension and (b) fixing the head dimension, one gains additional expressive power by using more heads. In response to such observations, the paper proposes Fixed Multihead Attention, where the constraint that `head_size * number_of_heads = embedding_size` in standard multihead attention is lifted; and it allows for using more attention heads without making each head smaller. One can control the total amount of parameters by using smaller embedding sizes, making it comparable (in terms of #parameters) to standard multihead attention. Empirical results on language modeling and NLI tasks confirms the arguments. 
Pros:
- The arguments on head size and head number tradeoff could be inspiring to future works.
- A simple approach that proves strong in several NLP tasks.
Cons:
- The theoretical discussion imposes too strong assumptions that might make it less interesting in practice.
- No NMT experiments.
- The takeaway seems a bit trivial.
Details:
- Theorem 1 presents a rank-based view of each attention head's capacity, which is nice. Yet it is still unclear whether it is the case that the more expressive are the heads the better. For example, several recent works argues for specialized attention heads, i.e., each head has specific "job," which may not require it being very expressive [1, 2]. Further, other works shows that a low-rank P matrix could be beneficial [3, 4, 5], which contradicts the argument in this work. It would be nice if the authors and discuss this in the revision
(To be clear, I do believe this is still an open question, and do not think presenting a different view from previous works hurts the contribution of this work in any way.)
- Theorem 2. I didn't carefully check the proof. Why is it required that the V matrices for each head have the same product. For both Thm.1 and 2, it would be nice to see some discussion on how they translate into the models in practice.
- Can the authors compare the training/inference speed? It probably will be the same as standard transformers, but it would be nice to confirm.
- Figure 1: the caption says trying out embedding sizes from 256 to 512. But it seems that only 4 values are tried. Can the authors comment on this? Also, it is a bit awkward to plot a line chart out of 4 points. Same for Figure 2.
- It would be nice to see some NMT experiments. 
- The proposed method is so straightforward that I'm actually very surprised that this paper is the first trying this. The authors might need justify the technical contribution more.
(I'm on the fence for this one, but the system doesn't allow me to. I'm happy to revise the score if the authors can address my concerns.)
[1] Analyzing Multi-Head Self-Attention: Specialized Heads Do the Heavy Lifting, the Rest Can Be Pruned. https://arxiv.org/abs/1905.09418
[2] Are Sixteen Heads Really Better than One? https://arxiv.org/abs/1905.10650
[3] Generating Long Sequences with Sparse Transformers. https://arxiv.org/abs/1904.10509
[4] Generating Long Sequences with Sparse Transformers. https://arxiv.org/pdf/1904.10509.pdf.
[5] Adaptively Sparse Transformers. https://arxiv.org/abs/1909.00015.