Attention mechanisms in sequence to sequence models have shown great ability and wonderful performance in various natural language processing (NLP) tasks, such as sentence embedding, text generation, machine translation, machine reading comprehension, etc. Unfortunately, existing attention mechanisms only learn either high-level or low-level features. In this paper, we think that the lack of hierarchical mechanisms is a bottleneck in improving the performance of the attention mechanisms, and propose a novel Hierarchical Attention Mechanism (Ham) based on the weighted sum of different layers of a multi-level attention.
Ham achieves a state-of-the-art BLEU score of 0.26 on Chinese poem generation task and a nearly 6.5% averaged improvement compared with the existing machine reading comprehension models such as BIDAF and Match-LSTM. Furthermore, our experiments and theorems reveal that Ham has greater generalization and representation ability than existing attention mechanisms.
The paper proposes to enhance existing multi-level attention (self-attention) mechanism by obtaining query and key vectors (= value vectors) from all levels after weighted-averaging them. The paper claims that this is also theoretically beneficial because the loss function will converge to zero as the number of layers increase. It claims that the proposed architecture outperforms existing attention-based models in English MRC test (SQuAD), Chinese MRC test, and Chinese poem generation task.
I find three major issues in the paper.
1.  I think the proposed hypothesis lacks the novelty that ICLR audience seeks for. Through many existing architectures (ResNet, ELMo), we already know that skip connection between CNN layers or weighted average of multiple LSTM layers could improve model significantly. Perhaps this could be an application paper that brings existing methods to a slightly different (attention) domain, but not only such paper is less suitable for ICLR, but also it would require strong experimental results. But as I will detail in the second point, I also have some worries about the experiments. 
2. The experimental results have problems. For English MRC experiment (SQuAD), the reproduced match-LSTM score is ~10% below the reported number in its original paper. Furthermore, it is not clear whether the improvement comes from having multiple attention layers (which is not novel) or weighted-averaging the attention layers (the proposed method). BiDAF and match-LSTM have single attention layers, so it is not fair to compare them with multi-layer attention. 
3. Lastly, I am not sure I understood the theoretical section correctly, but it is not much interesting that having multiple layers allow one to approach closer to zero loss. In fact, any sufficiently large model can obtain close-to-zero loss on the training data. This is not a sufficient condition for a good model. We cannot guarantee if the model has generalized well; it might have just overfit to the training data.
A few minor issues and typos  on the paper:
- First para second sentence: In -> in
- First para second sentence: sequence to sequence -> sequence-to-sequence
- Second last para of intro: sentence fragment
- Figure 3: would be good to have English translation.