Transformer has become ubiquitous in natural language processing (e.g., machine translation, question answering); however, it requires enormous amount of computations to achieve high performance, which makes it not suitable for real-world mobile applications since mobile phones are tightly constrained by the hardware resources and battery. In this paper, we investigate the mobile setting (under 500M Mult-Adds) for NLP tasks to facilitate the deployment on the edge devices. We present Long-Short Range Attention (LSRA), where some heads specialize in the local context modeling (by convolution) while the others capture the long-distance relationship (by attention). Based on this primitive, we design Mobile Transformer (MBT) that is tailored for the mobile NLP application. Our MBT demonstrates consistent improvement over the transformer on three well-established language tasks: IWSLT 2014 German-English, WMT 2014 English-German and WMT 2014 English-French. It outperforms the transformer by 0.9 BLEU under 500M Mult-Adds and 1.2 BLEU under 100M Mult-Adds on WMT'14 English-German. On WMT'14 English-French, our MBT reduces the computation of the transformer by 2.5× with negligible BLEU degradation. Without the costly architecture search that requires more than 250 GPU years, our MBT achieves 0.5 higher BLEU than the AutoML-based Evolved Transformer under the mobile setting.
This paper presents a new technique (LSRA) improving Transformer for constrained scenarios (e.g., mobile settings). It combines two attention modules to provide both global and local information separately for a translation task. In this manner, the authors place the attention and the convolutional module side by side, thus having different perspectives (globally and locally) of the sentence. They test their approach to 2 common translation benchmarks.
Enhancing deep learning model efficiency is very important, and the authors succeeded in reducing the computation costs and consequently, the CO2 emissions. But the evaluation results are not so impressive and go in line with other previous efficient deep learning approaches for different domains. I’m not an expert in NLP, but overall results of 10-1000x or more wall clock time reduction with <1-5% loss in accuracy are usually obtained for domains that have seen more optimization for mobile deployment (especially mobile-optimized CNNs like MobileNet). LSRA-based appraoch is slightly better than the original version of Transformer and its evolved version. From the latter (ET) authors seem to take the idea of parallel branches for their architecture. Also, adaptive attention span in Transformer models and all-attention layers have already been investigated to make networks more efficient and simpler for longer sentences. Include clearer ablation studies would be also interesting to support their findings and superior performance.
To summarize, the paper is addressing an important and interesting idea. It is, in general terms a nice engineering paper, but I am not sure about whether the developments and results are relevant/novel enough yet at this point to publish at ICLR. 
-------------------------------------------------
Looking at the other comments and the feedback provided by the authors, I have a more positive feeling about the contributions of the paper which are now sufficiently demonstrated. Therefore, I increase my original recommendation to "Weak Accept".