Published as a conference paper at ICLR 2018 DEPTHWISE SEPARABLE CONVOLUTIONS FOR NEURAL MACHINE TRANSLATION
Depthwise separable convolutions reduce the number of parameters and computation used in convolutional operations while increasing representational efficiency. They have been shown to be successful in image classification models, both in obtaining better models than previously possible for a given parameter count (the Xception architecture) and considerably reducing the number of parameters required to perform at a given level (the MobileNets family of architectures). Recently, convolutional sequence-to-sequence networks have been applied to machine translation tasks with good results. In this work, we study how depthwise separable convolutions can be applied to neural machine translation. We introduce a new architecture inspired by Xception and ByteNet, called SliceNet, which enables a significant reduction of the parameter count and amount of computation needed to obtain results like ByteNet, and, with a similar parameter count, achieves better results. In addition to showing that depthwise separable convolutions perform well for machine translation, we investigate the architectural changes that they enable: we observe that thanks to depthwise separability, we can increase the length of convolution windows, removing the need for filter dilation. We also introduce a new "super-separable" convolution operation that further reduces the number of parameters and computational cost of the models.
The paper proposes to use depthwise separable convolution layers in a fully convolutional neural machine translation model. The authors also introduce a new "super-separable" convolution layer, which further reduces the computational cost of depthwise separable convolutions. Results are presented on the WMT English to German translation task, where the method is shown to perform second-best behind the Transformer model.
The paper's greatest strength is in my opinion the quality of its exposition of the proposed method. The relationship between spatial convolutions, pointwise convolutions, depthwise convolutions, depthwise separable convolutions, grouped convolutions, and super-separable convolutions is explained very clearly, and the authors properly introduce each model component.
Perhaps as a consequence of this, the experimental section feels squeezed in comparison. Quantitative results are presented in two fairly dense tables (especially Table 2) which, although parsable after reading the paper carefully, could benefit from a little bit more information on how they should be read. The conclusions that are drawn in the text are stated without citing metrics or architectural configurations, leaving it up to the reader to connect the conclusions to the table contents.
Overall, I feel that the results presented make a compelling case both for the effectiveness of depthwise separable convolutions and larger convolution windows, as well as the overall performance achievable by such an architecture. I think the paper constitutes a good contribution, and adjustments to the experimental section could make it a great contribution.