The Transformer architecture is widely used in natural language processing. Despite its success, the design principle of the Transformer remains elusive. In this paper, we provide a novel perspective towards understanding the architecture: we show that the Transformer can be mathematically interpreted as a numerical Ordinary Differential Equation (ODE) solver for a convection-diffusion equation in a multi-particle dynamic system. In particular, how words in a sentence are abstracted into contexts by passing through the layers of the Transformer can be interpreted as approximating multiple particles' movement in the space using the Lie-Trotter splitting scheme and the Euler's method. Given this ODE's perspective, the rich literature of numerical analysis can be brought to guide us in designing effective structures beyond the Transformer. As an example, we propose to replace the Lie-Trotter splitting scheme by the Strang-Marchuk splitting scheme, a scheme that is more commonly used and with much lower local truncation errors. The Strang-Marchuk splitting scheme suggests that the self-attention and position-wise feed-forward network (FFN) sub-layers should not be treated equally. Instead, in each layer, two position-wise FFN sub-layers should be used, and the self-attention sub-layer is placed in between. This leads to a brand new architecture. Such an FFN-attention-FFN layer is "Macaron-like", and thus we call the network with this new architecture the Macaron Net. Through extensive experiments, we show that the Macaron Net is superior to the Transformer on both supervised and unsupervised learning tasks. The reproducible code can be found on http://anonymized
In this work, the authors show that the sequence of self-attention and feed-forward layers within a Transformer can be interpreted as an approximate numerical solution to a set of coupled ODEs. Based on this insight, the authors propose to replace the first-order Lie-Trotter splitting scheme by the more accurate, second-order Strang splitting scheme. They then present experimental results that indicate an improved performance of their Macaron Net compared to the Transformer and argue that this is due to the former being a more accurate numerical solution to the underlying set of ODEs.
The authors highlight an interesting connection between the Transformer architecture and ODEs. In particular, they derive a set of ODEs that is solved numerically by the Transformer and borrow from the body of literature on numerical ODE solvers to improve the architecture. I find that this is a very elegant and promising approach for finding better architectures. 
However, I also identified two major and a couple of minor shortcomings of the paper that are explained in detail below. Based on these shortcomings, I recommend rejecting the paper but I would be willing to increase the score if these points were addressed in sufficient detail.
Major points:
1) Replacing the first-order operator splitting scheme by a second-order scheme only guarantees a lower overall truncation error if the split ODEs are solved with sufficiently high accuracy. In particular, the overall accuracy of the numerical solution to the original ODE depends on the accuracy of the operator splitting and the accuracy of the integration scheme used to solve the split ODEs (e.g. Euler’s method). The authors improve the operator splitting, i.e. they make it second-order, but they keep Euler’s method to integrate the individual ODEs. Because of that, they actually do not get rid of the lowest-order error term of the overall scheme and therefore do not obtain a more accurate ODE solver. I think this is a crucial point that invalidates the authors’ claim that the Macaron Net employs a higher-order integration scheme. As far as I am aware, this is not commented on in the paper at all. To address this shortcoming, the authors could replace Euler’s method by a second-order integrator. 
2) The experiments considered in this paper are interesting and show competitive performance but, in my opinion, they do not sufficiently support the claim that the Macaron Net yields a more accurate solution to the underlying set of ODEs compared to a Transformer. For a more convincing support of this claim, the authors could consider a toy problem, i.e. a simple set of ODEs with known analytical solution, and actually show that the Macaron Net is more accurate. The accuracy of a numerical ODE solver is commonly assessed by plotting the absolute difference between the exact solution (or a high-resolution numerical approximation to it) and the numerical solution vs the timestep (here \gamma). I suspect that such an analysis would support my previous comment and show that the proposed new architecture is not more accurate ODE solver than the original one.  
Minor points and questions:
i) Eqs. (17-19) suggest that you apply two different FFN layers (doubling the number of parameters) instead of applying the same FFN layer twice. You comment on this in Sec. 4.1 when you say ‘..., we set the dimensionality of the inner-layer of the two FFN sub-layers in the Macaron layers to two times of the dimensionality of the hidden states’. It is not clear to me why consistency with Strang splitting requires two different layers rather than applying the same FFN layer twice. Is the reason for having a separate, trainable layer to account for the explicit time dependence of G in Eq. (16)? I think that this is a very important point that should be clarified.
ii) I think that this type of system is usually referred to as ‘dynamical system’ and not ‘dynamic system’. Please check that and, if applicable, update the title.  
iii) The authors say that Eq. (5) is a 'convection-diffusion equation’. As far as I am aware, the diffusion equation is a partial differential equation (PDE). Perhaps there is a different notion 'diffusion equation’ in ODE theory. If that’s the case, could the authors please clarify this point to avoid confusion, e.g. by adding a suitable reference in which this type of ODE is classified as a convection-diffusion equation? 
iv) In Sec. 2 (2nd paragraph),  you cite Vaswani et al. (2017) but in that work the quantity under the square-root in the denominator of Attention(Q, K, V) is actually d_k, the dimension of the key, and not d_model.
v) Figure 1 is a very vague illustration of the connection to ODEs and provides almost no explanation in the caption. I don’t think there is much value in having this figure there.
vi) There are a couple of mistakes in the paper (grammar and expressions) that should be fixed. For example, ‘the Euler’s method’ instead of ‘Euler’s method’, ‘movement in the space’ instead of ‘movement in space’, ‘dynamic system’ instead of ‘dynamical system’, ‘project parameter matrices’ instead of ‘parameter matrices’ or ‘projections’, ‘specially’ instead of ‘specifically’, etc. Please take a look at the relevant sections in the paper and revise them accordingly.
vii) You explain multiple times why the proposed architecture is called a ‘Macaron Net’ (Abstract, Sec 1, Sec. 3). To avoid repetition, I would only explain it once.