Residual networks are an Euler discretization of solutions to Ordinary Differential Equations (ODE). This paper explores a deeper relationship between Transformer and numerical ODE methods. We first show that a residual block of layers in Transformer can be described as a higher-order solution to ODE. Inspired by this, we design a new architecture, ODE Transformer , which is analogous to the Runge-Kutta method that is well motivated in ODE. As a natural extension to Transformer, ODE Transformer is easy to implement and efficient to use. Experimental results on the large-scale machine translation, abstractive summarization, and grammar error correction tasks demonstrate the high genericity of ODE Transformer. It can gain large improvements in model performance over strong baselines (e.g., 30.77 and 44.11 BLEU scores on the WMT'14 English-German and English-French benchmarks) at a slight cost in inference efficiency. Residual networks have been used with a great success as a standard method of easing information flow in multi-layer neural models (He et al., 2016; Vaswani et al., 2017). Given an input y t , models of this kind define the output of a layer t to be: y t +1 = y t + F ( y t ,  t ) (1) where F (  ,  ) is the function of the layer and  t is its parameter. Interestingly, recent work in machine learning (Weinan, 2017; Lu et al., 2018; Haber et al., 2018; Chang et al., 2018; Ruthotto and Haber, 2019) points out that Eq. (1) is an Euler discretization of the Ordinary Differential Equation (ODE), like this: d y ( t ) d t = F ( y ( t ) ,  ( t )) (2)  Corresponding author. where y ( t ) and  ( t ) are continuous with respect to t . In this way, we can call Eq. (1) an ODE block . This finding offers a new way of explaining residual networks in the view of numerical algorithms. Then, one can think of a multi-layer network as applying the Euler method (i.e., Eq. (1)) to solve Eq. (2) subject to the initial conditions y (0) = y 0 and  (0) =  0 . The solution of Eq. (2) has a sufficiently low error bound (call it a stable solution ) only if  ( t ) changes slow along t (Haber and Ruthotto, 2017; Chen et al., 2018). But this assumption does not always hold for state-of-the-art natural language processing (NLP) systems, in which models are non-linear and over-parameterized. For example, language modeling and machine translation systems learn quite different parameters for different layers, especially when the layers are close to the model input (Vaswani et al., 2017; Dai et al., 2019). Also, truncation errors are nonnegligible for the Euler method because it is a first-order approximation to the true solution (He et al., 2019). These problems make the situation worse, when more layers are stacked and errors are propagated through the neural network. It might explain why recent 8335 Machine Translation (MT) systems cannot benefit from extremely deep models (Wang et al., 2019; Liu et al., 2020a; Wei et al., 2020; Li et al., 2020). This paper continues the line of research on the ODE-inspired method. The basic idea is to use a high-order method for more accurate numerical solutions to the ODE. This leads to a larger ODE block that generates a sequence of intermediate approximations to the solution. We find that the larger ODE block is sufficient to take the role of several ODE blocks with first-order solutions. The benefit is obvious: the use of fewer ODE blocks lowers the risk of introducing errors in block switching, and the high-order method reduces the approximation error in each ODE block. See Figure 1 for a comparison of different models. Our method is parameter-efficient because  ( t ) is re-used within the same ODE block. As another bonus", the model can be improved by learning coefficients of different intermediate approximations in a block. We evaluate our method in strong Transformer systems, covering both the wide (and big) model and the deep model. For machine translation tasks, ODE Transformer achieves 30.77 and 44.11 BLEU scores on the WMT'14 En-De and En-Fr test sets, setting a new state-of-the-art on the WMT'14 En-Fr task. It also significantly outperforms baselines on abstractive summarization and grammar error correction tasks. 2 Transformer and ODEs We start with a description of Transformer, followed by its relationship with ODEs. We choose Transformer for our discussion and experiments because it is one of the state-of-the-art models in recent sentence generation tasks. 2.1 Transformer Transformer is an example of the encoder-decoder paradigm (Vaswani et al., 2017). The encoder is a stack of identical layers. Each layer consists of a self-attention block and a feedforward network (FFN) block. Both of them equip with a residual connection and a layer normalization unit. Note that the term block is used in many different ways. In this paper, the term refers to any neural network that is enhanced by the residual connection (occasionally call it a residual block ). Following the Pre-norm architecture (Wang et al., 2019), we define a block as y t +1 = y t + G ( LN ( y t ) ,  t ) (3) where LN (  ) is the layer normalization function, 1 and G (  ) is either the self-attention or feedforward network. The decoder shares a similar architecture, having an additional encoder-decoder attention block sandwiched between the self-attention and FFN blocks. An ordinary differential equation is an equation involving a function y ( t ) of a variable t and its derivatives. A simple form of ODE is an equation that defines the first-order derivative of y ( t ) , like d y ( t ) d t = f ( y ( t ) , t ) (4) where f ( y ( t ) , t ) defines a time-dependent vector field if we know its value at all points of y and all instants of time t . Eq. (4) covers a broad range of problems, in that the change of a variable is determined by its current value and a time variable t . This formulation also works with Pre-norm Transformer blocks. For notational simplicity, we re-define G ( LN ( y t ) ,  t ) as a new function F ( y t ,  t ) : F ( y t ,  t ) = G ( LN ( y t ) ,  t )) (5) We then relax y t and  t to continuous functions y ( t ) and  ( t ) , and rewrite Eq. where  t is the change of t , and is general called step size . Obviously, we have  t = 1 in Transformer. But we can adjust step size  t using a limit, and have lim  t  0 y ( t +  t )  y ( t )  t = F ( y ( t ) ,  ( t )) (7) Given the fact that lim  t  0 y ( t + t )  y ( t )  t = d y ( t ) d t , Eq. (7) is an instance of Eq. (4). The only difference lies in that we introduce  ( t ) into the righthand side of Eq. (4). Then, we say that a Pre-norm Transformer block describes an ODE. It has been found that Eq. (3) shares the same form as the Euler method of solving the ODE described in Eq. (7) (Haber and Ruthotto, 2017). This establishes a relationship between Transformer and ODEs, in that, given F (  ,  ) and learned parameters {  t } , the forward pass of a multi-block Transformer is a process of running the Euler method for several steps. In numerical methods of ODEs, we want to ensure the precise solutions to the ODEs in a minimum number of computation steps. But the Euler method is not precise because it is a first-order method, and naturally with local truncation errors. The global error might be larger if we run it for a number of times. 2 This is obviously the case for Transformer, especially when the multi-layer neural network arises a higher risk of instability in solving the ODEs (Haber and Ruthotto, 2017). Here we use the Runge-Kutta methods for a higher order solution to ODEs (Runge, 1895; Kutta, 1901; Butcher, 1996; Ascher and Petzold, 1998). They are a classic family of iterative methods with different orders of precision. 3 More formally, the explicit Runge-Kutta methods of an n -step solution is defined to be: y t +1 = y t + n (cid:88) i =1  i F i (8) F 1 = hf ( y t , t ) (9) F i = hf ( y t + i  1 (cid:88) j =1  ij F j , t +  i h ) (10) where h is the step size and could be simply 1 in most cases. F i is an intermediate approximation to the solution at step t +  i h . ,  and  are coefficients which can be determined by the Taylor series of y t +1 (Butcher, 1963). Eq. (10) describes a sequence of solution approximations { F 1 , ..., F n } over n steps { t +  1 h, ..., t +  n h } . These approximations are then interpolated to form the final solution, as in Eq. (8). The Runge-Kutta methods are straightforwardly applicable to the design of a Transformer block. All we need is to replace the function f (see Eq. (10)) with the function F (see Eq. (5)). The advantage is that the function F is re-used in a block. Also, the model parameter  t can be shared within the block. 4 In this way, one can omit t +  i h in Eq. 2 The global error is what we would ordinarily call the error: the difference between y ( t ) and the true solution. The local error is the error introduced in a single step: the difference between y ( t ) and the solution obtained by assuming that y ( t  1) is the true solution 3 A p -order numerical method means that the global truncation error is proportional to p power of the step size. 4 Although we could distinguish the parameters at different steps in a block, we found that it did not help and made the model difficult to learn. This makes the system more parameter-efficient. As would be shown in our experiments, the high-order Runge-Kutta methods can learn strong NMT systems with significantly smaller models. The Runge-Kutta methods are general. For example, the Euler method is a first-order instance of them. For a second-order Runge-Kutta (RK2) block, we have y t +1 = y t + 1 2( F 1 + F 2 ) (12) F 1 = F ( y t ,  t ) (13) F 2 = F ( y t + F 1 ,  t ) (14) This is also known as the improved Euler method. Likewise, we can define a fourth-order Runge-Kutta (RK4) block to be: y t +1 = y t + 1 6( F 1 + 2 F 2 + 2 F 3 + F 4 ) (15) F 1 = F ( y t ,  t ) (16) F 2 = F ( y t + 1 2 F 1 ,  t ) (17) F 3 = F ( y t + 1 2 F 2 ,  t ) (18) F 4 = F ( y t + F 3 ,  t ) (19) See Figure 2 for a comparison of different Runge-Kutta blocks. It should be noted that the method presented here can be interpreted from the perspective of representation refinement (Greff et al., 2017). It provides a way for a function to update the function itself. For example, Universal Transformer refines the representation of the input sequence using the same function and the same parameters in a block-wise manner (Dehghani et al., 2019). Here we show that inner block refinements can be modeled with good theoretical support. In our preliminary experiments, the RK2 and RK4 methods yielded promising BLEU improvements when the model was shallow. But it was found that the improvements did not persist for deeper models. To figure out why this happened, let us review the Runge-Kutta methods from the angle of training. Take the RK2 method as an example. We rewrite Eq. (12) by substituting F 1 and F 2 , as follow y t +1 = y t + 1 2 F ( y t ,  t ) + 1 2 F ( y t + F ( y t ,  t ) ,  t ) (20) Let E be the loss of training, L be the number blocks of the model, and y L be the model output. The gradient of E at y t is  E y t =  E y L  1 2 L  t  L  1 (cid:89) k = t (1 + g k ) (21) where g k = (cid:16) 1 + F ( y k ,  k ) y k (cid:17)  (cid:16) 1 + F ( y k + F ( y k ,  k ) ,  k ) y k + F ( y k ,  k ) (cid:17) (22) Seen from Eq. (21),  E y t is proportional to the factor 1 2 L  t . This leads to a higher risk of gradient vanishing when L is larger. The problem somehow attributes to the small coefficients of F i , that is,  1 =  2 = 12 . A natural idea is to empirically set  i = 1 to eliminate the product factor of less than 1 in gradient computation, although this is not theoretically grounded in standard Runge-Kutta methods. We rewrite Eq. (20) with the new coefficients, as follows y t +1 = y t + F ( y t ,  t ) + F ( y t + F ( y t ,  t ) ,  t ) (23) Then, we have the gradient, like this  E y t =  E y L  L  1 (cid:89) k = t g k (24) This model is easy to optimize because  E  yL can be passed to lower-level blocks with no scales. Note that, the methods here are instances of parameter sharing (Dehghani et al., 2019; Lan et al., 2020). For example, in each ODE block, we use the same function F with the same parameter  t for all intermediate steps. Setting  i = 1 is a further step towards this because F i is passed to the following computations with the same scale. Here we call it implicit parameter sharing. Another way of scaling F i to further improve ODE functions is to learn the coefficients automatically on the training data. The simplest method is to initialize  i = 1 and independently optimize each scale. It helps the system learn the way of flowing F i in a block. Based on it, scaling F i by a weighted gate mechanism (Srivastava et al., 2015) empirically achieves the best performance (see Section 4). Take RK2-block as an instance, the concatenation of F 1 and F 2 is transformed to a scalar (0 , 1) through a sigmoid gate, then the block output y t +1 is y t +1 = y t + g  F 1 + (1  g )  F 2 (25) g = sigmoid ([ F 1 , F 2 ]  W + b ) (26) where [ , ] denotes the concatenation operation and W, b are learnable parameters. We call it RK2-block (learnable  i ), and the architecture is shown in Figure 2 (d). This kind of formulation offers a more flexible way to decide which part contributes more and is also easy to be optimized. Moreover, we also summarize the comparison of various scaling functions in Appendix C. 8338 Model Layers WMT En-De WMT En-Fr #Param Steps BLEU SBLEU #Param Steps BLEU SBLEU Transformer (Vaswani et al., 2017) 6-6 213M 100K 28.40 222M 300K 41.00 MacaronNet (Lu et al., 2019) 6-6 -30.20 --Depth growing (Wu et al., 2019) 8-8 270M 800K 29.92 -43.27 Transformer-DLCL (Wang et al., 2019) 30-6 137M 50K 29.30 28.6 --Multiscale Collaborative (Wei et al., 2020) 18-6 512M 300K 30.56 --ADMIN (Liu et al., 2020a) 60-12 262M 250K 30.01 29.5 250K 43.80 41.8 SDT (Li et al., 2020) 48-6 192M 50K 30.21 29.0 198M 100K 43.28 41.5 BERT-fused model (Zhu et al., 2020) 6-6 -30.75 -43.78 Base and Deep Models Residual-block 6-6 61M 50K 27.89 26.8 69M 100K 41.05 39.1 RK2-block 6-6 61M 50K 28.67 27.5 69M 100K 42.08 40.1 RK2-block (learnable  i ) 6-6 61M 50K 28.89 27.7 69M 100K 42.31 40.3 RK4-block 6-6 61M 50K 29.03 27.9 69M 100K 42.56 40.6 Residual-block 24-6 118M 50K 29.43 28.3 123M 100K 42.67 40.6 RK2-block 24-6 118M 50K 29.85 28.7 123M 100K 43.04 41.1 RK2-block (learnable  i ) 24-6 118M 50K 30.29 29.2 123M 100K 43.48 41.5 RK4-block 24-6 118M 50K 29.80 28.8 123M 100K 43.28 41.3 Wide Models Residual-block-Big 6-6 211M 100K 29.21 28.1 221M 100K 42.89 40.9 RK2-block 6-6 211M 100K 30.11 29.0 221M 100K 43.34 41.3 RK2-block (learnable  i ) 6-6 211M 100K 30.53 29.4 221M 100K 43.59 41.6 RK4-block 6-6 211M 100K 30.39 29.3 221M 100K 43.55 41.6 Residual-block-Big 12-6 286M 100K 29.91 28.9 297M 100K 43.22 41.2 RK2-block 12-6 286M 100K 30.58 29.4 297M 100K 43.88 42.0 RK2-block (learnable  i ) 12-6 286M 100K 30.77 29.6 297M 100K 44.11 42.2 RK4-block 12-6 286M 100K 30.55 29.4 297M 100K 43.81 41.9 Table 1: Comparison with the state-of-the-arts on the WMT En-De and WMT En-Fr tasks. ODE Transformer is efficient to use. As we only apply the ODE design schema to the encoder side, it only brings minor impacts on the inference speed due to the autoregressive decoding schema. Another concern here is memory consumption. ODE Transformer consumes more memory than the baseline in the same depth since we need to store the intermediate approximations in the forward pass. But the additional consumption is less than that of the baseline who has the same computation cost, which is acceptable for most scenarios. We give a quantitative analysis in Section 5. We evaluated the ODE Transformer on three sequence generation tasks: machine translation, abstractive summarization and grammar error correction. The datasets we used are elaborated in the following section, and more details of experimental setups could be found in Appendix A and B. 4.1 Datasets Machine Translation We report results on three WMT benchmarks. For the WMT'14 English-German (En-De) task, the training data consisted of approximately 4 . 5 M tokenized sentence pairs, as in (Vaswani et al., 2017). All sentences were segmented into sequences of sub-word units (Sen-nrich et al., 2016) with 32 K merge operations using a shared vocabulary. We selected newstest2013 as the validation data and newstest2014 as the test data. For the WMT'14 English-French (En-Fr) task, we used the dataset provided within Fairseq, i.e., 36 M training sentence pairs from WMT'14. newstest2012+newstest2013 was the validation data and newstest2014 was the test data. For the WMT'16 English-Romanian (En-Ro) task, we replicated the setup of (Mehta et al., 2020), which used 600 K/ 2 K/ 2 K sentence pairs for training, evaluation and inference, respectively. Abstractive Summarization We also tested the models' ability to process long sequences on the CNN-DailyMail summarization task (Nallapati et al., 2016; Hermann et al., 2015). The prepro-8339 Model Params Epochs BLEU Transformer in Mehta et al. (2020) 62M 170 34.30 DeLight (Mehta et al., 2020) 53M 170 34.70 Int Transformer  (Lin et al., 2020) -32.60 Transformer (Our impl.) 69M 20 33.49 RK2-block (learnable  i ) 69M 20 34.94 RK2-block-Big (learnable  i ) 226M 20 35.28 Table 2: Results on the WMT En-Ro task. cessed method was the same as in (Ott et al., 2019). We used a shared BPE with 30 K operations, resulting in a vocabulary of 32 , 580 entries. The evaluation metric was F1-Rouge (Lin, 2004) (Rouge-1, Rouge-2 and Rouge-L). Grammar Error Correction We used the following datasets as the training data, including National University of Singapore Corpus of Learner English (NUCLE) (Dahlmeier et al., 2013), Lang-8 Corpus of Learner English (Lang-8) (Tajiri et al., 2012), FCE dataset (Yannakoudakis et al., 2011), and Write & Improve + LOCNESS Corpus (Bryant et al., 2019). We borrowed the setup from Chollampatt and Ng (2018) and used the provided preprocessed script. The word-level dropout technique was also applied to prevent the overfitting problem. Language Modeling The truncation error analysis is conducted on the Penn Treebank (Mikolov et al., 2011), which is a widely-used language model dataset. It contains 88 K, 3 , 370 and 3 , 761 sentences for training, validation and test. The vocabulary size was 10 K. We set the layer depth of the language model to 1 or 2 to make a fair comparison. Assume the layer depth is 1 , then the loss between the block output and the ground-truth could be regarded as the truncation error. It alleviates the influence of the error accumulation across different layers. Results of En-De and En-Fr Table 1 compares ODE Transformer with several state-of-the-art systems. Both RK2-block and RK4-block outperform the baselines by a large margin with different model capacities. For example, RK2-block obtains a +1 . 00 BLEU improvement with the base configu-ration when the depth is 6 . RK4-block yields a gain of 0 . 17 BLEU points on top of RK2-block. This observation empirically validates the conjecture that high-order ODE functions are more efficient. When we switch to deep models, our method is more parameter efficient. E.g., RK2-block is comparable with a strong 48-layer system (Li et al., 2020) with half of the encoder depth. Similarly, wide models can also benefit from the enlarging layer depth (Wei et al., 2020; Li et al., 2020). RK2-block achieves BLEU scores of 30 . 77 and 44 . 11 on the En-De and the En-Fr tasks, significantly surpassing the standard Big model by 1 . 32 and 0 . 70 BLEU points. This sets a new state-of-the-art on these tasks with fewer parameters. Results of En-Ro Table 2 exhibits model parameters, total training steps and BLEU scores of several strong systems on the En-Ro task. Again, ODE Transformer outperforms these baselines. As stated in (Mehta et al., 2020), they trained the model up to 170 epochs and obtained a BLEU score of 34 . 70 through the DeLight model. However, the observation here is quite different. The validation PPL begins to increase after 20 epochs. Thus, our baseline is slightly inferior to theirs, but matches the result reported in Lin et al. (2020). ODE blocks achieve even better performance with DeLight within much less training cost. For a bigger model (line 6), it obtains a BLEU score of 35 . 28 . Parameter Efficiency Table 3 summaries the results of several efficient Transformer variants, including Lite Transformer (Wu et al., 2020), DeLight (Mehta et al., 2020) and a light version of the Evolved Transformer (So et al., 2019). As expected, ODE Transformer is promising for smaller models. It is comparable in BLEU with DeLight but having 9 M fewer parameters. Under the same model capacity, it outperforms DeLight by 0 . 64 BLEU points. It may offer a new choice for deploying NMT systems on edge devices. Results of Summarization and Correction We also evaluated the ODE Transformer on another two sequence generation tasks. Table 4 shows that both RK2-block and RK4-block outperform the 8340 Model Summarization Correction RG-1 RG-2 RG-L Prec. baselines by a margin. Similarly, RK4-block is superior to RK2-block when the model is shallow. More results and case studies could be found in Appendix C. 5 Analysis Here we investigate some interesting issues. For simplicity, we call RK2-block with coefficients initialized by 1 as RK2-block-v1, and learnable coefficients (Eq. (25) ) as RK2-block-v2. Quantization of the Truncation Error In fact, we cannot obtain the true solution of each block output in NMT, because we mainly experimented on the encoder side. Instead, we tested our system on the language modeling task, where the perplexity between the single-layer model output and the ground truth could be regarded as the truncation error with no error propagations. Table 5 shows the perplexities on the Penn Treebank dataset (Mikolov et al., 2011). All ODE Transformer variants reduce the errors significantly. RK4-order achieves the lowest PPL on both settings. In addition, RK2-block can even obtain a lower PPL than a 2-layer residual-block. The observation here again verifies larger ODE blocks behave superior to the standard residual block. Inference Speed and Memory Consumption Table 6 shows the comparison of inference speed and memory consumption discussed in Section 3.3. Experimental results demonstrate the proposed ODE design schema results in acceptable inference speeds. And it is also memory-friendly through the memory comparison between the baseline and the RK variants in both base and big configurations. BLEU against Encoder Depth Figure 3 (left) depicts BLEU scores of several ODE Transformer variants and the baseline under different encoder depths. All ODE Transformer variants are significantly superior to the baseline when depth  24 . RK2-block-v2 almost achieves the best perfor-Model 1-Layer 2-Layer Residual-Block 142.33 136.07 RK2-block 131.80 123.12 RK2-block (  i = 1 ) 132.67 123.90 RK2-block (learnable  i ) 128.48 121.02 RK4-block 126.89 119.46 Table 5: Comparison of PPL on systems with different ODE blocks. mance over all depths, especially when the model becomes deeper. Interestingly, Figure 3 confirms again that ODE Transformer is parameter efficient, e.g., a 6-layer RK2-block is comparable with the 18-layer baseline system. Another finding here is RK4-block performs well on shallow models, but it is inferior to RK2-block when the depth is going deep. This is because original coefficients may cause the optimization problem in the backward propagation in deep models (see Section 3.2). Also, Figure 3 (right) plots BLEU as a function of the model size when the hidden size is 256 . The RK2 method significantly surpasses the baseline using much fewer parameters. Ablation Study on Different F (  ,  ) As stated in Section 3, the F (  ,  ) function can either be SAN, FFN or both of them (SAN+FFN). As shown in Figure 4, high-order ODE works better with FFN than SAN. An explanation might be that the FFN component has more parameters than the SAN component. 5 The model that treats FFN and SAN as a single ODE block behaves the best. Training and Validation Perplexity Figure 5 plots the training and validation PPL curves of RK blocks and the baseline enhanced by RPR (Shaw et al., 2018). RK2-block obtains lower training and validation PPLs in both configurations (base and wide models). Visualization of the Gradient Norm We also collect the gradient information of several well-trained systems during training. Figure 6 plots the gradient norm of RK2-block-v2, RK4-block and the standard residual-block (baseline). As we can see that Pre-Norm residual block is able to make the training stable (Wang et al., 2019). Both RK2-block-v2 and RK4-block provide richer signals due to the implicit parameter sharing among intermediate approximations. The two learning curves appear to be nearly the same, which is consistent with the results in Table 1. Comparison of Different ODE Design Schemas Then, we take a comprehensive analysis of several ODE design schemas. As stated in Lu et al. (2018)'s work, several models in computer vision, such as LeapfrogNet (He et al., 2019), PolyNet (Zhang et al., 2017) and MultistepNet (Lu et al., 2018), can also be interpreted from the ODE perspective. The related ODE functions are summarized in Table 7. We re-implemented these methods using the same codebase for fair comparisons. We conducted experiments following the base configu-ration on the En-De task. At the time t , Multistep Euler methods require previous states, e.g. y t  1 , to generate the current approximation, instead of iterative refinements based on the current-time state. So these methods are heavier than ODE Transformer. Note that DLCL (Wang et al., 2019) can also be regarded as a 2 6 10 14 18 22 4 . multistep Euler method, which is more competitive in deep Transformer. But there is just a modest improvement upon the shallow baseline. Theoretically, the Backward Euler method is slightly better than the Forward Euler method in numerical analysis, but the improvement is marginal. Note that our ODE Transformer achieves consistent BLEU improvements over the aforementioned methods. The reason is that such iterative refinements provide more efficient and effective parameter learning. Deep Transformer models Recently, deep Transformer has witnessed tremendous success in machine translation, especially on WMT news tasks (Li et al., 2019; Zhang et al., 2020; Zhou et al., 2021; Tran et al., 2021). A straightforward way is to shorten the path from upper-level layers to lower-level layers thus to alleviate the gradient vanishing or exploding problems (Bapna et al., 2018; Wang et al., 2019; Wu et al., 2019; Wei et al., 2020). For deeper models, the training cost is nonnegligible. To speed up the training, an alternative way is to train a shallow model first and progressively increase the model depth (Li et al., 2020; Dong et al., 2020). Apart from the model architecture improvements, another way of easing the optimization is to utilize carefully designed parameter initialization strategies (Zhang et al., 2019; Xu et al., 2020; Huang et al., 2020; Liu et al., 2020a). With the model capacity going larger, one can use Layer-8342 Model Information Flow Related ODEs BLEU Leapfrog (He et al., 2019) y t +1 = y t  1 + 2 F ( y t ,  t ) Multistep Euler 28.07 Multistep (Lu et al., 2018) y t +1 = k n  y t + (1  k n )  y t  1 + F ( y t ,  t ) Multistep Euler 28.17 DLCL (Wang et al., 2019) y t +1 = y 0 + (cid:80) tl =0 W l F ( y l ,  l ) Multistep Euler 27.78 PolyNet (Zhang et al., 2017) y t +1 = y t + F ( y t ,  t ) + F ( F ( y t ,  t ) ,  t ) Backward Euler 28.15 RK2-block y t +1 = y t + 12 F ( y t ,  t ) + 12 F ( y t + F ( y t ,  t ) ,  t ) Improved Euler 28.67 RK2-block (  i = 1 ) y t +1 = y t + F ( y t ,  t ) + F ( y t + F ( y t ,  t ) ,  t ) RK 2nd-order 28.77 RK2-block (learnable  i ) y t +1 = y t +  1  F ( y t ,  t ) +  2  F ( y t + F ( y t ,  t ) ,  t ) RK 2nd-order 28.86 RK4-block y t +1 = y t + 16 F 1 + 26 F 2 + 26 F 3 + 16 F 4 RK 4th-order 29.03 Table 7: Comparison of several ODE-inspired design schemas on the En-De task. Drop (Fan et al., 2020) or Skipping Sublayers (Li et al., 2021) to prevent deep models from the overfitting problem. Note that ODE Transformer is orthogonal to the aforementioned methods, and we will test it on these methods in future work. Ordinary Differential Equations The relationship between ResNet and ODEs was first proposed by Weinan (2017). This shows a brand-new perspective on the design of effective deep architectures. Moreover, the success of Neural ODENet (Chen et al., 2018) has attracted researchers. Some insightful architectures (Zhang et al., 2017; Lars-son et al., 2017; Lu et al., 2018; He et al., 2019; Zhu and Fu, 2018; Lu et al., 2019; Sander et al., 2021) can also be interpreted from the ODE perspective. But, in NLP, it is still rare to see studies on designing models from the ODE perspective. Zhang et al. (2021) proposed continuous self-attention models using the same merit with neural ODE. Perhaps the most relevant work with us is an (2021)'s work. They redesigned the Transformer architecture from a multi-particle dynamic system view in terms of efficiency. Unlike them, we show that the stacked first-order ODE blocks may cause error accumulation, thus hindering the model performance. We address this issue by introducing high-order blocks, and demonstrate significant performance improvements on three sequence generation tasks, which is complementary to Baier-Reinio and De Sterck (2020)'s work. This paper explores the relationship between Transformer and ODEs. We propose ODE Transformer to help the model benefit from high-order ODE solutions. Experimental results on the three representative sentence generations tasks (i.e., machine translation, abstractive summarization, and grammatical error correction) show the effectiveness and efficiency of ODE Transformer. It achieves 30 . 77 and 44 . 11 BLEU scores on the WMT'14 En-De and En-Fr benchmarks, setting a new state-of-the-art result on the En-Fr. Note that our code is publicly available at https://github. com/libeineu/ODE-Transformer . This work was supported in part by the National Science Foundation of China (Nos. 61732005 and 61876035), the National Key R&D Project of China (No. 2019QY1801), the China HTRD Center Project (No. 2020AAA0107904) and Yunnan Provincial Major Science and Technology Special Plan Projects (Nos. 201902D08001905 and 202103AA080015). The authors would like to thank anonymous reviewers for their valuable comments. And thank Yufan Jiang for his helpful advice to improve the paper.
abstain objective objective objective abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain objective abstain abstain method other abstain method abstain abstain abstain abstain abstain abstain other abstain abstain method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain method abstain other abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain other abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain other abstain abstain abstain abstain abstain other abstain method abstain abstain abstain abstain abstain abstain other abstain abstain abstain other abstain method abstain abstain abstain abstain abstain result abstain abstain abstain abstain abstain abstain abstain abstain result abstain result abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain result abstain abstain abstain abstain abstain abstain abstain abstain abstain result abstain abstain abstain abstain abstain abstain result abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain result abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain result abstain other other other other other other other abstain other other other other other other abstain other objective abstain objective objective abstain abstain abstain abstain other abstain other other other