In sequence modeling, certain tokens are usually less ambiguous than others, and representations of these tokens require fewer refinements for disambiguation. However, given the nature of attention-based models like Transformer and UT (universal transformer), all tokens are equally processed towards depth. Inspired by the equilibrium phenomenon, we present a lazy transition, a mechanism to adjust the significance of iterative refinements for each token representation. Our lazy transition is deployed on top of UT to build LT (lazy transformer), where all tokens are processed unequally towards depth. Eventually, LT is encouraged to oscillate around a relaxed equilibrium. Our experiments show that LT outperforms baseline models on several tasks of machine translation, pre-training, Learning to Execute, and LAMBADA. Attention-based models like Transformer (Vaswani et al., 2017) underly two core concepts: layer and refinement . With layer-stacked structure, the model entirely relies on attention mechanisms (Parikh et al., 2016; Lin et al., 2017) to refine token representations (e.g.,vectors) layer-by-layer from context-informed representations by using residual connections (He et al., 2016; Ebski et al., 2018). However, layer is not necessary. UT (universal transformer) (Dehghani et al., 2019) and its variants (Bai et al., 2019; Lan et al., 2020) iteratively run single-layer but wide Transformer for sequence modeling that token representations are refined at each step by attending to the sequence representation of the previous step, showing higher performance than layer-stacked Transformer does on NMT (Dehghani et al., 2019), pre-training (Lan et al., 2020), language modeling (Bai et al., 2019), and other tasks of varying complexity (Dehghani et al., 2019) with the same number of parameters. The computational bound of recurrence is not the number of tokens in the sequence (like RNN) or the number of stacked layers in the model but is the maximum number of refinements made to token representations, e.g., a pre-defined maximum number of iteration steps. We follow this line but further ponder refinement . Concretely, the model refines token representations iteratively that tokens are equally processed towards depth for disambiguation. Essentially, different token representations are refined at a step equally and concurrently. Thus, regardless of the ambiguous state, a step shows the same importance for all the tokens. However, certain tokens are less ambiguous than others in sequence modeling, especially in NMT. It raises two questions: 1) do deep representations have to be learned for less-ambiguous tokens; 2) do tokens must consider the contributions or importance of a step equally? Meanwhile, our work derives its motivation from the combination of UT and ACT (adaptive computation time) (Graves, 2016) that can dynamically estimate the importance of iteration steps for understanding different sentences in the bAbI task (Weston et al., 2016). However, we consider the importance of every iteration step for all the tokens in the same sequence, sharing a similar motivation with depth-adaptive Transformer (Elbayad et al., 2020). On the other hand, these methods learn a halting probability for each step without considering the correspondence and interdependence throughout the iteration and the model's convergence. We attempt a mechanism like the reset gate in GRU (Chung et al., 2014) that the model is allowed to dynamically forget some information based on the correspondence and interdependence, where in our case, this information is the further refinement at the current step for the corresponding token representation. In this way, we weaken the importance of an iteration step for less-ambiguous token representations to prevent over-refining but retain the significance for more-ambiguous token 2904 representations to avoid under-refining, refining token representations at the same step unequally. To this end, we present a lazy transition that performs between two consecutive steps when running UT iteratively. It forms iterative refinements (Eb-ski et al., 2018) for token representations with a residual structure: h t i i = h t i  1 i + (1   t i i ) UT ( h t i  1 i ) (1) where h t i i is the representation for the i -th token in the sequence at step t i (token-wise), UT yields the deep refinement UT ( h t i  1 i ) for h t i i by consuming h t i  1 i , and (1   t i i ) is a refining rate for adjusting the contribution of UT ( h t i  1 i ) . In this way, we model an equilibrium (Bai et al., 2019; Lan et al., 2020) between two representations of the same token from consecutive steps. Significantly, when off (1   t i i ) = 0 , the model is allowed to dismiss the further refinement UT ( h t i  1 i ) for h t i i at the step t i . Then, the model reaches a local equilibrium for the corresponding token that the input h t i  1 i is similar to the output h t i i . In other words, the model ponders the significance of the further refinement via our lazy transition that a token representation is lazy to absorb the information if the further refinement is trivial. In sequence modeling, all the equilibrium token representations are learned and concatenated for the required sequence representation that the model oscillates around a relaxed equilibrium. Our contributions are: 1) we present a lazy transition that is placed on top of UT to build LT (lazy transformer). Our lazy transition dynamically forms a refinement path for a token at each step by pondering the step importance and the local equilibrium. Eventually, LT is encouraged to oscillate around a relaxed equilibrium in sequence modeling; 2) we provide an empirical study to quantitatively analyze how the relaxed equilibrium impacts NMT in performance, deep models, and zero-shot inferring; 3) we show our model can consistently improve the performance of pre-training and two tasks of varying complexity, where standard Transformer fails; 4) our empirical study shows that stable and smooth refinements at the early iteration steps are significant, which results in a strong equilibrium and a stable oscillation. Iterative Refinement in Transformer, Tied Transformer, and Universal Transformer One core backbend of Transformer (Vaswani et al., 2017) is iterative refinements (Ebski et al., 2018) that are finished by forming a residual structure around each sub-layer: h li = h l  1 i + f l ( h l  1 i ) , where l denotes depth and f (  ) represents a sublayer (e.g., an attention layer (Parikh et al., 2016; Lin et al., 2017)). Following tied Transformer (Gul-cehre et al., 2018; Xia et al., 2019; Dabre and Fujita, 2019), which share parameters across some layers, (Dehghani et al., 2019) introduce token-wise recurrence (Graves et al., 2014; Joulin and Mikolov, 2015; Kaiser and Sutskever, 2016) to signal-layer but wide Transformer and then present UT, a Turing-complete model. Theoretically, UT yields a similar recurrent inductive bias of RNN's because each token representation is refined by attending to the sequence representation of the previous step with tied parameters. We present a comparison in Appendix A. Significantly, the iterative refinement in UT is reformed to : h ti = h t  1 i + f ( h t  1 i ) , where f is the same for every iteration step t , i.e., a single-layer structure that the parameters are tied throughout the iteration. In this work, we apply our lazy transition on top of a UT block. To reform iterative refinements, we are inspired by (Zhang et al., 2021; Escolano et al., 2021; Bapna et al., 2018; Wang et al., 2019) that f (or f l ) could be a more complicated network rather than a simple sub-layer, defining f as the entire UT block and reforming the iterative refinement as Eq.1. Adaptive Computation While a token-wise recurrent model like UT can theoretically run in-finitely, the model is commonly trained by setting a maximum number of iteration steps. However, the number of inference steps is flexible. (Graves, 2016) first introduce ACT (Adaptive Computation Time ) to compute a scalar halting probability predicted by the model at each step for each token in RNN. Then, (Dehghani et al., 2019) adapt ACT for UT to halt the iteration before reaching the maximum number of training steps 1 in inferring. Furthermore, (Elbayad et al., 2020) present depth-adaptive decoding for tokens to model the distribution of exiting with the probability of computing each layer/step and then emitting token predictions. Our lazy transition follows this line somewhat that (1   t i i ) in Eq.1 dynamically approximates the computation time for tokens. It reflects on two characteristics: 1) (1   t i i ) = 0  stop iteration; 1 In some papers, training steps refer to how many batches we use for training. 2) (1   t i i ) computed at each step. We present LT (Lazy Transformer), a token-wise recurrent model. LT only consists of a UT block and our lazy transition. In training, we iteratively run LT but constrain the computational bound to control the training process by setting a maximum number of iteration steps. However, iteration steps are dynamic in inferring. In sequence modeling, DEQ(Bai et al., 2019) and ALBERT (Lan et al., 2020) report that a UT-based model tends to oscillate around an equilibrium. Concretely, each additional step has a smaller and smaller contribution to the current sequence representation until the model oscillates around a fixed point. Formally, we have: lim t  +  h t 1: N = lim t  +  UT ( h t 1: N )  h E 1: N = UT ( h E 1: N ) , where t is the step, h t 1: N is the sequence representation of a N length sequence X 1: N at t , and h E 1: N is the equilibrium sequence representation. Empirically, the equilibrium phenomenon could be observed from the difference norm of sequence representations: || UT ( h t 1: N )  h t 1: N ||  || UT ( h t +11: N )  UT ( h t 1: N ) ||  || UT ( h t :1: N )  h t :1: N || < (cid:15) , where (cid:15) depends on the model and t : denotes steps after t . Intuitively, UT, DEQ, and ALBERT are encouraged to reach a global and sequence-level equilibrium 2 at a specific iteration step t in a dynamic programming style for outputting the equilibrium sequence representation. By contrast, we attempt a greedy strategy to independently find a local equilibrium for a token, modeling the locally optimal choice. Then, for an input sequence, we model all the local equilibriums to find a relaxed equilibrium instead of naively finding a global and sequence-level equilibrium. Formally, in sequence modeling, we model a relaxed equilibrium to obtain an equilibrium sequence representation h R 1: N :  i  N : lim t i  +  h t i i = lim t i  +  UT ( h t i i )   i  N : h R i i = UT ( h R i i )  h R 1: N = UT ( h R 1: N ) = [ h R 1 1 , h R 2 2 , ..., h RNN ] (2) where h t i i denotes i -th token representation at t i (token-wise), h R i i is the equilibrium token representation, R stands the step of reaching the relaxed 2 Our preliminary experiment confirms this intuition, which will be further discussed in Experiment. equilibrium, and [  ] denotes concatenation. Similarly, we can evaluate the local equilibrium from the difference norm of token representations: || UT ( h t i : i )  h t i : i || <  (cid:15) (3) Note that, in our preliminary experiment, we find that h E 1: N (cid:54) = [ h R 1 1 , h R 2 2 , ..., h RNN ] , i.e., the equilibrium sequence representation is not equivalent to the combination of all the equilibrium token representations because tokens have different ambiguous states. For the relaxed equilibrium, we do not have to update tokens equally at an iteration step so that an iteration step can show varying importance for different tokens, subject to the ambiguous state of the token representations. Thus, we consider adjusting the impact of the current step for different tokens throughout the iteration. Although we can easily and immediately observe the equilibrium or the local equilibrium phenomenon from difference norms, we have no prior knowledge in practice that we cannot analyze the equilibrium quantitatively throughout the iteration. Hence, we consider the linear CKA exam (centered kernel alignment): CKA ( X, Y ) = || XTY || 2 F / ( || XTX || F || YTY || F )  (0 , 1] that is introduced by (Kornblith et al., 2019) to identify correspondences between representations in models trained from different initializations and is invariant to orthogonal transform and isotropic scaling but is not invariant to any linear transforms. For this exam, we are inspired by (Wu et al., 2020), who measure the degree of a layer's multilinguality with a CKA exam between two averages of outputted sequence representations. However, we run the exam for two averages of sequence representations h 1: N emerged from two consecutive steps 3 . Precisely, we majorly rewrite the equilibrium to: lim t  +  CKA ( h t , (cid:80) Ni =1 UT ( h ti ) N ) = 1  CKA (  h E , (cid:80) Ni =1 UT ( h Ei ) N ) = 1 (4) 3 Note that, in this way, the feature space is the channel of representations, not the representations itself in the original CKA exam. where  h t = (cid:80) Ni =1 h ti N . For our relaxed equilibrium in sequence modeling, we have:  i  N : lim t i  +  CKA ( h t i i , UT ( h t i i )) = 1   i  N : CKA ( h R i i , UT ( h R i i )) = 1  N (cid:88) i =1 CKA ( h R i i , UT ( h R i i )) = N (5) Essentially, we can dynamically evaluate the local equilibrium by giving the exam to token representations throughout the iteration: CKA ( h t i : i , UT ( h t i : i ))  1 (6) It reflects the correspondence and interdependence between UT's input and output. Note that, although the global equilibrium does not expect the local equilibrium, this exam also gives an intuition of how a token representation changes throughout the iteration. To leverage the relaxed equilibrium (Eq.2 and Eq.5), our lazy transition uses a residual structure to form iterative refinements (Ebski et al., 2018) for h i at each step in order to obtain h R i i . Recall that, UT ( h t i  1 i ) is the deep refinement we can obtain at step t i , and CKA (  ) returns (0 , 1] . We form the iterative refinement (Eq.1) to: h t i i = h t i  1 i + (1   t i i ) UT ( h t i  1 i )  t i i = CKA ( h t i  1 i , UT ( h t i  1 i )) (7) Concretely, the model is based on the correspondence and interdependence between UT's input h t i  1 i and output UT ( h t i  1 i ) . When  t i i is close to 1 at step t i , h t i : i is only oscillating around h t i i , and UT cannot provide useful information for better representations anymore. Then, the model is encouraged to dismiss UT and then outputs h i 's equilibrium token representation: h R i i (  h t i  1 i  h t i i  h t i : i ), for the the relaxed equilibrium. It is similar to the reset gate in GRU that dynamically forget the previous state, but our lazy transition attempts to dismiss the newly obtained information UT ( h t i  1 i ) that is unimportant. Therefore, our lazy transition provides implicit step information, similar to GRU that can identify the current position and handle input sentences of varying length. Since there is no variable for identifying steps, the model can run varying steps in inferring. On the other hand, this could be viewed as a linear interpolation: h t i i =  t i i h t i  1 i + (1   t i i )( h t i  1 i + UT ( h t i  1 i )) , where  t i i decides how much the model updates h t i  1 i at step t i from the pre-refined representation 4 : h t i  1 i + UT ( h t i  1 i ) . Meanwhile, since h t i i is informed by UT ( h t i  1 i ) without any step-specific parameter, the benefits are twofold: 1) our lazy transition does not hurt the global receptive field of UT, only adjusting the contribution of a step; 2) the recurrent inductive bias of UT is inherited because all the parameters are tied throughout the iteration. Our empirical study confirms these two benefits (see Experiment). In sequence modeling, since our model does not provide any position information, in order for the model to make use of the order of the sequence, we inject position information at each step. Therefore, our LT (lazy Transformer) is formed as: h t 1: N 1: N = h t 1: N  1 1: N +(1   t 1: N 1: N ) UT ( h t 1: N  1 1: N + P E 1: N ) , where h t i i computed by Eq.7 is the i -th token representation of a N length sequence representation h 1: N and P E 1: N is the sinusoidal position encoding for identifying positions as defined in (Vaswani et al., 2017). Recall that 1   t i i collapses to 0 when the model is researching the local equilibrium of h i . The model can simply copy the equilibrium token representation to the next step for speed. Instantiation LT can be instantiated, used, and trained as the same as a vanilla Transformer block or a UT block. Concretely, we can instantiate and train: 1) a LT encoder with the objective of MLM (masked language modeling) (Devlin et al., 2019; Lan et al., 2020); 2) a LT decoder with the objective of GPT (generative pre-training) (Radford et al., 2018; Alec Radford, 2020); 3) a LT encoder-decoder (consisting of a LT encoder and a LT decoder) in a seq2seq (Graves, 2013) manner. Lazy Transition vs. GRU They have different motivations. GRU aims to learn a segment-level representation by accumulating the information from all the tokens, whereas LT is encouraged to ponder the importance of a step for different tokens and then to oscillate around the relaxed equilibrium. Meanwhile, compared to GRU, which forgets the previously computed state via the reset gate, our lazy transition is allowed to dismiss the newly computed information that is trivial. Lazy Transition vs. Adaptive Computation Adaptive computation methods like ACT (Graves, 2016) and Adaptive-depth Transformer (Elbayad et al., 2020) learn a generator to output a probability of exiting based on the step output. We argue that these methods are agonistic for the model's convergence because they do not consider the information flow from the input to the corresponding output. By contrast, our method leverages the model's convergence and applies CKA to ponder the correspondence and interdependence between inputs and outputs, where in our case, the model's convergence is the relaxed equilibrium. Lazy Transformer vs. UT, DEQ, and ALBERT LT is parallel to UT (Dehghani et al., 2019), DEQ (Bai et al., 2019), and ALBERT (Lan et al., 2020). We share the idea of recurrence over depth, but we have three main differences: 1) previous methods require an explicit step encoding for each iteration, whereas we let our lazy transition handle iterations implicitly; 2) we ponder the significance of a step for different tokens, whereas previous methods refine tokens equally at a step; 3) we consider the local and token-level equilibrium in addition to the sequence-level equilibrium. We divide our empirical studies and experiments into two genres: 1) we experiment with NMT (our main task) to confirm the effectiveness of our methods for large-scale sequence modeling and further quantitatively justify our hypotheses and assumptions; 2) we attempt pre-training tasks and two somewhat rare but challenging tasks: the Learning to Execute task (Zaremba and Sutskever, 2014) and the LAMBADA (Paperno et al., 2016) task, to observe the performance on tasks of varying complexity. All the links of datasets, libraries, scripts, and tools marked with (cid:5) are listed in Appendix H. We open source code on GitHub. Training Our code is implemented on Tensor-flow 2.6 (Abadi et al., 2016) with 4 NVIDIA TI-TAN Xp 12G GPU. We implement our model based on the codebase of official UT from tensor2tensor (cid:5) 5 and official CKA (cid:5) . We use the default setting: universal_transformer_base from tensor2tensor. Concretely, we use Adam optimizer (Kingma and Ba, 2015) with parameters  1 = 0 . 9 ,  2 = 0 . 997 and (cid:15) = 10  9 , and a dynamic learning rate with warm _ up = 8000 (Vaswani et al., 2017) ( learning _ rate  (0 , 7 e  4 ] ) is employed. We set dropout regularization with a drop rate rate = 0 . 1 and label smoothing with gamma = 0 . 1 (Mezzini, 2018). For data feeding efficiency, each batch of similar-length sequences are padded to the same length and may have a different number of elements in each batch. Reimplementation and Reconfiguration We reimplement some models on our machine with the same batch size. We compare the reimplemented results to the reported results on the same test set to ensure the difference is less than 5% (or 1 in BLEU). Then, we can confirm the reimplementation and reconfiguration. Dataset and Preprocessing We train a LT encoder-decoder model for machine translation. To be comparable, we share two NMT tasks: 1) English  German of WMT 2014 (cid:5) (Bojar et al., 2014); 2) English  Romanian of WMT 2016 (cid:5) (Bojar et al., 2016). Following the standard evaluation, the model is evaluated on newstest2014 for English  German and newstest2016 for English  Romanian . We use the Moses tokenizer (cid:5) developed by (Koehn et al., 2007) for tokenization and use fastBPE (cid:5) to learn shared 32K BPE (Sennrich et al., 2016) for a language pair. Data filtering is finished by FAIR tool (cid:5) (Ng et al., 2019). We use sacreBleu (cid:5) (Post, 2018) with standard settings 6 to evaluate the quality of translation. Model Configuration The model configurations are identical to base-UT (Dehghani et al., 2019). Specifically, we set model dimension, word embedding, head, and FFN filter to 1024, 1024, 16, and 4096, which results in the same number of parameters (62M) as base-Transformer 5 Note that, the newest UT implementation uses pre-normalization y = x + f ( ln ( x )) instead of reported post-normalization y = ln ( x + f ( x )) . UT with pre-normalization shows slight degradation in performance (  1% ) but improves stability in training, initialization, and scalability, where x denotes the input, f ( . ) stands for a sub-layer, and ln is a layer-normalization unit. 6 {nrefs:1|case:mixed|eff:no|tok:13a|smooth:exp| version:2.0.0} 2908 newstest2014 newstest2016 # Model En  De En  Ro basemodel: { 6 , 6 } =  { 6 , 6 } 1 base-Transformer(Vaswanietal.,2017) 27.50 32.31 2 (cid:63) base-UTw/oSE 27.85 3 (cid:63) base-UT 28.73 33.97 4 base-UT(Dehghanietal.,2019) 28.90 5 (cid:63) base-UT+ACT 29.11 6 (cid:63) base-UT+GRU 26.59 7 OURS:base-LT 29.81 35.02 deepmodel: { 20 / 40 , 6 } =  { 20 / 40 , 6 } 9 (cid:63) 20-Transformer(Bapnaetal.,2018) 28.72 33.59 10 20-Transformer(Wangetal.,2019) 28.90 11 (cid:63) 20-UT 29.69 34.32 12 OURS:20-LT 30.54 35.62 13 OURS:40-LT 31.05 36.04 Table 1: Performance of translation. (Vaswani et al., 2017). Beam search is con-figured with beam size 4 and length penalty 0.6. We train the model for 100k iterations. We use { T _ enc _ step, T _ dec _ step } =  { I _ enc _ step, I _ dec _ step } to denote a model that runs the maximum T _ enc _ step and T _ dec _ step steps in the encoder and decoder respectively for training and runs the maximum I _ enc _ step and I _ dec _ step steps in the encoder and decoder respectively for inferring. For instance, { 6 , 6 } =  { 6 , 6 } means we set the maximum step to 6 both in the encoder and decoder for training and inferring. In this experiment, we set { 6 , 6 } =  { 6 , 6 } ( base-LT ), which is identical to the baseline model: base-UT. Also, it is equivalent to base-Transformer that has 6 layers in both the encoder and decoder. For comparison, we place GRU 7 on top of the UT block for evaluation (base-UT + GRU), similar to that we use our lazy transition, and we also train another UT with ACT (base-UT + ACT). All of these models require step encodings for the identification of steps. For the evaluation of how our lazy transition handles iterations without explicit encodings, we instantiate another UT without step encodings (base-UT w/o SE), a similar model that naively repeats one layer without step identifications (Dabre and Fujita, 2019). Table 1 shows the results on NMT tasks. base-LT outperforms base-UT (row 3&7) by 4% . By observing row 2&3, the performance of UT significantly degrades without using step encodings, which indicates a mechanism for step identification is beneficial for UT. Meanwhile, we find that applying ACT to UT (row 5) can improve the per-7 We use orthogonal kernels for GRU to solve an optimization problem. no effect. We observe the equilibrium phenomena in models by giving the CKA exam (Eq.6) in the encoder and decoder. Visualizations are presented in Appendix E. In our case study, all the tokens in base-UT (Appendix E.1 (c,d)) run to an equilibrium synchronously because every step shows similar importance to all the tokens, i.e., all the tokens have similar CKA scores at every step throughout the iteration. It confirms that base-UT tends to find a global equilibrium (Eq.4) for all the tokens, as discussed before. Furthermore, this process is unstable that CKA scores change dramatically at the early steps 8 , resulting in unstable refinements (under-refining or over-refining) for some tokens and hurting tokens' local equilibrium. Meanwhile, we are aware that base-UT + ACT (Appendix E.3), base-UT w/o SE (Appendix E.2), and base-UT + GRU (Appendix E.4) show a similar behavior to base-UT throughout the iteration. base-UT + GRU is even more irregular than base-UT at the early steps. Also, we suspect that ACT and GRU have no power to adjust the importance of a step for each token throughout the iteration in NMT. Compared to that, base-LT (Appendix E.1 (a,b)) enables each token to smoothly find its local equilibrium independently and then turns to oscillate around the relaxed equilibrium (Eq.2) together. Specifically, we observe that CKA scores of a step (  t i i in Eq.7) differ from one token to the others at every step in the early 4 iterations, which answers our question in Introduction that a step can show varying importance for all the tokens. Then, base-LT tends to refine token representations based on the sequence-level characteristics at the late 2 steps to oscillate around the relaxed equilibrium, resulting in similar CKA scores. Given the nature of the iteration-based model, we can run infinite steps in the encoder and decoder. Therefore, we test the performance beyond 6 steps. In this experiment, we share a challenge with (Bapna et al., 2018; Wang et al., 2019) to train deep models. According to their works, NMT can get significant benefits from many layers in the encoder but not in the decoder. Similarly, we assume we can obtain benefits from running many steps in the encoder. For comparison, we config-8 e.g., from step 1 to step 2, the average of absolute difference is 0 . 221 , which is only 0 . 105 in base-LT. ure models with { 20 , 6 } =  { 20 , 6 } ( 20-LT ), which is similar to the 20-layer model in (Bapna et al., 2018; Wang et al., 2019). Beyond 20 steps in the encoder, we further configure a model with { 40 , 6 } =  { 40 , 6 } ( 40-LT ). We show the results of deep models in Table 1. Running a large number of steps in the encoder can consistently improve the performance of LT, and deep LT outperforms baseline models. Similar to base-LT, we also consider the CKA exam for the equilibrium phenomena (Appendix E.5 ) that we obtain similar conclusions to base-LT. It indicates our method is a potential new development for deep models without introducing additional parameters. Besides, we find UT can get benefits from deep settings, but the performance is not very strong. Intuitively, it is caused by the global equilibrium strategy observed in base-UT that CKA scores change dramatically at the early steps. Therefore, 20-UT cannot smoothly run to an equilibrium, which results in a suboptimal choice. The CKA exam confirms our intuition and draws a similar conclusion of base-UT. By contrast, instead of searching a global equilibrium from scratch, LT searches local equilibriums for tokens at the early steps, which results in varying contributions for different token representations, and then turns to find a global equilibrium for all tokens that we observe a similar behavior to UT. We conjure that LT tends to focus on token-specific characteristics first and then on sequence-specific characteristics. In the above experiments, we set the inference steps as the same as the training steps, sharing the same strategy with previous works (Dehghani et al., 2019; Lan et al., 2020; Bai et al., 2019). We are interested in an asymmetric strategy that the inference steps is different from the training steps. Thus, some steps are zero-shot inferring without explicit training. Recall that the model oscillates around an equilibrium point and outputs similar representations (if iterating). We assume these similar representations can be used for prediction, and we expect similar performance. On the other hand, we visualize the equilibrium phenomena to observe the equilibrium, and zero-shot inferring can quantitatively examine the equilibrium. For this test, we reuse our trained models 9 9 Interestingly, we find the reported sinusoidal step encodings for UT support this test without bringing noises to untrained inference steps. for En  De but change the setting to { 6 , 6 } =  { [1 , 12] , [1 , 12] } , { 20 , 6 } =  { [15 , 26] , [1 , 12] } , and { 40 , 6 } =  { [35 , 46] , [1 , 12] } respectively. For metrics, we compute avg and std for nonzero ( > 1 ) outputs, where avg indicates how strong the equilibrium point is and std tells us how far the model is oscillating around the point. We show a part of experimental results in Table 2 (see all the results in Appendix F). We observe that models can achieve competitive performance from zero-shot inferring for some steps close to the training step, and the best performance does not precisely exist at the training step, except for base-UT + ACT. Essentially, we observe and confirm some conclusions as mentioned earlier. 1) LT's behavior is regularized around the equilibrium when comparing base-LT std : 5 . 03 with base-UT std : 6 . 32 and 20-LT std : 1 . 65 with 20-UT std : 4 . 62 . Specifically, LT's lazy transition controls the refinements throughout the iteration that token representations from one step can jointly compose a sequence representation yielding similar performance to the others, whereas UT is look-up methods are problematic. encouraged to compose the final sequence representations at the training step and lacks a mechanism to control the refinements explicitly. 2) UT does not oscillate around a strong equilibrium point, i.e., having a suboptimal equilibrium, because base-UT avg : 22 . 16 and 20-UT avg : 26 . 03 are not strong. By contrast, LT is stably oscillating around a strong equilibrium and can compose relatively strong sequence representations in different inference steps because base-LT and 20-LT achieve avg : 24 . 52 and avg : 28 . 89 respectively. 3) In both UT and LT, deep models generally find a stronger and more stable equilibrium than base models. 4) base-UT + GRU fails in this test, which results in low avg : 17 . 46 and high std : 9 . 13 . We suspect this failure causes by a very irregular behavior at the first step, as the CKA exam indicates in Appendix E.4. 5) base-UT + ACT outperforms base-UT on avg . However, base-UT is slightly more stable than base-UT + ACT. Besides, base-UT + ACT generally has a larger gradient norm than others, which may impact the stability and convergence in training. 6) Due to the halting mechanism in ACT, base-UT + ACT seems to halt the process instead of oscillating around the equilibrium because the performance is constant after some steps. 7) base-UT significantly outperforms base-UT w/o SE on stability, which indicates a mechanism for step identifications is an essential component for stability. Limitation We do not find a solution to pick the inference step for the best performance, because we do not recognize any pattern. Meanwhile, we could dynamically select a step for a tradeoff between speed and performance by simply copying equilibrium token representations from the previous steps to the next steps. We will leave this for our future work. Besides, we find LT and UT are not stable when translating sentences longer than 50 words. The BLEU score varies from 5 to 60 for different sentences (see Appendix G). We will conduct further experiments to probe this problem. ALBERT (Lan et al., 2020) study the application of UT in pre-training. Since LT is extended from UT, we study LT in pre-training, sharing the framework of ALBERT. Concretely, our setting is identical to base ALBERT, denoting it as 12-base-ALBERT. We set the model dimension, word embedding dimension, and the maximum number of steps to 768, Model SQuAD1.1 SQuAD2.0 MNLI (F1) (F1) (Acc) 12-base-ALBERT (Lan et al., 2020) 89.3 80.0 81.6 (cid:63) 12-base-ALBERT 89.4 80.0 81.4 (cid:63) 12-base-ALBERT-ACT 89.5 80.5 81.6 (cid:63) 12-base-ALBERT-GRU 86.9 77.8 78.6 OURS: 12-base-LT 89.8 81.1 82.1 (cid:63) 24S-base-ALBERT 89.6 80.9 81.7 OURS: 24-base-LT 90.1 81.7 82.6 Table 3: LT in pre-training. 128, and 12. Note that in the original ALBERT, they denote steps as layers. As recommended, we generate masked span for the MLM targets using the random strategy from (Joshi et al., 2020), and we use LAMB optimizer (cid:5) with a learning rate of 0.00176 (You et al., 2020) instead of Adam optimizer. The only change is that we use our LT to replace UT in 12-base-ALBERT, and we denote our model as 12-base-LT. Following the instructions, we pre-train models on BooksCorpus (cid:5) (Zhu et al., 2015) and English Wikipedia (cid:5) (Devlin et al., 2019) for 140k steps. Then, we fine-tune on MNLI (cid:5) (Williams et al., 2018) and SQuAD(v1.1 and v2.0) (cid:5) (Rajpurkar et al., 2016, 2018). We report the performance on the dev set as the same as (Devlin et al., 2019; Lan et al., 2020). Result We report the result in Table 3. In this test, we run all models 12 steps, and we implement 12-base-ALBERT-ACT and 12-base-ALBERT-GRU, similar to the experiment of the translation task. 12-base-LT significantly outperforms 12-base-ALBERT, 12-base-ALBERT-ACT, and 12-base-ALBERT-GRU. These observations confirm the effectiveness of our lazy transition. For further tests, we train all models in 24 steps. Our model gets benefits from a large number of steps, improving the performance from the base model significantly. By contrast, 24-base-ALBERT cannot obtain significant improvements. LTE (Learning to Execute) (Zaremba and Sutskever, 2014) including program evaluation tasks (program, control, and addition) and memorization tasks (copy, double, and reverse) is an algorithmic task of varying complexity. The goal is to train models on short snippets of python code to predict the output of the generated programs, which is parameterized by their length and nesting . Specifically, length is the number of digits in the integers that appear in the programs, and nesting 2911 programevaluation memorization Model program control addition copy double reverse (cid:63) LSTM 54.1 69.2 83.8 78.1 51.9 92.1 (cid:63) Transformer(Vaswanietal.,2017) 72.0 92.9 99.8 98.2 94.8 81.8 DNC(Gravesetal.,2016) 69.5 83.8 99.4 100.0 100.0 100.0 Entet(Henaffetal.,2017) 73.4 83.8 98.4 91.8 62.3 100.0 RMC(Santoroetal.,2018) 79.0 99.6 99.9 100.0 99.8 100. is the number of times we are allowed to combine the operations. Following the instructions from the official repository of LTE (cid:5) , we use the mix-strategy to generate the datasets for training. Result Table 4 shows the results on LTE. Our method yields benchmark performance on the program task of program evaluation (column 2) and reaches SOTA performance on the other tasks. We attempt the LAMBADA (Paperno et al., 2016) task to evaluate our model on language modeling tasks of varying complexity. The goal of the LAMBADA task is to predict the target word of the target sentence, based on a narrative passage. In this test, we only use the setting of the standard setup as language modeling that is more challenging. Following the instructions (Parikh et al., 2016), we download the dataset from the official repository of LAMBADA (cid:5) , and then we train the model to predict the next word as a general language modeling task on the training dataset but only predict the target word at test time. Note that we do not compare our method with pre-training-based SOTA (Radford et al., 2018; Brown et al., 2020; Alec Radford, 2020). Readers can refer to Appendix D or the authors' paper for more introduction. Result Table 5 shows the results on the LAMBADA task. 1) We first observe that Transformer fails in this test. Specifically, Transformer shows strong performance on control but weak performance on test . The low performance on test cannot be attributed simply to poor language modeling because control is used to evaluate Transformer in standard language modeling before test . We suspect that the low performance on test can be attributed to a lack of inductive bias in training. Concretely, Transformer is trained to predict the next word as a general language modeling task but only predict the target word at test time. The varying complexity leads to failure on test , similar to the report in (Dehghani et al., 2019). 2) Our method significantly improves the performance on Ppl. test , which means our method does not hurt the recurrent inductive bias inherited from UT 10 . Also, our method is robust to the maximum number of steps we choose (base-LT achieves strong results), whereas UT seems a bit sensitive to the maximum number of steps. In this work, we place our lazy transition on top of UT to build LT. Our lazy transition leverages the relaxed equilibrium for sequence modeling and provides step identifications that the model ponders the importance of every step for different tokens throughout the iteration. Our main experiment shows that LT can achieve strong performance on translation tasks, facilitate the training of deep models, and tackle the challenge of zero-shot inferring. Our method retains the recurrent inductive bias learned by its UT component, which is confirmed by our secondary experiments. LT tends to focus on token-specific characteristics at the early steps and then turns to find sequence-specific ones at the late steps, especially in deep settings. Meanwhile, stable and smooth behaviors in the early iteration are significant. Although there are some practical limitations, as we mentioned in this paper, we believe our lazy transition is a novel perspective to reconsider the models based on iterative refinements in sequence modeling. 10 As mentioned before, LT can inherit the recurrent inductive bias of UT for handling varying complexity.
abstain abstain method method abstain result abstain abstain abstain other abstain method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain method other abstain abstain abstain abstain abstain objective method result other other other other method method method other other other other other method method other method method method abstain abstain abstain other other abstain abstain abstain other abstain other abstain method abstain method method abstain other abstain method abstain abstain abstain method abstain abstain abstain method abstain other abstain abstain method method abstain abstain abstain other abstain abstain method abstain method abstain abstain method result abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain result abstain method abstain abstain abstain abstain method abstain other abstain other abstain abstain method method abstain abstain abstain method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain method abstain abstain abstain abstain abstain abstain method abstain abstain abstain method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain method abstain abstain abstain method abstain abstain method method method abstain abstain abstain method abstain method method abstain abstain abstain abstain other method method abstain abstain method abstain method method method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain method result method abstain abstain objective abstain