It is a common belief that training deep transformers from scratch requires large datasets. Consequently, for small datasets, people usually use shallow and simple additional layers on top of pre-trained models during fine-tuning. This work shows that this does not always need to be the case: with proper initialization and optimization, the benefits of very deep transformers can carry over to challenging tasks with small datasets, including Text-to-SQL semantic parsing and logical reading comprehension. In particular, we successfully train 48 layers of transformers, comprising 24 fine-tuned layers from pre-trained RoBERTa and 24 relation-aware layers trained from scratch. With fewer training steps and no task-specific pre-training, we obtain the state-of-the-art performance on the challenging cross-domain Text-to-SQL parsing benchmark Spider 1 . We achieve this by deriving a novel D ata-dependent T ransformer Fix ed-up date initialization scheme (DT-Fixup), inspired by the prior T-Fixup work (Huang et al., 2020). Further error analysis shows that increasing depth can help improve generalization on small datasets for hard cases that require reasoning and structural understanding. In recent years, large-scale pre-trained language models (Radford et al., 2019; Devlin et al., 2018; Liu et al., 2019b) trained with transformers (Vaswani et al., 2017) have become standard building blocks of modern NLP systems to help improve generalization when task-specific annotations are limited. In practice, it has been found that deeper transformers generally yield better results with sufficient training data (Lan et al., 2019),  Work done while the author was an intern in Borealis AI. 1 The code to reproduce our results can be found in: https://github.com/BorealisAI/DT-Fixup especially on tasks involving reasoning and structural understanding. This suggests that additional transformer layers should be employed in conjunction with pre-trained models, instead of simple and shallow neural components, such as a classifier head, currently used by models of many NLP tasks. However, the common belief in the literature is that training deep transformers from scratch requires large datasets, and few attempts have been made on small datasets, to the best of our knowledge. One implication is that although extra transformer layers on top of pre-trained models should help with more challenging problems in principle, it does not work in practice due to limited training data. We show that after resolving several optimization issues with the method proposed in this work, it is possible to train very deep transformers with improved generalization even on small datasets. One advantage of pre-trained models is the reduced computational resources needed when fine-tuning on small datasets. For instance, it allows practitioners to finetune on a single GPU and obtain strong performance on a downstream task. However, the large size of pre-trained models limits the batch size that can be used in training new transformer layers on a small computational budget. Despite their broad applications, training transformer models is known to be difficult (Popel and Bojar, 2018). The standard transformer training approach leverages learning rate warm-up, layer normalization (Ba et al., 2016) and a large batch size, and models typically fail to learn when missing any one of these components. The restricted batch size aggravates the training difficulties. Even if a large batch size can be feasibly employed, poorer generalization results are often observed (Keskar et al., 2016), especially when the dataset size is only several times larger than the batch size. Furthermore, many recent works noticed a performance gap in this training approach due to layer normalization (Xu et al., 2019; Nguyen and Salazar, 2019; Zhang et al., 2019a; Wang et al., 2019b; Liu et al., 2020; Huang et al., 2020). Inspired by the recent T-Fixup by Huang et al. (2020), which eliminates the need for learning rate warm-up and layer normalization to train vanilla transformers, we derive a data-dependent initialization strategy by applying different analyses to address several key limitations of T-Fixup. We call our method the D ata-dependent T ransformer Fix edup date initialization scheme, DT-Fixup . In the mixed setup of additional yet-to-be-trained transformers on top of pre-trained models, DT-Fixup enables the training of significantly deeper transformers, and is generally applicable to different neural architectures. Our derivation also extends beyond vanilla transformers to transformers with relational encodings (Shaw et al., 2018), allowing us to apply the results to one variant called relation-aware transformer (Wang et al., 2019a). By applying DT-Fixup on different tasks, we show that the impression that deep transformers do not work on small datasets stems from the optimization procedure rather than the architecture. With proper initialization and optimization, training extra transformer layers is shown to facilitate the learning of complex relations and structures in the data. We verify the effectiveness of DT-Fixup on Spider (Yu et al., 2018), a complex and cross-domain Text-to-SQL semantic parsing benchmark, and ReColr (Yu et al., 2020b), a reading comprehension dataset requiring logical reasoning. While Text-to-SQL semantic parsing is inherently different from reading comprehension, they share similar characteristics which require certain levels of reasoning and structural understanding ability. Meanwhile, the sizes of both datasets are less than 10k training samples, which is tiny by deep learning standards and renders large-batch training undesirable due to poor generalization 2 . On both datasets, DT-Fixup consistently outperforms the standard approach with better generalization and allows the training of significantly deeper transformer models. For Spider, we successfully apply DT-Fixup to train a Text-to-SQL parser containing 48 transformer layers, with 24 relation-aware layers trained from scratch on top of 24 pre-trained layers from pre-trained RoBERTa 2 For a comparison, T-Fixup applies batch sizes of more than 1k on machine translation to stabilize the training, which would hurt the generalization significantly on our datasets whose sizes are less than 10k. (Liu et al., 2019b). Our parser achieves 70 . 9% exact match accuracy on the Spider test set, which is the state of the art at the time of writing. At the same time, it requires less training steps and no task-specific pre-training as compared to the prior art (Yu et al., 2020a). For ReClor, we rank the second on the public leaderboard by simply adding 4 transformer layers on top of RoBERTa. Further error analysis shows that the performance improvements by increasing the depth mainly come from better generalization on the harder cases requiring reasoning and structural understanding. Even the failed predictions from the deep models are more reasonable than from the shallow ones. In this section, we present the necessary background by first introducing the relation-aware transformer layer, which outperforms the vanilla transformer layer with limited data by injecting additional inductive bias (Wang et al., 2019a). Then, we introduce the T-Fixup technique (Huang et al., 2020) for optimizing deeper vanilla transformers and discuss why it does not directly apply in the mixed transformer optimization setup. Consider a set of inputs X = [ xxx 1 , . . . ,xxx n ] where xxx i  R d x . A transformer , introduced by Vaswani et al. (2017), is a stack of blocks, with each block consisting of a multi-head self-attention layer , layer normalizations, a multi-layer perceptron and skip connections. Each block (with one head in self-attention for notational simplicity) transforms each xxx i into yyy i  R d x as follows:  ij = softmax (cid:16) xxx i qqq ( xxx j kkk ) (cid:62) (cid:46)(cid:112) d z (cid:17) (1) zzz i = (cid:80) nj =1  ij xxx j vvv ; (2)  y  y  y i = LayerNorm ( xxx i + zzz i www (cid:62) ) (3) yyy i = LayerNorm ( y  y  y i + MLP ( y  y  y i )) (4) where the softmax operation is applied across the index j , MLP is a two-layer perceptron, LayerNorm is a layer normalization (Ba et al., 2016) layer, and qqq,kkk,vvv  R d x  d z ,www  R d x  d z . In order to bias the transformer toward some pre-existing relational features between the inputs, Shaw et al. (2018) described a way to represent relative position information in a self-attention layer by changing Equation 1-2 as follows:  ij = softmax (cid:32) xxx i qqq ( xxx j kkk + rrr kij ) (cid:62)  d z (cid:33) zzz i = (cid:80) nj =1  ij ( xxx j vvv + rrr vij ) (5) Here the rrr ij  R d z terms encode the known relationship between two elements xxx i and xxx j in the input. Wang et al. (2019a) adapted this framework to effectively encode the schema information using rrr ij 's for Text-to-SQL parsers, and called it relation-aware transformer (RAT). Huang et al. (2020) found that the requirement for the warmup during the early stage training of the transformers comes from a combined effect of high variance in the Adam optimizer and back-propagation through layer normalization. Bounding the gradient updates would reduce the variance and make training stable, which can be achieved by appropriately initializing the model weights. They derived a weight initialization scheme called T-Fixup for the vanilla transformer that fully eliminates the need for layer normalization and learning rate warmup, and stabilizes the training to avoid harmful plateaus of poor generalization. T-Fixup requires the inputs xxx to be Gaussian randomly initialized embeddings with variance d  12 where d is the embedding dimension. Then, the input and parameters of the encoder, xxx , vvv , www in the vanilla self-attention blocks as well as the weight matrices in the MLP blocks defined in Eq. 1-4 are re-scaled by multiplying with a factor of 0 . 67 N  14 , where N are the number of transformer layers. However, there are two restrictions of T-Fixup narrowing down the range of its application. First, T-Fixup is only designed for vanilla transformer but not other variants like the relative position or relation-aware version described previously. Second, they make the critical assumption that the inputs xxx can be freely initialized then scaled to the same magnitude as vvv , www and MLP weights. This renders the method inapplicable for the mixed setup where the inputs to the yet-to-be-trained transformer layers depend on the outputs from the pretrained models. The first issue can be addressed by re-deriving the scaling factor following the methodology of T-Fixup but taking into account the additional relational term. However, to lift the second restriction requires changing the assumption and more dramatic modification to the analysis. We now follow the analysis framework of T-Fixup (Huang et al., 2020), but derive the conditions to bound the gradient updates of the self-attention block in the presence of a pre-trained model. Based on the derivation, we propose a data-dependent initialization strategy for the mixed setup of the new transformers on pre-trained encodings. Our analysis applies to the general architecture type illustrated in Figure 1, where the input passes through a pre-transformer, a main transformer, and a post-transformer module before outputting. The pre and post transformer modules can be any architectures that can be stably trained with Adam (Kingma and Ba, 2014), including MLP, LSTM, CNN, or a pre-trained deep transformer module which can be stably fine-tuned with a learning rate significantly smaller than the main learning rate used for the main transformer module. For this work, we will just consider the case of the main transformer containing only the encoder for simplicity, while our decoder will be an LSTM which can be viewed as part of the post-transformer module. Extending our analysis to include deep transformer decoder is straightforward following the framework of Huang et al. (2020). We use f e to denote the pre-transformer module ( e for pre-trained encoder), and its parameters  e ; similarly f o for post-transformer module ( o for output) with parameters  o . The main transformer module f G is a stack of L transformer blocks, each consisting of a self-attention block and a MLP block. Let G l , l = 1 , . . . , 2 N denote individual self-attention or MLP layers in the blocks ( G l 's do not include the skip connections), with parameters  l and let L = 2 N , f G 's parameters are denoted by  G = L (cid:83) l =1  l . Let the whole model with the output softmax layer(s) and all layer normalization blocks removed be denoted by f (  ;  ) and the loss function by L , where  are all the learnable parameters. Following Huang et al. (2020), we aim to derive a condition under which, per each SGD update with learning rate  , the model output changes by (  ) , i.e. (cid:107)  f (cid:107) = (  ) where  f = f (  ;     L   )  f (  ;  ) . By Taylor expansion, the SGD update is:  f = f  o   o + f  G   G + f  e   e + O ( (cid:107)  o (cid:107) 2 + (cid:107)  G (cid:107) 2 + (cid:107)  e (cid:107) 2 ) =   ( f o  o f o  o (cid:62)  L f o (cid:62) + f o f G f G  G f G  G (cid:62) f o f G (cid:62)  L f o (cid:62) + f o f G f G f e f e  e f e  e (cid:62) f G f e (cid:62) f o f G (cid:62)  L f o (cid:62) ) + O (  2 ) (6) As assumed in Sec. 3.1, we can stably train f e and f o coupled with L , i.e, (cid:107)  L f o (cid:107) = (cid:107) f o   o (cid:107) = (cid:107) f e   e (cid:107) = (cid:107) f o f G (cid:107) = (cid:107) f G f e (cid:107) = (1) , we only need to bound the magnitudes of f G   G to bound the overall SGD update. Since what we care is the magnitude of the update as it relates to the depth, we can assume all parameters to be scalars, i.e, qqq l ,kkk l ,vvv l ,www l ,rrr kl ,rrr vl reduce to scalars q l , k l , v l , w l , r kl , r vl  R . The next theorem states the condition under which, (cid:107) f G   G (cid:107) is bounded by (1) , achieving the overall (cid:107)  f (cid:107) = (  ) . Theorem 3.1 Assuming (cid:107) xxx (cid:107) = (  ) for some  (cid:29) 1 , then (cid:107) f G   G (cid:107) = (1) if (cid:107) v l (cid:107) = (cid:107) w l (cid:107) = (cid:107) r vl (cid:107) =  (cid:16) ((4  2 + 2  + 2) N )  12 (cid:17) for all encoder layers l in relation-aware transformers; and (cid:107) v l (cid:107) = (cid:107) w l (cid:107) =  (cid:16) (4  2 N )  12 (cid:17) in the case of vanilla transformers. The proof is in Appendix A. One important immediate observation is that our scaling as the depth N is to the power of  1 / 2 , whereas T-Fixup has a scaling with power of  1 / 4 . While this theorem is all we need for deriving our DT-Fixup approach, it is not immediately intuitive. So next we inspect what it takes to bound the change in a individual layer output (cid:107)  G l (cid:107) to ( /L ) in each gradient update. This will shine some light on the particular form of the expressions in Theorem 3.1: Theorem 3.2 Let xxxxxxxxx l = [ x l 1 , . . . , x ln ] be the input into l -th layer, and assume that (cid:107)  L /G l (cid:107) = (1) , i.e. the gradient signal from the layers above is bounded, then  G l = G l ( xxx l    L  xxx l ;  l    L   l )  G l ( xxx l ;  l ) satisfies (cid:107)  G l (cid:107) = ( /L ) when for all i = 1 , . . . , n : 2 (cid:107) v l (cid:107) 2 (cid:107) x li (cid:107) 2 + 2 (cid:107) v l (cid:107)(cid:107) r vl (cid:107)(cid:107) x li (cid:107) + (cid:107) r vl (cid:107) 2 + (cid:107) w l (cid:107) 2 (1 + 2 (cid:107) x li (cid:107) 2 ) = (1 /N ) (7) for relation-aware transformers. Alternatively, in the case of vannilla transformers: (cid:107) v l (cid:107) 2 (cid:107) x li (cid:107) 2 + (cid:107) w l (cid:107) 2 (cid:107) x li (cid:107) 2 = (1 /L ) (8) In this case, the proof is straightforward by taking partial derivatives of G l with respect to each parameter, and keep the terms with the lowest powers as they dominate the norm when the scale is smaller than one. Appendix B gives the detailed proof. The insight from this theorem is: if the input xxx l has the same norm as xxx , setting parameters v l , w l , r vl to have the same norm and solve the equations would yield the scale factors in Theorem 3.1. Remark: In T-Fixup, the corresponding condition to Eq. 8 keeps the term (cid:107) v l (cid:107) 2 (cid:107) w l (cid:107) 2 which is dropped by ours. It is due to the fact that T-Fixup assumes (cid:107) x i (cid:107) can be controlled to be the same scale as v l and w l , so the lowest power terms (which are dominating the norms here) are the quartic ( 4 th power) ones. For us, (cid:107) xxx (cid:107) is treated separately by a constant to be estimated from data, so the lowest power terms are the quadratic ones in v l , w l , r vl in Eq. 7 and 8, and (cid:107) v l (cid:107) 2 (cid:107) w l (cid:107) 2 are dropped. Another important distinction from T-Fixup is that we assume the estimated (cid:107) xxx (cid:107) to be much larger than the scale of v l and w l , unlike the case when they are also controlled to be the same scale. As we will see next, these changes imply our proposed method employs more aggressive scaling for initialization as compared to T-Fixup, and the assumption that (cid:107) xxx (cid:107) has larger scale is satisfied naturally. Unlike previous works (Zhang et al., 2019b; Huang et al., 2020), appropriate initialization is not enough to ensure Eq. 7 and 8 during the early stage of the training. This is due to the fact that the input xxx often depends on the pre-trained model weights instead of being initialized by ourselves. Empirically, we observe that the input norm (cid:107) xxx (cid:107) are relatively stable throughout the training but difficulty to control directly by re-scaling. Based on this observation, we treat (cid:107) xxx (cid:107) as a constant and estimate it by a forward pass on all the training examples as  = max j [ (cid:107) xxx j (cid:107) ] . We then use this estimated  in the factors of Theorem 3.1 to obtain the scaling needed for initialization. Since parameters of all layers are initialized to the same scale, we drop index l for brevity in this section. In practice,  is on the order of 10 for pre-trained models, hence v , w and r vi are naturally two orders of magnitude smaller. DT-Fixup is described as follows:  Apply Xavier initialization (Glorot and Ben-gio, 2010) on all free parameters except loaded weights from the pre-training models;  Remove the learning rate warm-up and all layer normalization in the transformer layers, except those in the pre-trained transformer;  Forward-pass on all the training examples to get the max input norm  = max j [ (cid:107) xxx j (cid:107) ] ;  Inside each transformer layer, scale v, w, r v in the attention block and weight matrices in the MLP block by ( N  (4  2 + 2  + 2))  12 for relation-aware transformer layer; or scale v, w in the attention block and weight matrices in the MLP block by N  12 / (2  ) for vanilla transformer layer. We first apply DT-Fixup on the task of cross-domain Text-to-SQL semantic parsing. Given an unseen schema S for a database during training, our goal is to translate the natural question Q to the target SQLT . The correct prediction depends on the interplay between the questions and the schema structures and the generalization over unseen schemas during inference. As a result, reasoning and structural understanding are crucial to perform well on this task, especially for the more challenging cases. We denote our baseline model as SQL-SP 3 and henceforth. Implementation. For modeling Text-to-SQL generation, we adopt the encoder-decoder framework which can be directly fit into the architecture shown in Fig. 1. First, the pre-transformer module f e is a pre-trained language model which embeds the inputs Q and S into joint representations xxx i for each column, table s i  S and question word q i  Q respectively. The joint representations are passed into a sequence of N relation-aware transformer layers. The post-transformer module f o is a grammar-guided LSTM decoder, which uses the transformer output yyy i to predict the target SQL T . We follow prior arts (Wang et al., 2019a; Guo et al., 2019; Yin and Neubig, 2018) to implement SQL-SP. The implementation details and hyperparameter settings are described in Appendix C. Dataset. We evaluate SQL-SP on Spider (Yu et al., 2018), a complex and cross-domain Text-to-SQL semantic parsing benchmark. The dataset size is relatively small by deep learning standards, with only 10 , 181 questions and 5 , 693 queries covering 200 databases in 138 domains. The second task where we apply DT-Fixup is multi-choice reading comprehension requiring logical reasoning. Given a context, a question and four options, the task is to select the right or most suitable answer. Rather than extracting relevant information from a long context, this task relies heavily on the logical reasoning ability of the models. Implementation. On top of the pre-trained encodings of the input context, question and options, a stack of N vanilla transformer layers are added before the final linear layer which gives the predictions. The implementation details and hyper-paramter settings are described in Appendix D Dataset. We evaluate on ReClor (Yu et al., 2020b), a newly curated reading comprehension dataset requiring logical reasoning. The dataset contains logical reasoning questions taken from 3 SQL S emantic P arser. standardized exams (such as GMAT and LSAT) that are designed for students who apply for admission to graduate schools. Similar to Spider, this dataset is also small, with only 6 , 139 questions. As the test set of Spider is only accessible through an evaluation server, most of our analyses are performed on the development set. We use the exact match accuracy 4 on all examples following Yu et al. (2018), which omits evaluation of generated values in the SQL queries. We present our results on the Spider leaderboard 5 in Table 1, where SQL-SP trained with DT-Fixup outperforms all the other approaches and 4 We use the evaluation script provided in this repo: https://github.com/taoyds/spider 5 https://yale-lily.github.io/spider achieves the new state of the art performance. Notably, the top four submissions on the previous leaderboard are all occupied by models leveraging relation-aware transformers and task-specific pretraining. Table 2 compares our proposed models with the publicly available works. With enough training steps, our baseline model trained with the standard optimization strategy achieves the same level of performance as compared to RAT-SQL. However, models trained with standard optimization strategy obtain much lower performance with the same epochs 6 of training as compared to models trained with DT-Fixup and require more training steps to achieve the best accuracy. At the same time, by adding more relation-aware transformer layers, further gains can be obtained for models trained with DT-Fixup, which achieves the state-of-the-art performance without any task-specific pre-training on additional data sources. As mentioned in Section 2.2, in the mixed setup, there is no way to apply T-Fixup as it was originally proposed. The closest thing to compare is to drop its constraints on the inputs, but training then becomes highly unstable and fails to converge 4 times out of 5 runs. These results demonstrate the necessity and effectiveness of DT-Fixup to improve and accelerate the transformer training for Text-to-SQL parsers. Table 3 shows the accuracy of our best model as compared to other approaches 7 with different level of hardness defined by Yu et al. (2018). We can see that a large portion of the improvement of our model comes from the medium level on both dev and test set. Interestingly, while our model obtains similar performance for the extra hard level on the dev set, our model performs significantly better on the unseen test set. As most of the extra 6 One epoch iterates over the whole training set once. Wang et al. (2019a) trained with a batch size of 20 for 90 , 000 steps, which is around 200 epochs on the Spider training set. Yu et al. (2020a) trained with a batch size of 24 for 40 , 000 steps, which is around 100 epochs on the Spider training set. 7 We choose the top two submissions which also report the breakdown of the accuracy on the test set. hard cases involves implicit reasoning steps and complicated structures, it shows that our proposed models possess stronger reasoning and structural understanding ability, yielding better generalization over unseen domains and database schemas. For ReClor, we choose the best model in Yu et al. (2020b) as the baseline which employs a linear classifier on top of RoBERTa. From the results presented in Table 4, we can see that simply stacking additional vanilla transformer layers outperforms the baseline and adding DT-Fixup further improves the accuracy, which ranks the second on the public leaderboard at the time of this submission 8 . The result further validates the benefit of adding extra transformer layers and the effectiveness of DT-Fixup. For fair comparisons and better understanding, we conduct multiple sets of ablation with the same architecture and implementation to validate the advantages of DT-Fixup over the standard optimization strategy. Note that, the batch sizes in our experiments are relatively small (16 for Spider and 24 for ReClor) due to the size of the pre-trained models, while batch sizes for masked language modelling (Liu et al., 2019b) and machine translation (Huang et al., 2020) are commonly larger than 1024 . Deeper Models. As we can see from Table 5, the standard optimization strategy fails completely to train deep transformers whose depths are larger than 8 on both Spider and ReClor, showing that it struggles to properly train the transformer model as the depth increases. At the same time, DT-Fixup can successfully train deeper transformers up to 32 layers and consistently achieves better performance than models trained by the standard optimization strategy with the same depth on both Spider and ReClor. With DT-Fixup, deep models generally 8 https://eval.ai/web/challenges/challenge-page/503/ achieve better performance than the shallow ones even there are only thousands of training examples. It contradicts the common belief that increasing depth of the transformer model is helpful only when there are enough training data. Faster Convergence. Demonstrated by the validation curves on Spider plotted in Figure 2, models trained with DT-Fixup converges to the same level of performance much faster than models trained with the standard optimization strategy. While standard optimization strategy struggles as the models become deeper, DT-Fixup can keep the model training smooth, showing that DT-Fixup can effectively accelerate the convergence of the transformer training, especially for the deep ones. Batch Sizes When Dataset Size is Small. As shown in Table 7, increasing batch size on Spider from 16 to 120, the average performance from five runs drops from 73.24 to 71.08 and the gap with the standard training approach becomes much narrower. It empirically verifies that large-batch training has a negative impact on the generalization when the dataset size is small, confirming the need to stablize small batch training. From the results on the Spider benchmark, we can see significant improvements by applying DT-Fixup and increasing the depth of the transformer model. However, why and where they help Text-to-SQL semantic parsing are still unclear. As an attempt to answer these questions, we investigate into the predicted results from three variants of our proposed model: Baseline , the best model ( N = 4 ) trained with the standard training approach; Shallow , a shallow model ( N = 4 ) trained with DT-Fixup; Deep , our best model ( N = 24 ) trained with DT-Fixup, which is much deeper. To better understand the models' behavior, we manually examine all the failed cases predicted by these models and classify the errors into four categories: 1) Correct: equivalent in meaning but with different SQL syntax ( e.g. , ORDER BY X LIMIT 1 and SELECT MIN(X) ); 2) Column: the SQL structure is correct but there existed mispredicted columns; 3) Sketch: the SQL structure is predicted different from the ground truth, while the aligned column prediction are correct; 4) Both: there exist both sketch and column errors in the prediction. Table 6 presents the overall statistics of our error analysis. Due to logically equivalent N Standard DT-Fixup Spider 2 69 . queries, there are a number of false negatives for all three models, confirming that the current Spider evaluation metric is not ideal. At first glance, the improvements by applying DT-Fixup and increasing the depth seem to come from correcting Sketch and Both errors, while the three models make similar number of Column only errors. It provides evidence that applying DT-Fixup and increasing the depth can help the transformer model handle hard examples which are mispredicted completely (errors in Both category) by the baseline model. Typically, correct predictions on these hard examples require a certain level of reasoning and structural understanding ability. Fine-grained Error Analysis. In order to better understand the errors made, we look into the composition of error types by each model on mistaken examples common to all models, as well as on examples where at least one model is wrong. In Fig. 3-4, Column means proportion with column errors ( i.e. , Column or Both ); Sketch means proportion with sketch errors ( i.e. , Sketch or Both ). There are 190 examples mispredicted by all the three models and 387 examples which at least one of the three models mispredict. Fig. 3-4 exclude false negatives due to equivalent logic queries, we can see the real improvements from the deep model are even more significant than what the exact match accuracy shows. Furthermore, among the common mistakes to all three models, the deep model has a much smaller proportion in the sketch mistakes which usually involve more logic and structure understanding. Some of column mistakes are due to missing domain knowledge or common sense, which is harder to improve without external data or knowledge. This shows that even among the failed cases, deeper transformer model can make more reasonable predictions. Many research efforts have been devoted to understanding the training and improving the optimization optimization of the transformer models. In particular, transformer models often fail to learn unless a gradual learning rate warm-up is applied at the beginning of training. Chen et al. (2018); Nguyen and Salazar (2019); Wang et al. (2019b) noticed a performance gap due to layer normalization, and introduced various architecture changes as remedy. Zhang et al. (2019b,a); Liu et al. (2020) proposed initialization schemes to stabilize training, allowing either to remove layer normalization or learning rate warmup. Liu et al. (2019a) demonstrated the instability of the Adam optimizer during early stages of optimization. Based on these results, Huang et al. (2020) proposed a weight initialization schema for the transformer that eliminates the need for layer normalization and warmup completely. Despite the broad applications of the transformer model, it struggles to perform well for some NLP tasks with limited training data. In this work, we propose a theoretically justified optimization strategy DT-Fixup to train deeper transformer model with improved generalization and faster convergence speed on small datasets, which is generally applicable to different neural architectures. On two important tasks, Text-to-SQL semantic parsing and logical reading comprehension that require reasoning and structural understanding, applying DT-Fixup achieves SOTA or near-SOTA results by simplying using extra transformer layers on top of the pre-trained models. Such observations suggest even boarder applicability of deeper transformers. We thank all the anonymous reviewers and area chair for their valuable inputs.
abstain abstain result method result abstain abstain abstain abstain other abstain method abstain objective abstain abstain abstain abstain abstain abstain abstain abstain method abstain abstain abstain result abstain result abstain abstain abstain method abstain result abstain abstain method abstain abstain abstain method abstain abstain other other abstain other other other other other other other other other other other other other method method method abstain method method method abstain abstain abstain method method method abstain other method method method other other abstain abstain abstain abstain abstain method abstain method method abstain abstain abstain method method method method abstain other abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain method abstain method abstain method abstain abstain abstain abstain abstain method abstain abstain abstain abstain abstain other abstain abstain result abstain abstain abstain abstain abstain result result result abstain abstain abstain abstain abstain abstain result abstain abstain abstain abstain result abstain other abstain abstain abstain abstain abstain abstain abstain result abstain abstain other abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain result abstain abstain abstain other other other other other other other abstain objective abstain abstain other