Transformers (Vaswani et al., 2017) have gradually become a key component for many state-of-the-art natural language representation models. A recent Transformer based modelBERT (Devlin et al., 2018) achieved state-of-the-art results on various natural language processing tasks, including GLUE, SQuAD v1.1, and SQuAD v2.0. This model however is computationally prohibitive and has a huge number of parameters. In this work we revisit the architecture choices of BERT in efforts to obtain a lighter model. We focus on reducing the number of parameters yet our methods can be applied towards other objectives such FLOPs or latency. We show that much efficient light BERT models can be obtained by reducing algorithmically chosen correct architecture design dimensions rather than reducing the number of Transformer encoder layers. In particular, our schuBERT gives 6 . 6% higher average accuracy on GLUE and SQuAD datasets as compared to BERT with three encoder layers while having the same number of parameters. Transformer (Vaswani et al., 2017) based models have achieved state-of-the-art performance for many natural language processing tasks (Dai and Le, 2015; Peters et al., 2018; Radford et al., 2018; Howard and Ruder, 2018). These include machine translation (Vaswani et al., 2017; Ott et al., 2018), question-answering tasks (Devlin et al., 2018), natural language inference (Bowman et al., 2015; Williams et al., 2017) and semantic role labeling (Strubell et al., 2018). A recent Transformer based model BERT (De-vlin et al., 2018) achieved state-of-the-art results on various natural language processing tasks including GLUE, SQuAD v1.1 and SQuAD v2.0. BERT's model architecture is a multi-layer bidirectional Transformer encoder based on the original implementation described in Vaswani et al. (2017). Following the seminal results obtained by the BERT model, several follow up studies explored methods for improving them further. XLNet (Yang et al., 2019) adds autoregressive capabilities to BERT, improving its quality, though at the cost of additional compute requirements. RoBERTa (Liu et al., 2019) modifies the training procedure of BERT and provides pre-training methods that significantly improve its performance. Two notable papers exploring the architecture design of the BERT are following. Michel et al. (2019) examines the importance of attention heads in BERT architecture, highlighting scenarios where attention heads may be pruned. The main objective of the paper is to provide techniques for pruning attention head, and as such the amount of experiments performed on BERT is limited to a single task (MNLI). ALBERT (Lan et al., 2019) proposes two methods for reducing the number of parameters in BERT. The first is via parameter sharing across layers, and the second is by factorizing the embedding layers. We note (this was mentioned in the conclusion section of the paper) that while these methods are efficient in reducing the number of parameters used by the model, they do not help in reducing its latency. These studies provide some advancement towards a more efficient architecture design for BERT but leave much to be explored. In this paper we take a broader approach examining multiple design choices. We parameterize each layer of BERT by five different dimensions, as opposed to Devlin et al. (2018) that parameterizes a layer with two dimensions and suggests a fixed value for the remaining three. We then (pre-)train multiple variants of BERT with different values chosen for these dimensions by applying pruning-based architecture search technique that jointly optimizes the architecture of the model with the objective of minimizing both the pre-training loss and the number of model parameters. Our experiments result in the following findings:  The ratio of the architecture design dimensions within a BERT encoder layer can be modified to obtain a layer with better performance. Transformer design dimensions suggested in Vaswani et al. (2017) are suboptimal. When we aim to obtain a computationally lighter model, using a tall and narrow' architecture provides better performance than a wide and shallow' architecture. The fully-connected component applied to each token separately plays a much more significant role in the top layers as compared to the bottom layers. Following BERT's notations, we use (cid:96) to denote the number of encoder layers (i.e. Transformer blocks), h to denote the hidden size, and a to denote the number of self attention heads. The BERT paper (Devlin et al., 2018) primarily reports results on two models: BERTBASE ( (cid:96) = 12 , h = 768 , a = 12) and BERTLARGE ( (cid:96) = 24 , h = 1024 , a = 16) . BERT base has 108 M parameters and BERT large has 340 M parameters. Though BERT large achieves higher accuracy than BERT base, due to its prohibitively large size it finds limited use in practice. Since BERT base achieves higher accuracy compared to previous state-of-the-art modelsPre-OpenAI SOTA, BiL-STM+ELMo+Attn and OpenAI GPTon most of the benchmark datasets, it is widely used in practice. BERT base and OpenAI GPT have the same number of model parameters. Given its broad adoption for NLP tasks, an immediate question is: can we reduce the size of BERT base without incurring any significant loss in accuracy? The BERT paper (Devlin et al., 2018) provides an ablation study, Table 1, over the number of model parameters by varying the number of layers (cid:96) , the hidden size h , and the number of attention heads a . It can be observed that the accuracy decreases drastically when the number of encoder layers (cid:96) is reduced, and also when the number of attention heads is reduced. We ask the following question: are there any other design dimensions that can be reduced without incurring huge loss in accuracy? As noted above, the three primary design dimensions of the BERT architecture are the number of encoder layers (cid:96) , the hidden size h , and the number of attention heads a . BERT's Transformer encoder layers are based on the original Transformer implementation described in Vaswani et al. (2017). Vaswani et al. (2017) fixed dimension of key, query, and value in multi-head attention, and filter dimension in feed-forward networks as a function of the hidden size and the number of attention heads. However, these are variable design dimensions and can be optimized. Moreover, BERT architecture uses the same number of attention heads for all the encoder layers and hence all the layers are identical. In this work, we jointly optimize all these design dimensions of BERT architecture while allowing each encoder layer to have different design dimensions. In order to explore the parameter space efficiently we chose to optimize the design dimensions in a pruning framework rather than launching a pretraining job for each of these choices. This allows a speedup of several orders of magnitude that is crucial in order to obtain meaningful conclusions. We parameterize the different dimensions one can modify and jointly optimize them with a mixed target of both accuracy and parameter reduction. We look at how the accuracy of BERT evolves on various downstream datasets like GLUE, SQuAD v1.1, and SQuAD v2.0 when we reduce the model size via an optimization procedure. There is a vast literature on pruning trained neural networks. Starting with the classical works Le-Cun et al. (1990); Hassibi and Stork (1993) in the early 90 's to the recent works Han et al. (2015), pruning deep neural networks has received a lot of attention. There have been two orthogonal approaches in pruning networks: structured pruning (Li et al., 2016; Molchanov et al., 2016) and unstructured pruning (Anwar et al., 2017). Structured pruning gives smaller architecture whereas unstructured pruning gives sparse model parameters. In natural language processing, Murray and Chiang (2015) explored structured pruning in feed-forward language models. See et al. (2016) and Kim and Rush (2016) provided pruning approaches for machine translation. A closely related line of work is Neural Architecture Search (NAS). It aims to efficiently search the space of architectures (Pham et al., 2018; Liu et al., 2018; Singh et al., 2019). Quantization is another technique to reduce the model size. This is done by quantizing the model parameters to binary (Rastegari et al., 2016; Hubara et al., 2017), ternary (Zhu et al., 2016), or 4 or 8 bits per parameter (Han et al., 2015). Recently published DistilBERT (Sanh et al., 2019) shows that a BERT model with fewer number of layers can be efficiently pre-trained using knowledge distillation to give much higher accuracy as compared to the same model pre-trained in a regular way. We note that the distillation technique is complimentary to our work and our schuBERTs can be pre-trained using distillation to boost their accuracy. The ablation study in Table 1, BERT (De-vlin et al., 2018), and the above explained works (Michel et al., 2019; Lan et al., 2019) look at the problem of reducing the BERT model size by reducing one or the other design dimensions number of encoder layers, hidden size, number of attention heads, and embedding size in isolation and in a sub-optimal way. In this work, we address this problem comprehensively. In this section, we present detailed architecture of the original BERT model and explain which design dimensions of it can be optimized. Figure 1 shows BERT pre-training architecture. First, the tokenized inputs are embedded into a vector of dimension h through an embedding layer E . The embedded inputs pass through a sequence of encoder layers 1 to (cid:96) . Each encoder layer is identical in its architecture. The output of the last encoder layer is decoded using the same embedding layer E and softmax cross-entropy loss is computed on the masked tokens. A special token CLS from the last encoder layer is used to compute next-sentence-prediction (NSP) loss. For further details of the loss Encoder layer  2 o Encoder layer  1 o Encoder layer  L o NSP MaskLM MaskLM Embeddingo classification head decoder Tokenizero Masked sentence A Masked sentence B Pre-training Figure 1: BERT pre-training BERT-base number of encoder layers (cid:96) 12 hidden size h 768 number of self-attention heads a 12 feed forward dimension f 4 h key-query dimension for attention k h/a value dimension for attention v h/a Table 2: Elements of BERT corresponding to masked tokens and the NSP loss, we refer the readers to the BERT paper (Devlin et al., 2018). We follow BERT notation conventions and denote the number of encoder layers as (cid:96) , the hidden size as h , and the number of attention heads as a . Following the original Transformer implementation described in Vaswani et al. (2017) BERT sets key-query dimension for multi-head attention k to h/a . Following the same Transformer implementation it sets value dimension for multi-head attention v equal to k , and feed-forward filter size f equal to 4 h . In total, there are three design dimensions in BERT(cid:96) , h and a , they are listed in Table 2. For BERT base, the number of encoder layers (cid:96) is set to 12 , the hidden size h is set to 768 , and the number of attention heads a is set to 12 . The other three dimensions f, k, v are function of h and a . Further, each encoder layer of BERT is identical and uses same value of a, f, k, v . First of all, BERT has no architectural constraint that requires all the encoder layers to be identical. This aspect of design can be optimized and in full-generality it might result in highly nonidentical layers. This implies that a generalized schuBERT (cid:96) (cid:96) h h a a 1 , a 2 ,    , a (cid:96) f f 1 , f 2 ,    , f (cid:96) k k 1 , k 2 ,    , k (cid:96) v v 1 , v 2 ,    , v (cid:96) Table 3: Elements of schuBERT BERT will have a 1 , a 2 ,    , a (cid:96) number of heads, f 1 , f 2 ,    , f (cid:96) filter sizes in the feed forward networks, k 1 , k 2 ,    , k (cid:96) key sizes and v 1 , v 2 ,    , v (cid:96) value sizes in the attention heads, in the layers 1 , 2 ,    , (cid:96) respectively. Table 3 lists all the design dimensions of BERT that can be optimized without changing the architecture. Note that we abuse the term architecture to refer to the entire BERT network and the layer operations except sizes of the parameter matrices. In this work, our goal is to optimize (by pruning) all these dimensions to maximize accuracy for a given size of the model. We refer the BERT with optimized dimensions as schuBERTSize Constricted Hidden Unit BERT. Now, we show which parameter matrices are tied with each of these design dimensions. Each design dimension is tied with more than one parameter matrix. This is explained by providing a detail view of an encoder cell of the BERT. Figure 2 shows architecture of an encoder layer of BERT. The notations in the figure have subscript 1 that represent first encoder layer. Input to an encoder layer is the hidden representation of a token which is of dimension h . Input first goes through a multi-head attention cell. Note that multi-head attention cell processes hidden representation of all the tokens in a combined way. For simplicity, in Figure 2 we have shown only one hidden representation. The multi-head attention cell consists of three parameter tensors, namely key K 1 , query Q 1 and value V 1 . K 1 is of size k 1  a 1  h . Key vector for each head of the attention is of dimension k 1 and a 1 represents the number of heads. Hidden representation of dimension h is projected on the key tensor K 1 to get a 1 key vectors each of dimension k 1 . Similarly the query tensor Q 1 is used to get a 1 query vectors each of dimension k 1 for a 1 heads of the multi-head attention cell. The value tensor V 1 is of dimension v 1  a 1  h . The hidden representation is projected on the value tensor V 1 to get a 1 value vectors each of dimension v 1 . Note that k 1 and v 1 can be different. The inner product of key and query vectors after passing through softmax layer give weights for combining value vectors. For details of multi-head attention cell we refer the readers to Vaswani et al. (2017). In nutshell, using three parameter tensorsK 1 , Q 1 , V 1 , a multi-head attention cell transforms hidden representation of size h to a vector of dimension ( v 1  a 1 ) . This vector is projected back to the same dimension h through a proj matrix P 1 . Which is then added element-wise to the hidden representation that was input to the encoder cell and layer norm is applied on the addition. The output is passed sequentially through two fully-connected layers namely D 1 and G 1 . D 1 consists of a parameter matrix of dimension f 1  h and G 1 consists of a parameter matrix of dimension h  f 1 . The output of G 1 is added element-wise to the input of D 1 and layer norm is applied to it. This is the output of the encoder cell and is input to the next encoder cell. The color coding in Figure 2 shows which vectors need to be of the same dimension. The hidden representation size h needs to be same throughout all the encoder layers. In a multi-head attention cell, in each head key and query vectors must have the same dimension. Therefore, key and query tensors, K 1 , Q 1 must be of the same size k 1  a 1  h . The value vector can be of different dimension v 1 . Therefore the value tensor V 1 should be of dimension v 1  a 1  h . Further, the filter size f 1 in the two fully-connected layers D 1 , G 1 is a variable and can take any integral value. subsequent improvements such as XLNet (Yang et al., 2019) and RoBERTa (Liu et al., 2019), we set the WordPiece embedding size e equal to the hidden layer size h , i.e. e  h . However, factorization of the embedding matrix can be incorporated as demonstrated in ALBERT (Lan et al., 2019). We optimize BERT design dimensions listed in Table 3 by pruning the original BERT base architecture. All the design dimensions are upper bounded by their original value in the BERT base as given in the Table 2. Since we keep the architecture same, that is we do not remove any layer, the design dimensions are lower bounded by one. For each design dimension that we seek to optimize, we introduce a prune-parameter vector  of size equal to the original dimension. We take pre-trained original BERT base network, and multiply all the parameter tensors/matrices that are associated with the particular design dimension with the corresponding prune-parameter vector. For example, filter size of the feed-forward layer in the first encoder layer is f 1 = 3072 . To optimize f 1 , we introduce a prune-parameter vector  f 1  R 3072 and initialize it with all ones. In the original BERT base, the two parameter matrices D 1 and G 1 are associated with the design dimension f 1 . We replace D 1 by diag(  f 1 )  D 1 and G 1 by G 1  diag(  f 1 ) in the BERT pre-trained model. Table 4 lists all the prune parameters. Table 5 lists all the parameter tensors/matrices for which design dimensions are optimized by multiplying prunable parameters on all the sides. key and query tensors K i , Q i for i  { 1 , 2 ,    , (cid:96) } are multiplied on all the three sides with prunable parameters corresponding to key-vector, number of attention heads, and hidden size. Similarly multiplications are performed on value tensor V i with a different value-vector prunable parameter. proj tensor has same multiplication as value tensor. The two feed-forward matrices D i , G i have same multiplications. We denote the so obtained prunable tensors with tilde on their top. Note that we do not have prune parameters for pruning encoder layers. We find the optimal number of encoder layers (cid:96) by running experiments for different values of (cid:96) . Our approach is to optimally find which individual elements of prunable parameters {  h , {  a i ,  v i ,  k i ,  f i } i  [ (cid:96) ] } can be set to zero while incurring minimal increase in the preh  R h {  f i  R f } i =1 , 2 ,  ,(cid:96) {  a i  R a } i =1 , 2 ,  ,(cid:96) {  k i  R k } i =1 , 2 ,  ,(cid:96) {  v i  R v } i =1 , 2 ,  ,(cid:96) Table 4: Prunable parameters. K i  K i [diag(  k i )diag(  a i )diag(  h )]  (cid:102) K i Q i  Q i [diag(  k i )diag(  a i )diag(  h )]  (cid:102) Q i V i  V i [diag(  v i )diag(  a i )diag(  h )]  (cid:101) V i P i  P i [diag(  h )diag(  v i )diag(  a i )]  (cid:101) P i D i  D i [diag(  f i )diag(  h )]  (cid:102) D i G i  G i [diag(  h )diag(  f i )]  (cid:102) G i Table 5: Prunable BERT parameter matrices/tensors. training loss. After we have sparse prunable parameter vectors, we remove the corresponding rows/columns from the BERT parameter matrices { K i , Q i , V i , P i , D i , G i } i  [ (cid:96) ] , and get a small-er/faster BERT model. Below we explain the algorithm to find the sparse prunable parameters. We start with the pre-trained BERT base trained on BooksCorpus ( 800 M words) and English Wikipedia ( 2500 M words) following the BERT pretraining procedure given in Devlin et al. (2018). Particularly, we minimize the loss given in Equation (1) to learn the optimal parameter tensors { K i , Q i , V i , P i , D i , G i } i  [ (cid:96) ] and the embedding matrix E . Next, we introduce the prunable parameters given in Table 4 and initialize them with all ones. We create prunable BERT parameter matrices by multiplying the prunable parameters to the learned BERT parameter matrices, as given in Table 5. Then, we optimize the prunable parameters  's while fixing the learned parameters matrices as given in Equation 2. In addition to the MLM and NSP loss, we add sparsity inducing loss on the prunable parameters with a regularization coefficient  . It is well known that (cid:96) 1 penalty induces sparsity in the parameters. Further, since our goal is to minimize the number of parameters, to account for the fact that each element of prune parameters  when set to zero reduces different number of BERT parameters, we multiply the (cid:96) 1 loss terms with the cost terms  's. For example,  a i is proportional to the number of model parameters that will be removed when an element of the prune parameter  a i is set to zero. It is critical to incorporate  's. Their values are significantly different from each other. The  values are 1 . 0 , 0 . 73 , 0 . 093 , 0 . 093 , 0 . 0078 for a, h, k, v and f respectively. After training the prunable BERT model for a fixed number of steps, we truncate the smallest prune parameters to zero, and remove the corresponding rows/columns from the BERT parameter matrices { K i , Q i , V i , P i , D i , G i } i  [ (cid:96) ] . Then we fine-tune the so obtained smaller schuBERT model. Algorithm 1 summarizes our approach. If we want to reduce the number of parameters by a fraction  , we do so in T steps. In each step, we prune /T fraction of parameters, and at the end of the step we fine-tune the network and repeat these steps T times. Though we have explained the algorithm in terms of (cid:96) 1 penalty on the prunable parameters, in our experiments we tried alternative sparsity inducing penalties as well(cid:96) 0 regularization, and proximal gradient descent on prunable parameters. arg min { E, { K i ,Q i ,V i ,P i ,D i ,G i } i  [ (cid:96) ] } L MLM+NSP ( E, { K i , Q i , V i , P i , D i , G i } i  [ (cid:96) ] ) . (1) arg min {  h , {  ai , vi , ki , fi } i  [ (cid:96) ] } L MLM+NSP ( E, { (cid:102) K i , (cid:102) Q i , (cid:101) V i , (cid:101) P i , (cid:102) D i , (cid:102) G i } i  [ (cid:96) ] ) +  {  h (cid:107)  h (cid:107)} +  (cid:96) (cid:88) i =1 {  a i (cid:107)  a i (cid:107) +  v i (cid:107)  v i (cid:107) +  k i (cid:107)  k i (cid:107) +  f i (cid:107)  f i (cid:107)} . (2) 6 Experimental Results In this section, we present our experimental results. We apply Algorithm 1 on BERT base. For pre-training BERT base we use MXNET based gluon-nlp repository that uses the hyper-parameters suggested in the original BERT paper. Besides pretraining, our algorithm has three hyper-parameters: regularization coefficient  , learning rate for prunable parameters, and the number of steps for regularizing prune parametersEquation (2). We run hyper-parameter optimization on these parameters to get the best results. For regularization loss (2), we use the same training data that we use for pre-training, BooksCorpus (800M words) and English Wikipedia (2,500M words). However, we run the regularization step for 1 / 1000 th steps as used for pre-training. We finet-une the pruned BERT Algorithm 1 Pruning Transformers Input: A Transformer model, minimization objective (FLOPs/Params/Latency), target fraction  , number of iterations T . pretraining. We provide accuracy results for schuBERT on the following downstream tasksquestion answering datasetsSQuAD v1.1, SQuAD v2.0; and GLUE datasets MNLI, MRPC, SST-2 and RTE. For these downstream tasks, we use the fine-tuning hyper-parameters as suggested in the BERT paper. We create six schuBERTs by pruning one or all of the design dimensions. Accuracy of the downstream tasks on these schuBERTs are given in Tables 6-13. The BERT base has 108 million parameters. The schuBERT sizes 88 , 66 , 43 million are chosen to match the number of parameters in BERT with (cid:96)  { 9 , 6 , 3 } layers. We use schuBERTx notation for x  { h, f, a } to denote a schuBERT obtained by only pruning h -hidden size, f -filter size of feed-forward, a number of attention heads respectively. We use schuBERT-all to denote the case when all the design dimensionsh, f, a, k, v , except (cid:96) are pruned. We compare our results with original BERT base, and by varying its number of encoder layers (cid:96)  { 12 , 9 , 6 , 3 } . We denote these results by BERT(cid:96) . Since ALBERT reduces parameters by factorizing the embedding matrix, we denote its results by model SQuAD v1.1 SQuAD v2.0 MNLI MRPC SST-2 RTE Avg BERT-base ( 108 M) 90 . ALBERTe . ALBERT provided results only for 88 million parameter model, not for any smaller models. Further, we also compare with the baseline case when all the design dimensions are pruned uniformly. We denote these results by BERT-all uniform. For 99 M model, Table 6, schuBERT-all beats the baseline BERT-all uniform by 0 . 4% higher average accuracy and performs better than schuBERT-f/h/a . Moreover, the loss in performance in comparison to BERT base with 108 million parameters is only 0 . 2% . Table 7 gives exact design dimensions for schuBERT-all with 99 million parameters. We see that number of hidden units remain same as in BERT base, h = 768 . Parameter reduction primarily comes from feed-forward layers. Moreover, filter size of feed-forward layer f has a clear increasing pattern across the layers. For 88 M model, Table 8, again schuBERT-all beats all the other models. It gives 1 . 1% higher average accuracy than BERT(cid:96) with 9 layers. ALBERTe performs better on SQuAD datasets, but performs significantly worse on MNLI and SST-2 datasets. Note ALBERT's approach is complementary to our approach and it can be incorporated into our schuBERTs. schuBERTa performs significantly worse than schuBERT-all which implies that pruning only number of attention heads is highly sub-optimal, as is recently done in Michel et al. (2019). Table 9 provides the exact design dimensions for schuBERT-all with 88 million parameters. Similar to 99 M model, filter size of feed-forward layer f has a clear increasing pattern across the layers. For heavily pruned models 77 M, 66 M, 55 M and 43 M models accuracy results are shown in Table 10, Table 11, Table 12 and Table 13 respectively. In all these models schuBERTh beats all the other models. For 66 M model, schuBERTh gives 1 . 9% higher average accuracy than BERT(cid:96) with 6 layers. For 43 M model, schuBERTh gives 6 . 6% higher average accuracy than BERT(cid:96) with 3 layers. That is reducing the hidden units is way better than to reduce the number of layers to create a light BERT model. Ideally, we would expect schuBERT-all to perform better than schuBERTh , but marginally worse performance of schuBERT-all can be attributed to the high complexity of pruning all the design dimensions together. Table 14 provides best schuBERT architectures when the number of model parameters are restricted to different values. For smaller models, schuBERTh outperforms all other schuBERTs including schuBERT-all. Note that our schuBERT architectures are smaller in size as well as they yield lower latency. Based on the above described experimental results, we provide following insights on the design dimensions of schuBERT architecture. connected component applied to each token separately plays a much more significant role in the top layers as compared to the bottom layers. Figure 3 shows pattern of filter size of feed-forward layer across the encoder cells for various schuBERT-all models. In each of them, filter size follows an increasing pattern with min-max ratio ranging from 1 . 5 to 4 , as opposed to same value across all the layers. Tall and Narrow BERT. When we aim to obtain a computationally lighter model, using a tall and narrow' architecture provides better performance than a wide and shallow' architecture. Our results in Tables 8, 11, 13 demonstrate that schuBERT with (cid:96) = 12 encoder layers significantly outperforms BERT with (cid:96)  { 9 , 6 , 3 } layers for the same number of parameters. Expansive Multi-head Attention. The ratio of the design dimensions within a BERT encoder layer can be modified to obtain a better performing layer architecture. Transformer design dimensions suggested in (Vaswani et al., 2017) are sub-optimal. Following the original Transformer architecture described in (Vaswani et al., 2017), BERT and other Transformer based models set key-query k and value v dimension for multi-head attention to k = v = h/a , where h is the size of the hidden representation, and a is the number of attention heads. Also, following the same architecture (Vaswani et al., 2017), BERT sets feed-forward filter size f = 4 h . Although there is no restriction in using different output dimensions k, v and filter size f , without changing the behaviour of the attention mechanism, we are not aware of any study questioning this default value' of k = v = h/a and f = 4 h . Our schuBERT architecture for various model sizes given in Table 14, show that for smaller models k, v should be much larger than h/a . For 43 M schuBERT model h/a = 25 . 3 whereas k = v = 64 . Also, f should be much larger than 4 h . For the same 43 M schuBERT model 4 h = 936 whereas f = 3072 . Table 13 shows that 43 M schuBERT ( (cid:96) = 12 , h = 304 , a = 12 , k = v = 64 , f = 3072 ) significantly outperforms BERT(cid:96) ( (cid:96) = 3 , h = 768 , a = 12 , k = v = h/a, f = 4 h ).
abstain abstain abstain result objective result abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain method abstain method abstain objective result objective objective objective method other other other other other abstain other other method other other other other other method objective other method method other other other other other other other other other other other method other abstain method abstain abstain abstain abstain abstain abstain other method abstain abstain abstain abstain abstain abstain abstain abstain other abstain abstain abstain method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain method abstain method abstain method method method abstain method abstain method abstain abstain abstain abstain abstain abstain method method method method other abstain method method method method method method method method abstain method abstain abstain abstain abstain abstain abstain abstain abstain abstain method method method method method method abstain other method method method method method method method method abstain method method method abstain abstain abstain method method method method method abstain abstain method method abstain abstain abstain abstain abstain method abstain abstain abstain abstain abstain abstain method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain method abstain abstain method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain method abstain abstain abstain abstain abstain abstain