Microsoft Azure AI wzchen@microsoft.com Abstract Model ensemble is a popular approach to produce a low-variance and well-generalized model. However, it induces large memory and inference costs, which are often not affordable for real-world deployment. Existing work has resorted to sharing weights among models. However, when increasing the proportion of the shared weights, the resulting models tend to be similar, and the benefits of using model ensemble diminish. To retain ensemble benefits while maintaining a low memory cost, we propose a consistency-regularized ensemble learning approach based on perturbed models, named CAMERO. Specifically, we share the weights of bottom layers across all models and apply different perturbations to the hidden representations for different models, which can effectively promote the model diversity. Meanwhile, we apply a prediction consistency regularizer across the perturbed models to control the variance due to the model diversity. Our experiments using large language models demonstrate that CAMERO significantly improves the generalization performance of the ensemble model. Specifically, CAMERO outperforms the standard ensemble of 8 BERT-base models on the GLUE benchmark by 0 . 7 with a significantly smaller model size ( 114 . 2 M vs. 880 . 6 M). Deep Neural Networks (DNNs) have achieved remarkable success in various fields and have become very powerful in learning complicated models (De-vlin et al., 2018; Brown et al., 2020; He et al., 2020). However, their remarkable representation powers come at the expense of large model variance, which may hurt the model generalization performance. A popular approach for reducing such variance is model ensemble, where the weights or predictions of a set of models are aggregated to produce the  Work was done during an internship at Microsoft Azure AI. predictions (Yang and Lv, 2021; Dong et al., 2020). For example, Zhang et al. (2018) show that a simple 2 -model ensemble leads to notable improvement over a single model in computer vision tasks. Despite such notable benefits, model ensemble has not been widely applied to large language models. The major barriers are its enormous storage and expensive inference cost, which linearly scales with the size and the number of models. Therefore, it is often not affordable to ensemble large language models for deployment using memory-constrained and low-latency edge devices. To alleviate the memory burden, recent works have resorted to a weight-sharing strategy, where all models share the same set of bottom-layer weights, on top of which branches out a set of parallel, un-shared top-layer weights (Lan et al., 2018; Chen et al., 2020; Li et al., 2020). Since the shared weights are optimized to accommodate multiple diverse un-shared branches, they can learn shared representations with better generalization (Liu et al., 2020; Luong et al., 2015; Ruder et al., 2019). For large models, however, such a weight-sharing strategy no longer enjoys the same benefits. Due to memory constraints, a significant proportion of bottom-layer weights need to be shared. Accordingly, top-layer branches have only limited capacity, and therefore the resulting models tend to be similar (Chen et al., 2020; Rame and Cord, 2021; Feng et al., 2020; Yang et al., 2021; Wu and Gong, 2021) (Figure 1 (Left)). Due to the lack of the model diversity, their ensemble cannot gain much improvement in generalization. As shown in Figure 1 (Right), both the generalization performance and model variance of the ensemble model are similar to those of a single model when the branch size is small. To retain a light memory cost while maintaining the ensemble benefits, we propose a new Consistency-regulArized enseMble lEarning ap-7162 2 4 6 8 10 Branch Size (Layers) 98 . proach based on peRturbed mOdels  CAMERO. Specifically, we share the bottom-layer weights across all models and apply different perturbations to the hidden representations for different models. Such a perturbation strategy effectively promotes the model diversity. Accordingly, the weights at each layer are optimized to produce consistent outputs given diverse input representations from differently perturbed models. In other words, the shared weights are essentially an on-the-fly ensemble of all perturbed models. In the end, we ensemble all branches on top of the shared weights to produce the final model, which has both a low variance and good generalization performance. Since we apply perturbations in large models with significant depth, different models' hidden representations may end up being extremely diverse, especially in upper layers. As a result, optimizing the shared weights to accommodate such perturbations can be very challenging. To prevent the models from being over-diverse, we apply a consistency regularizer to reduce variance across different models. Specifically, such a consistency regularizer can be viewed as collaborative distillation across models (Guo et al., 2020; Lan et al., 2018; Zhang et al., 2018; Kim et al., 2021; Chen et al., 2020; Li et al., 2020). By regularizing the discrepancy between each model's output logits and the ensemble of these logits, it encourages all models to be consistent in their predictions. We thus adopt consistency regularization to control the perturbed models' diversity from being too large, and thus ease the optimization of the shared weights. We conduct thorough experiments to demonstrate CAMERO's effectiveness and efficiency in ensembling large number of models with more than hundreds of millions of learning parameters. Specifically, our experiments in fine-tuning the BERT-base model on the GLUE benchmark achieve 0 . 7 points of gain in terms of task-average score with a significantly smaller parameters over the vanilla ensemble approach ( 114 . 2 M vs. 880 . 6 M) and achieve 1 . 2 points of gain with the same amount of learning parameters over the single model. CAMERO also achieves significant improvements in neural machine translation on both low-resource and high-resource language pairs. Furthermore, we verify that CAMERO can learn shared layers with better generalization and ensemble model with smaller variance. We also investigate the effects of using different types and strengths of perturbation and consistency regularization techniques. In particular, we observe that models created with virtual adversarial perturbation (Jiang et al., 2019a) and neuron dropout (Srivastava et al., 2014) lead to ensemble models with the best generalization performance. Lastly, we demonstrate CAMERO's effectiveness on a larger-scale model, RoBERTa-large (Liu et al., 2019), where it achieves 0 . 8 and 0 . 9 points of gain over the vanilla ensemble approach and single model performance, respectively. Our codes are released at https: //github.com/cliang1453/CAMERO . Notations. We use f (  ;  ) to denote a mapping f associated with the parameter  from the input sample to an output space, where the output is a multi-dimensional probability simplex for classification tasks and a scalar for regression tasks. We denote the model's final logits as g (  ;  ) , where f (  ;  ) =  ( g (  ;  )) and  (  ) is the Softmax function. We denote n pairs of data samples of the target task as { ( x i , y i ) } ni =1 . The training loss of f (  ;  ) is computed as (cid:96) ( f ( x i ;  ) , y i ) for any given training instance ( x i , y i ) where (cid:96) (  ;  ) denotes the loss function. We use DKL ( P || Q ) = (cid:80) k p k log( p k /q k ) to denote the KL-divergence of two discrete distributions P and Q with the associated parameters of p k 's and q k 's, respectively. Collaborative Distillation. Collaborative distillation approaches train two or more models in parallel while regularizing the consistency of their final prediction distributions (Guo et al., 2020; Lan et al., 2018; Zhang et al., 2018; Kim et al., 2021; Chen et al., 2020; Li et al., 2020). Specifically, we use { f (  ;  j ) } mj =1 to denote m individual models with the same architectures with parameters by  1 , ...,  m , respectively, and denote 7163  = {  1 , ...,  m } . A typical collaborative distillation approach solves the following optimization problem: min  L () +  R () , where  > 0 is a tuning parameter, and L () and R () are defined as L () = 1 m m (cid:88) j =1 (cid:96) ( f ( x ;  j ) , y ) , R () = 1 m m (cid:88) j =1 D ( f ( x ;  j ) , E ( x ; )) . (1) For notational simplicity, we will omit the subscript i throughout the rest of the paper. Here, E ( x ; ) de-fines a mapping function associated with  , which maps the input sample x to a multi-dimensional probability simplex or a scalar depending on the tasks. A commonly adopted ensemble-distillation approach makes E (  ; ) =  ( (cid:80) mj =1 w j g ( x ;  j )) , where { w j } mj =1 are non-negative scalars summing to one. D (  ,  ) denotes the distance metric of two discrete distributions P and Q or two scalars p and q . D ( P, Q ) can take the form of symmetric KL-Divergence, 12 ( DKL ( P || Q )+ DKL ( Q || P )) . D ( p, q ) or euclidean distance (cid:107) p  q (cid:107) 22 . Weight-Sharing. Weight-sharing technique has been adopted in several representation learning scenarios, e.g., multi-task learning (Liu et al., 2020; Luong et al., 2015; Ruder et al., 2019), multi-domain learning (Britz et al., 2017; Zeng et al., 2018; Tars and Fishel, 2018; Jiang et al., 2019b) and multi-lingual tasks (Gu et al., 2018; Aharoni et al., 2019). Weight-sharing strategy can reduce the number of free parameters in the model, which helps prevent overfitting in the training and lead to better generalization abilities. We introduce CAMERO, a weight-sharing ensemble learning approach based on consistency-regularized perturbed models. Based on the multi-layer structures of neural networks, we divide each model into two parts: the bottom-layers and the top-layers. The model parameters in bottom-layers are shared across all models. Specifically, the parameters of the j -th model is denoted as  j = [  0 ,  (cid:48) j ] , where  0 denotes the Figure 2: Illustration of CAMERO during training. shared weights in bottom-layers, and  (cid:48) j denotes the top-layer weights of the j -th model. Based on such a compositional structure, the j -th model's output can be denoted as f ( x ;  j ) = f K ( f K  1 ( . . . f 1 ( x ;  (1) j ) . . . ;  ( K  1) j );  ( K ) j ) , where f k (  ;  ( k ) j ) is the mapping associated with the k -th layer parameter  ( k ) j , and  0 consists of {  ( k ) j } K (cid:48) k =1 for some K (cid:48)  { 1 , ..., K } , which shares across models. A significant proportion of shared weights leads to the models' similarity, which accordingly impairs the ensembled model's performance. To in-crease the models' diversity, we consider perturbing each layer's hidden representations for different models during training (Figure 2). Specifically, the j -th model's output is denoted as f ( x ;  j ,  j ) = f K ( f K  1 ( . . . f 1 ( x ;  (1) j ) +  (1) j . . . ;  ( K  1) j ) +  ( K  1) j ;  ( K ) j ) , where  ( k ) j is the perturbation applied at the k th layer's hidden representations, and  j = {  ( k ) j } K  1 k =1 , which is sampled from a distribution P . We then train m models with SGD-type algorithms using the following loss: L  () = 1 m m (cid:88) j =1 E  j P [ (cid:96) ( f ( x ; [  0 ,  (cid:48) j ] ,  j ) , y )] . Remark 1. We can consider a wide variety of perturbations for hidden representations, input embed-dings or data samples, e.g., random perturbation (Aghajanyan et al., 2020), virtual adversarial perturbation (Miyato et al., 2018; Jiang et al., 2019a), neuron dropout (Srivastava et al., 2014) and word dropout (Wei and Zou, 2019). In large models with significant depth, different models' hidden representations may end up being extremely diverse, especially in upper layers. As a result, optimizing the shared weights to accommodate such diverse inputs can be very challenging. To address this issue, we propose to control model variability through consistency regularization. Specifically, we regularize the consistency among m models' final prediction distributions by minimizing the following loss, R  () = 1 m m (cid:88) j =1 E  j P [ D ( f ( x ; [  0 ,  (cid:48) j ] ,  j ) , E ( x ;  , {  j } mj =1 ))] , (3) where E ( x ;  , {  j } mj =1 ) denotes the final prediction distribution produced by some ensemble method applied upon models with perturbed representations. For example, commonly adopted ensemble methods include logits ensemble, where E ( x ;  , {  j } mj =1 ) =  ( (cid:80) mj =1 w j g ( x ;  j , {  j } mj =1 )) . In summary, we train m models by minimizing the following overall loss function: L  () +  R  () , where L  () is defined in Eq. (2) and R  () is defined in Eq. (3). We adjust the strength of consistency regularization via  , a non-negative hyper-parameter. Remark 2. Different from existing weight-sharing strategies, which control models' diversity via the amount of shared and un-shared weights, CAMERO controls model diversity via the strength of perturbation and regularization. Such a difference renders significant memory benefits. In practice, we safely share all layers except a single top layer. Accordingly, the memory storage is reduced to that of a single model. This allows us to explore the behaviors of ensemble learning under a larger number of models. We verify the effectiveness of CAMERO on widely used benchmarks for natural language understanding and neural machine translation. Model and data. We evaluate the fine-tuning performance of BERT-base (110M) (Devlin et al., 2018) and RoBERTa-large (335M) (Liu et al., 2019) on the General Language Understanding Evaluation (GLUE, Wang et al. (2018)) benchmark. GLUE contains nine NLU tasks, including textual entailment, question answering, sentiment analysis, and text similarity. Details about the benchmark are deferred to Appendix A.1.1. Baseline methods. We compare CAMERO with Vanilla, where all models are independently trained without consistency regularization. We also compare CAMERO with representative collaborative distillation methods: Deep Mutual Learning (DML, Zhang et al. (2018)), On-the-fly Native Ensemble Learning (ONE, Lan et al. (2018)) and Knowledge Distillation via Collaborative Learning (KDCL, Guo et al. (2020)) 1 . DML trains two models with alternating updates while regularizing the consistency between their final prediction distributions. KDCL extends two models to multiple models, training all models concurrently while regularizing the consistency between the prediction distribution of each individual model and of the ensemble of all models. ONE adopts the traditional weight-sharing strategy with a learnable gating factor assigned to each individual branch, which helps to control the model diversity. Perturbation. We demonstrate the effectiveness of CAMERO using neuron dropout (Srivastava et al., 2014), one of the most straightforward perturbation techniques which randomly zeros out neurons based on a small, fixed ratio. In particular, the ratio adopted in our experiments is 0 . 1 . In Section 5.3, we further demonstrate that a wide variety of perturbations, including virtual adversarial perturbation (Jiang et al., 2019a), random perturbation (Agha-janyan et al., 2020) and word dropout (Wei and Zou, 2019), can all serve the role. Consistency regularization. We demonstrate the effectiveness of CAMERO using the ensemble consistency defined in Eq. (1). In Section 5.4, we further investigate the effectiveness of different types of consistency regularization techniques. Initialization. To fine-tune the BERT encoder on downstream tasks, the common initialization approach is to append a randomly initialized, fully connected classification layer on top of the encoder 1 We do not include the data augmentation technique proposed in KDCL for a fair comparison. (Devlin et al., 2018). For ONE and CAMERO, we append m differently initialized, parallel classification layers on top of the encoder. For other methods, we initialize m individual encoders and append a differently initialized classification layer on top of each. Inference. For ONE and CAMERO, we conduct a single pass through the encoder and average the predicted logits of m classification layers. For other methods, we average the predicted logits of m models. All results in the following experiments are evaluated based on such a logits ensemble. Implementation details. Our implementation is based on the MT-DNN code-base 2 . We follow the suggested training and hyper-parameters settings from Liu et al. (2020). Specifically, we adopt Adamax (Kingma and Ba, 2014) as the optimizer with  = (0 . 9 , 0 . 999) . We tune  in range of { 0 . 5 , 1 , 2 , 5 } for all methods. Comprehensive training details are reported in Appendix A.1.2. Results of BERT-base. Table 1 shows the evaluation results of BERT-base on the GLUE development set. The results are averaged over five random 2 https://github.com/namisan/mt-dnn seeds, and all gains are statistically significant 3 . We have the following observations: 1) With significantly less learning parameters, CAMERO achieves a prominent and consistent margin over Vanilla, DML and KDCL. This suggests that CAMERO can produce better-generalized ensemble model with higher parameter efficiency. 2) CAMERO significantly outperforms ONE, suggesting that applying perturbations to models effectively improves the performance of weight-sharing strategy. 3) As the number of models increases from 2 to 8 , CAMERO's performance steadily increases for 6 out of 8 tasks, while Vanilla and KDCL fail to do so. Results of RoBERTa-large. We further verify that CAMERO can benefit an even larger model, RoBERTa-large. As shown in Table 2, CAMERO achieves consistent gains across all tasks 4 . Worth noticing, Vanilla shows limited improvements upon the single model performance (e.g., the gains are 3 All results have passed a paired student t-test with p-values less than 0 . 0 . 0 , 0 . 1 and  0 . 4 on QNLI, SST-2 and RTE, re-spectively). We conjecture that the high model variance in large models compromises the ensemble benefits. In contrast, by control the model variance with regularization, CAMERO achieves gains of 0 . 6 , 0 . 6 and 1 . 1 on these tasks. Model and data. We further evaluate CAMERO on the Transformer-base NMT model (Vaswani et al., 2017) using widely used IWSLT (Cet-tolo et al., 2016) 5 and WMT (Bojar et al., 2016) 6 datasets. Specifically, we adopt IWSLT'14 En  De, IWSLT'16 En  Fr and WMT'14 En  De. IWSLT En  De and En  Fr are low-resource datasets containing 160k and 236k sentence pairs. WMT En  De is a rich-resource dataset containing 4.5M sentence pairs. Model and dataset details are deferred to Appendix A.2.1. Implementation details. Our implementation is based on the fairseq code-base and follows the training and hyper-parameters settings from Ott et al. (2018, 2019). Specifically, we use 5  10  4 as the learning rate and employ Adam (Kingma and Ba, 2014) as the optimizer with  = (0 . 9 , 0 . 98) . We select  in range of { 1 , 2 , 5 } . For ONE and CAMERO, we randomly initialize multiple parallel decoders's last layers as the un-shared branches. Comprehensive training details are reported in Appendix A.2.2. Main results. Table 3 shows the BLEU scores on the IWSLT test set and the SacreBLEU scores (Post, 2018) with compound splitting on the WMT test set 7 . WMT's corresponding BLEU scores are reported in Appendix A.2.3. With a number of learning parameters similar to a single model, CAMERO achieves around 2 and 1 points upon ONE, and improves around 0 . 4 and 0 . 4 points upon KDCL, on low-resource and rich-resource datasets, respectively. This suggests that other than fine-tuning, CAMERO also improves the generalization of training-from-scratch models in both low-resource and rich-resource datasets. We first verify that CAMERO leads to a well-generalized and low-variance ensemble model. We 5 https://wit3.fbk.eu/ 6 http://data.statmt.org/wmt16/translation-task/ 7 We evaluate the SacreBLEU score on the average of last 10 checkpoints. then demonstrate how the perturbation and consistency regularization strength influences the model diversity and performance. Finally, we demonstrate CAMERO's effectiveness on various types of perturbation and regularization techniques. We verify that CAMERO allows the shared weights to learn better-generalized representations. Specifi-cally, we attach a randomly initialized classifier on top of a BERT-base encoder trained by CAMERO. We then fix the encoder and fine-tune the attached classifier only. As shown in Table 4, the encoder trained by CAMERO learns better representations than ONE's consistently across different tasks and under different numbers of models. We verify that CAMERO produces an ensemble model that both generalizes well and has a low variance across different random seeds under a light parameter budget. Figure 3 plots the prediction accuracy of 2 -model and 4 -model ensemble across five seeds. For example, in MNLI, CAMERO's 2 model ensemble ( 110 . 7 M) achieves similar performance to KDCL's 4 -model ensemble (440.2M) and CAMERO's 4 -model ensemble (111.9M) achieves an even better performance. Across different tasks, CAMERO's ensemble model has a similar or lower variance than all others. Complete variance statistics are presented in Appendix A.1.3. Types of perturbation. We verify that CAMERO produces well-generalized ensemble models under various types of perturbations. Specifically, we apply virtual adversarial perturbation (Jiang et al., 2019a) and random noise perturbation (Aghajanyan et al., 2020) on the first layer input embeddings, neuron dropout on all layers' input representations (Srivastava et al., 2014), and word dropout on the input sentences (Wei and Zou, 2019). Specifi-cally, we set the dropout ratio to be 0 . 1 for neuron dropout and 0 . 05 for word dropout. We set the norm constraint (cid:15) = 1  10  5 for both virtual adversarial perturbation and random noise perturbation. The random noise is sampled from a normal distribution. As shown in Table 5, CAMERO leads to significant margin of improvement under all types of perturbation. In particular, virtual adversarial 7167 # of Method IWSLT WMT Models En-De De-En En-Fr Fr-En Avg. perturbation and neuron dropout perform consistently well on all tasks. Random noise perturbation performs well on larger tasks (e.g., MNLI, QNLI, SST-2) while the gains shrink on smaller tasks. Strength of perturbation. We then verify that a larger perturbation strength improves the perturbed models' diversity during training. As consistency loss is computed as the average distance between all perturbed models' output logits to the ensembled logits at each iteration, it directly reflects the model diversity during training. As shown in Figure 4 (Left), a larger neuron dropout ratio leads to larger consistency loss, thus higher model diversity. Furthermore, we observe that a larger perturbation strength leads to a lower-variance ensemble model. As shown in Figure 4 (Right), as the neuron dropout ratio grows, CAMERO's ensemble model variance decreases. In contrast, ONE has a large variance under all ratios. Types of consistency regularization. We then investigate the effects of using different types of consistency regularization techniques. Specifically, we compare the existing ensemble consistency , as defined in Eq. (1), and a newly proposed pairwise consistency , which is defined as R () = 2 m ( m  1) m (cid:88) j =1 m (cid:88) p = j +1 D ( f ( x ;  j ) , f ( x ;  p )) . The pairwise consistency measures the average distance between each pair of models' output logits, thus we expect it to capture the discrepancies among models more accurately. As shown 7168 Perturbation Types MNLI QNLI SST-2 MRPC CoLA Avg. in Table 6, CAMERO shows consistent improvements under both types of regularization. In particular, pairwise consistency shows larger advantages on smaller tasks (e.g., 0 . 3 on MRPC and 0 . 4 on CoLA). Strength of consistency regularization. We further investigate how the strength of the regularization factor  affects the ensemble model's performance. As shown in Figure 5, as  increases, the generalization performance of the ensemble model first increases, then decreases. This suggests that regularization can effectively benefits the generalization performance through balancing the model diversity. We propose CAMERO, a consistency-regularized ensemble learning approach based on perturbed models. Such a strategy significantly improves the parameter efficiency of model ensemble in large language models, making it an accessible and powerful technique for learning ensemble models with better generalization performances.
abstain abstain abstain abstain abstain abstain method method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain objective abstain method abstain abstain abstain method result abstain method abstain abstain method objective result abstain abstain abstain result objective result abstain abstain abstain other other method abstain abstain other method other other method other method other other other other other other other other method method abstain abstain abstain other abstain method other method abstain method abstain abstain method method abstain method abstain abstain method abstain abstain abstain method abstain method result abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain other result abstain abstain abstain abstain result abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain other abstain abstain abstain method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain method abstain method abstain abstain abstain abstain abstain abstain abstain objective abstain