Building huge and highly capable language models has been a trend in the past years. Despite their great performance, they incur high computational cost. A common solution is to apply model compression or choose light-weight architectures, which often need a separate fixed-size model for each desirable computational budget, and may lose performance in case of heavy compression. This paper proposes an effective dynamic inference approach, called E-LANG, which distributes the inference between large accurate Super-models and light-weight Swift models. To this end, a decision making module routes the inputs to Super or Swift models based on the energy characteristics of the representations in the latent space. This method is easily adoptable and architecture agnostic. As such, it can be applied to black-box pre-trained models without a need for architectural manipulations, reassembling of modules, or re-training. Unlike existing methods that are only applicable to encoder-only backbones and classification tasks, our method also works for encoder-decoder structures and sequence-to-sequence tasks such as translation. The E-LANG performance is verified through a set of experiments with T5 and BERT backbones on GLUE, SuperGLUE, and WMT. In particular, we outperform T5-11B with an average computations speed-up of 3.3  on GLUE and 2.9  on SuperGLUE. We also achieve BERT-based SOTA on GLUE with 3.2  less computations. Code and demo are available here. With the introduction of influential language models such as BERT (Devlin et al., 2019), a trend in natural language processing (NLP) research has been to develop high capacity models and push their performance to new levels. Consequently, state-of-the-art (SOTA) results were achieved on various benchmarks using these models; GPT-3 (Brown et al., 2020), XLNet (Yang et al., 2019), RoBERTa (Liu et al., 2019), T5 (Raffel et al., 2020), ELECTRA (Clark et al., 2020), and DeBERTa (He et al., 2021) to name a few. A potential down-side, however, is that the number of parameters or float-ing point operations (FLOPs) for these models can get extremely large. For example, Gshard (Lep-ikhin et al., 2021) comes with 600B parameters with an enormous amount of computation. This in turn results in a higher inference latency, which is not desirable for latency-sensitive applications. A common solution to speed-up the large language models is to apply model compression (Gupta et al., 2020). Although generally successful, compression does come with a trade-off on accuracy, and may lose performance if compression is heavy. In addition, these methods usually compress a model to a fixed smaller size, where a separate model is required for each possible computational budget. An alternative approach explored in the literature is to leverage dynamic inferencing in a way that examples may be routed to different (potentially lower cost) paths throughout the network. For example, a temporal early-exit model (Shen et al., 2017; Yu et al., 2018) terminates the procedure of reading the input sequence when sufficient evidence has been found for accurate predictions. Instance-wise early-exiting (Xin et al., ACL 2020) is another technique, which allows a sample to adaptively choose from multiple available exit nodes if some conditions are met. Consequently, earlier exists require less computation and lead to a lower latency. Adjusting the size of the model at the inference time by choosing adaptive width and depth is also another approach employed for dynamic inference (Kim and Cho, 2021; Hou et al., 2020). There is a variety of adaptive/dynamic inference approaches proposed, however, a general down-side for many of these methods is that often times they require a careful architecture design, manipulation of network modules, or even re-training. In this paper, we propose a simple but rather effective approach of dynamically distributing the inference between the original large model (called the Super model) and a light-weight (e.g., compressed) model referred to as the Swift model. To this end, we design an energy-based decision making module that routes examples to the appropriate model based on the negative free energy of the latent space representations, such that the Swift model attains a high accuracy on the examples sent to it. The remaining samples are then forwarded to the Super model that is supposed to have a good performance on all examples. Since the Swift model can make highly accurate predictions over the majority of the samples, E-LANG significantly reduces the overall computational cost, while maintains the high accuracy of the Super model. Although simple, this strategy achieves SOTA results on multiple structures (e.g., T5 and BERT) and benchmarks (e.g., GLUE and SuperGLUE). Due to its desirable practical characteristics, this method is a strong candidate for the practical application of Super models. The main contributions of the paper are as follows:  Combining Super models with high accuracy and latency and Swift models with lower accuracy and latency, to achieve high accuracy and low latency . In other words, by employing our method, we can achieve the high levels of accuracy provided by Super models, but at a lower computational cost. Our method is easily adoptable, architecture agnostic, and orthogonal to many other existing methods. It can be applied to black-box pre-trained models without a need for architectural manipulations, careful reassembling of modules, or re-training. An energy-based routing mechanism for directing examples to the Super or Swift. This provides a dynamic trade-off between the accuracy and computational cost that outperforms the previous works in both fixed-size and dynamic inference (with zero overhead for real-time adjustment of speed/accuracy). As such, E-LANG acts like a knob for adjusting the accuracy-latency trade-off in real-time during model serving. To the best of our knowledge, our method is the first generic approach to apply dynamic inference on both encoder-only and encoder-decoder architectures (e.g., T5) and also can extend the usage beyond classification tasks, to sequence-to-sequence tasks such as translation. As mentioned, compression is a widely used strategy to speed-up the large language models (Gupta et al., 2020; Gupta and Agrawal, 2022). This involves incorporating techniques such as quantization of weights and activations (Bai et al., 2021; Shen et al., 2020; Kim et al., 2021; Zhang et al., 2020; Jin et al., 2021), knowledge distillation (KD) (Hinton et al., 2015; Jiao et al., 2020; Sanh et al., 2019), pruning/sharing (Gordon et al., 2020; Chen et al., 2020), multi-device distribution (Banitalebi-Dehkordi et al., 2021), or a combination of these techniques (Cheng et al., 2017; Polino et al., 2018). Among all the compression techniques, creating a fixed-size small version of large models along with distillation has been popular in the recent years. Sanh et al. (2019) introduced DistillBERT, which was a smaller version of BERT trained with distillation for general purposes. Another compact variant of BERT was proposed by MobileBERT (Sun et al., 2020) in which inverted bottleneck structures and progressive knowledge transfer were used. TinyBERT (Jiao et al., 2020) also presented a novel two-stage transformer distillation for both pre-training and task-specific fine-tuning. In (Iandola et al., 2020), the usage of grouped convolutions was studied to design SqueezeBERT. ELM (Jiao et al., 2021), a layer mapping search framework, was also proposed for improving downstream BERT distillation. A recent method, GhostBERT (Huang et al., 2021), employed softmax-normalized 1D convolutions as ghost modules to generate more features with cheap operations. Although compression techniques in general are effective, they come with a trade-off on accuracy, and may lose performance in case of high ratio compression. In addition, an individual fixed-size model is required for each possible computational budget. As stated in the introduction, the alternative solution is dynamic inference, which can be achieved with either early-exit or length/depth-adaptive models. One of the first temporal early-exit strategies was proposed by ReasoNet (Shen et al., 2017), which stops its reading procedure when sufficient evidence has been found for answering a question. Similarly, in (Yu et al., 2018), an early stopping method applicable to classification tasks was presented. DeeBERT (Xin et al., ACL 2020) also proposed an instance-wise multi-exit method via the entropy of the output probability distribution to speed-up BERT inference. As a length-adaptive method, Kim and Cho (2021) introduced a dynamic inference framework with one-shot training of transformers for both sequenceand token-level classification. Also, in (Hou et al., 2020), an architecture named DynaBERT was proposed for adaptively adjusting the computations by choosing sub-networks of different widths and depths. Both Length-Adaptive and DynaBERT utilized knowledge distillation and data augmentation to improve their performance. Although early-exit and adaptive methods have made significant progress and work well in practice, they often require architectural manipulation and re-training. In addition, they are only applicable to encoder-only backbones and classification tasks. In contrast, our method can work with out-of-the-box pre-trained models without a need for re-training and are also applicable for encoder-decoder structures and sequence-to-sequence tasks. We propose a new energy-based joint inference method called E-LANG, where a large/accurate language model ( Super ) is jointly employed with a small/fast one ( Swift ) to achieve efficient inference without sacrificing the accuracy. To this end, inspired by the method in (Akbari et al., 2021), a routing mechanism empowered by energy-based models (EBM) is introduced to dynamically distribute the input samples between the Super and Swift models. Similar to the out-of-distribution (OOD) detection problem, our goal is to identify the OOD samples that are hard to handle for the Swift and forward them to the Super model. On the other hand, we have the in-distribution data for which the Swift can make highly reliable and accurate predictions. In other words, the routing mechanism needs to detect whether or not the input data fits in the Swift's distribution (i.e., the one the Swift has been trained with). Inspired by the success of EBMs in dealing with OOD detection problems (Lee et al., 2019), the energy characteristics of data samples for an efficient and effective routing are investigated in our work. The overall framework of E-LANG is shown in Figure 1. 3.1 Energy-Based Models The goal of EBM is to build an energy function denoted by E ( x ) : RD  R that maps an input data x  RD to a non-probabilistic energy value y  R . To turn a collection of arbitrary energies for all possible outputs (denoted by Y ) into a normalized probability distribution, Gibbs distribution can be used as follows (LeCun et al., 2006): p ( y | x ) = e  E ( x ,y ) (cid:82) y (cid:48)  Y e  E ( x ,y (cid:48) ) , (1) where the negative log of the denominator expresses the Helmholtz free energy (LeCun et al., 2006) defined as F ( x ) =  log (cid:0) (cid:82) y (cid:48)  Y e  E ( x ,y (cid:48) ) (cid:1) . In machine learning, there is a deep relationship between the EBMs and discriminative models, which can be seen by connecting the Gibbs distribution in Equation (1) and the categorical distribution derived for a discriminative model. A discriminative classifier is defined as a function for mapping the input x to C real-valued logits (i.e., for C number of class labels): f ( x ) : RD  RC . In order to derive a categorical distribution over C possible outputs, the softmax function is utilized: p ( y | x ) = e f y ( x ) (cid:80) Ci e f i ( x ) , (2) where f y ( x ) denotes the logit (probability) of the y th class label. between the Gibbs and categorical distributions defined in (1) and (2), the energy function for a given input ( x , y ) can be defined as E ( x , y ) =  f y ( x ) . The free energy function F ( x ; f ) can then be obtained by taking the negative log of the categorical distribution denominator as: F ( x ; f ) =  log C (cid:88) i e f i ( x ) . Our goal is to detect the easy samples suitable for the Swift, which are indeed the ones with high likelihood in the density function. The energy-based density function for Swift is then defined as: p ( x ) = e  F ( x ; f ) (cid:82) x e  F ( x ; f ) , (4) where the denominator is the normalized densities, which can be intractable to compute or estimate. The log ( (cid:82) x e  F ( x ; f ) ) term has no effect on the distribution of the overall energy values because it is constant for all x . As a result,  F ( x ; f ) , i.e., the negative free energy, has a linear alignment with the log likelihood function, which makes it a well-suited solution to the easy vs. hard detection problem in our framework. To this end, lower energy values indicate higher likelihood and represent easier (more fit) samples for the Swift model. More precisely, for a threshold  on the density function such that p ( x ) <  , then a threshold t on the negative free energy can be calculated according to (5) as  F ( x ; f ) < t = log (  (cid:82) x e  F ( x ; f ) ) . In practice, for a given input, an energy function is applied to the outputs of the Swift model during inference time to calculate the energy score. Then, if the negative energy value is smaller than a threshold, the input is identified as a bad sample for the Swift, and is sent to the Super model. Given the energy threshold t , the Swift classifier f ( x ) , and the Super classifier defined as g ( x ) : RD  RC , the joint inference function J ( x ; f, g, t )  [1 , C ] for a classification task with C classes can then be expressed by: J ( x ; f, g, t ) = (cid:40) f ( x ) if  F ( x ; f )  t g ( x ) otherwise. The proposed energy-based joint inference solution can be directly applied to the encoder-only models such as BERT that are designed for text classification tasks. To this end, the energy scores corresponding to the BERT-based Swift model are obtained using Equation (3) and the joint inference is performed based on Equation 6. On the other hand, for the encoder-decoder (auto-encoder) architectures such as T5, which are usually considered as generative models, some mod-ifications are required. Encoder-decoder models are basically designed for sequence-to-sequence (e.g., text-to-text) problems such as translation or summarization. Although such models can also be employed for classification tasks, they still consider the task as a text generation (sequence-to-sequence) problem, where the target labels and the output predictions are treated as a sequence or a piece of text. In Section 3.1, it was discussed that there is an inherent connection between the discriminative clas-sifiers and the EBMs. In order to benefit from this characteristic for encoder-decoder architectures, we consider adding an extra classification head (i.e., a single linear layer) to the Swift model. As encoders are commonly considered as better feature extractors for training a classifier rather than the decoders, we place the extra head after the Swift encoder. While freezing the pre-trained encoder model (denoted by f E ), the extra energy head (de-noted by h ) is trained as a regular classifier head with C class labels. Note that the decoder is not required for training the head. The corresponding free energy function is then defined as follows: F ( x ; f E , h ) =  log C (cid:88) i e h i (cid:0) f E ( x ) (cid:1) , (7) where f E ( x ) denotes the outputs of the encoder's last hidden state. These features are then fed to the extra head h to obtain the logits for the i th class required for computing the energy scores. In this approach, as the decoder part of the Swift model is not required for calculating the energy scores, less computations are involved and the joint inference is performed more efficiently. For text-to-text (or sequence-to-sequence) problems such as translation, the output is a sequence of M word-pieces from a vocabulary/dictionary of size N . To still utilize the relationship of discriminative models and EBMs in designing and training the extra energy head, we can treat the text-to-text 5232 models as M multi-class classifiers. In this case, the number of class labels, i.e., C in (7), is equal to N . The final energy score is then calculated as the average of M energy values as follows: F ( x ; f E , h ) =  1 M (cid:80) Mm (cid:0) log (cid:80) Ci e h m,i (cid:0) f E ( x ) (cid:1)(cid:1) , (8) where h m,i ( . ) denotes the logits corresponding to the m th word in the sequence and i th class label. Denote the Swift's decoder by f D , the joint inference function, J ( x ; f, g, h, t ) , based on energy scores in either Equation (7) or (8) is expressed as: J = (cid:40) f D (cid:0) f E ( x ) (cid:1) if  F ( x ; f E , h )  t g ( x ) otherwise. In addition to energy, softmax and entropy (Xin et al., ACL 2020) scores can also be used for analyzing the Swift model's performance in the routing mechanism. In this sub-section, we study the mathematical connection of them with the energy score and their potential to solve our problem. The softmax score for a classifier is expressed by: max y p ( y | x ) = max y e f y ( x ) (cid:80) Ci e f i ( x ) = e f max ( x ) (cid:80) Ci e f i ( x ) . (10) By taking the logarithm of both sides, we see the connection between the log of the softmax and the free energy score formulated in Equation (3): log max y p ( y | x ) = log ( e f max ( x ) )  log C (cid:88) i e f i ( x ) = f max ( x ) + F ( x ; f ) , (11) where all logits are shifted by their maximum f max ( x ) . Plugging in the energy term to (5) yields: log max y p ( y | x ) =  log ( p ( x )) + f max ( x )  log (cid:0) (cid:90) x e  F ( x ; f ) (cid:1) . (12) It is observed that for the samples with high likelihood of being in the Swift's distribution, the free energy goes lower, but the max logit tends to go higher. Due to this shifting, unlike the energy score, the softmax score is not well-aligned with the probability density p ( x ) . As a result, the softmax score is less reliable for our routing module to analyze the performance of the Swift. The entropy score is a measure of randomness in the processed information, and is calculated as: where f i ( x ) is the probability (logit) corresponding to the i th class label. Let U be the internal energy, i.e., the expectation value of the energy function (Oh et al., 2020), defined by: U ( x ; f ) = C (cid:88) i E ( x , i ) f i . According to Oh et al. (2020), the entropy can be defined in terms of the internal and free energy functions as: H ( x ; f ) = U ( x ; f )  F ( x ; f ) , where all logits are shifted by the internal energy U . Substituting the free energy from (5) yields: H ( x ; f ) = log ( p ( x )) + U ( x ; f ) + log (cid:0) (cid:82) x e  F ( x ; f ) (cid:1) , (15) which shows, due to the shifting caused by internal energy, the entropy is not reliably aligned with the probability density p ( x ) . Thus, it is a less suitable routing mechanism unlike the energy score. In this section, the performance of E-LANG on different architectures such as T5 and BERT; and benchmarks such as GLUE (Wang et al., 2019b), SuperGLUE (Wang et al., 2019a), and WMT (Bojar et al., 2016) is evaluated and compared with the Super models and previous works. In Table 1, the T5-based results on GLUE, SuperGLUE, and WMT benchmarks are reported. For all the tasks, we use T5-11B (with 87  10 11 FLOPs) and T5-large (with 4.25  10 11 FLOPs) as our Super and Swift models, respectively. The average GPU-based running time and accuracy of both models compared with E-LANG are also summarized in the table. Note that the T5 models used in this experiment have been separately fine-tuned on each of the downstream tasks given in Table 1. The extra energy head for each of these tasks was also separately trained and used based on the task-specific number of classes, i.e., C in Equation (7). where N su and N sw are respectively the number of samples processed by the Super (with F su FLOPs) and Swift (with F Esw , F Dsw , and F h FLOPs for the encoder, decoder, and energy head). Note that F h is equal to  0.00001  10 11 , which has a very in-significant overhead in our framework. As presented in Table 1, E-LANG can reach the Super model's accuracy on all GLUE tasks with an average 3.3X in FLOPs and 1.8X in running time speed-ups. For some tasks such as QNLI, MRPC, and COLA, we even outperform the Super model, which leads to a higher average accuracy of 89.7% than the Super model with 89.5% on GLUE. For the SuperGLUE benchmark, with an average FLOPs and running time speed-up of 2.9X and 2.0X, our method achieves the same accuracy as the Super model on MRC and CB; and better accuracy on RTE and WIC. On BoolQ and COPA, although 99% and 97% of the Super's accuracy are respectively obtained, it is with 1.7X and 1.4X less FLOPs and latency, on average. In order to analyze the generality of E-LANG to other NLP problems rather than text classification (Section 3.2.1), we also apply our method to two text-to-text tasks including SuperGLUE's WSC and WMT's English-to-Romanian (EnRo) translation. As given in the table, our method achieves the Super model's accuracy on both WSC and EnRo with 4.2X and 1.4X less FLOPs, respectively. GLUE benchmarks. The curves related to all tasks are given in the supplementary materials. The tradeoff points on the curves are dynamically achieved at the inference time by selecting different thresholds, i.e., t in Equations (6) and (9). Larger values for t will result in routing more input data to the Super model, which consequently provides more accurate, but slower inference. As the Swift is able to make accurate predictions for the majority of input data, the dynamic inference with a small enough t can reach the Super model's accuracy but with a much lower computational cost and latency. Figure 3 illustrates the distribution of the energy scores across the input samples in GLUE tasks. For each task, the distributions of the samples processed by the Super and the Swift models are plotted. As shown, the samples routed to the Super model tend to have lower energy scores that are indeed considered as out-of-distribution samples for the Swift. On the other hand, in overall, higher scores are observed for the Swift distribution, that is for the samples handled by the Swift only. For some tasks such as MRPC and QNLI, the Swift is shown to be highly capable of handling the majority of the input samples. This is also supported by the results in Table 1 and Figure 2, where 91% (for MRPC) and 75% (for QNLI) of the samples are accurately processed by the Swift. In contrast, for other datasets including RTE and MNLI with Swift ratio of less than 50%, most of the samples are hard for the Swift, which are transferred to the Super model. Based on our experiments, the most optimal results for our joint inference framework is achieved when the crossing point of the two distributions (highlighted in green in the figures) is chosen as the threshold t in Equation (9). In Sections 3.3.1 and 3.3.2, the possibility of using softmax and entropy scores instead of energy score was theoretically analyzed. To support that analysis and also experimentally evaluate the performance of different routing mechanisms, an ablation study on GLUE is performed, which is presented in Table 2. In this study, we report the joint inference results based on softmax, entropy, and random scores (i.e., randomly distributing the samples between Super and Swift). Our experiments show that, compared to the random score, softmax and entropy can result in satisfactory performance on routing the samples. However, as also discussed in Sections 3.3.1 and 3.3.2, the energy score is still a better mechanism with about 14% less FLOPs. Another potential mechanism is the perplexity (Chen et al., 1998), but since it provides the same information as entropy, we did not add any extra experiment on it. The results with the usage of different Swift models including T5-small (with 0.33  10 11 FLOPs) and T5-base (with 1.24  10 11 FLOPs) are also given in Table 2. Using these models as Swifts can lead to good performance on some tasks, but not all of them. For example, on SST2, the joint inference with T5-small and T5-base Swifts can respectively reach the Super's accuracy with 1.9X and 2.X less computations. In general, although these models are smaller and require less FLOPs, our results in Table 2 indicate that they perform worse than T5-large in our joint inference structure. In Figure 2, the trade-off curves for different Swift models are shown for GLUE and SuperGLUE. Moreover, to show the effectiveness of the extra energy head for the Swift encoder, the E-LANG results based on last linear layer of the Swift decoder is also given and compared in Table 2. As reported, the E-LANG empowered by the energy head on the Swift encoder outperforms the case with the decoder's head in both FLOPs (36.8% less) and accuracy (0.7% better). As explained in Section 3.2.1, this shows the deep connection between the encoder's features, discriminative models, and the proposed routing mechanism via the energy head. We observed that E-LANG can achieve a high performance even when applied to individually pre-trained Super and Swift models. But, more improvement can still be obtained by performing KD from the Super to the Swift model, especially at the fine-tuning process for downstream tasks. To study this, we apply the KD technique in (Sanh et al., 2019) to the Super and Swift models for some GLUE tasks. As summarized in Table 3, the Super model's accuracy for QNLI, SST2, and COLA is respectively attained by the distillation-based E-LANG with 29.2%, 48.5%, and 14.3% less FLOPs than E-LANG (without distillation). The results show the effectiveness of E-LANG along with other compression techniques such as distillation. The trade-off curves for this experiment will be provided in the supplementary materials. In this section, the proposed energy-based joint inference method is applied to the BERT architecture (Devlin et al., 2019) and compared with Super (11B) 87.0 / 91.7 87.0 / 95.9 87.0 / 96.6 87.0 / 92.4 87.0 / 91.7 87.0 / 69.1 87.0 / 89.5 Random (Encoder) 78.5 / 91.5 61.9 / 95.3 58.7 / 96.3 60.2 / 91.2 47.5 / 91.9 61.6 / 67.2 61.4 / 88.9 Softmax (Encoder) 57.7 / 91.6 36.5 / 95.9 34.6 / 96.5 52.0 / 92.3 13.8 / 92.1 45.7 / 69.3 40.1 / 89.6 Entropy (Encoder) 55.7 / 91.6 27.1 / 96.0 40.2 / 96.5 50.7 / 92.0 23.0 / 92.2 48.1 / 69.3 40.8 / 89.6 Energy (Swift small ) 71.3 / 91.0 58.8 / 95.6 47.0 / 96.6 71.2 / 88.5 55.0 / 91.4 75.3 / 68.3 63.1 / 88.5 Energy (Swift base ) 54.5 / 91.5 50.5 / 95.8 35.9 / 96.6 55.8 / 90.6 44.0 / 91.9 50.6 / 68.4 48.5 / 89.1 Energy (Decoder) 57.9 / 90.6 68.1 / 95.5 75.8 / 96.3 60.5 / 91.5 20.2 / 90.9 45.1 / 69.3 54.6 / 89.0 Energy (Encoder) 47.8 / 91.7 25.7 / 96.0 32.0 / 96.6 50.4 / 92.4 11.5 / 92.2 39.9 / 69.5 34.5 / 89.7 BERT-based SOTA in both fixed-size and dynamic inference. The majority of the previous methods employ knowledge distillation and data augmentation techniques for training their student models. For a fair comparison, we follow the same practice and use the transformer distillation and augmentation strategies in TinyBERT (Jiao et al., 2020) to train and prepare our Swift model (i.e., BERT Tiny with 1 . 2  10 9 FLOPs). Moreover, similar to the other works, we use BERT Base (with 21 . 8  10 9 FLOPs) as our Super (i.e., teacher) model. In Table 4, the comparison results with the baseline BERT Base and SOTA on GLUE benchmark are presented in terms of accuracy, FLOPs, and latency. Compared to the Super model, E-LANG delivers better accuracy on SST2 and RTE with 3.5X and 2.0X FLOPs speed-up; and the same accuracy on QNLI, MRPC, and QQP with 2.4X, 2.7X, and 7.0X FLOPs speed-up, respectively. On MNLI and COLA, 99.8% and 97.3% of the Super model's accuracy are achieved, but with an average FLOPs speed-up of 2.3X. On average, E-LANG outperforms the Super model with 0.1% higher accuracy, 3.2X less FLOPs, and 1.6X less latency. Compared with SOTA, our method achieves the best performance on all GLUE tasks, except MRPC for which SqueezeBERT outperforms all due to having a more accurate teacher (Iandola et al., 2020). There are some works such as ELECTRA (Clark et al., 2020) and MobileBERT (Sun et al., 2020) that require less FLOPs than our method, but they only reach 95% of the baseline's accuracy. Compared to other methods, GhostBERT (Huang et al., 2021) and DynaBERT (Hou et al., 2020) give the closest performance to the baseline and even the same as ours on some tasks such as QNLI. However, on average, they still need about 30% more FLOPs on GLUE compared to E-LANG. The E-LANG accuracy vs. FLOPs trade-off curves compared to SOTA on some of GLUE tasks are shown in Figure 4. The trade-off curves for all the tasks are reported in the supplementary materials. Among the SOTA methods presented in Table 5236 MNLI (m/mm) QNLI SST2 RTE MRPC COLA QQP Avg. 4 and Figure 4, only DeeBERT (Xin et al., ACL 2020), Length-Adaptive (Kim and Cho, 2021), and DynaBERT (Hou et al., 2020) are in the category of dynamic inference, where a single model can operate at different trade-off points between accuracy and computational cost. The other approaches propose fixed-size smaller versions of BERT Base , which require re-training for every trade-off point. To investigate the orthogonality of E-LANG with others, we integrate our energy-based joint inference strategy with DynaBERT that is SOTA in BERT-based adaptive inference. In other words, we analyze whether E-LANG can be added on top of other efficient methods to benefit both from their designs and our approach. In this experiment, the DynaBERT configurations with the highest accuracy (i.e., width=0.75 & depth=1.0) and the lowest FLOPs (i.e., width=0.5 & depth=0.25) are respectively employed as the Super and Swift models in our framework. The corresponding joint inference results on MNLI, SST2, and QQP are reported in Table 5. As observed, we accomplish the DynaBERT Super's accuracy for MNLI and SST2 with 1.7X and 3.1X less FLOPs. For QQP, our method combined with DynaBERT even outperforms DynaBERT by 0.1% with 2.6X FLOPs speed-up. In this paper, we introduced E-LANG, an energy-based joint inference approach, which integrates Super and Swift language models for achieving efficient inference without sacrificing the accuracy. Our method can work with both encoder-only (e.g., BERT) and encoder-decoder (e.g., T5) architectures, and is also applicable for text classification and sequence-to-sequence problems. The proposed joint inference strategy was theoretically and experimentally analyzed with an extensive set of experiments and ablation studies. Our results showed that E-LANG outperforms SOTA in both fixed-size and dynamic inference over different benchmarks such as GLUE and SuperGLUE. One future direction to this work is to apply E-LANG to multiple Super and Swift models with different sizes.
abstain abstain abstain objective abstain abstain abstain abstain abstain result result abstain abstain other abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain objective method abstain abstain abstain abstain abstain result abstain objective objective objective objective objective other other other other other other other other other other other other other other other other other other other other objective method abstain method method abstain method abstain abstain other abstain abstain abstain abstain abstain method abstain abstain method abstain abstain abstain abstain other abstain abstain abstain abstain abstain abstain method method abstain abstain abstain abstain abstain abstain method abstain other other abstain method abstain abstain method abstain abstain abstain abstain method abstain abstain abstain abstain other abstain abstain abstain abstain abstain abstain abstain abstain result abstain result result abstain abstain result abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain result abstain abstain result result abstain abstain abstain abstain abstain result abstain abstain abstain abstain result abstain abstain abstain abstain abstain abstain other abstain abstain abstain abstain abstain abstain abstain abstain result abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain result result result abstain abstain result method