Fast and reliable evaluation metrics are key to R&D progress. While traditional natural language generation metrics are fast, they are not very reliable. Conversely, new metrics based on large pretrained language models are much more reliable, but require significant computational resources. In this paper, we propose FrugalScore, an approach to learn a fixed, low cost version of any expensive NLG metric, while retaining most of its original performance. Experiments with BERTScore and MoverScore on summarization and translation show that FrugalScore is on par with the original metrics (and sometimes better), while having several orders of magnitude less parameters and running several times faster. On average over all learned metrics, tasks, and variants, FrugalScore retains 96.8% of the performance, runs 24 times faster, and has 35 times less parameters than the original metrics. We make our trained metrics publicly available 1 and easily accessible via Hugging Face, to benefit the entire NLP community and in particular researchers and practitioners with limited resources. Automatic evaluation metrics are the only way to monitor the training of, evaluate, and compare across models in a systematic, large-scale way, and are thus a critical component of the research and development ecosystem in machine learning. To get adopted in practice, evaluation metrics need to be both reliable and affordable, i.e., fast and easy to compute. While some metrics meet these criteria, such as precision and recall in information retrieval, root mean square error in regression, etc., finding suitable metrics is still an open problem in the field of Natural Language Generation (NLG) (Novikova et al., 2017). *Equal contribution 1 https://github.com/moussaKam/FrugalScore Indeed, historical n -gram matching metrics such as ROUGE (Lin, 2004) for summarization, BLEU (Papineni et al., 2002) and METEOR (Banerjee and Lavie, 2005) for translation, while affordable, are not very reliable, as they are based on surface-form matching only, i.e., lexical similarity, and have thus no sense of semantic similarity. For instance, it makes little sense to use ROUGE for the evaluation of abstractive summarization systems (which are becoming the norm), or whenever the generated text paraphrases the original text. Following the advent of transfer learning in NLP, new NLG metrics based on large pretrained language models have recently been proposed, such as BERTScore (Zhang et al., 2019) and MoverScore (Zhao et al., 2019). By relying on contextual embeddings, these metrics capture semantics and are therefore much more reliable. However, due to the sheer size of the underlying models, these metrics pose environmental issues (Strubell et al., 2019), take time to compute, and require access to significant computational resources, so they are not accessible by everyone in the NLP community. For example, we were not able to run some of the best variants of BERTScore 2 , based on DeBERTa-Large and DeBERTa-XLarge (He et al., 2020) on a 12GB GPU. Even when enough GPU memory is available, relying on such large models is still associated with extended runtimes, which can impede the progress of experiments when used once or more per epoch for validation and monitoring purposes. To address this problem, we propose in this paper FrugalScore, an approach to learn a lightweight version of BERTScore, MoverScore, and more generally any metric based on a large pretrained language model. Our contributions can be summarized as follows: 1) Our compact models have several orders of magnitude less parameters than the original metrics and 2 From BERTScore's authors: https://tinyurl.com/8cwyter2 1305 run several times faster, while retaining most of their original performance. We even outperform the original metrics in some cases 3 . 2) Our metrics are not only faster because of the much smaller amount of parameters, but also because they do not rely on any similarity function. 3) Regardless of how expensive the original metric is, querying our trained metrics always has the same low, fixed cost. This decoupling is a major advantage as the size of the pretrained language models has recently been growing tremendously (e.g., Brown et al. (2020)). Related work falls into two categories: unsupervised and supervised metrics. To address the limitations of ROUGE and BLEU, variants based on static word embeddings (Mikolov et al., 2013) were developed, e.g., ROUGE-WE (Ng and Abrecht, 2015), BLEU2VEC (Tttar and Fishel, 2017), and MEANT 2.0 (Lo, 2017). While using word vectors is a progress over strict n -gram matching, static embeddings are still very limited as they do not capture polysemy, i.e., the fact that words have different meanings in different contexts. More recently, the focus has shifted to harnessing the power of the contextualized embeddings produced by large pretrained language models. For instance, the Sentence Mover's Similarity (Clark et al., 2019) represents sentences as the average of their ELMo word embeddings (Peters et al., 2018) and measures the minimum cost of transforming one summary into the other, using a modified version of the Word Mover's Distance (Kusner et al., 2015). BERTR (Mathur et al., 2019) computes approximate recall based on the pairwise cosine similarity between the BERT embeddings (Devlin et al., 2018) of the words in automatic and reference translations. Mark-Evaluate (Mordido and Meinel, 2020) is a family of metrics that consider contextualized word or sentence embeddings derived from BERT as population samples, to evaluate language generation with population estimation methods used in ecology. Finally, the recently introduced BERTScore (Zhang et al., 2019) and MoverScore (Zhao 3 Hence the name FrugalScore, as frugal engineering is defined as achieving more with fewer resources. et al., 2019) are general-purpose NLG evaluation metrics that are becoming widely used. The main difference between BERTScore and MoverScore lies in the function used to compute the similarity between the representations of the two sequences x =  x 1 , ..., x k  and y =  y 1 , ..., y l  . We experimented with these two metrics, so we provide more details about them in what follows. BERTScore first computes the pairwise cosine similarity between the representations of the tokens in each sequence, and uses greedy matching to match each token to the most similar one in the other sequence. Given two pre-normalized vector sequences x and y , BERTScore computes: RBERT = 1 | x | (cid:88) x i  x max y j  y x Ti y j (1) and: PBERT = 1 | y | (cid:88) y i  y max x j  x y Ti x j (2) The F1-score is classically obtained as: FBERT = 2 PBERTRBERTPBERT + RBERT (3) MoverScore uses an n -gram generalization of the Word Mover's Distance (WMD) (Kusner et al., 2015) as their (dis)similarity function. More specifically, they solve for the optimal transportation flow matrix F  R | x || y | between the two weighted sequences of n -grams: W MD ( x , y ) = min F  C, F  (4) s.t. F 1 = f x , FT 1 = f y Where C is the transportation cost matrix ( C ij is the Euclidean distance between x i and y j ) and f x  R | x | + and f y  R | y | + are the n -gram weight vectors. Note that by directly learning BERTScore's and MoverScore's full internal mapping (from sequence pairs to final scalar scores), FrugalScore internalizes their similarity functions. This does not only provide a speedup at inference time, but also improves performance, as shown in section 5. Related to our work are also supervised metrics, which are directly trained on human evaluations. ROSE (Conroy and Dang, 2008) is a linear combination model of different variants of ROUGE using canonical correlation. BEER (Stanojevic and Sima'an, 2014) is a learning-to-rank approach using word and character n-gram matching, and token ordering, as features to maximize correlation with human rankings of machine translation systems. S 3 (Peyrard et al., 2017) trains a regression model that takes the evaluation scores of several existing metrics and many hand-crafted features as input, and learns the best combination of them to approximate human summary judgments. DPMFcomb (Yu et al., 2015) and Blend (Ma et al., 2017) are combined metrics incorporating a vast amount of lexical, syntactic and semantic based translation evaluation metrics using ranking and regression SVMs respectively. RUSE (Shimanaka et al., 2018) evaluates machine translation with a neural regressor based on universal sentence embeddings (e.g., InferSent (Conneau et al., 2017)). NUBIA (Kane et al., 2020) consists of three modules: a feature extractor based on RoBERTa (Liu et al., 2019) and GPT-2 (Rad-ford et al., 2019) fine-tuned on language evaluation tasks, an aggregator trained to predict the quality of the hypothesis given the reference using the extracted features, and a calibrator mapping all predictions between 0 and 1. Differences . Like the aforementioned efforts, FrugalScore is a learned metric. However, it does not rely on any intermediate or handcrafted features, and, most importantly, it does not require training on human annotations. Supervision in FrugalScore is conducted on a synthetic dataset, as a trick to expose and learn the internal mapping of the unsupervised metrics to be learned. Last but not least, unlike all aforementioned methods, compression is central to FrugalScore, which is based on miniature versions of the models used by the original metrics. Knowledge distillation (KD) (Hinton et al., 2015) is the process of transferring knowledge from a large teacher model to a smaller student model to accomplish model compression (Bucilua et al., 2006). It was originally proposed in the domain of computer vision and speech recognition, then successfully adapted to NLP (Sanh et al., 2019). Distillation can be accomplished in three ways: (1) offline, where a teacher is first pre-trained, then a student is trained under the guidance of the teacher (Hinton et al., 2015); (2) online, where the student and the teacher are trained simultaneously (Zhang et al., 2018); and (3) self, where the same model plays the role of student and teacher, e.g., transferring the knowledge of a later exit layer into an earlier one of the same multi-exit network (Phuong and Lampert, 2019). Previous studies on KD mainly focused on classification problems (Gou et al., 2021). A few attempts have been made on regression problems (Chen et al., 2017; Saputra et al., 2019; Takamoto et al., 2020), in which special losses were proposed to train the student with respect to both the teacher's regression outputs and ground truth scores. Different from conventional distillation, our work is more similar to data-free KD (Kang and Kang, 2021), where the student is trained in the absence of the dataset used to train the teacher. To transfer knowledge, we first create a synthetic dataset by annotating sequence pairs with a large model (teacher), and then train a miniature model (student) on that dataset, in an offline and regression setting. A work closely related to ours is BLEURT (Sellam et al., 2020). However, there are a number of significant differences with our approach. First, BLEURT continues the pretraining of an already pretrained BERT-based model on a synthetic dataset in a self-supervised way, whereas FrugalScore is directly trained to learn the scores of the metric of interest, in a supervised fashion. Also, BLEURT's synthetic dataset is made by perturbing Wikipedia sentences with mask-filling, backtranslation, and word dropping, whereas we use other data sources than Wikipedia such as summarization and translation datasets, and only NLG models to induce perturbations. When creating its synthetic dataset, BLEURT automatically annotates the (original, perturbed) sequence pairs with numerical and categorical sig-nals: BLEU, ROUGE, BERTscore, backtranslation likelihood, textual entailment (probability of three labels: entail, contradict, and neutral, given by BERT fine-tuned on MNLI), and backtranslation flag. On the other hand, FrugalScore simply and directly annotates the sequence pairs with the metric to be learned. metrics described in subsection 2.2. BLEURT does not learn to generate a scalar until that final fine-tuning phase, so it cannot be used as a metric before that. Conversely, FrugalScore is trained from the start to be a metric, and the fine-tuning phase is optional. Also, BLEURT was designed for the evaluation of translation. The authors only test whether it can be applied to a different task by experimenting on the WebNLG (data-to-text) dataset (Gardent et al., 2017). Conversely, we focus on learning general text similarity metrics (e.g., BERTscore and MoverScore), so FrugalScore is task-agnostic by design. Finally, and above all, the objective of FrugalScore is model compression, whereas that of BLEURT is metric learning. of which is optional. Phase 1 . We create a synthetic dataset (see subsection 3.1) by sampling pairs of more or less related sequences and annotating them with the expensive metrics to be learned. This is a one-time operation that does not need to be repeated regardless of the model used in Phase 2. Phase 2 . We continue the pretraining (see subsection 3.2) of a miniature pretrained language model on the synthetic dataset built by Phase 1. Here, the miniature model learns the internal mapping of the expensive metric, including any similarity function applied to the representations. Note that a different miniature is trained for each metric to be learned (we leave learning metric combinations as future work). The miniature can then be used in inference mode to generate scores for any never-seen pair of sequences. Phase 3 (optional). We fine-tune the miniature on human annotations, which, as shown in section 6, can boost performance. The objective here was to generate pairs of sequences mimicking the (reference, candidate) pairs found in NLG datasets, which are usually semantically related and in many cases paraphrasing one another. We sampled our sequences from a variety of data sources, listed next. Summarization . For each document in the wellknown CNN/DailyMail dataset (Nallapati et al., 2016), our goal was to generate several summaries differing in terms of structure and quality. To this purpose, we used different pretrained seq2seq summarization models: BART-base and BART-large (Lewis et al., 2019), mBART (Liu et al., 2020), and BARThez (Kamal Eddine et al., 2021). BART is a seq2seq autoencoder with a Transformer architecture. The four models were fine-tuned for one epoch on 50k examples randomly sampled from the training set of CNN/DM, and were used to generate summaries for the whole training set of 287,112 documents, using greedy decoding. Note that we kept the 50K documents used for fine-tuning in the final generation pool, in order to create quality differences among summaries. Indeed, models are expected to better summarize the documents used for training than never-seen documents. We also used the human reference summaries, so that in the end, each document was associated with 5 summaries, resulting in 10 pairs of summaries per document. Backtranslation . We also generated paraphrases with backtranslation, by sampling sentences from the OpenSubtitle English monolingual corpus (Li-son and Tiedemann, 2016), and translating them to French, Arabic and German with OPUS-MT (Tiedemann and Thottingal, 2020), before translating them back to English. We used OPUS-MT because of its ready-to-use checkpoints available for many language pairs. We ended up with 4 variations for each sentence (including the original one), resulting in 6 paraphrase pairs per sentence. Denoising . To avoid bias towards summarization and translation, we also generated pairs of related sequences such that the first element in the pair was a Wikipedia segment and the second element was a BART-denoised version of it (Lewis et al., 2019). More precisely, we sampled 2M segments from Wikipedia such that the number of unigrams in these segments was uniformly distributed between 1 and 200. Our assumption was that enforcing variations in sequence length would help the learned metric to generalize. We then applied BART's text infilling and sentence permutation perturbation strategies to each segment. That is, multiple text spans were sampled and replaced with a [MASK] special token. The 1308 lengths of the spans were sampled from a Poisson distribution (  = 3 ). 50% of the tokens within the input segment were masked and 20% of the masked text was replaced with random tokens (cre-ating pathological examples to increase the robustness of the learned metric). The sentences in the input segment were then shuffled. We finally used a BART-Base checkpoint 4 from the Fairseq library (Ott et al., 2019) to try to reconstruct the perturbed versions of the original sequences, hence creating variants of them. Annotating pairs . We sampled 4.5M sequence pairs uniformly from each aforelisted source. These pairs were then annotated with the metrics to be learned. Note that this is a one-time operation that does not need to be repeated regardless of which models are trained downstream. In this work, we experimented with two recent expensive NLG metrics that rely on large pretrained language models, BERTScore (Zhang et al., 2019) and MoverScore (Zhao et al., 2019), presented in section 2. However, it is important to note that our method can be used with any other NLG metric. Note that for BERTScore, we used the F-1 score FBERT , as recommended by the authors (Zhang et al., 2019). For MoverScore, still following the authors (Zhao et al., 2019), we used the variant operating on unigrams and the IDF to compute the vectors of weights. We continue the pretraining of three BERT miniatures 5 on our synthetic dataset: BERT-Tiny ( L = 2 , H = 128 ), BERT-Small ( L = 4 , H = 512 ) and BERT-Medium ( L = 8 , H = 512 ), where L is the number of layers and H is the dimension of the embedding space. These models have respectively 25 times, 3.78 times, and 2.64 times less parameters than BERT-base. The concept of BERT miniatures was introduced by Turc et al. (2019) to test whether pretraining small models from scratch was competitive to distilling very large models. The miniature models have already been pretrained on masked language model and next sentence prediction objectives. We continue pretraining using the standard method introduced by Devlin et al. (2018). We concatenate the two sequences x =  x 1 , ..., x k  and y =  y 1 , ..., y l  in a given pair, separating them 4 https://dl.fbaipublicfiles.com/fairseq/models/bart.base.tar.gz 5 https://huggingface.co/google with a special [SEP] token. A special [CLS] token is also added at the beginning of the resulting sequence. The sequence of contextualised embeddings  z [ CLS ] , x 1 , ... x k , z [ SEP ] , y 1 , ..., y l  is then obtained. We finally add a fully connected layer on top, that linearly projects the z [ CLS ] vector to a scalar s . The model is trained to minimize the mean square error (MSE) loss between the learned metric s i and the metric to be learned  s i (i.e., the annotation of the pair): l = 1 NN (cid:88) i =1 || s i   s i || 2 (5) When pretraining is over, the models can be further fine-tuned on smaller human-annotated datasets as shown in section 6, or directly used to generate scores for unseen examples as shown in section 4. Setup . We use a batch size of 32 and the Adam optimizer (Kingma and Ba, 2014) with a learning rate of 3  10  5 , linear decay, and a warm-up for 6% of the total training steps, and we train the model for three epochs. We conducted the pretraining on a single TITAN RTX GPU (24GB). It took 10, 24 and 33 hours, respectively for the tiny, small, and medium miniatures. We rely on the Transformers library (Wolf et al., 2019) for all pretraining and fine-tuning experiments. In this section, FrugalScore is used in inference mode to generate scores directly after pretraining, i.e., no fine-tuning is performed (see section 6 for fine-tuning results). We evaluate on two text generation tasks: summarization and translation. We use evaluation datasets containing (reference, candidate) sequence pairs annotated with human scores assessing the quality of the candidates given the references. We measure the effectiveness of FrugalScore by measuring the Pearson correlation of its scores with the human judgments and comparing it to that of the original metrics. We also take the number of parameters and the runtime into account. Text summarization . We use 4 different multi-document summarization datasets from the Text Analysis Conference (TAC) 6 : TAC-2008, TAC-2009, TAC-2010 and TAC-2011. These datasets respectively contain 48, 44, 46 and 44 clusters of documents and 58, 55, 43 and 51 systems are used to generate summaries. Each cluster forms a topic to be summarized and has 4 reference summaries. There are approximately 10k pairs in each dataset. Each pair is annotated with two human judgment scores: the Pyramid Score (Harnly et al., 2005) and the Responsiveness (Dang et al., 2008). The former measures the proportion of important semantic units (SCUs) in the reference summaries captured by the system summary, while the latter reflects the content coverage and the readability of each summary. Machine translation . Our evaluation corpus is from the WMT-2019 7 shared task (Li et al., 2019). We consider all the to-English pairs: Chinese, Czech, German, Finnish, Russian, Lithuanian and Kazakh to English. For each language, we use the test set that contains several thousands of reference-candidate pairs annotated with human ratings that assess the translation quality. Table 1 reports the results averaged over the 4 TAC datasets and the 7 WMT to-English language pairs. Details are provided in Appendices A and B. We benchmarked the metrics in terms of Pearson correlations with human scores, runtimes, and numbers of parameters. We used two approaches to compute the Pearson correlations: summary-level (or segment-level) and system-level. In the former approach, a score is attributed to each of the output candidates, while in the latter approach, one single overall score is attributed to the system (by averaging its individual scores). Rows a to c correspond to BERTScore with BERT miniatures as the underlying model. They are simple baselines added for the sake of comparison, to see what we get when BERTScore is used with the same number of parameters as FrugalScore. Rows d to g correspond to the expensive metrics that are learned by FrugalScore (in the respective sections of the bottom half of the table). They 1310 are BERTScore and MoverScore metrics where the underlying model is a large pretrained language model: BERT-Base ( L = 12 , H = 512 ), RoBERTa-Large ( L = 24 , H = 1024 ) (Liu et al., 2019), and DeBERTa-XLarge ( L = 24 , H = 1536 ) (He et al., 2020). Finally, rows i to xii correspond to FrugalScore. Subscripts refer to row labels and indicate which metric-model combination was used to annotate pairs. I.e., FrugalScore d learned the metric of row d , i.e., BERTScore with BERT-Base. First, results show that all FrugalScores, regardless of which metric they learned, significantly outperform the BERTScores with miniature models. These results suggest that FrugalScore is a better approach than using an existing metric with a lightweight underlying model. The reason for this is probably that in FrugalScore, the knowledge of the original unsupervised metric (based on a large model) is explicitly transferred to the miniature via the continuation of its pretraining on the synthetic dataset. That is, the miniature is actually learning a metric. Whereas, on the other hand, plugging a compressed version of a general-purpose language model into the original unsupervised metric just makes it lose expressiveness and capacity. Second, we can clearly see that FrugalScore retains most of the performance of the original metric, while running several times faster and reducing the number of parameters by several orders of magnitude. On average over all metrics, tasks, and miniatures, FrugalScore retains 96.8% of the original performance, runs 24 times faster, and has 35 times less parameters. More precisely, on average across all metrics, FrugalScore-Tiny retains 97.7/94.7% of the original performance on TAC (pyramid score/responsiveness), while running 54 times faster and having 84 times less parameters. Its small and medium versions retain near full performance in terms of responsiveness (98 and 97.7%) and even slightly outperform the original metrics in terms of pyramid score, while at the same time reducing the runtime and the number of parameters by 32 (resp. 21) and 13 (resp. 9) times. On WMT, FrugalScore-Tiny retains 88.58% of the performance of the original metrics, while running 14 times faster (and still having 84 times less parameters), while the small and medium versions of FrugalScore retain 95.71 % and 98.06% of the original performance while still offering a 32 times (resp. 21) speedup and having 13 times (resp. 9) less parameters, on average. Interestingly, FrugalScore even improves the performance of the original metrics in some cases. For example, on TAC, FrugalScore g with BERT-Tiny (row x ) improves the performance of the original MoverScore metric based on BERT-Base (row g ) from 66.5 to 67.3 in terms of pyramid score, while reducing the number of parameters by 25 and running 50 times faster. Other examples, also for TAC with the pyramid score, include FrugalScore f with BERT-Small (row viii , +1.5 point) and FrugalScore f with BERT-medium (row ix , +1 point). Finally, the results of FrugalScore for different miniature sizes show that, on WMT, using larger models always improves performance (e.g., row x  xi  xii ). But interestingly, on TAC, this observation does not hold (e.g., row vi  viii  ix ), and sometimes, FrugalScore with the smallest miniature (BERT-Tiny) is superior (e.g. rows i and x ). This finding suggests that the impact of the pretrained language model size is task-dependent. To sum up, results clearly show the effectiveness of FrugalScore in learning a cheaper, lighter, and faster version of the original metrics, while retaining most of their original performance. The system-level correlations, provided in Appendices C and D, corroborate these positive results. We also provide the correlations between the original and the learned metrics in Appendices E and F. It is interesting to note that a greater correlation with the original metric is not always associated with a better performance. E.g., the tiny version of FrugalScore g is the best (row x ), while it is the less correlated with the original metric. We test two hypotheses in this section: (1) whether fine-tuning on a human-annotated dataset is beneficial, and (2) when fine-tuning on human annotations, whether continuing pretraining on our synthetic dataset is useful. Because we cannot use the same dataset for fine-tuning and evaluation, we fine-tune a BERT-Small on each year of TAC 2008-2011 for 4 epochs, using two other years as the validation set, and the remaining year as the test set. The best epoch is selected based on validation performance. We use a batch size of 32 and a learning rate of 2e-5 that linearly decreases to zero. Finally, we experi-1311 PretrainingContinued TAC-2008 TAC-2009 TAC-2010 TAC-2011 Average TAC-2008 no -67.7 0 . ment with two scenarios: fine-tuning the miniature directly without continuing its pretraining on our synthetic dataset, and fine-tuning it after the pretraining continuation (with annotations generated by BERTScore-BERT-Base). Results . Results are reported in Table 2 in terms of summary-level Pearson correlations with human evaluations (Pyramid), averaged over 3 runs with different random seeds. First, it is obvious that everywhere, continuing the pretraining on our synthetic dataset leads to a significant boost in performance. This is in accordance with Sellam et al. (2020), who found that pretraining was beneficial even in a supervised setting. Second, even if a direct comparison is not possible, we can remark when looking at the TAC Pyramid score of row ii) in Table 1 (FrugalScore d -BERT-Small) that fine-tuning after pretraining seems very beneficial too. Indeed, after fine-tuning, we reach on average 71, 67.5, 68.7, and 69.2 (de-pending on the split), which represents overall a gain of 4.4 points over the non-fine-tuned model (score of 64.7). To test the importance of each data source introduced in subsection 3.1, we created a training set containing sequence pairs uniformly and equally sampled from each source. We annotated these pairs with the BERTScore-BERT-Base metric and we used them to continue the pretraining of a BERT-Small miniature. We also considered pairs drawn at random from the pairs generated with the other strategies. The motivation for random pairs was to sample negative examples, as seeing only positive examples (pairs of related sequences) could bias the learned n o _ s u m m . We then continued the pretraining of the BERT-Small miniature four times, excluding each time the pairs coming from a specific data source. We evaluated the learned metric on TAC-2008 to 2011 and on WMT-2019. Figure 1 shows the average improvements in the Pearson correlation with human judgments relative to training a model on all sources. Note that when training on all four sources, we sampled 30k pairs from each source (120k to-tal), and when excluding a source, we sampled 40k pairs from each source (120k total). We can clearly see that excluding the random pairs improves performance while excluding any of the other data sources decreases performance. In other words, all our data sources are beneficial, and it is not necessary to add negative examples. We hypothesise that this is due to the fact that NLG datasets typically do not contain completely unrelated pairs of sentences. Interestingly, the pairs generated with the backtranslation strategy have the greatest impact on performance. We proposed FrugalScore, an approach to learn a fixed, low-cost version of any expensive NLG evaluation metric. Experiments on summarization and translation tasks show that our FrugalScore versions of BERTScore and MoverScore retain most of the original performance in terms of the correlation with human judgments, while running several times faster and having several orders of magnitude less parameters. On average over all learned metrics, tasks, and variants, FrugalScore retains 96.8% of the performance, runs 24 times faster, and has 35 times less parameters than the original metrics. This work was supported by the SUMM-RE project (ANR-20-CE23-0017).
abstain abstain abstain objective abstain abstain method abstain abstain abstain other abstain abstain abstain abstain abstain abstain objective other result abstain method abstain other abstain other other other other other other other method other other other other other method other other other other other other other other other other other other other other other other method objective other abstain other method other other other other other other other method other abstain abstain method abstain abstain method abstain method abstain abstain method abstain method abstain method method abstain abstain method abstain method abstain method method method abstain method method method method abstain abstain abstain abstain method abstain method abstain abstain method method method method method abstain abstain abstain method other abstain abstain method other abstain method method abstain method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain result abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain result abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain method abstain method abstain method abstain abstain abstain abstain abstain abstain method method method abstain abstain method abstain method abstain method abstain abstain objective result abstain other