We study the settings for which deep contextual embeddings (e.g., BERT) give large improvements in performance relative to classic pretrained embeddings (e.g., GloVe), and an even simpler baseline random word embeddingsfocusing on the impact of the training set size and the linguistic properties of the task. Surprisingly, we find that both of these simpler baselines can match contextual embeddings on industry-scale data, and often perform within 5 to 10% accuracy (absolute) on benchmark tasks. Furthermore, we identify properties of data for which contextual embeddings give particularly large gains: language containing complex structure, ambiguous word usage, and words unseen in training. In recent years, rich contextual embeddings such as ELMo (Peters et al., 2018) and BERT (Devlin et al., 2018) have enabled rapid progress on benchmarks like GLUE (Wang et al., 2019a) and have seen widespread industrial use (Pandu Nayak, 2019). However, these methods require significant computational resources (memory, time) during pretraining, and during downstream task training and inference. Thus, an important research problem is to understand when these contextual embeddings add significant value vs. when it is possible to use more efficient representations without significant degradation in performance. As a first step, we empirically compare the performance of contextual embeddings with classic embeddings like word2vec (Mikolov et al., 2013) and GloVe (Pennington et al., 2014). To further understand what performance gains are attributable to improved embeddings vs. the powerful downstream models that leverage them, we also compare with a simple baseline fully random embedEqual contribution. dings which encode no semantic or contextual information whatsoever. Surprisingly, we find that in highly optimized production tasks at a major technology company, both classic and random embeddings have competitive (or even slightly better!) performance than the contextual embeddings. 1 , 2 To better understand these results, we study the properties of NLP tasks for which contextual embeddings give large gains relative to non-contextual embeddings. In particular, we study how the amount of training data, and the linguistic properties of the data, impact the relative performance of the embedding methods, with the intuition that contextual embeddings should give limited gains on data-rich, linguistically simple tasks. In our study on the impact of training set size, we find in experiments across a range of tasks that the performance of the non-contextual embeddings (GloVe, random) improves rapidly as we increase the amount of training data, often attaining within 5 to 10% accuracy of BERT embeddings when the full training set is used. This suggests that for many tasks these embeddings could likely match BERT given sufficient data, which is precisely what we observe in our experiments with industry-scale data. Given the computational overhead of contextual embeddings, this exposes important trade-offs between the computational resources required by the embeddings, the expense of labeling training data, and the accuracy of the downstream model. To better understand when contextual embeddings give large boosts in performance, we identify three linguistic properties of NLP tasks which help explain when these embeddings will provide gains:  Complexity of sentence structure : How interdependent are different words in a sentence? 1 This aligns with recent observations from experiments with classic word embeddings at Apple (R  e et al., 2020). 2 These tasks are proprietary, so we share these results anecdotally as motivation for our study. Ambiguity in word usage : Are words likely to appear with multiple labels during training? Prevalence of unseen words : How likely is encountering a word never seen during training? Intuitively, these properties distinguish between NLP tasks involving simple and formulaic text (e.g., assistant commands) vs. more unstructured and lexically diverse text (e.g., literary novels). We show on both sentiment analysis and NER tasks that contextual embeddings perform significantly better on more complex, ambiguous, and unseen language, according to proxies for these properties. Thus, contextual embeddings are likely to give large gains in performance on tasks with a high prevalence of this type of language. We discuss the different types of word embeddings we compare in our study: contextual pretrained embeddings, non-contextual pretrained embeddings, and random embeddings; we also discuss the relative efficiency of these embedding methods, both in terms of computation time and memory (Sec. 2.1). Pretrained contextual embeddings Recent contextual word embeddings, such as BERT (De-vlin et al., 2018) and XLNet (Yang et al., 2019), consist of multiple layers of transformers which use self-attention (Vaswani et al., 2017). Given a sentence, these models encode each token into a feature vector which incorporates information from the token's context in the sentence. Pretrained non-contextual embeddings Noncontextual word embeddings such as GloVe (Pen-nington et al., 2014), word2vec (Mikolov et al., 2013), and fastText (Mikolov et al., 2018) encode each word in a vocabulary as a vector; intuitively, this vector is meant to encode semantic information about a word, such that similar words (e.g., synonyms) have similar embedding vectors. These embeddings are pretrained from large language corpora, typically using word co-occurrence statistics. Random embeddings In our study, we consider random embeddings (e.g., as in Limsopatham and Collier (2016)) as a simple and efficient baseline that requires no pretraining. Viewing word embeddings as n -byd matrices ( n : vocabulary size, d : embedding dimension), we consider embedding matrices composed entirely of random values. To reduce the memory overhead of storing these n  d random values to O ( n ) , we use circulant random matrices (Yu et al., 2017) as a simple and efficient approach (for more details, see Appendix A.1). 3 , 4 2.1 System Efficiency of Embeddings We discuss the computational and memory requirements of the different embedding methods, focusing on downstream task training and inference. 5 Computation time For deep contextual embeddings, extracting the word embeddings for tokens in a sentence requires running inference through the full network, which takes on the order of 10 ms on a GPU. Non-contextual embeddings (e.g., GloVe, random) require negligible time ( O ( d ) ) to extract an embedding vector. Memory Using contextual embeddings for downstream training and inference requires storing all the model parameters, as well as the model activations during training if the embeddings are being fine-tuned (e.g., 440 MB to store BERTBASE parameters, and on the order of 5-10 GB to store ac-tivations). Pretrained non-contextual embeddings (e.g., GloVe) require O ( nd ) to store a n -byd embedding matrix (e.g., 480 MB to store a 400k by 300 GloVe embedding matrix). Random embeddings take O (1) memory if only the random seed is stored, or O ( n ) if circulant random matrices are used (e.g., 1.6 MB if n = 400 k). We provide an overview of our experimental protocols (Section 3.1), the results from our study on the impact of training set size (Section 3.2), and the results from our linguistic study (Section 3.3). We show that the gap between contextual and noncontextual embeddings often shrinks as the amount of data increases, and is smaller on language that is simpler based on linguistic criteria we identify. To study the settings in which contextual embeddings give large improvements, we compare 3 Note that one could also simply store the random seed, though this requires regenerating the embedding matrix every time it is accessed. 4 We provide an efficient implementation of circulant random embedding matrices here: https://github.com/ HazyResearch/random_embedding . 5 Pretrained contextual and non-contextual embeddings also require significant computational resources during pretraining. For example training BERTBASE takes 4 days on 16 TPU chips. them to GloVe and random embeddings across a range of named entity recognition (NER) (Tjong Kim Sang and De Meulder, 2003), sentiment analysis (Kim, 2014), and natural language understanding (Wang et al., 2019a) tasks. We choose these lexically diverse tasks as examples of word, sentence, and sentence-pair classification tasks, respectively. For our embeddings, we consider 768-dimensional pretrained BERTBASE word embeddings, 300-dimensional publicly available GloVe embeddings, and 800-dimensional random circulant embeddings. We keep the embedding parameters fixed during training for all embedding types (no fine-tuning), to isolate the benefits of pretraining from the benefits of task training. We use a CNN model (Kim, 2014) for sentiment analysis and a BiLSTM (Akbik et al., 2018; Wang et al., 2019a) for the NER and General Language Understanding Evaluation (GLUE) tasks. For more details on the tasks, models, and training protocols, please see Appendix A. 3.2 Impact of Training Data Volume We show that the amount of downstream training data is a critical factor in determining the relative performance of contextual vs. non-contextual embeddings. In particular, we show in representative tasks in Figure 1 that the performance of the non-contextual embedding models improves quickly as the amount of training data is increased (plots for all tasks in Appendix B). 6 As a result of this improvement, we show in Table 1 that across tasks when the full training set is used, the non-contextual embeddings can often (1) perform within 10% absolute accuracy of the contextual 6 We provide theoretical support for why random embeddings perform strongly given sufficient data in Appendix B.3. embeddings, and (2) match the performance of the contextual embeddings trained on 1x-16x less data, while also being orders of magnitude more computationally efficient. In light of this, ML practitioners may find that for certain real-world tasks the large gains in efficiency are well worth the cost of labeling more data. Specifically, in this table we show for each task the difference between the accuracies attained by BERT vs. GloVe and random (note that random sometimes beats GloVe!), as well as the largest integer n  { 1 , 4 , 16 , 64 , 256 } such that BERT trained on 1 n of the training set still outperforms non-contextual embeddings trained on the full set. In this section, we aim to identify properties of the language in a dataset for which contextual embeddings perform particularly well relative to noncontextual approaches. Identifying such properties would allow us to determine whether a new task is likely to benefit from contextual embeddings. As a first step in our analysis, we evaluate the different embedding types on the GLUE Diagnostic Dataset (Wang et al., 2019a). This task defines four categories of linguistic properties; we observe that the contextual embeddings performed similarly to the non-contextual embeddings for three categories, and significantly better for the predicate-argument structure category (Matthews correlation coefficients of .33, .20, and .20 for BERT, GloVe, and random, respectively. See Appendix C.2.1 for more detailed results). This category requires understanding how sentence subphrases are composed together (e.g., prepositional phrase attachment, and identifying a verb's subject and object). Motivated by the observation that contextual embeddings are systematically better on specific types of linguistic phenomena, we work to identify simple and quantifiable properties of a downstream task's language which correlate with large boosts in performance from contextual embeddings. In the context of both word-level (NER) and sentence-level (sentiment analysis) classification tasks, we define metrics that measure (1) the complexity of text structure, (2) the ambiguity in word usage, and (3) the prevalence of unseen words (Sec-tion 3.3.1), and then show that contextual embeddings attain significantly higher accuracy than noncontextual embeddings on inputs with high metric values (Section 3.3.2, Table 2). We now present our metric definitions for NER and sentiment analysis, organized by the above three properties (detailed definitions in Appendix C). Complexity of text structure We hypothesize that language with more complex internal structure will be harder for non-contextual embeddings. We define the metrics as follows:  NER : We consider the number of tokens spanned by an entity as its complexity metric (e.g., George Washington spans 2 tokens), as correctly labeling a longer entity requires understanding the relationships between the different tokens in the entity name. Sentiment analysis : We consider the average distance between pairs of dependent tokens in a sentence's dependency parse as a measure of the sentence's complexity, as long-range dependencies are typically a challenge for NLP systems. Ambiguity in word usage We hypothesize that non-contextual embeddings will perform poorly in disambiguating words that are used in multiple different ways in the training set. We define the metrics as follows:  NER : We consider the number of labels (person, location, organization, miscellaneous, other) a token appears with in the training set as a measure of its ambiguity (e.g., Washington appears as a person, location, and organization in CoNLL-2003). Sentiment analysis : As a measure of a sentence's ambiguity, we take the average over the words in the sentence of the probability that the word is positive in the training set, and compute the entropy of a coin flip with this probability. 7 Prevalence of unseen words We hypothesize that contextual embeddings will perform significantly better than non-contextual embeddings on words which do not appear at all in the training set for the task. We define the following metrics:  NER : For a token in the NER input, we consider the inverse of the number of times it was seen in the training set (letting 1 / 0 :=  ). Sentiment analysis : Given a sentence, we consider as our metric the fraction of words in the sentence that were never seen during training. In Table 2 we show that for each of the metrics defined above, the accuracy gap between BERT and random embeddings is larger on inputs for which the metrics are large. In particular, we split each of the task validation sets into two halves, with points with metric values below the median in one half, and above the median in the other. We see that in 19 out of 21 cases, the accuracy gap between BERT and random embeddings is larger on the slice of the validation set corresponding to large metric values, validating our hypothesis that contextual embeddings provide important boosts in accuracy on these points. In Appendix C.2.2, we present a similar table comparing the performance of BERT and GloVe embeddings. We see that the gap between GloVe and BERT errors is larger above the median than below it in 11 out of 14 of the complexity and am-7 For sentiment tasks with C -labels ( C = 6 for the TREC dataset), we consider the entropy of the average label distribution 1 n (cid:80) ni =1 p ( y | w i )  RC over the sentence words w i . biguity results, which is consistent with our hypothesis that context is helpful for structurally complex and ambiguous language. However, we observe that GloVe and BERT embeddingswhich can both leverage pretrained knowledge about unseen wordsperform relatively similarly to one another above and below the median for the unseen metrics. The original work on ELMo embeddings (Peters et al., 2018) showed that the gap between contextual and non-contextual embeddings narrowed as the amount of training data increased. Our work builds on these results by additionally comparing with random embeddings, and by studying the linguistic properties of tasks for which the contextual embeddings give large gains. Our work is not the first to study the downstream performance of embeddings which do not require any pretraining. For example, in the context of neural machine translation (NMT) it is well-known that randomly-initialized embeddings can attain strong performance (Wu et al., 2016; Vaswani et al., 2017); the work of Qi et al. (2018) empirically compares the performance of pretrained and randomly-initialized embeddings across numerous languages and dataset sizes on NMT tasks, showing for example that the pretrained embeddings typically perform better on similar language pairs, and when the amount of training data is small (but not too small). Furthermore, as mentioned in Section 2, random embeddings were considered as a baseline by Limsopatham and Collier (2016), to better understand the gains from using generic vs. domain-specific word embeddings for text classification tasks. In contrast, our goal for using random embeddings in our study was to help clarify when and why pretraining gives gains, and to expose an additional operating point in the trade-off space between computational cost, data-labeling cost, and downstream model accuracy. We compared the performance of contextual embeddings with non-contextual pretrained embeddings and with an even simpler baselinerandom embeddings. We showed that these non-contextual embeddings perform surprisingly well relative to the contextual embeddings on tasks with plentiful labeled data and simple language. While much recent and impressive effort in academia and industry has focused on improving state-of-the-art performance through more sophisticated, and thus increasingly expensive, embedding methods, this work offers an alternative perspective focused on realizing the trade-offs involved when choosing or designing embedding methods. We hope this work inspires future research on better understanding the differences between embedding methods, and on designing simpler and more efficient models. We gratefully acknowledge the support of DARPA under Nos. FA87501720095 (D3M), FA86501827865 (SDH), and FA86501827882 (ASED); NIH under No. U54EB020405 (Mo-bilize), NSF under Nos. CCF1763315 (Be-yond Sparsity), CCF1563078 (Volume to Ve-locity), and 1937301 (RTML); ONR under No. N000141712266 (Unifying Weak Supervision); the Moore Foundation, NXP, Xilinx, LETI-CEA, Intel, IBM, Microsoft, NEC, Toshiba, TSMC, ARM, Hitachi, BASF, Accenture, Ericsson, Qualcomm, Analog Devices, the Okawa Foundation, American Family Insurance, Google Cloud, Swiss Re, the Stanford Graduate Fellowship in Science and Engineering, and members of the Stanford DAWN project: Teradata, Facebook, Google, Ant Financial, NEC, VMWare, and Infosys. The U.S. Government is authorized to reproduce and distribute reprints for Governmental purposes notwithstanding any copyright notation thereon. Any opinions, findings, and conclusions or recommendations expressed in this material are those of the authors and do not necessarily reflect the views, policies, or endorsements, either expressed or implied, of DARPA, NIH, ONR, or the U.S. Government.
method result method abstain abstain abstain abstain objective abstain result result method result result abstain method abstain objective objective objective abstain result abstain method other other other other method method abstain method other other other other other result result abstain abstain abstain other abstain abstain abstain abstain abstain abstain abstain result result result abstain abstain result abstain abstain abstain result abstain abstain result abstain abstain abstain abstain abstain abstain abstain abstain result result result abstain result abstain result result result other abstain objective other other objective method result result method other other other other other other other