Contextualized representations (e.g. ELMo, BERT) have become the default pretrained representations for downstream NLP applications. In some settings, this transition has rendered their static embedding predecessors (e.g. Word2Vec, GloVe) obsolete. As a side-effect, we observe that older interpretability methods for static embeddings  while more mature than those available for their dynamic counterparts  are underutilized in studying newer contextualized representations. Consequently, we introduce simple and fully general methods for converting from contextualized representations to static lookup-table embeddings which we apply to 5 popular pretrained models and 9 sets of pretrained weights. Our analysis of the resulting static embeddings notably reveals that pooling over many contexts significantly improves representational quality under intrinsic evaluation. Complementary to analyzing representational quality, we consider social biases encoded in pretrained representations with respect to gender, race/ethnicity, and religion and find that bias is encoded disparately across pretrained models and internal layers even for models that share the same training data. Concerningly, we find dramatic inconsistencies between social bias estimators for word embeddings. Word embeddings (Bengio et al., 2003; Collobert and Weston, 2008; Collobert et al., 2011) have been a hallmark of modern natural language processing (NLP) for many years. Embedding methods have been broadly applied and have experienced parallel and complementary innovations alongside neural network methods for NLP. Advances in embedding quality in part have come from integrating additional information such as syntax (Levy and Goldberg, 2014a; Li et al., 2017), morphology (Cot-terell and Schutze, 2015), subwords (Bojanowski et al., 2017), subcharacters (Stratos, 2017; Yu et al., 2017) and, most recently, context (Peters et al., 2018; Devlin et al., 2019). Due to their tremendous representational power, pretrained contextualized representations, in particular, have seen widespread adoption across myriad subareas of NLP. The recent dominance of pretrained contextualized representations such as ELMo (Peters et al., 2018) and BERT (Devlin et al., 2019) has served as the impetus for exciting and diverse interpretability research: Liu et al. (2019a); Tenney et al. (2019a) study what is learned across the layers of these models, Tenney et al. (2019b); Ethayarajh (2019) consider what is learned from context, Clark et al. (2019); Michel et al. (2019) look at specific attention heads, Hewitt and Manning (2019); Ettinger (2020) address linguistic understanding such as syntax and negation, and Wallace et al. (2019); Tan and Celis (2019) address ethical concerns such as security (adversarial robustness) and social bias. In fact, the neologism BERTology was coined specifically to describe this flurry of interpretability research. 1 While these works have provided nuanced fine-grained analyses by creating new interpretability schema/techniques, we instead take an alternate approach of trying to re-purpose methods developed for analyzing static word embeddings. In order to employ static embedding interpretability methods to contextualized representations, we begin by proposing a simple strategy for converting from contextualized representations to static embeddings. Crucially, our method is fully general and assumes only that the contextualized model maps word sequences to vector sequences. Given this generality, we apply our method to 9 popular pretrained contextualized representations. The resulting static embeddings serve as proxies for the original contextualized model. We initially examine the representational quality of these embeddings under intrinsic evaluation. Our evaluation produces several insights regarding layer-wise lexical semantic understanding and representational variation in contextualized representations. Importantly, our analyses suggest constructive improvements to potentially improve downstream practices in using contextualized models. Simultaneously, we find that our static embeddings substantially outperform Word2Vec and GloVe and therefore suggests our method serves the dual purpose of being a lightweight mechanism for generating static embeddings that track with advances in contextualized representations. Since static embeddings have significant advantages with respect to speed, computational resources, and ease of use, these results have important implications for resource-constrained settings (Shen et al., 2019), environmental concerns (Strubell et al., 2019), and the broader accessibility of NLP technologies. 2 Alongside more developed methods for embedding analysis, the static embedding setting is also equipped with a richer body of work regarding social bias. In this sense, we view understanding the encoded social bias in representations as a societally critical special-case of interpretability research. We employ methods for identifying and quantifying gender, racial/ethnic, and religious bias (Bolukbasi et al., 2016; Garg et al., 2018; Manzini et al., 2019) to our static embeddings. These experiments not only shed light on the properties of our static embeddings for downstream use but can also serve as a proxy for understanding latent biases in the original pretrained contextual representations. We find that biases in different models and across different layers are quite disparate; this has important consequences on model and layer selection for downstream use. Further, for two sets of pretrained weights learned on the same training data, we find that bias patterns still remain fairly distinct. Most surprisingly, our large-scale evaluation makes clear that existing bias estimators are dramatically inconsistent with each other. In order to use a contextualized model like BERT to compute a single context-agnostic representation for a given word w , we define two operations. 2 A humanist's outlook on the (in)accessibility of BERT: https://tedunderwood.com/2019/07/15/do-humanists-need-bert/ The first is subword pooling : the application of a pooling mechanism over the k subword representations generated for w in context c in order to compute a single representation for w in c , i.e. { w 1 c , . . . , w kc } (cid:55) w c . Beyond this, we define context combination to be the mapping from representations w c 1 , . . . , w c n of w in different contexts c 1 , . . . , c n to a single static embedding w that is agnostic of context. Subword Pooling. The tokenization procedure for BERT can be decomposed into two steps: performing a simple word-level tokenization and then potentially deconstructing a word into multiple subwords, yielding w 1 , . . . , w k such that cat ( w 1 , . . . , w k ) = w where cat (  ) indicates concatenation. Then, every layer of the model computes vectors w 1 c , . . . , w kc . Given these vectors, we consider four pooling mechanisms to compute w c : w c = f ( w 1 c , . . . , w kc ) f  { min , max , mean , last } min(  ) , max(  ) are element-wise min/max pooling, mean(  ) is the arithmetic mean and last(  ) indicates selecting the last vector, w kc . Context Combination. Next, we describe two approaches for specifying contexts c 1 , . . . , c n and combining the associated representations w c 1 , . . . , w c n . Decontextualized: For a word w , we use a single context c 1 = w . That is, we feed the single word w into the pretrained model and use the outputted vector as the representation of w (applying subword pooling if the word is split into multiple subwords). Aggregated: Since the Decontextualized strategy presents an unnatural input to the pretrained encoder, which likely never encountered w in isolation, we instead aggregate representations of w across multiple contexts. In particular, we sample n sentences from a text corpus D (see  A.2) each of which contains the word w , and compute the vectors w c 1 , . . . , w c n . Then, we apply a pooling strategy to yield a single representation that aggregates representations across contexts: w = g ( w c 1 , . . . , w c n ); g  { min , max , mean } 3 Setup We begin by verifying that the resulting static embeddings that we derive retain their representational strength, to some extent. We take this step to ensure that properties we observe of the static embeddings can be attributed to, and are consistent with, the original contextualized representations. Inspired by concerns with probing methods/diagnostic classifiers (Liu et al., 2019a; Hewitt and Liang, 2019) regarding whether learning can be attributed to the classifier and not the underlying representation, we employ an exceptionally simple parameter-free method for converting from contextualized to static representations to ensure that any properties observed in the latter are not introduced via this process. When evaluating static embedding performance, we consider Word2Vec (Mikolov et al., 2013) and GloVe (Pennington et al., 2014) embeddings as baselines since they have been the most prominent pretrained static embeddings for several years. Similarly, we begin with BERT as the contextualized model as it is currently the most prominent in downstream use among the growing number of alternatives. We provide identical analyses for 4 other contextualized model architectures (GPT-2 (Radford et al., 2019), XLNet (Yang et al., 2019), RoBERTa (Liu et al., 2019b), DistilBERT (Sanh et al., 2019)) and, in total, 9 sets of pretrained weights. All models, weights, and naming conventions used are enumerated in Appendix C and Table 9. Additional representation quality results appear in Tables 47 and Figures 410. We primarily report results for bert-base-uncased ; further results for bert-large-uncased appear in Figure 3. To assess the representational quality of our static embeddings, we evaluate on several word similarity and word relatedness datasets. 3 We consider 4 such datasets: RG65 (Rubenstein and Goodenough, 1965), WS353 (Agirre et al., 2009), SIMLEX 999 (Hill et al., 2015) and SIMVERB 3500 (Gerz et al., 2016) (see  A.4 for more details). Taken together, these datasets contain 4917 examples and specify a vocabulary V of 2005 unique words. Each example is a pair of words ( w 1 , w 2 ) with a gold-standard annotation (provided by one or more humans) of the semantic similarity or relatedness between w 1 and w 2 . A word embedding is evaluated by the relative correctness of its ranking 3 Concerns with this decision are discussed in  A.3. of the similarity/relatedness of all examples in a dataset with respect to the gold-standard ranking using the Spearman  coefficient. Embedding predictions are computed using cosine similarity. Pooling Strategy. In Figure 1, we show the performance on all 4 datasets for the resulting static embeddings. For embeddings computed using the Aggregated strategy, representations are aggregated over N = 100 K sentences where N is the number of total contexts for all words (  A.5). Across all four datasets, we see that g = mean is the best-performing pooling mechanism within the Aggregated strategy and also outperforms the Decontexualized strategy by a substantial margin. Fixing g = mean , we further observe that mean pooling at the subword level also performs best (the dark green dashed line in all plots). We further find that this trend consistently holds across pretrained models. Number of Contexts. In Table 1, we see that performance for both BERT-12 and BERT-24 steadily increases across all datasets with increas-Figure 1: Layer-wise performance of distilled BERT-12 embeddings for all pairs ( f, g ) with N = 100 K. ing N ; this trend holds for the other 7 pretrained models. In particular, in the largest setting with N = 1 M, the BERT-24 embeddings distilled from the best-performing layer for each dataset drastically outperform both Word2Vec and GloVe. However, this can be seen as an unfair comparison given that we are selecting specific layers for specific datasets. As the middle band of Table 1 shows, we can fix a particular layer for all datasets and still outperform both Word2Vec and GloVe on all datasets. Relationship between N and model layer. In Figure 1, there is a clear preference towards the first quarter of the model's layers (layers 0-3) with a sharp drop-off in performance immediately thereafter. A similar preference for the first quarter of the model is observed in models with a different number of layers (Figure 3, Figure 10). Given that our intrinsic evaluation is centered on lexical semantic understanding, this appears to be largely consistent with the findings of Liu et al. (2019a); Tenney et al. (2019a) regarding where lexical semantic information is best encoded in pretrained contextualized models. However, as we pool over a larger number of contexts, Table 1 reveals an interesting relationship between N and the best-performing layer. The best-performing layer monotonically (with a single exception) shifts to be later and later within the pretrained model. Since the later layers did not perform better for smaller values of N , these layers demonstrate greater variance with respect to the layer-wise distributional mean and reducing this variance improves performance. 4 Since later layers of the 4 Shi et al. (2019) concurrently propose a different ap-model are generally preferred by downstream practitioners (Zhang et al., 2020), our findings suggest that downstream performance could be further improved by considering variance reduction as we suggest; Ethayarajh (2019) also provides concrete evidence of the tremendous variance in the later layers of deep pretrained contextualized models. Cross-Model Results. Remarkably, we find that most tendencies we observe generalize well to all other pretrained models we study (specifically the optimality of f = mean , g = mean , the improved performance for larger N , and the layer-wise tendencies with respect to N ). This is particularly noteworthy given that several works have found that different contextualized models pattern substantially differently (Liu et al., 2019a; Ethayarajh, 2019). In Table 2, we summarize the performance of all models we studied. All of the models considered were introduced during a similar time period and have comparable properties in terms of downstream performance. In spite of this, we observe that their static analogues perform radically differently. For example, several do not reliably outperform Word2Vec and GloVe despite outperforming Word2vec and GloVe reliably in downstream evaluation. Future work may consider whether the reduction to static embeddings affects different models differently and whether this is reflective of the quality of context-agnostic lexical semantics from other types of linguistic knowledge (e.g. context modelling, syntactic understanding, and semantic composition). In general, these results proach with similar motivations. provide further evidence to suggest that linguistic understanding captured by different pretrained weights may be substantially different, even for models with near-identical Transformer (Vaswani et al., 2017) architectures. Somewhat surprisingly, in Table 2, DistilBert-6 outperforms BERT-12 on three out of the four datasets despite being distilled (Ba and Caruana, 2014; Hinton et al., 2015) from BERT-12. Analogously, RoBERTa, which was introduced as a direct improvement over BERT, does not reliably outperform the corresponding BERT models. Bias is a complex and highly relevant topic in developing representations and models in NLP and ML. In this context, we study the social bias encoded within our static word representations as a proxy for understanding biases of the source contextualized representations. As Kate Crawford argued for in her NIPS 2017 keynote, while studying individual models is important given that specific models may propagate, accentuate, or diminish biases in different ways, studying the representations that serve as the starting point and that are shared across models (which are used for possibly different tasks) allows for more generalizable understanding of bias (Baro-cas et al., 2017). In this work, we simultaneously consider multiple axes of social bias (i.e. gender, race, and religion) and multiple proposed methods for computationally quantifying these biases. We do so precisely because we find that existing NLP literature has primarily prioritized gender (which may be a technically easier setting and is starkly incomplete in terms of social biases of interest). Further, as we will show, different computational specifi-cations of bias that evaluate the same underlying social phenomena yield markedly different results. As a direct consequence, we strongly caution that the results must be taken with respect to the definitions of bias being applied. Further, we note that an embedding which receives low bias scores cannot be assumed to be (nearly) unbiased. Instead, it satisfies the significantly weaker condition that under existing definitions the embedding exhibits low bias and perhaps additional (more nuanced) definitions are needed. Bolukbasi et al. (2016) introduced a measure of gender bias which assumes access to a set P = { ( m 1 , f 1 ) , . . . , ( m n , f n ) } of (male, female) word pairs where m i and f i only differ in gender (e.g. men' and women'). They compute a gender direction g : g = PCA (cid:0) [ m 1  f 1 , . . . , m n  f n ] (cid:1) [0] where [0] indicates the first principal component. Then, given a set N of target words that we are interested in evaluating the bias with respect to, Bolukbasi et al. (2016) specifies the bias as: bias BOLUKBASI ( N ) = mean w N | cos ( w , g ) | This definition is only inherently applicable to binary bias settings, i.e. where there are exactly two protected classes . Multi-class generalizations are difficult to realize since constructing P requires aligned k -tuples whose entries only differ in the underlying social attribute and this becomes increasingly challenging for increasing k . Further, this definition assumes the first principal component explains a large fraction of the observed variance. Garg et al. (2018) introduced a different definition that is not restricted to gender and assumes access to sets A 1 = { m 1 ,    , m n } and A 2 = { f 1 ,    , f n (cid:48) } of representative words for each of the two protected classes. For each class,  i = mean w A i w is computed. Garg et al. (2018) computes the bias in two ways: bias GARG-EUC ( N ) = mean w N (cid:107) w   1 (cid:107) 2  (cid:107) w   2 (cid:107) 2 bias GARG-COS ( N ) = mean w N cos( w ,  1 )  cos( w ,  2 ) Compared to the definition of Bolukbasi et al. (2016), these definitions may be more general as constructing P is strictly more difficult than constructing A 1 , A 2 (as P can always be split into two such sets but the reverse is not generally true) and Garg et al. (2018)'s definition does not rely on the first principal component explaining a large fraction of the variance. However, unlike the first definition, Garg et al. (2018) computes the bias in favor of/against a specific class (meaning if N = { programmer', homemaker' } and pro-grammer' was equally male-biased as homemaker' was female-biased, then under the definition of Garg et al. (2018), there would be no bias in ag-gregate). To permit comparison, we insert absolute values around each term in the mean over N . to sets of representative words A 1 , . . . , A k 5 bias MANZINI ( N ) = mean w N mean i { 1 ,...,k } mean a A i cos( w , a ) Inspired by the results of Nissim et al. (2020), in this work we transparently report social bias in existing static embeddings as well as the embeddings we produce. In particular, we exhaustively report the measured bias for all 3542 valid (pretrained model, layer, social attribute, bias definition, target word list) 5-tuples  all possible combinations of static embeddings and bias measures considered. The results for models beyond BERT appear in Figures 1118. We specifically report results for binary gender (male, female), two-class religion (Christianity, Islam) and three-class race (white, Hispanic, and Asian), directly following Garg et al. (2018). We study bias with respect to target word lists of professions N prof and adjectives N adj . These results are by no means intended to be comprehensive with regards to the breadth of bias socially and only address a restricted subset of social biases which notably does not include intersectional biases. The types of biases being evaluated for are taken with respect to specific word lists (which are sometimes subjective albeit being peer-reviewed) that serve as exemplars and definitions of bias are grounded in the norms of the United States. All word lists are provided in Appendix B and are sourced in  A.6. Layer-wise Bias Trends. In Figure 2, we report layer-wise bias across all ( attribute, definition ) pairs. We clearly observe that for every social attribute, there is a great deal of variation across the layers in the quantified amount of bias for a fixed bias estimator. Further, while we are not surprised that different bias measures for the same social attribute and the same layer assign different absolute scores, we observe that they also do not agree in relative judgments. For gender, we observe that the bias estimated by the definition of Manzini et al. (2019) steadily increases before peaking at the penultimate layer and slightly decreasing thereafter. In contrast, under bias GARG-EUC 5 We slightly modify the definition of Manzini et al. (2019) by (a) using cosine similarity where they use cosine distance and (b) inserting absolute values around each term in the mean over N . We make these changes to introduce consistency with the other definitions and to permit comparison. we see a distribution with two peaks corresponding to layers at the start or end of the pretrained model with less bias within the intermediary layers. For estimating the same quantity, bias GARG-COS is mostly uniform across the layers. Similarly, in looking at the religious bias, we see similar inconsistencies with the bias increasing monotonically from layers 2 through 8 under bias MANZINI , decreasing monotonically under bias GARG-EUC , and remaining roughly constant under bias GARG-COS . In general, while the choice of N (and the choice of A i for gender) does affect the absolute bias estimates, the relative trends across layers are fairly robust to these choices for a specific definition. Consequences. Taken together, our analysis suggests a concerning state of affairs regarding bias quantification measures for (static) word embeddings. In particular, while estimates are seemingly stable to some types of choices regarding word lists, bias scores for a particular word embedding are tightly related to the definition being used and existing bias measures are markedly inconsistent with each other. We find this has important consequences beyond understanding the social biases in our representations. Concretely, we argue that without certainty regarding the extent to which embeddings are biased, it is impossible to properly interpret the meaningfulness of debiasing procedures (Bolukbasi et al., 2016; Zhao et al., 2018a,b; Sun et al., 2019) as we cannot reliably estimate the bias in the embeddings both before and after the procedure. This is further compounded with the existing evidence that current intrinsic measures of social bias may not handle geometric behavior such as clustering (Gonen and Goldberg, 2019). Cross-Model Bias Trends. In light of the above, next we compare bias estimates across different pretrained models in Table 3. Given the conflicting scores assigned by different definitions, we retain all definitions along with all social attributes in this comparison. However, we only consider target words given by N prof due to the aforementioned stability (and for visual clarity) with results for N adj appearing in Table 8. Since we do not preprocess or normalize embeddings, the scores using bias GARG-EUC are incomparable (and may be improper to compare in the layer-wise case) as Figure 2: Layer-wise bias of distilled BERT-12 embeddings for f = mean , g = mean , N = 100 K. Gender Race Religion B, P GE, P GC, P M, P GE GC M M GE GC M Word2Vec 0.0503 0.1758 0.075 0.2403 0.1569 0.0677 0.2163 0.0672 0.0907 0.053 0.14 GloVe 0.0801 0.3534 0.0736 0.1964 0.357 0.0734 0.1557 0.1171 0.2699 0.0702 0.0756 BERT-12 0.0736 0.3725 0.0307 0.3186 0.2868 0.0254 0.3163 0.2575 1.2349 0.0604 0.2955 BERT-24 0.0515 0.6418 0.0462 0.234 0.4674 0.0379 0.2284 0.1956 0.6476 0.0379 0.2316 GPT2-12 0.4933 25.8743 0.0182 0.6464 2.0771 0.0062 0.7426 0.6532 4.5282 0.0153 0.776 GPT2-24 0.6871 40.1423 0.0141 0.8514 2.3244 0.0026 0.9019 0.8564 8.9528 0.0075 0.9081 RoBERTa-12 0.0412 0.2923 0.0081 0.8546 0.2077 0.0057 0.8551 0.8244 0.4356 0.0111 0.844 RoBERTa-24 0.0459 0.3771 0.0089 0.7879 0.2611 0.0064 0.783 0.7479 0.5905 0.0144 0.7636 XLNet-12 0.0838 1.0954 0.0608 0.3374 0.6661 0.042 0.34 0.2792 0.8537 0.0523 0.318 XLNet-24 0.0647 0.7644 0.0407 0.381 0.459 0.0268 0.373 0.328 0.8009 0.0505 0.368 DistilBERT-6 0.0504 0.5435 0.0375 0.3182 0.3343 0.0271 0.3185 0.2786 0.8128 0.0437 0.3106 Table 3: Social bias encoded within different pretrained models with respect to a set of professions N prof . they are sensitive to the absolute norms of the embeddings. 6 Further, we note that bias BOLUKBASI may not be a reliable indicator since the first principal component explains less than 35% of the variance for the majority of distilled embedding (Zhao et al. (2019a) show similar findings for ELMo). For bias MANZINI and bias GARG-COS , we find that all distilled static embeddings have substantially higher scores under bias MANZINI but generally lower scores under bias GARG-COS when compared to Word2Vec and GloVe. Interestingly, we see that under bias MANZINI both GPT-2 and RoBERTa embeddings consistently get high scores when compared to other distilled embeddings but under bias GARG-COS they are deemed the least biased. Data alone does not determine bias. Comparing the results for BERT-12 and BERT-24 (full layer-wise results for BERT-24 appear in Figure 11) reveals that bias trends for BERT-12 and BERT-24 are starkly different for any fixed 6 When we normalized using the Euclidean norm, we found the relative results to reliably coincide with those for bias GARG-COS which is consistent with Garg et al. (2018). bias measure. What this indicates is the bias observed in contextualized models is not strictly determined by the training data (as these models share the same training data as do all other 12 and 24 model pairs) and must also be a function of the architecture, training procedure, and/or random initialization. Takeaways. Ultimately, given the aforementioned issues regarding the reliability of bias measures, it is difficult to arrive at clear consensus of the how the bias encoded compares between our distilled representations and prior static embeddings. What our analysis does resolutely reveal is a pronounced and likely problematic effect of existing bias definitions on the resulting bias estimates. Contextualized  Static. Recently, Akbik et al. (2019) introduced an approach that gradually aggregates representations during training to accumulate global information and demonstrated improvements over only contextualized representations for NER. May et al. (2019) instead synthetically construct a single semantically-bleached sentence which is fed into a sentence encoder to yield a static representation. In doing so, they introduce SEAT as a means for studying biases in sentence encoders by applying WEAT (Caliskan et al., 2017) to the resulting static representations. This approach appears inappropriate for quantifying bias in sentence encoders 7 as sentence encoders are trained on semantically-meaningful sentences and semantically-bleached constructions are not representative of this distribution and their templates heavily rely on deictic expressions which are difficult to adapt for certain syntactic categories such as verbs (as required for SIMVERB 3500 especially). Given these concerns, our reduction method may be preferable for use in estimation of bias in contextualized representations. Due to the fact that we use mean-pooling, our approach may lend itself to interpretations of the bias in a model on average across contexts. Ethayarajh (2019) considers a similar method to ours where pooling is replaced by PCA . While this work demonstrated contextualized representations are highly contextual, our work naturally explores the complementary problem of what value can be extracted from the static analogue of these representations. Bias. Social bias in NLP has been primarily evaluated in three ways: (a) using geometric similarity between embeddings (Bolukbasi et al., 2016; Garg et al., 2018; Manzini et al., 2019), (b) adapting psychological association tests (Caliskan et al., 2017; May et al., 2019), and (c) considering downstream behavior (Zhao et al., 2017, 2018a, 2019a; Stanovsky et al., 2019). 8 Our bias evaluation is in the style of (a) and we consider multi-class social bias in the lens of gender, race, and religion whereas prior work has centered on binary gender. Additionally, while most prior work has discussed the static embedding setting, recent work has considered sentence encoders and contextualized models. Zhao et al. (2019a) consider gender bias in ELMo when applied to coreference systems and Kurita et al. (2019) extend these results by leveraging the masked language modeling objective of BERT. Similarly, Basta et al. (2019) considers intrinsic gender bias in ELMo via gender-swapped 7 The authors also identified several empirical concerns that draw the meaningfulness of this method into question. 8 Sun et al. (2019) provides a taxonomy of the work towards understanding gender bias within NLP. sentences. When compared to these approaches, we study a broader class of biases under more than one bias definition and consider more than one model. Further, while many of these approaches generally neglect reporting bias values for different layers of the model, we show this is crucial as bias is not uniformly distributed throughout model layers and practitioners often do not use the last layer of deep Transformer models (Liu et al., 2019a; Zhang et al., 2020; Zhao et al., 2019b). 9 7 Future Directions Our work furnishes multiple insights about pretrained contextualized models that suggest changes (subword pooling, layer choice, beneficial variance reduction via averaging across contexts) to improve downstream performance. Recent models have combined static and dynamic embeddings (Peters et al., 2018; Bommasani et al., 2019; Akbik et al., 2019) and our representations may also support drop-in improvements in these settings. While not central to our goals, we discovered that our static embeddings substantially outperform Word2Vec and GloVe under intrinsic evaluation. Future research may consider downstream gains as improved static embeddings are critical for resource-constrained settings and may help address environmental concerns in NLP (Strubell et al., 2019), machine learning (Canziani et al., 2016), and the broader AI community (Schwartz et al., 2019). Future research could explore weighting schema in the averaging process analogous to SIF (Arora et al., 2017) for sentence representations computed via averaging (Wieting et al., 2016). The generality of the proxy analysis method implies that other interpretability methods for static embeddings can also be considered. Further, post-processing approaches beyond analy-sis/interpretability such as dimensionality reduction may be particularly intriguing given that this is often challenging to perform within large multilayered networks like BERT (Sanh et al., 2019) but has been successfully demonstrated for static embeddings (Nunes and Antunes, 2018; Mu and Viswanath, 2018; Raunak et al., 2019). Future work may revisit the choice of the corpus D from which contexts are drawn. For downstream use, setting D to be the target domain may serve as a lightweight domain adaptation strategy similar to findings for averaged word representations for 9 This is the only layer studied in Kurita et al. (2019). While our work demonstrates that contextualized representations retain substantial representational power even when reduced to be noncontextual, it is unclear what information is lost. After all, contextualized representations have been so effective precisely because they are tremendously contextual (Ethayarajh, 2019). As such, the validity of treating the resulting static embeddings as reliable proxies for the original contextualized model still remains open. On the other hand, human language processing has often been conjectured to have both context-dependent and context-independent properties (Barsalou, 1982; Rubio-Fernandez, 2008; De-praetere, 2014, 2019). Given this divide, our approach may provide an alternative mechanism for clarifying how these two properties interact in the computational setting from both an interpretability standpoint (i.e. comparing results for analyses on the static embeddings and the original contextualized representations) and a downstream standpoint (i.e. comparing downstream performance for models initialized using the static embeddings and the original contextualized representations). However, the precise relationship between the role of context in human language processing and computational language processing remains unclear. Theoretical explanation for the behavior we observe in two settings is also needed. First, it is unclear why learning contextualized representations and then reducing them to static embeddings drastically outperforms directly learning static embeddings. In particular, the GloVe embeddings we use are learned using 6 billion tokens whereas the BERT representations were trained on roughly half as much data (3.3 billion tokens). Perhaps the behavior is reminiscent of the benefits of modelling in higher dimensional settings temporarily as is seen in other domains (e.g. the kernel trick and Mercer's theorem for learning non-linear classifiers using inner product methods): begin by recasting the problem in a more expressive space (contextual-ized representations) and then project/reduce to the original space (static embeddings). Second, the reason for the benefits of the variance reduction that we observe are unclear. Given that best-performing mechanism is to average over many contexts, it may be that approaching the asymptotic mean of the distribution across contexts is desirable/helps combat the anisotropy that exists in the original contextualized space (Ethayarajh, 2019). In this work, we consider how methods developed for analyzing static embeddings can be re-purposed for understanding contextualized representations. We introduce simple and effective procedures for converting from contextualized representations to static word embeddings. When applied to pretrained models like BERT, we find the resulting embeddings are useful proxies that provide insights into the pretrained model while simultaneously outperforming Word2Vec and GloVe substantially under intrinsic evaluation. We further study the extent to which various social biases (gender, race, religion) are encoded, employing several different quantification schemas. Our large-scale analysis reveals that bias is encoded disparately across different popular pretrained models and different model layers. Our findings also have significant implications with respect to the reliability of existing protocols for estimating bias in word embeddings. All data, code and visualizations are made publicly available. 10 Further details are explictly and comprehensively reported in Appendix A. Acknowledgments We thank Ge Gao, Marty van Schijndel, Forrest Davis, and members of the Mozilla DeepSpeech and Cornell NLP groups for their valuable advice. We especially thank the reviewers and area chairs for their articulate and constructive feedback.
abstain abstain objective method result result result abstain abstain other abstain other abstain objective objective method method abstain abstain method result objective abstain abstain method method abstain result result abstain method other method abstain abstain abstain method abstain method method method method method method method method method method method abstain abstain method method abstain abstain abstain abstain abstain abstain abstain result abstain result result result abstain result abstain result result abstain abstain abstain abstain result abstain abstain abstain abstain result abstain abstain abstain result abstain abstain abstain abstain abstain abstain abstain method abstain abstain abstain abstain abstain method abstain abstain abstain abstain method abstain abstain abstain abstain other abstain abstain abstain abstain result abstain abstain result abstain abstain abstain abstain abstain abstain result result result abstain abstain abstain abstain result abstain result abstain abstain abstain abstain result abstain abstain abstain abstain abstain result other abstain abstain result result abstain result abstain abstain abstain abstain result other other other other other abstain method other objective other other other other other abstain method other other other other other method abstain abstain method objective other other other other other other abstain abstain abstain abstain result abstain result abstain method abstain result abstain objective method result method result result abstain abstain abstain