Transformer language models have shown remarkable ability in detecting when a word is anomalous in context, but likelihood scores offer no information about the cause of the anomaly. In this work, we use Gaussian models for density estimation at intermediate layers of three language models (BERT, RoBERTa, and XLNet), and evaluate our method on BLiMP, a grammaticality judgement benchmark. In lower layers, surprisal is highly correlated to low token frequency, but this correlation diminishes in upper layers. Next, we gather datasets of morphosyntactic, semantic, and commonsense anomalies from psycholinguistic studies; we find that the best performing model RoBERTa exhibits surprisal in earlier layers when the anomaly is morphosyntactic than when it is semantic, while commonsense anomalies do not exhibit surprisal at any intermediate layer. These results suggest that language models employ separate mechanisms to detect different types of linguistic anomalies. Transformer-based language models (LMs) have achieved remarkable success in numerous natural language processing tasks, prompting many probing studies to determine the extent of their linguistic knowledge. A popular approach is to formulate the problem as a multiple-choice task, where the LM is considered correct if it assigns higher likelihood to the appropriate word than an inappropriate one, given context (Gulordava et al., 2018; Ettinger, 2020; Warstadt et al., 2020). The likelihood score, however, only gives a scalar value of the degree that a word is anomalous in context, and cannot distinguish between different ways that a word might be anomalous. It has been proposed that there are different types of linguistic anomalies. Chomsky T h e c a t w o n ' t e a t i n g t h e f oo d 0 1 2 3 4 5 6 7 8 9 10 11 12 L a y e r T h e p l a n e l a u g h e d a t t h e r un w a y 0 1 2 3 4 5 6 7 8 9 10 11 12 Figure 1: Example sentence with a morphosyntactic anomaly (left) and semantic anomaly (right) (anoma-lies in bold). (1957) distinguished semantic anomalies ( color-less green ideas sleep furiously ) from ungrammaticality ( furiously sleep ideas green color-less ). Psycholinguistic studies initially suggested that different event-related potentials (ERPs) are produced in the brain depending on the type of anomaly; e.g., semantic anomalies produce negative ERPs 400 ms after the stimulus, while syntactic anomalies produce positive ERPs 600 ms after (Kutas et al., 2006). Here, we ask whether Transformer LMs show different surprisals in their intermediate layers depending on the type of anomaly. However, LMs do not compute likelihoods at intermediate layers  only at the final layer. In this paper, we introduce a new tool to probe for surprisal at intermediate layers of BERT (De-vlin et al., 2019), RoBERTa (Liu et al., 2019), and XLNet (Yang et al., 2019), formulating the problem as density estimation. We train Gaussian models to fit distributions of embeddings at each layer of the LMs. Using BLiMP (Warstadt et al., 2020) for evaluation, we show that this model is effective at grammaticality judgement, requiring only a small amount of in-domain text for training. Figure 1 shows the method using the RoBERTa model on two example sentences. We apply our model to test sentences drawn from BLiMP and 7 psycholinguistics studies, exhibiting morphosyntactic, semantic, and commonsense anomalies. We find that morphosyntactic anomalies produce out-of-domain embeddings at earlier layers, semantic anomalies at later layers, and no commonsense anomalies, even though the LM's final accuracy is similar. We show that LMs are internally sensitive to the type of linguistic anomaly, which is not apparent if we only had access to their softmax probability outputs. Our source code and data are available at: https://github.com/SPOClab-ca/ layerwise-anomaly . Soon after BERT's release, many papers invented probing techniques to discover what linguistic knowledge it contains, and how this information is distributed between layers (e.g., Rogers et al. (2021) provides a comprehensive overview). Tenney et al. (2019) used edge probing to determine each layer's contribution to a task's performance, and discovered that the middle layers contributed more when the task was syntactic, and the upper layers more when the task was semantic. Several papers found that BERT's middle layers contain the most syntactic information. Kelly et al. (2020) found that BERT's middle layers are best at distinguishing between sentences with direct and indirect object constructions. Hewitt and Manning (2019) used a structural probe to recover syntax trees from contextual embeddings, and found the performance peaked in middle layers. Probing results are somewhat dependent on the choice of linguistic formalism used to annotate the data, as Kulmizev et al. (2020) found for syntax, and Kuznetsov and Gurevych (2020) found for semantic roles. Miaschi et al. (2020) examined the layerwise performance of BERT for a suite of linguistic features, before and after fine tuning. Our work further investigates what linguistic information is contained in different layers, with a focus on anomalous inputs. Many recent probing studies used grammaticality judgement tasks to test the knowledge of spe-cific phenomena in LMs. Warstadt et al. (2019) gathered sentences from linguistic publications, and evaluated by Matthews Correlation with the ground truth. More commonly, the model is presented with a binary choice between an acceptable and unacceptable sentence: BLiMP (Warstadt et al., 2020) used templates to generate 67k such sentence pairs, covering 12 types of linguistic phenomena. Similarly, Hu et al. (2020) created syntactic tests using templates, but defined success criteria using inequalities of LM perplexities. In contrast with artificial templates, Gulordava et al. (2018) generated test cases by perturbing natural corpus data to test long-distance dependencies. Most grammaticality studies focused on syntactic phenomena, but Rabinovich et al. (2019) tested LMs' sensitivity to semantic infelicities involving indefinite pronouns. Violations of selectional restrictions are one type of linguistic unacceptability, defined as a semantic mismatch between a verb and an argument. Sasano and Korhonen (2020) examined the geometry of word classes (e.g., words that can be a direct object of the verb play') in word vector models; they compared single-class models against discriminative models for learning word class boundaries. Chersoni et al. (2018) tested distributional semantic models on their ability to identify selectional restriction violations using stimuli from two psycholinguistic datasets. Finally, Metheniti et al. (2020) tested how much BERT relies on selectional restriction information versus other contextual information for making masked word predictions. The N400 response is a negative event-related potential that occurs roughly 400ms after a stimulus in human brains, and is generally associated with the stimulus being semantically anomalous with respect to the preceding context (Kutas and Federmeier, 2011). Although many studies have been performed with a diverse range of linguistic stimuli, exactly what conditions trigger the N400 response is still an open question. Frank et al. (2015) found that the N400 response is correlated with surprisal, i.e., how unlikely an LM predicts a word given the preceding context. Recently, several studies have investigated relationships between surprisal in neural LMs and the N400 response. Michaelov and Bergen (2020) compared human N400 amplitudes with LSTM-based models using stimuli from several psycholinguistic studies. Ettinger (2020) used data from three psycholinguistic studies to probe BERT's knowledge of commonsense and negation. Our work is similar to the latter  we leverage psycholinguistic studies for their stimuli, but we do not use the their N400 amplitude results. We use the transformer language model as a contextual embedding extractor (we write this as BERT for convenience). Let L be the layer index, which ranges from 0 to 12 on all of our models. Using a training corpus { w 1 ,    , w T } , we extract contextual embeddings at layer L for each token: x ( L ) 1 ,    , x ( L ) T = BERTL ( w 1 ,    , w T ) . (1) Next, we fit a multivariate Gaussian on the extracted embeddings: x ( L ) 1 ,    , x ( L ) T  N ( b  L , b  L ) . (2) For evaluating the layerwise surprisal of a new sentence s = [ t 1 ,    , t n ] , we similarly extract contextual embeddings using the language model: y 1 ,    , y n = BERTL ( t 1 ,    , t n ) . (3) The surprisal of each token is the negative log likelihood of the contextual vector according to the multivariate Gaussian: G i =  log p ( y i | b  L , b  L ) for i = 1 . . . n. (4) Finally, we define the surprisal of sentence s as the sum of surprisals of all of its tokens, which is also the joint log likelihood of all of the embeddings: surprisal L ( t 1 ,    , t n ) = n X i =1 G i =  log p ( y 1 ,    , y n | b  L , b  L ) . The theoretical motivation for using the sum of log likelihoods is that when we fit a Gaussian model with full covariance matrix, low likelihood corresponds exactly to high Mahalanobis distance from the in-distribution points. The score given by the Gaussian model is: G =  log p ( y | b  L , b  L ) =  log 1 (2  ) D/ 2 | b  L | 1 / 2 exp(  1 2 d 2 ) ! , (6) where D is the dimension of the vector space, and d is the Mahalanobis distance: d = q ( y  b  L ) T b   1 L ( y  b  L ) . b thus the negative log likelihood is the squared Mahalanobis distance plus a constant. Various methods based on Mahalanobis distance have been used for anomaly detection in neural networks; for example, Lee et al. (2018) proposed a similar method for out-of-domain detection in neural classification models, and Cao et al. (2020) found the Mahalanobis distance method to be competitive with more sophisticated methods on medical out-of-domain detection. In Transformer models, Podolskiy et al. (2021) used Mahalanobis distance for out-of-domain detection, outperforming methods based on softmax probability and likelihood ratios. Gaussian assumptions. Our model assumes that the embeddings at every layer follow a multivariate Gaussian distribution. Since the Gaussian distribution is the maximum entropy distribution given a mean and covariance matrix, it makes the fewest assumptions and is therefore a reasonable default. Hennigen et al. (2020) found that embeddings sometimes do not follow a Gaussian distribution, but it is unclear what alternative distribution would be a better fit, so we will assume a Gaussian distribution in this work. For all of our experiments, we use the base' versions of pretrained language models BERT (De-vlin et al., 2019), RoBERTa (Liu et al., 2019), and XLNet (Yang et al., 2019), provided by Hugging-Face (Wolf et al., 2020). Each of these models have 12 contextual layers plus a 0 th static layer, and each layer is 768-dimensional. We train the Gaussian model on randomly selected sentences from the British National Corpus (Leech, 1992), representative of acceptable English text from various genres. We evaluate on BLiMP (Warstadt et al., 2020), a dataset of 67k minimal sentence pairs that test acceptability judgements across a variety of syntactic and semantic phenomena. In our case, a sentence pair is considered correct if the sentence-level surprisal of the unacceptable sentence is higher than that of the acceptable sentence. How much training data is needed? We experiment with training data sizes ranging from 10 to 10,000 sentences (Figure 2a). Compared to the massive amount of data needed for pretraining the LMs, we find that a modest corpus suf-fices for training the Gaussian anomaly model, and a plateau is reached after 1000 sentences for all three models. Therefore, we use 1000 training sentences (unless otherwise noted) for all subsequent experiments in this paper. Which layers are sensitive to anomaly? We vary L from 0 to 12 in all three models (Figure 2b). The layer with the highest accuracy differs between models: layer 9 has the highest accuracy for BERT, 11 for RoBERTa, and 6 for XLNet. All models experience a sharp drop in the last layer, likely because the last layer is specialized for the MLM pretraining objective. Comparisons to other models. Our best-performing model is RoBERTa, with an accuracy of 0.830. This is slightly higher the best model reported in BLiMP (GPT-2, with accuracy 0.801). We do not claim to beat the state-of-the-art on BLiMP: Salazar et al. (2020) obtains a higher accuracy of 0.865 using RoBERTa-large. Even though the main goal of this paper is not to maximize accuracy on BLiMP, our Gaussian anomaly model is competitive with other transformer-based models on this task. In Appendix A, we explore variations of the Gaussian anomaly model, such as varying the type of covariance matrix, Gaussian mixture models, and one-class SVMs (Scholkopf et al., 2000). However, none of these variants offer a significant improvement over a single Gaussian model with full covariance matrix. We notice that surprisal scores in the lower layers are sensitive to token frequency: higher frequency tokens produce embeddings close to the center of the Gaussian distribution, while lower frequency tokens are at the periphery. The effect gradually diminishes towards the upper layers. To quantify the sensitivity to frequency, we compute token-level surprisal scores for 5000 sentences from BNC that were not used in training. We then compute the Pearson correlation between the surprisal score and log frequency for each token (Figure 3). In all three models, there is a high correlation between the surprisal score and log frequency at the lower layers, which diminishes at the upper layers. A small positive correlation persists until the last layer, except for XLNet, in which the correlation eventually disappears. There does not appear to be any reports of this phenomenon in previous work. For static word vectors, Gong et al. (2018) found that embeddings for low-frequency words lie in a different region of 0.0 0.2 0.4 0.6 0.8 1.0 0 1 2 3 4 5 6 7 8 9 10 11 12 Layer P ea r s on C o rr e l a t i on BERT RoBERTa XLNet Figure 3: Pearson correlation between token-level surprisal scores (Equation 4) and log frequency. the embedding space than high-frequency words. We find evidence that the same phenomenon occurs in contextual embeddings (Appendix B). In this scenario, the Gaussian model fits the high-frequency region and assigns lower likelihoods to the low-frequency region, explaining the positive correlation at all layers; however, it is still unclear why the correlation diminishes at upper layers. We turn to the question of whether LMs exhibit different behaviour when given inputs with different types of linguistic anomalies. The task of partitioning linguistic anomalies into several distinct classes can be challenging. Syntax and semantics have a high degree of overlap  there is no widely accepted criterion for distinguishing between ungrammaticality and semantic anomaly (e.g., Abrusan (2019) gives a survey of current proposals), and Poulsen (2012) challenges this dichotomy entirely. Similarly, Warren et al. (2015) noted that semantic anomalies depend somewhat on world knowledge. Within a class, the anomalies are also heterogeneous (e.g., ungrammaticality may be due to violations of agreement, wh -movement, negative polarity item licensing, etc), which might each affect the LMs differently. Thus, we define three classes of anomalies that do not attempt to cover all possible linguistic phenomena, but captures different levels of language processing while retaining internal uniformity: sandwich ), or incorrect verb tense or aspect inflection ( *the boy eaten the sandwich ). In each case, the sentence can be corrected by changing the inflectional form of one word. 2. Semantic anomaly : a violation of a selectional restriction, such as animacy ( #the house eats the sandwich ). In these cases, the sentence can be corrected by replacing one of the verb's arguments with another one in the same word class that satisfies the verb's selectional restrictions. 3. Commonsense anomaly : sentence describes an situation that is atypical or implausible in the real world but is otherwise acceptable ( #the customer served the waitress ). We use two sources of data for experiments on linguistic anomalies: synthetic sentences generated from templates, and materials from psycholinguistic studies. Both have advantages and disadvantages  synthetic data can be easily generated in large quantities, but the resulting sentences may be odd in unintended ways. Psycholinguistic stimuli are designed to control for confounding factors (e.g., word frequency) and human-validated for acceptability, but are smaller (typically fewer than 100 sentence pairs). We curate a set of 12 tasks from BLiMP and 7 psycholinguistic studies 1 . Each sentence pair consists of a control and an anomalous sentence, so that all sentences within a task differ in a consistent manner. Table 1 shows an example sentence pair from each task. We summarize each dataset: 1. BLiMP (Warstadt et al., 2020): we use subject-verb and determiner-noun agreement tests as morphosyntactic anomaly tasks. For simplicity, we only use the basic regular sentences, and exclude sentences involving irregular words or distractor items. We also use the two argument structure tests involving animacy as a semantic anomaly task. All three BLiMP tasks therefore have 2000 sentence pairs. 1 Several of these stimuli have been used in natural language processing research. Chersoni et al. (2018) used the data from Pylkkanen and McElree (2007) and Warren et al. (2015) to probe word vectors for knowledge of selectional restrictions. Ettinger (2020) used data from Federmeier and Kutas (1999) and Chow et al. (2016), which were referred to as CPRAG-102 and ROLE-88 respectively. 2. Osterhout and Nicol (1999): contains 90 sentence triplets containing a control, syntactic, and semantic anomaly. Syntactic anomalies involve a modal verb followed by a verb in -ing form; semantic anomalies have a selectional restriction violation between the subject and verb. There are also double anomalies (simultaneously syntactic and semantic) which we do not use. 3. Pylkkanen and McElree (2007): contains 70 sentence pairs where the verb is replaced in the anomalous sentence with one that requires an animate object, thus violating the selectional restriction. In half the sentences, the verb is contained in an embedded clause. 4. Warren et al. (2015): contains 30 sentence triplets with a possible condition, a selectional restriction violation between the subject and verb, and an impossible condition where the subject cannot carry out the action, i.e., a commonsense anomaly. 5. Osterhout and Mobley (1995): we use data from experiment 2, containing 90 sentence pairs where the verb in the anomalous sentence is semantically inappropriate. The experiment also tested gender agreement errors, but we do not include these stimuli. 6. Federmeier and Kutas (1999): contains 34 sentence pairs, where the final noun in each anomalous sentence is an inappropriate completion, but in the same semantic category as the expected completion. 7. Chow et al. (2016): contains 44 sentence pairs, where two of the nouns in the anomalous sentence are swapped to reverse their roles. This is the only task in which the sentence pair differs by more than one token. 8. Urbach and Kutas (2010): contains 120 sentence pairs, where the anomalous sentence replaces a patient of the verb with an atypical one. Let D = { ( s 1 , s  1 ) ,    , ( s n , s  n ) } be a dataset of sentence pairs, where s i is a control sentence and s  i is an anomalous sentence. For each layer L , we define the surprisal gap as the mean difference of surprisal scores between the control and anomalous sentences, scaled by the standard deviation: surprisal gap L ( D ) = E { surprisal L ( s  i )  surprisal L ( s i ) } ni =1  { surprisal L ( s  i )  surprisal L ( s i ) } ni =1 (9) The surprisal gap is a scale-invariant measure of sensitivity to anomaly, similar to a signal-to-noise ratio. While surprisal scores are unitless, the surprisal gap may be viewed as the number of standard deviations that anomalous sentences trigger surprisal above control sentences. This is advantageous over accuracy scores, which treats the sentence pair as correct when the anomalous sentence has higher surprisal by any margin; this hard cutoff masks differences in the magnitude of surprisal. The metric also allows for fair comparison of surprisal scores across datasets of vastly different sizes. Figure 4 shows the surprisal gap for all 12 tasks, using the RoBERTa model; the results for BERT and XLNet are in the Appendix C. Next, we compare the performance of the Gaussian model with the masked language model (MLM). We score each instance as correct if the masked probability of the correct word is higher than the anomalous word. One limitation of the MLM approach is that it requires the sentence pair to be identical in all places except for one token, since the LMs do not support modeling joint probabilities over multiple tokens. To ensure fair comparison between GM and MLM, we exclude instances where the differing token is out-of-vocabulary in any of the LMs (this excludes approximately 30% of instances). For the Gaussian model, we compute accuracy using the best-performing layer for each model (Section 3.2). The results are listed in Table 2. 5 Discussion 5.1 Anomaly type and surprisal Morphosyntactic anomalies generally appear earlier than semantic anomalies (Figure 4). The surprisal gap plot exhibits different patterns depending on the type of linguistic anomaly: morphosyntactic anomalies produce high surprisal relatively early (layers 3-4), while semantic anomalies produce low surprisals until later (layers 9 and above). Commonsense anomalies do not result in surprisals at any layer: the surprisal gap is near zero for all of the commonsense tasks. The observed difference between morphosyntactic and semantic Commonsense  Urbach and Kutas Commonsense  Chow et al. anomalies is consistent with previous work (Ten-ney et al. , 2019), which found that syntactic information appeared earlier in BERT than semantic information. One should be careful and avoid drawing conclusions from only a few experiments. A similar situation occurred in psycholinguistics research (Kutas et al., 2006): early results suggested that the N400 was triggered by semantic anomalies, while syntactic anomalies triggered the P600  a different type of ERP. However, subsequent experiments found exceptions to this rule, and now it is believed that the N400 cannot be categorized by any standard dichotomy, like syntax versus semantics (Kutas and Federmeier, 2011). In our case, Pylkkanen and McElree (2007) is an exception: the task is a semantic anomaly, but produces surprisals in early layers, similar to the morphosyntactic tasks. Hence it is possible that the dichotomy is something other than syntax versus semantics; we leave to future work to determine more precisely what conditions trigger high surprisals in lower versus upper layers of LMs. The masked language model (MLM) usually outperforms the Gaussian anomaly model (GM), but the difference is uneven. MLM performs much better than GM on commonsense tasks, slightly better on semantic tasks, and about the same or slightly worse on morphosyntactic tasks. It is not obvious why MLM should perform better than GM, but we note two subtle differences between the MLM and GM setups that may be contributing factors. First, the GM method adds up the surprisal scores for the whole sequence, while MLM only considers the softmax distribution at one token. Second, the input sequence for MLM always contains a [MASK] token, whereas GM takes the original unmasked sequences as input, so the representations are never identical between the two setups. MLM generally outperforms GM, but it does not solve every task: all three LMs fail to perform above chance on the data from Warren et al. (2015). This set of stimuli was designed so that both the control and impossible completions are not very likely or expected, which may have caused the difficulty for the LMs. We excluded the task of Chow et al. (2016) for MLM because the control and anomalous sentences differed by more than one token 2 . RoBERTa is the best-performing of the three LMs in both the GM and MLM settings: this is expected since it is trained with the most data and performs well on many natural language benchmarks. Surprisingly, XLNet is ill-suited for this task and performs worse than BERT, despite having a similar model capacity and training data. XL-2 Sentence pairs with multiple differing tokens are inconvenient for MLM to handle, but this is not a fundamental limitation. For example, Salazar et al. (2020) proposed a modifi-cation to MLM to handle such cases: they compute a pseudolog-likelihood score for a sequence by replacing one token at a time with a [MASK] token, applying MLM to each masked sequence, and summing up the log likelihood scores. Net (Appendix C) show some differences from RoBERTa: only morphosyntactic tasks produce out-of-domain embeddings in these two models, and not semantic or commonsense tasks. Evidently, how LMs behave when presented with anomalous inputs is dependent on model architecture and training data size; we leave exploration of this phenomenon to future work. We use Gaussian models to characterize out-of-domain embeddings at intermediate layers of Transformer language models. The model requires a relatively small amount of in-domain data. Our experiments reveal that out-of-domain points in lower layers correspond to low-frequency tokens, while grammatically anomalous inputs are out-of-domain in higher layers. Furthermore, morphosyntactic anomalies are recognized as out-of-domain starting from lower layers compared to syntactic anomalies. Commonsense anomalies do not generate out-of-domain embeddings at any layer, even when the LM has a preference for the correct cloze completion. These results show that depending on the type of linguistic anomaly, LMs use different mechanisms to produce the output softmax distribution. We thank Julian Salazar and our anonymous reviewers for their helpful suggestions. YX is funded through an NSERC Discovery Grant, a SSHRC Insight Grant, and an Ontario ERA award. FR is supported by a CIFAR Chair in Artificial Intelligence.
abstain method abstain result abstain abstain abstain abstain abstain other abstain abstain result abstain abstain method result abstain method result result other other other other other other other other objective other other other other other other other other other other other other other other other other other abstain method abstain method method abstain abstain method abstain abstain abstain abstain abstain abstain abstain method abstain abstain method abstain abstain method method method abstain method abstain method abstain abstain abstain abstain abstain method abstain abstain abstain abstain abstain method abstain method method abstain abstain abstain other abstain abstain abstain abstain abstain abstain abstain abstain method abstain abstain abstain abstain abstain method abstain abstain abstain abstain abstain abstain method method method abstain abstain abstain abstain abstain abstain method abstain abstain abstain method abstain abstain abstain abstain abstain abstain method abstain abstain abstain abstain abstain abstain abstain method abstain abstain abstain abstain abstain abstain abstain abstain abstain method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain method method abstain result abstain abstain abstain other other other