Existing pre-trained transformer analysis works usually focus only on one or two model families at a time, overlooking the variability of the architecture and pre-training objectives. In our work, we utilize the oLMpics benchmark and psycholinguistic probing datasets for a diverse set of 29 models including T5, BART, and ALBERT. Additionally, we adapt the oLMpics zero-shot setup for autoregressive models and evaluate GPT networks of different sizes. Our findings show that none of these models can resolve compositional questions in a zero-shot fashion, suggesting that this skill is not learnable using existing pre-training objectives. Furthermore, we find that global model decisions such as architecture, directionality, size of the dataset, and pre-training objective are not predictive of a model's linguistic capabilities. The code for this study is available on GitHub 1 . After the initial success of transfer learning in natural language processing (Howard and Ruder, 2018; Peters et al., 2018), the number of pre-trained models in NLP has increased dramatically (Radford and Narasimhan, 2018; Devlin et al., 2018; Lewis et al., 2019; Liu et al., 2019b; Raffel et al., 2019; Lan et al., 2019; Dong et al., 2019). However, there is a limited understanding of why certain models perform better than others and what linguistic capabilities they acquire through pre-training. While a lot of work has been done to evaluate these models on general natural language understanding datasets (Wang et al., 2018, 2019; Lai et al., 2017), such datasets do not allow researchers to identify the specific linguistic capabilities of a model. Furthermore the performance on these  The first two authors made equal contribution to this work. datasets results from a combination of pre-trained knowledge and task-specific information learned through fine-tuning. Probing tasks (Talmor et al., 2019; Zagoury et al., 2021; McCoy et al., 2019; Goldberg, 2019) give a promising solution to this problem, as they evaluate specific capabilities of pre-trained models, and in many cases, these tasks are designed for zero-shot evaluation, which reveals the knowledge that models have actually learned purely through the upstream task. Currently, most in-depth analysis studies focus on one or two model families. Many analysis papers only probe BERT and similar models (Ettinger, 2020; Kobayashi et al., 2020; Gar Soler and Apidianaki, 2020; Ravichander et al., 2020; Zagoury et al., 2021; Kassner et al., 2020; Mohebbi et al., 2021; Clark et al., 2020; Liu et al., 2021). Fortunately, this trend is changing and now we see more papers that probe models such as ALBERT , T5 or BART (Mosbach et al., 2020; Phang et al., 2021; Jiang et al., 2021). However, only a small number of analysis papers have probed multiple (three or more) model families (Zhou et al., 2021; Ilharco et al., 2021). In our work, we test 8 families of models on oLMpics tasks (Talmor et al., 2019) and 6 families on psycholinguistic tasks from Ettinger (2020). These models differ in size, architecture, pretraining objective, dataset size, and have other small yet important differences. Such a diverse set of models provides a broader view of what linguistic capabilities are affected by the change of any of these properties. We also include several distilled models in our analysis. We find that different models excel in different symbolic reasoning tasks, suggesting that slight differences related to optimization or masking strategy might be more important than the pre-training approach, dataset size, or architecture . Furthermore, in contrast to Radford et al. (2019), we find that for oLMpics tasks, model size rarely correlates with the model 3180 performance. In addition, we observe that all models fail on composition tasks when evaluated in a zero-shot fashion. Pre-trained model analysis is a rapidly growing area in NLP today. There exists a number of methods for analyzing internal representations of a model, including structured head and FCN pruning (Michel et al., 2019; Voita et al., 2019; Prasanna et al., 2020), residual connection and LayerNormal-ization analysis (Kovaleva et al., 2021; Kobayashi et al., 2021), and analyzing attention patterns (Clark et al., 2019; Kovaleva et al., 2019). Compared to these methods, probing tasks (Con-neau et al., 2018; Tenney et al., 2019) provide a more direct way to evaluate what a model can and cannot accomplish. While it is possible to probe embeddings or hidden representations directly (Tenney et al., 2019; Liu et al., 2019a), the adoption of pre-trained language models has made it possible to evaluate such models by framing probing tasks close to the original model objective (Rad-ford et al., 2019; Talmor et al., 2019; Ettinger, 2020; Goldberg, 2019). However, when a research area moves this quickly, it can be hard to keep up with many new models. Most of the existing research (Gar Soler and Apidianaki, 2020; Zagoury et al., 2021; Kassner et al., 2020) papers compare only one or two model families. Even some of the most recent works only probe BERT or very similar models (Zagoury et al., 2021; Liu et al., 2021). Only a small number of analysis papers have probed multiple (three or more) model families (Zhou et al., 2021; Ilharco et al., 2021). In contrast to existing work, we perform a large-scale probing of 29 models across 8 different model families. We apply the existing probing benchmarks, namely, oLMpics (Talmor et al., 2019) and psycholinguistic datasets (Ettinger, 2020), to models that differ in the pre-training objective, datasets, size, architecture, and directionality. We use 8 different model families in this study. All of them are based on the transformer architecture and pre-trained on general-domain texts, but this 2 GPTNEO is trained on a 800Gb dataset. is where the similarities end. We summarize their major differences in Table 1. In this section, we discuss and highlight the details that distinguish models, from the major ones to the ones that might appear very minor. BERT (Devlin et al., 2018) is pre-trained on Book Corpus and Wikipedia using a combination of Masked Language Modeling (MLM) and Next Sentence Prediction (NSP). It uses GELU activations (Hendrycks and Gimpel, 2016) for fully-connected layers. For the first 90% of the training iterations, the maximum length is 128, but then it is increased to 512. RoBERTa (Liu et al., 2019b) is the most similar to BERT in this study; however, it differs from it in many small but important details: the pre-training dataset is considerably larger and includes Open-WebText (Gokaslan and Cohen, 2019), Stories (Trinh and Le, 2018), and CC-News. RoBERTa does not use Next Sentence Prediction; applies masking dynamically; always trains with 512 max tokens; uses a smaller ADAM  = 0 . 98 ; 8 times larger batch size than BERT ; and has a larger, byte-level BPE vocabulary (50K instead of 31K). DistilBERT (Sanh et al., 2019) is a distilled version of BERT. It has half the layers of BERT and is trained using soft targets produced by BERT . ALBERT (Lan et al., 2019) shares parameters across transformer layers and uses an extra projection between the embedding and the first transformer layer. It replaces NSP with the sentence-order prediction. ALBERT uses n-gram masking and the LAMB (You et al., 2019) optimizer. The training setup is similar to BERT , but it trains 90% of the time using the sequence length 512 and randomly reduces it in 10% of iterations. Parameter sharing allows ALBERT to achieve performance similar to BERT with much fewer trainable parameters. The smallest ALBERT model has 12M trainable parameters and the largest has 235M. ALBERTv2 is a minor modification of ALBERT that was trained without dropout, for twice as many training steps with additional training data 3 . GPT-2 (Radford et al., 2019) is a unidirectional transformer language model trained on the WebText dataset. Unlike other models, it is a Pre-Norm transformer. Similar to RoBERTa , GPT2 has a 50K vocabulary and a byte-level BPE but treats spaces as a separate symbol. It also comes in multiple sizes from 124M parameters up to 2.8B parame-3 github.com/google-research/albert 3181 Model Parameters Pre-training Data Size Enc-Dec Autoregressive Tokenization Vocab. ters. There exist several popular reimplementations of this model, such as GPT-Neo (Black et al., 2021), which generally follow the original paper but differ in dataset (Gao et al., 2020), model, and training hyperparameters. UniLM (Dong et al., 2019) utilizes several attention masks to control the access to context for each word token. It uses a multitask objective that is modeled by applying different attention masks. The mix of tasks includes masked language modeling, unidirectional language modeling, and sequence-to-sequence language modeling. Additionally, it employs the NSP objective and is initialized using BERT model weights. In optimization, it generally follows BERT but always uses 512 as the maximum sequence length. BART (Lewis et al., 2019) is an encoder-decoder model that is trained on text infilling and sentence permutation tasks. It is trained on the same dataset as RoBERTa . Compared to BERT , BART does not use an additional projection when predicting word logits. In optimization, it closely follows RoBERTa , but disables dropout for the final 10% of training. T5 (Raffel et al., 2019) is also an encoder-decoder model. It is trained using a text infilling task on the C4 dataset. However, it only generates the text in place of the [MASK] token and not the full input sentence. Architecturally, it is a Pre-Norm model and T5 LayerNorm does not use bias. Output projection weights are tied with the input embedding matrix. It uses 128 relative positional embeddings that are added at every layer. Unlike most of the models in this study, it uses the ReLU activation. The smallest T5 model used in this study has 233M parameters and the largest has 2.8B. We have not evaluated the 11B T5 model due to hardware limitations. Unlike the original T5 , T5v1 . 1 4 is trained on different data, does not tie logit layer with input embeddings, uses GEGLU activations (Shazeer, 2020) and no dropout. It also slightly changes model shapes. The oLMpics benchmark consists of eight tasks that test multiple specific skills, such as a model's ability to draw comparisons, understand negation, and perform simple linguistic composition tasks. Table 2 shows examples for every task in oLMpics. Zero-Shot vs. Multi-Shot A major advantage of the oLMpics tasks is that zero-shot evaluation can be performed for most tasks due to the task format. Zero-shot evaluation eliminates the ambiguity of whether a model's knowledge is stored in its pre-trained representations or learned during fine-tuning. However, a model may possess the necessary information but fail during zero-shot evaluation due to the wording of the task. Therefore, multi-shot evaluation can also be informative, allowing the model to adapt to the input format and possibly learn task-specific features. OLMpics tasks include training sets specifically for this reason, in order to separate the impact of fine-tuning from pre-training. MC-MLM vs. MC-QA The oLMpics tasks are framed in one of two ways: MC-MLM (Multi-ple Choice-Masked Language Modeling) and MC-QA (Multiple Choice-Question Answering). MC-MLM tasks are formulated as a masked language modeling task (Devlin et al., 2018), where the model needs to predict the word replaced by the MASK token. An example of an Age Comparison sentence is A 41 year old is [MASK] a 42 year 4 huggingface.co/google/t5-v1_1-base 3182 Task Name Example Question Choices Age Comparison A 41 year old person age is [MASK] than a 42 year old person. younger, older Object Comparison The size of a nail is usually [MASK] than the size of a fork. smaller, larger Antonym Negation It was [MASK] a fracture, it was really a break. not, really Taxonomy Conjunction A ferry and a biplane are both a type of [MASK]. airplane, craft, boat Property Conjunction What is related to vertical and is related to honest? straight, trustworthy, steep Encyclopedic Composition When did the band where Alan Vega played first form? 1970, 1968, 1969 Hypernym Conjunction A basset and a tamarin are both a type of [MASK] primate, dog, mammal Multi-hop Composition When comparing a 21 year old, 15 year old, and 19 year old, the [MASK] is oldest. third, first, second Table 2: Examples of oLMpics questions, with the correct answer underlined. old. A model's prediction is determined by the probabilities assigned to the [MASK] token, with younger being selected if its probability is higher than older, and older otherwise. MC-MLM restricts the possible answers to single tokens. Tasks with longer answers require MC-QA. In this method, a new feedforward network maps the [CLS] token embedding to a single logit. For prediction, answer choices are individually concatenated to the original question, forming a new sentence for each choice. This set of sentences is input into the model, and the choice corresponding to the sentence with the largest logit is selected. While the MC-QA method allows for longer choices, the added feedforward network must be trained; therefore, zero-shot evaluation is not possible. Extending Beyond MLM The oLMpics MC-MLM method relies on the model giving probabilities of individual words in a bidirectional context. However, models like GPT2 do not have access to the future context, which makes it impossible to directly predict the token in an example like A 41 year old is [MASK] than 42 year old. For these models, we sum the log-probabilities of individual words to find the probability of the whole sentence. We do this for every possible answer, e.g., a sequence with younger instead of [MASK] and older. Then, we select the one with the highest total probability. Extending BART and T5 is more straightforward because their objectives and architecture are very flexible. For both of these models, we use the original oLMpics input format. T5 has multiple [MASK]-tokens and we always use <extra_id_0> token in our evaluation. The biggest difference is that BART produces the full sentence and we need to extract the probabilities for the masked words and T5 produces only the tokens in the place of [MASK]. Similar to oLMpics, the datasets used by Ettinger (2020) are framed as fill in the blank tasks. Unlike oLMpics, the model always needs to predict only the last word, so both bidirectional and unidirectional models can be evaluated on these tasks directly. The biggest distinction of this dataset is its source. The datasets CPRAG-102 (Federmeier and Kutas, 1999), ROLE-88 (Chow et al., 2016), and NEG-136 (Fischler et al., 1983) come from the psycholinguistics and neuroscience studies and were originally evaluated on humans. CPRAG-102 targets commonsense and pragmatic inference e.g. Justin put a second house on Park Place. He and his sister often spent hours playing __ , Target: monopoly , other labels: chess, baseball . ROLE-88 aims at evaluating event knowledge and semantic roles. NEG-136 tests how well models understand the meaning of negation and consists of two subsets: simple (SIMP) and natural (NAT). For example, SIMP: Salmon is a fish/dog versus Salmon is not a fish/dog. NAT: Rockets and missiles are very fast/slow versus Rockets and missiles aren't very fast/slow . Evaluation of this dataset is performed in two ways: affirmative statements and negative statements. For affirmative ones, the model needs to complete a sentence like A robin is a with the expected answer bird . For negative, A robin is not a should not be completed with a bird . (Ettinger, 2020) finds that this type of error is very common in BERT , which suggests that the model cannot handle negation correctly. Ettinger (2020) tests BERT models in two ways: using a pre-defined set of answers, similar to oLMpics MC-MLM, or computing top-k accuracy from the whole model vocabulary. We adopt the same approach in this study. We evaluate eight models families on the oLMpics (29 models in total) and six families on psycholinguistic data (17 models). This extends the Talmor et al. (2019) results with six new model families and Ettinger (2020) with four. Zero-shot evaluation It has been shown that language models can implicitly learn downstream tasks (Radford et al., 2019; Brown et al., 2020). However, it is still not obvious what tasks are learnable in this manner without explicit supervision. In our study, similar to Talmor et al. (2019), we find that none of the models can solve Multi-Hop Composition or Always-Never tasks substantially better than a majority baseline (see Table 4). This holds true not only for masked language models but also for unidirectional language models such as GPT2 and text-infilling models such as T5 or BART . Only small and base versions of T5v1 . 1 outperform the majority baseline on MultiHop Composition by a small margin. Multi-shot evaluation Not surprisingly, fine-tuning models on oLMpics improves the scores across the board. This is true even for the tasks on which zero-shot performance is extremely poor. For example, while all models fail on Multi-hop Composition during zero-shot evaluation, most models can reach perfect or near-perfect accuracy on this task after fine-tuning. However, AlwaysNever and Taxonomy Conjunction remain challenging for all models. For the full multi-shot evaluation, see Table 7 in the Appendix. To check how the size of a model affects the performance, we evaluated different versions of GPT2 , T5 , and ALBERT models on the oLMpics tasks ranging from 14M (smallest ALBERT ) to 2.8B (largest T5 ) parameters. All of the models perform near-random on 3 out of the 6 tasks, suggesting that Multi-Hop Composition, Antonym Negation, and Always-Never are hard to learn via the (masked) language modeling objective. On the rest of the tasks, we observe no clear improvement trend for GPT models based on the model size. In most of the tasks, GPT large either performs on par or has higher accuracy than GPT xl while being twice as small. We also compute Spearman correlation between model accuracy and model size for GPT2 , ALBERT , and T5 models. 5 For all GPT2 and ALBERT (v1 and v2) tests, the p-value is (cid:29) 0 . 05 , suggesting that there is no rank-correlation between model size and task performance. However, in the case of T5 models, there is a strong (1.0) and significant correlation (p-value  10  6 ) for all tasks except Always-Never . We account for multiple hypothesis testing using Bonferroni's method. For Taxonomy Conjunction , the correlation is negative. Contrary to the common knowledge, with rare exceptions (Section 4.1), we do not observe that parameter count, dataset size, model architecture or directionality are predictive of model performance on zero-shot oLMpics (Table 4). RoBERTa large usually performs amongst the best models, while having a very similar architecture and objective to BERT large . Reasonable explanations would be the dataset size, but this does not align with the BART large results. Encoder-decoder architecture does seem not to be indicative of the performance either, as T5 large and BART large have vastly different results. Psycholinguistic datasets (Table 5) demonstrate similar behaviour. RoBERTa large is generally the stongest model followed by T5 xl . We would like to note that these datasets have less than 100 examples and their statistical power (Card et al., 2020) is very small. Our intuitions about the relative suitability of different model classes are based on their performance on standard benchmarks (Wang et al., 2018, 2019) and existing investigations of scaling laws (Rad-ford et al., 2019; Kaplan et al., 2020). In contrast to this received wisdom, our experiments suggest that this does not in fact lead to better performance on specific linguistic skills. Ettinger (2020) observed that BERT is not sensitive to negation in non-natural (SIMP) or less-natural cases. In our experiments (Table 6), we find that the only model with zero accuracy outside of BERT is a distilled version of BERT itself. Multiple models achieve non-zero accuracy 5 Note that sample size for each test is  4, so these results should be taken as anecdotal. on NEG-SIMP (neg), but the numbers might be misleading. For example, while ALBERTv1 xlarge has 27.8% accuracy on NEG-SIMP (neg), this accuracy is mainly caused by mistakes in language modeling while still being insensitive to negation (e.g., it predicts vegetable for both An ant is a and An ant is not a ). Specifically, ALBERTv1 xlarge only changes its predictions in 5.5% cases. However, unlike other models, RoBERTa large actually changes its predictions in 33% cases, suggesting that sensitivity to negation might be possible to learn via masked language modeling. One drawback of datasets from Ettinger (2020) that we have noticed was the ambiguity of answers. For 3185 example, many models predict words like this, that, it as the next word for Checkmate, Rosaline announced with glee. She was getting to be really good at [MASK] instead of the word chess. In fact, for T5 xl predictions, we found that 79.4% of predictions are semantically and grammatically plausible, while this model has only achieved 58.8% top-5 accuracy on the CPRAG-126 dataset (Table 5). Another example would be I'm an animal like Eeyore! the child exclaimed. His mother wondered why he was pretending to be a [MASK] . CPRAG expects the answer donkey, which assumes that the reader (or model) is familiar with the English names of Winnie-the-Pooh book characters. 6 4.6 Antonym Negation: Impact of prompt variation While there is clear evidence that models pretrained with the MLM objective have trouble with negation (Ettinger, 2020), no such evidence has been available for models trained autoregressively. At the same time, a number of studies have shown that autoregressive models can be significantly improved with prompting. Our question is whether we can make a language model (GPT-2) understand negation via an alternative wording of the task (prompt engineering). We tested four different prompts for the Antonym Negation task. Table 3 shows the patterns and the corresponding accuracies of GPT models. All experiments use yes/no verbalizers. While some prompts improve the oLMpics prompt results (up to +6%), this improvement is not consistent across models showing that even very similar models are sensitive to prompt variation in different ways. Additionally, prompt #4 (Table 3) improves the smallest model, GPT2 base , so significantly that it outperforms the largest model by approximately 10%, demonstrating once again that parameter count is not a reliable predictor of the model performance. For one oLMpics task, Age Comparison, we observe that models do not perform equally well on all age ranges, similar to the findings of Talmor et al. (2019). Figure 1 shows that with the exception of GPT2 base , all GPT2 variants perform well on 10-20 year olds and poorly on the 30-40 age group, with a significant drop in performance from 80% to 20%. Generally, GPT2 seems to predict younger ages more accurately. However, the smallest model, GPT2 base , exhibits a different trend than other models as age increases. We find that model performance can change significantly on both oLMpics and psycholinguistic datasets if we add a period to the end of the sequence. For example, BERT and DistilBERT achieve an accuracy of 3% without a period on CPRAG as compared to 52.9% when a '. is appended. We observe a similar trend on the ROLE and NEG datasets and for other models including RoBERTa , where the accuracy on CPRAG jumped from 47.1% to 70.1%. For oLMpics, the change of performance is less dramatic, but still noticeable. We observe that in 6% of cases (across all 3186 CPRAG-126 ROLE-88 NEG-136SIMP(Aff) NEG-136NAT(Aff) BERT base 52.9 27.3 100 43.8 BERT large 52.9 37.5 100 31.3 RoBERTa base 70.1 46.6 94.4 56.3 RoBERTa large 82.4 55.7 94.4 50 DistilBERT base 55.9 28.4 94.4 43.8 AlBERTv1 base 17.6 17.1 72.2 25.0 AlBERTv1 large 35.3 26.1 83.3 25 AlBERTv1 xlarge 41.2 34.1 55.5 18.8 AlBERTv1 xxlarge 82.4 53.4 72.2 50 AlBERTv2 base 41.4 26.1 33.3 31.1 AlBERTv2 large 47.1 29.5 83.3 37.5 AlBERTv2 xlarge 61.8 37.5 94.4 25 AlBERTv2 xxlarge 85.3 50 100 37.5 T5 small 20.6 9.1 44.4 18.8 T5 base 41.1 27.3 88.9 31.3 T5 large 50.0 36.4 94.4 43.8 T5 xl 58.8 44.3 83.3 62.5 Table 5: Zero-shot top-5 word prediction accuracy. models and all tasks), model performance changes by more than 10 absolute percentage points if a full stop is added to the end of sentence. Figure 2 shows the histogram of accuracy changes for oLMpics tasks. In this work, we apply a large and diverse set of models to oLMpics and psycholinguistic tasks. The variety of models allows us to investigate the performance of different architectures and pre-training methods on a variety of linguistic tasks. Contrary to received wisdom, we find that parameter count within a given model family does not correlate with model performance on these tasks. We find that none of the models, even the 2.8B-sized ones, can resolve Multi-Hop Composition and Always-Never tasks in a zero-shot manner, suggesting that the existing pre-training methods cannot learn such tasks. Finally, we find that different models excel in different symbolic reasoning tasks, suggesting that slight differences related to optimization or masking strategy might be more important than the pre-training approach, dataset size, or architecture. This work is funded in part by the NSF award number IIS-1844740.
abstain method method objective objective abstain other abstain abstain objective abstain abstain abstain abstain result abstain abstain abstain abstain abstain result abstain result other other other other other other other other objective abstain method other other objective abstain other other other abstain other other other other other other other other other other other other other other other other other other other other other other other other other other other other other other other other method method method other other other other other other other other other other other other other other other other other other other other other other abstain abstain abstain other method method method other other other other other other other other other other other other other other other method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain result abstain abstain abstain abstain abstain abstain abstain result abstain abstain abstain abstain abstain abstain abstain abstain abstain result abstain abstain abstain abstain abstain abstain abstain abstain result abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain result abstain abstain abstain abstain result abstain abstain result abstain result abstain abstain method objective result result result other