Prompt-based probing has been widely used in evaluating the abilities of pretrained language models (PLMs). Unfortunately, recent studies have discovered such an evaluation may be inaccurate, inconsistent and unreliable. Furthermore, the lack of understanding its inner workings, combined with its wide applicability, has the potential to lead to unforeseen risks for evaluating and applying PLMs in real-world applications. To discover, understand and quantify the risks, this paper investigates the prompt-based probing from a causal view, highlights three critical biases which could induce biased results and conclusions, and proposes to conduct debiasing via causal intervention. This paper provides valuable insights for the design of unbiased datasets, better probing frameworks and more reliable evaluations of pretrained language models. Furthermore, our conclusions also echo that we need to rethink the criteria for identifying better pretrained language models 1 . During the past few years, the great success of pretrained language models (PLMs) (Devlin et al., 2019; Liu et al., 2019; Brown et al., 2020; Raffel et al., 2020) raises extensive attention about evaluating what knowledge do PLMs actually entail. One of the most popular approaches is prompt-based probing (Petroni et al., 2019; Davison et al., 2019; Brown et al., 2020; Schick and Schtze, 2020; Ettinger, 2020; Sun et al., 2021), which assesses whether PLMs are knowledgable for a specific task by querying PLMs with task-specific prompts. For example, to evaluate whether BERT knows the birthplace of Michael Jordan, we could query BERT with Michael Jordan was born in [MASK]. Recent studies often construct prompt-based probing datasets, and take PLMs' perforCorresponding Authors 1 We openly released the source code and data at https: //github.com/c-box/causalEval . mance on these datasets as their abilities for the corresponding tasks. Such a probing evaluation has been wildly used in many benchmarks such as SuperGLUE (Wang et al., 2019; Brown et al., 2020), LAMA (Petroni et al., 2019), oLMpics (Tal-mor et al., 2020), LM diagnostics (Ettinger, 2020), CAT (Zhou et al., 2020), X-FACTR (Jiang et al., 2020a), BioLAMA (Sung et al., 2021), etc. Unfortunately, recent studies have found that evaluating PLMs via prompt-based probing could be inaccurate, inconsistent, and unreliable. For example, Poerner et al. (2020) finds that the performance may be overestimated because many instances can be easily predicted by only relying on surface form shortcuts. Elazar et al. (2021) shows that semantically equivalent prompts may result in quite different predictions. Cao et al. (2021) demonstrates that PLMs often generate unreliable predictions which are prompt-related but not knowledge-related. In these cases, the risks of blindly using prompt-based probing to evaluate PLMs, without understanding its inherent vulnerabilities, are significant. Such biased evaluations will make us overestimate or underestimate the real capabilities of PLMs, mislead our understanding of models, and result in 5796 wrong conclusions. Therefore, to reach a trustworthy evaluation of PLMs, it is necessary to dive into the probing criteria and understand the following two critical questions: 1) What biases exist in current evaluation criteria via prompt-based probing? 2) Where do these biases come from? To this end, we compared PLM evaluation via prompt-based probing with conventional evaluation criteria in machine learning. Figure 1 shows their divergences. Conventional evaluations aim to evaluate different hypotheses (e.g., algorithms or model structures) for a specific task. The tested hypotheses are raised independently of the train-ing/test data generation. However, this indepen-dence no longer sustains in prompt-based probing. There exist more complicated implicit connections between pretrained models, probing data, and prompts, mainly due to the bundled pretraining data with specific PLMs. These unaware connections serve as invisible hands that can even dominate the evaluation criteria from both linguistic and task aspects. From the linguistic aspect, because pretraining data, probing data and prompts are all expressed in the form of natural language, there exist inevitable linguistic correlations which can mislead evaluations. From the task aspect, the pretraining data and the probing data are often sampled from correlated distributions. Such invisible task distributional correlations may significantly bias the evaluation. For example, Wikipedia is a widely used pretraining corpus, and many probing data are also sampled from Wikipedia or its extensions such as Yago, DBPedia or Wikidata (Petroni et al., 2019; Jiang et al., 2020a; Sung et al., 2021). As a result, such task distributional correlations will inevitably confound evaluations via domain overlapping, answer leakage, knowledge coverage, etc. To theoretically identify how these correlations lead to biases, we revisit the prompt-based probing from a causal view. Specifically, we describe the evaluation procedure using a structural causal model (Pearl et al., 2000) (SCM), which is shown in Figure 2a. Based on the SCM, we find that the linguistic correlation and the task distributional correlation correspond to three backdoor paths in Figure 2b-d, which lead to three critical biases:  Prompt Preference Bias, which mainly stems from the underlying linguistic correlations between PLMs and prompts, i.e., the performance may be biased by the fitness of a prompt to PLMs' linguistic preference. For instance, semantically equivalent prompts will lead to different biased evaluation results. Instance Verbalization Bias, which mainly stems from the underlying linguistic correlations between PLMs and verbalized probing datasets, i.e., the evaluation results are sensitive and inconsistent to the different verbalizations of the same instance (e.g., representing the U.S.A. with the U.S. or America). Sample Disparity Bias, which mainly stems from the invisible distributional correlation between pretraining and probing data, i.e., the performance difference between different PLMs may due to the sample disparity of their pretraining corpus, rather than their ability divergence. Such invisible correlations may mislead evaluation results, and thus lead to implicit, unaware risks of applying PLMs in real-world applications. We further propose to conduct causal intervention via backdoor adjustments, which can reduce bias and ensure a more accurate, consistent and reliable probing under given assumptions. Note that this paper not intends to create a universal cor-rect probing criteria, but to remind the underlying invisible risks, to understand how spurious correlations lead to biases, and to provide a causal toolkit for debiasing probing under specific assumptions. Besides, we believe that our discoveries not only exist in prompt-based probing, but will also influence all prompt-based applications to pretrained language models. Consequently, our conclusions echo that we need to rethink the criteria for identifying better pretrained language models with the above-mentioned biases. Generally, the main contributions of this paper are:  We investigate the critical biases and quantify their risks of evaluating pretrained language models with widely used prompt-based probing, including prompt preference bias, instance verbalization bias, and sample disparity bias. We propose a causal analysis framework, which can be used to effectively identify, understand, and eliminate biases in prompt-based probing evaluations. We provide valuable insights for the design of unbiased datasets, better probing frameworks, and more reliable evaluations, and echo that we should rethink the evaluation criteria for pretrained language models. Causal inference is a promising technique for identifying undesirable biases and fairness concerns in benchmarks (Hardt et al., 2016; Kilbertus et al., 2017; Kusner et al., 2017; Vig et al., 2020; Feder et al., 2021). Causal inference usually describes the causal relations between variables via Structural Causal Model (SCM), then recognizes confounders and spurious correlations for bias analysis, finally identifies true causal effects by eliminating biases using causal intervention techniques. SCM The structural causal model (Pearl et al., 2000) describes the relevant features in a system and how they interact with each other. Every SCM is associated with a graphical causal model G = { V, f } , which consists of a set of nodes representing variables V , as well as a set of edges between the nodes representing the functions f to describe the causal relations. Causal Intervention To identify the true causal effects between an ordered pair of variables ( X, Y ) , Causal intervention fixes the value of X = x and removes the correlations between X and its precedent variables, which is denoted as do ( X = x ) . In this way, P ( Y = y | do ( X = x )) represents the true causal effects of treatment X on outcome Y (Pearl et al., 2016). Backdoor Path When estimating the causal effect of X on Y , the backdoor paths are the noncausal paths between X and Y with an arrow into X , e.g., X  Z  Y . Such paths will confound the effect that X has on Y but not transmit causal influences from X , and therefore introduce spurious correlations between X and Y . Backdoor Criterion The Backdoor Criterion is an important tool for causal intervention. Given an ordered pair of variables ( X, Y ) in SCM, and a set of variables Z where Z contains no descendant of X and blocks every backdoor path between X and Y , then the causal effects of X = x on Y can be calculated by: P ( Y = y | do ( X = x )) = (cid:88) z P ( Y = y | X = x, Z = z ) P ( Z = z ) , (1) where P ( Z = z ) can be estimated from data or priorly given, and is independent of X . Task This paper investigates prompt-based probing on one of the most representative and well-studied tasks  factual knowledge probing (Liu et al., 2021b). For example, to evaluate whether BERT knows the birthplace of Michael Jordan, factual knowledge probing queries BERT with Michael Jordan was born in [MASK], where Michael Jordan is the verbalized subject mention, was born in is the verbalized prompt of relation birthplace , and [MASK] is a placeholder for the target object. Data We use LAMA (Petroni et al., 2019) as our primary dataset, which is a set of knowledge triples sampled from Wikidata. We remove the N-M relations (Elazar et al., 2021) which are unsuitable for the P@1 metric and retain 32 probing relations in the dataset. Please refer to the appendix for detail. Pretrained Models We conduct probing experiments on 4 well-known PLMs: BERT (Devlin et al., 2019), RoBERTa (Liu et al., 2019), GPT-2 (Rad-ford et al., 2019) and BART (Lewis et al., 2020), which correspond to 3 representative PLM architectures, including autoencoder (BERT, RoBERTa), autoregressive (GPT-2) and denoising autoencoder (BART). In this section, we formulate the SCM for factual knowledge probing procedure and describe the key variables and causal relations. The SCM is shown in Figure 2a, which contains 11 key variables: 1) Pretraining corpus distribution D a ; 2) Pretraining corpus C , e.g., Webtext for GPT2, Wikipedia for BERT; 3) Pretrained language model M ; 4) Linguistic distribution L , which guides how a concept is verbalized into natural language expression, e.g., relation to prompt, entity to mention; 5) Relation R , e.g., birthplace , capital , each relation corresponds to a probing task; 6) Verbalized prompt P for each relation , e.g, x was born in y ; 7) Task-specific predictor I , which is a PLM combined with a prompt, e.g., <BERT, was born in > as a birthplace predictor; 8) Probing data distribution D b , e.g., fact distribution in Wikidata; 9) Sampled probing data T such as LAMA, which are sampled entity pairs (e.g., <Q41421, Q18419> in Wikidata) of relation R ; 10) Verbalized instances X , (e.g., <Michael Jordan, Brooklyn> from <Q41421, Q18419>); 11) Performance E of the predictor I on X . PLM Pretraining. The path { D a , L }  C  M represents the pretraining procedure for language model M , which first samples pretraining corpus C according to pretraining corpus distribution D a and linguistic distribution L , then pretrains M on C . Prompt Selection. The path { R, L }  P represents the prompt selection procedure, where each prompt P must exactly express the semantics of relation R , and will be influenced by the linguistic distribution L . generation procedure of verbalized probing instances X , which first samples probing data T of relation R according to data distribution D b , then verbalizes the sampled data T into X according to the linguistic distribution L . Performance Estimation. The path { M, P }  I  E  X represents the performance estimation procedure, where the predictor I is first derived by combining PLM M and prompt P , and then the performance E is estimated by applying predictor I on verbalized instances X . To evaluate PLMs' ability on fact extraction, we need to estimate P ( E | do ( M = m ) , R = r ) . Such true causal effects are represented by the path M  I  E in SCM. Unfortunately, there exist three backdoor paths between pretrained language model M and performance E , as shown in Figure 2b-d. These spurious correlations make the observation correlation between M and E cannot represent the true causal effects of M on E , and will inevitably lead to biased evaluations. In the following, we identify three critical biases in the prompt-based probing evaluation and describe the manifestations, causes, and casual interventions for each bias. In prompt-based probing, the predictor of a specific task (e.g., the knowledge extractor of relation birthplace ) is a PLM M combined with a prompt P (e.g., BERT + was born in ). However, PLMs are pretrained on specific text corpus, therefore will inevitably prefer prompts sharing the same linguistic regularity with their pretraining corpus. Such implicit prompt preference will confound the true causal effects of PLMs on evaluation performance, i.e., the performance will be affected by both the task ability of PLMs and the preference fitness of a prompt. In the following, we investigate prompt preference bias via causal analysis. In factual knowledge probing, we commonly assign one prompt for each relation (e.g., X was born in Y for birthplace ). However, different PLMs may prefer different prompts, and it is unable to disentangle the influence of prompt preference from the final performance. Such invisible 5799 Language Continent Religion Owned by 0 10 20 30 40 50 60 70 80 BERT-large RoBERTa-large GPT2-xl BART-large Figure 3: The variances of P@1 performance of 4 PLMs on 4 relations using semantically equivalent prompts. prompt preference will therefore lead to inconsistent conclusions. To demonstrate this problem, we report the performance variance on LAMA using different prompts for each PLM. For each relation, we follow Elazar et al. (2021); Jiang et al. (2020b) and design at least 5 prompts that are semantically equivalent and faithful but vary in linguistic expressions. Prompt selection significantly affects performance. Figure 3 illustrates the performance on several relations, where the performances of all PLMs vary significantly on semantically equivalent prompts. For instance, by using different prompts, the Precision@1 of relation languages spoken dramatically changing from 3.90% to 65.44% on BERT-large, and from 0.22% to 71.94% on BART-large. This result is shocking, because the same PLM can be assessed from knowing nothing to sufficiently good by only changing its prompt. Table 1 further shows the quantitative results, for BERT-large, the averaged standard deviation of Precision@1 of different prompts is 8.75. And the prompt selection might result in larger performance variation than model selection: on more than 70% of relations, the best and worst prompts will lead to >10 point variation at Precision@1, which is larger than the majority of performance gaps between different models. Prompt preference also leads to inconsistent comparisons. Figure 4 demonstrates an example, where the ranks of PLMs are significantly changed when applying diverse prompts. We also conduct quantitative experiments, which show that the PLMs' ranks on 96.88% relations are unstable when prompt varies. mance. Such inconsistent performance will further lead to unstable comparisons between different PLMs, and therefore significantly undermines the evaluations via prompt-based probing. Figure 2b shows the cause of the prompt preference bias. When evaluating the ability of PLMs on specific tasks, we would like to measure the causal effects of path M  I  E . However, because the prompt P and the PLMM are all correlated to the linguistic distribution L , there is a backdoor path M  C  L  P  I  E between PLMM and performance E . Consequently, the backdoor path will confound the effects of M  I  E with P  I  E . Based on the above analysis, the prompt preference bias can be eliminated by blocking this backdoor path via backdoor adjustment, which requires a prior formulation of the distribution P ( P ) . In Section 7, we will present one possible causal intervention formulation which can lead to more consistent evaluations. Apart from the prompt preference bias, the underlying linguistic correlation can also induce bias in the instance verbalization process. Specifically, an instance in probing data can be verbalized into different natural language expressions (e.g., verbalize 5800 Relation Mention Prediction Capital of America Chicago the U.S. Washington China Beijing Cathay Bangkok Birthplace Einstein Berlin Albert Einstein Vienna Isaac Newton London Sir Isaac Newton town Table 2: Different verbalized names of the same entity lead to different predictions on BERT-large. Q30 in Wikidata into America or the U.S. ), and different PLMs may prefer different verbalizations due to mention coverage, expression preference, etc. This will lead to instance verbalization bias. In factual knowledge probing, each entity is verbalized to its default name. However, different PLMs may prefer different verbalizations, and such underlying correlation is invisible. Because we couldn't measure how this correlation affects probing performance, the evaluation may be unstable using different verbalizations. Table 2 shows some intuitive examples. When we query BERT The capital of the U.S. is [MASK], the answer is Washington . Meanwhile, BERT would predict Chicago if we replace the U.S. to its alias America . Such unstable predictions make us unable to obtain reliable conclusions on whether or to what degree PLMs actually entail the knowledge. To quantify the effect of instance verbalization bias, we collect at most 5 verbalizations for each subject entity in LAMA from Wikidata, and calculate the verbalization stability on each relation, i.e., the percentage of relation instances whose predictions are unchanged when verbalization varies. The results in Figure 5 show the average verbalization stabilities of all four PLMs are < 40%, which demonstrate that the instance verbalization bias will bring unstable and unreliable evaluation. Figure 2c shows the cause of instance verbalization bias: the backdoor path M  C  L  X  E , which stems from the confounder of linguistic distribution L between pretraining corpus C and verbalized probing data X . Consequently, the ob-20 40 60 80 100 BERT RoBERTa GPT2 BART Figure 5: The verbalization stabilities of 4 PLMs on all relations, which is measured by the percentage of relation instances whose predictions are unchanged when verbalization varies. served correlation between M and E couldn't faithfully represent the true causal effect of M on E , but is also mixed up the spurious correlation caused by the backdoor path. The instance verbalization bias can be eliminated by blocking this backdoor path via causal intervention, which requires a distribution formulation of the instance verbalization, i.e., P ( X ) . We will present a possible intervention formulation in Section 7. Besides the biases induced by linguistic correlations, the distributional correlations between pretraining corpus and task-specific probing data can also introduce sample disparity bias. That is, the performance difference between different PLMs may due to the sample disparity of their pretraining corpus, rather than their ability divergence. In conventional evaluation, the evaluated hypotheses are independent of the train/test data generation, and all the hypotheses are evaluated on training data and test data generated from the same distribution. Therefore, the impact of correlations between training data and test data is transparent, controllable, and equal for all the hypotheses. By contrast, in prompt-based probing, each PLM is bundled with a unique pretraining corpus, the correlation between pretraining corpus distribution and probing data distribution cannot be quantified. In the following we investigate this sample disparity bias in detail. is commonly used to compare different PLMs. Previous work claims that GPT-style models are with weaker factual knowledge extraction abilities than BERT because they perform worse on LAMA (Petroni et al., 2019; Liu et al., 2021c). However, because PLMs are pretrained on different pretraining corpus, the performance divergence can stem from the spurious correlation between pretraining corpus and LAMA, rather than their ability difference. For example, BERT's superior performance to GPT-2 may stem from the divergence of their pretraining corpus, where BERT's pretraining corpus contains Wikipedia, while GPT-2's pretraining corpus doesn't. To verify the effect of sample disparity bias, we further pretrain BERT and GPT-2 by constructing pretraining datasets with different correlation degrees to LAMA, and report their new performances on LAMA. Specifically, we use the Wikipedia snippets in LAMA and collect a 99k-sentence dataset, named WIKI-LAMA. Then we create a series of pretraining datasets by mixing the sentences from WIKI-LAMA with WebText 2 (the pretraining corpus of GPT2). That is, we fix all datasets' size to 99k, and a parameter  is used to control the mixture degree: for each dataset, there are  % instances sampled from WIKI-LAMA and 1   % instances sampled from WebText. Please refer to the appendix for pretraining detail. Table 3 demonstrates the effect of sample disparity bias. We can see that 1) Sample disparity significantly influences the PLMs' performance: the larger correlation degree  will result in better performance for both BERT and GPT-2; 2) Sample disparity contributes to the performance difference. We can see that the performance gap between GPT2 and BERT significantly narrows down when they 2 http://Skylion007.github.io/ OpenWebTextCorpus are further pretrained using the same data. Besides, further pretraining BERT on WebText (  =0) would significantly undermine its performance. These results strongly confirm that the sample disparity will significantly bias the probing conclusion. The cause of sample disparity bias may diverge from PLMs and scenarios due to the different causal relation between pretraining corpus distribution D a and probing data distribution D b . Nevertheless, sample disparity bias always exist because the backdoor path will be M  C  D a  D b  T  X  E when D a is the ancestor of D b , or M  C  D a  D b  T  X  E when D a is the descendant of D b . Figure 2d shows a common case when the pretraining corpus distribution D a is an ancestor of probing data distribution D b . For example, the pretraining data contains Wikipedia and probing data is a sampled subset from Wikipedia (e.g., LAMA, X-FACTR, BioLAMA). As a result, there is a backdoor path between M and E , which will mislead the evaluation. This section describes how to eliminate the abovementioned biases by blocking their corresponding backdoor paths. According to the Backdoor Criterion in Section 2.1, we need to choose a set of variables Z that can block every path containing an arrow into M between M and E . Since the linguistic distribution L , pretraining corpus distribution D a and probing data distribution D b are unobservable, we choose Z = { P, X } as the variable set for blocking all backdoor paths between ( M, E ) in the SCM by conducting backdoor adjustment: P ( E | do ( M = m ) , R = r ) = (cid:88) p  P (cid:88) x  XP ( p, x ) P ( E | m, r, p, x ) . (2) Equation 2 provides an intuitive solution. To eliminate the biases stemming from the spurious correlations between pretraining corpus, probing data and prompts, we need to consider the natural distribution of prompts and verbalized probing data regardless of other factors. Consequently, the overall causal effects between PLM and evaluation result are the weighted averaged effects on all valid prompts and probing data. Unfortunately, the exact distribution of P ( x, p ) is intractable , which needs to iterate over all valid prompts and all verbalized probing data. To address this problem, we propose a sampling-based approximation. Specifically, given a specific as-sumption about P ( x, p ) (we assume uniform distribution in this paper without the loss of generality), we sample K p prompts for each relation and K x kinds of verbalization for each instance according to P ( x, p ) , and then these samples are used to estimate the true causal effects between M and E according to Equation 2. To verify whether causal intervention can improve the evaluation consistency and robustness, we conduct backdoor adjustment experiments on 8 different PLMs. We randomly sample 1000 subsets with 20 relations from LAMA, and observe whether the evaluation conclusions were consistent and stable across the 1000 evaluation runtimes. Specifically, we use rank consistency as the evaluation metric, which measures the percentage of the most popular rank of each model in 1000 runtimes. For example, if BERT ranks at 3 rd place in 800 of the 1000 runtimes, then the rank consistency of BERT will be 80% . Table 4 shows the results. We can see that causal intervention can significantly improve the evaluation consistency: 1) The consistency of current prompt-based probing evaluations is very poor on all 8 PLMs: when we randomly select prompts and verbalizations during each sampling, the overall rank consistency is only 5.5%; 2) Causal intervention can significantly improve overall rank consistency: from 5.5% to 68.5%; 3) Casual intervention can consistently improve the rank consistency of different PLMs: the rank of most PLMs is very stable after backdoor adjustment. Prompt-based Probing Prompt-based probing is popular in recent years (Rogers et al., 2020; Liu et al., 2021b) for probing factual knowledge (Petroni et al., 2019; Jiang et al., 2020a; Sung et al., 2021), commonsense knowledge (Davison et al., 2019), semantic knowledge (Ettinger, 2020; Sun et al., 2021; Brown et al., 2020; Schick and Schtze, 2020) and syntactic knowledge (Ettinger, 2020) in PLMs. And a series of prompt-tuning studies consider optimizing prompts on training datasets with better performance but may undermine interpretability (Jiang et al., 2020b; Shin et al., 2020; Haviv et al., 2021; Gao et al., 2021; Qin and Eisner, 2021; Li and Liang, 2021; Zhong et al., 2021). Because such prompt-tuning operations will introduce additional parameters and more unknown correlations, this paper does not take prompt-tuning into our SCM, delegate this to future work. Biases in NLP Evaluations Evaluation is the cornerstone for NLP progress. In recent years, many studies aim to investigate the underlying biases and risks in evaluations. Related studies include investigating inherent bias in current metrics (Cough-lin, 2003; Callison-Burch et al., 2006; Li et al., 2017; Sai et al., 2019, 2020), exploring dataset artifacts in data collection and annotation procedure (Lai and Hockenmaier, 2014; Marelli et al., 2014; Chen et al., 2018; Levy and Dagan, 2016; Schwartz et al., 2017; Cirik et al., 2018; McCoy et al., 2019; Liu et al., 2021a; Branco et al., 2021), and identifying the spurious correlations between data and label which might result in catastrophic out-of-distribution robustness of models (Poliak et al., 2018; Rudinger et al., 2018; Rashkin et al., 2018). Most previous studies demonstrate the evaluation biases empirically, and interpret the underlying reasons intuitively. However, intuitive explanations are also difficult to critical and extend. In contrast, 5803 this paper investigates the biases in prompt-based probing evaluations from a causal view. Based on the causal analysis framework, we can identify, understand, and eliminate biases theoretically, which can be extended and adapted to other evaluation settings in a principled manner 3 . We believe both the causal analysis tools and the valuable insights can benefit future researches. This paper investigates the critical biases and quantifies their risks in the widely used prompt-based probing evaluation, including prompt preference bias, instance verbalization bias, and sample disparity bias. A causal analysis framework is proposed to provide a unified framework for bias identification, interpretation and elimination with a theoretical guarantee. Our studies can promote the understanding of prompt-based probing, remind the risks of current unreliable evaluations, guide the design of unbiased datasets, better probing frameworks, and more reliable evaluations, and push the bias analysis from empirical to theoretical. Another benefit of this paper is to remind the evaluation criteria shifts from conventional machine learning algorithms to pretrained language models. As we demonstrate in Figure 1, in conventional evaluation, the evaluated hypotheses (e.g., algorithms, architectures) are raised independently of the train/test dataset generation, where the impact of correlations between training data and test data is transparent, controllable, and equal for all the hypotheses. However, in evaluations of pretrained language models, the pretraining corpus is bundled with the model architecture. In this case, it is significant to distinguish what you need to assess (architecture, corpus, or both), as well as the potential risks raised by the correlations between pretraining corpus and test data, which most current benchmarks have ignored. Consequently, this paper echoes that it is necessary to rethink the criteria for identifying better pretrained language models, especially under the prompt-based paradigm. In the future, we would like to extend our causal analysis framework to fit prompt-tuning based probing criteria and all PLM-based evaluations. We sincerely thank all anonymous reviewers for their insightful comments and valuable sugges-3 tions. This research work is supported by the National Natural Science Foundation of China under Grants no. 62122077, the Strategic Priority Research Program of Chinese Academy of Sciences under Grant No. XDA27020200, and the National Natural Science Foundation of China under Grants no. 62106251 and 62076233. This paper has no particular ethic consideration.
abstain abstain abstain objective method objective abstain abstain method other abstain other abstain abstain abstain abstain abstain result abstain abstain method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain method result result objective objective objective objective objective method method objective objective objective objective abstain abstain abstain abstain abstain abstain abstain abstain abstain other method abstain method method abstain method method other abstain abstain abstain abstain abstain abstain abstain abstain method abstain abstain abstain method abstain abstain abstain abstain abstain abstain abstain abstain abstain method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain method abstain abstain abstain method abstain other abstain abstain abstain method abstain abstain method abstain method abstain abstain abstain abstain abstain method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain method method method abstain abstain abstain other abstain abstain abstain abstain abstain abstain abstain abstain method method abstain method abstain abstain abstain method abstain abstain method abstain abstain other other other method other other other other other objective objective method objective abstain abstain abstain objective abstain abstain objective objective other other other other abstain