Named entity recognition (NER) is a fundamental task to recognize specific types of entities from a given sentence. Depending on how the entities appear in the sentence, it can be divided into three subtasks, namely, Flat NER, Nested NER, and Discontinuous NER. Among the existing approaches, only the generative model can be uniformly adapted to these three subtasks. However, when the generative model is applied to NER, its optimization objective is not consistent with the task, which makes the model vulnerable to the incorrect biases. In this paper, we analyze the incorrect biases in the generation process from a causality perspective and attribute them to two confounders: pre-context confounder and entity-order confounder. Furthermore, we design Intraand Inter-entity Deconfounding Data Augmentation methods to eliminate the above confounders according to the theory of backdoor adjustment. Experiments show that our method can improve the performance of the generative NER model in various datasets. Named entity recognition (NER) is a task aimed at identifying distinct and independent entities from a given text while classifying them into predefined types. As a fundamental work in Natural Language Processing (NLP), its research facilitates the application of many downstream tasks (Ganea and Hofmann, 2017; Miwa and Bansal, 2016; Shen et al., 2021b). In previous work (Sang and Meul-der, 2003; Pradhan et al., 2013a; Doddington et al., 2004; Kim et al., 2003; Karimi et al., 2015), three kinds of different NER subtasks were raised (as shown in Figure 1), which are Flat NER, Nested NER and Discontinuos NER. (Ju et al., 2018; Strakov et al., 2019), span-based (Luan et al., 2019a; Shen et al., 2021a) and generative-based (Strakov et al., 2019; Paolini et al., 2021; Yan et al., 2021a) methods. Nongenerative methods have different problems when applied to all three different subtasks: the labeling-based methods need to design different tagging schema for various types (Ratinov and Roth, 2009; Metke-Jimenez and Karimi, 2016; Strakov et al., 2019; Dai et al., 2020) while the span-based methods suffers from ambiguity of boundary when applied to discontinuous task. Although generative-based methods are able to model all NER subtasks uniformly (Yan et al., 2021a), the training objective differ significantly from NER task due to the autoregressive generation mannner, resulting in some incorrect biases learned by the model during the training process. From a causal perspective, the incorrect biases stem from two confounders: pre-context confounder and entity-order confounder. Pre-context confounder means that the model is affected by pre-context words that may be extra-entity words when generating a particular entity word. For example, in S3 of Figure 1, the autoregressive generation mannner causes the model to generate the word "fa-tigue" of the entity "muscle fatigue" conditioned on the extra-entity words "muscle" and "pain" . This causes the model to mistakenly establish dependen-808 Figure 2: Structural Causal Model of the generative NER method. cies between the intra-entity word "fatigue" and the extra-entity words "muscle" and "pain" , while ignoring the dependency between the intra-entity words "muscle" and "fatigue" . Therefore, when only the entity "muscle fatigue" is in the input sentence, the model cannot predict the entity accurately and completely due to the learned incorrect dependency bias. Entity-order confounder refers to the fact that the model is affected by a predetermined order of entities when generating an entity sequence. The entities in a sentence are essentially a set structure without decoding order among them. In contrast, the generative NER model pre-specifies the decoding order of entities, which introduces incorrect bias and ignores the bidirectional dependency between entities. As in S1 of Figure 1, after fixing the set of entities as "Stallone"  "Rocky"  "Rambo" , the model only models the unidirectional dependency of "Rambo" on "Stallone" and "Rocky" , without considering the reverse dependency of "Stallone" on "Rocky" and "Rambo" . In this case, if "Rambo" is decoded first, it is difficult for the model to decode the other two entities "Stal-lone" and "Rocky" due to the lack of the reverse dependency. We can formulate the causalities in the process of entity sequence generation with a Structural Causal Model (SCM). As illustrated in Figure 2, the direct links denote the causality between the two nodes: cause  effect. X  Y represents the generation process of the target sequence, which can be divided into two cases according to the location of the generated words: intra-entity generation and inter-entity generation. In the former case, N denotes the pre-context words, which can affect the generation of the next word ( N  Y ). While in the latter case, N denotes the entity decoding order, and can affect the generation of the next entity ( N  Y ). In both cases, the representation of input X is contaminated by the backdoor path X  N  Y . Therefore, N is a confounder for the X  Y process which introduces a incorrect bias to the model. In order to eliminate the bias caused by confounders N in both cases, we designed the Intra-and Inter-entity Deconfounding Data Augmentation method from the theory of backdoor adjustment. Our contributions are as follows:  We analyzed the incorrect bias of the generative model on the NER task from a causal perspective, concluding that the pre-context confounder and the entity-order confounder are the main causes of the bias. Based on the backdoor adjustment theory, we designed the Intraand Inter-entity Deconfounding Data Augmentation methods to remove the pre-context confounder and the entity-order confounder, respectively, to eliminate the incorrect bias of the generative model on the NER task. Experiments on three kinds of NER tasks show that our proposed method can de-bias the generative NER model and thus improve the model performance. For subsequent analysis, in this section we first illustrate how the NER task is modeled as a generative task, after which we illustrate the training and inference process of the generative model. The three kinds of NER tasks can all be formulated as follows, given an input sentence of l tokens x = { x 1 , x 2 , ..., x l } , the target sequence y = { [ ss ] , E 1 ,    , EM , [ ee ] } , where E i = { [ s ] , y e i 1 , ..., y e i E , [ e ] } is a word sequence of entity e i , M denotes the number of the entities, E denotes the length of the entity, [ ss ] and [ ee ] are the start and end tags for the sequence, [ s ] and [ e ] are the start and end tags for the entity, and y e ij is the j -th word of i -th target entity. In general, given an input sentence x , the generative model will return a sequence consisting of a collection of entities arranged in fixed order y = { [ ss ] , E 1 ,    , EM , [ ee ] } . To this end, 809 we first computes the hidden vector representation H = h 1 , ..., h l of the input via a multi-layer transformer encoder: H = Encoder ( x 1 , ..., x l ) (1) where each layer of Encoder(  ) is a transformer block with multi-head attention mechanism. After the input sentence is encoded, the decoder predicts the output token-by-token according to the sequential inputs' hidden vectors. At the step i of generation, the self-attention decoder predicts the i th token y i in the linearized form and decoder state h di as: y i , h di = Decoder ([ H ; h d 1 , ..., h di  1 ] , y i  1 ) (2) where each layer of Decoder(  ) is a transformer block that contains self-attention with decoder hidden state h d<i and cross-attention with encoder state H . Specifically, the optimization objective of the generated model is to maximize the conditional probability of the entire output sequence p ( y | x ) , which is progressively combined by the probability of each step p ( y i | y <i , x ) : p ( y | x ) = | y | (cid:89) i p ( y i | y <i , x ) (3) 3 The Proposed Solution In the above, we have analyzed that the bias in the traditional generative NER model P ( Y | X ) is introduced by two kinds of confounders: the pre-context confounder and the entity-order confounder. Now we need to perform the deconfounding using backdoor adjustment to obtain a debi-ased model P ( Y | do ( X )) . Deconfounding seeks the true causal effect of one variable on another, and it is appealing to the objective of NER: given a sentence X , we hope Y extracted by the model being faithful only to the content of the input X itself. And the backdoor adjustment promotes the posterior probability P ( Y | do ( X )) from passive observation to active intervention as shown below: P ( Y | do ( X )) = (cid:88) n P ( Y | X, n ) P ( n ) (4) where n is the stratum for the confounder N . every stratum n , only subject to a prior P ( n ) listening to no one, and hence the model is decon-founded. In the next sections, we apply Equation 4 to design two data augmentation (DA) methods, Intra-entity Deconfounding DA and Inter-entity Deconfounding DA, for the pre-context confounder and the entity-order confounder, respectively. We first focus on the generation of words inside the entity. The autoregressive decoder needs to decode the word at the current step conditioned on the pre-context words, i.e., the already generated word sequence. The pre-context words may be in other entities that are not associated with the entity currently being generated. Thus it will learn the wrong dependencies and bring in bias to the model. From the SCM in Figure 2, the pre-context words are the confounder in the generation of words inside the entity, causing the spurious correlation X  N  Y to mislead the model from the true objective X  Y . Next we implement Intra-entity Deconfounding by data augmentation to eliminate pre-context confounder. As the backdoor adjustment shown in Equation 4, we stratify the confounder N , pre-context words, and train the model on each stratum. To avoid the influence of other entity words, we split the target sequences of the samples by entity and construct separate target sequences for each entity. Specifically, we randomly sample a context word [ CW ] of an entity e i from X and concatenate it in front of the entity as a target sequence Y  , denoted as: { [ CW ] , y e i 1 , y e i 2 ,    , y e i E } where E denotes the length of the entity e i . If there are M entities in a sentence X , we can construct M augmented samples ( X, Y  ) . It is worth noting that, compared to the target sequence Y of the original sample, the target sequence in the augmented sample does not contain tags denoting the beginning and end of the sequence, i.e., [ ss ] and [ ee ] . This is to tell the model to generate only a single entity on the augmented sample instead of all the entities, as a way to prevent the model trained by the augmented samples from exiting early in the practical prediction. Another generation case is that after the current entity is generated, the model is expected to generate the first word of the next entity. In traditional generative NER models (Paolini et al., 2021; Yan et al., 2021a), the target sequence is fixed in the order of entities, for example, Yan et al. (2021a) prespecified entity order according to the occurrence. However, entities are essentially set structures and the decoding sequence is not supposed to be fixed. A pre-specified entity order can make the optimization target inconsistent with the task and introduce an incorrect bias to the model. As shown in the SCM of Figure 2, entity order is the confounder N who affects the generation X  Y through the backdoor path X  N  Y . According to Equation 4, we design an Inter-entity Deconfounding data augmentation to eliminate entity-order confounder. Similar to Section 3.1, we construct augmented samples by sampling from all possible entity orders. Specifically, for the original sample (X,Y), we keep the last entity of its target sequence fixed and permute the order of the other entities. The target sequence Y  of the augmented sample can be represented as: { [ ss ] , Perm ( E 1 ,    , EM  1 ) , EM , [ ee ] } where Perm(  ) represents the permutation operation. During the training, we only compute the loss for the first token of the last entity, while the other entities are fed directly to the decoder as decoded sequences. As the model uses a token-by-token approach for prediction, in order to reduce the search space and the impact of exposure bias, we restrict the model to generating only tokens from the original sentence at generation time, and control the entire generation process by limiting tokens that can be generated at each step. Specifically, we add special start and end tokens for the generation of each entity and the generation of the whole sequence. At the time of prediction, the generation of the sequence must start from the sequence start token, and the generation of the entity must start from the entity start token, and when the end token of the entity is generated, the next token that could be generated can only be the sequence end token and entity start token. Also, when generating each entity, we restrict the category of the entity to be generated only after the entity is generated, and the category can only be followed by the [e]. In this section, we first describe the dataset we used, then we present related implementation details and experimental results, after which we make an analysis based on the experimental results. As same as (Yan et al., 2021b), to show that our proposed method can be used in various NER subtasks, we conducted experiments on eight datasets. We selected the CoNLL2003 (Sang and Meul-der, 2003) and OntoNotes (Pradhan et al., 2013b) datasets to do the experiments of Flat NER subtask. For CoNLL2003, we follow (Lample et al., 2016; Yu et al., 2020) to train our model on the concatenation of the train and development sets. For OntoNotes, we use the same train, development and test splits as (Pradhan et al., 2012; Yu et al., 2020). For Nested NER subtask, we adopt ACE2004 (Dod-dington et al., 2004), ACE2005 and Genia datasets (Kim et al., 2003). In experiment conducted on ACE2004 and ACE2005, we use the same data split as (Lu and Roth, 2015; Muis and Lu, 2017; Yu et al., 2020), the ratio between train, development and test is 8:1:1. For Genia, we follow (Wang et al., 2020b; Shibuya and Hovy, 2020) to use five types of entities and split the train, development and test as 8.1:0.9:1.0. We follow (Dai et al., 2020) to use CADEC (Karimi et al., 2015), ShARe13 (Pradhan et al., 2013a) and ShARe14 (Mowery et al., 2014) datasets to do our experiment. Since only the Adverse Drug Events (ADEs) entities include discontinuous annotation, only this kind of entity is considered. (Karimi et al., 2015; Metke-Jimenez and Karimi, 2016; Tang et al., 2018). Because of the use of special tokens, we use the pre-trained language model T5 (Raffel et al., 2020) as our encoder-decoder generative architecture. The 811 Model CoNLL2003 OntoNotes Prec. T5 pre-trained model provides 100 default sentinel tokens for unsupervised training, here we use these special tokens to control the sequence generation process for avoiding the occupation of real tokens in the word list. Specifically, we use <extra_id_2> and <extra_id_3> to represent [ s ] and [ e ] , <ex-tra_id_0> and <extra_id_1> to represent [ ss ] and [ ee ] , <extra_id_11> to <extra_id_30> to represent 812 different NER categories, and <extra_id_50> to mark the sample of inter-entity deconfounding samples. In addition, we use the AdamW (Loshchilov and Hutter, 2019) optimizer with a linear learning rate schedule (with peak learning rate of 1e-4). For simplicity, we assume that entities are unique, and for words with referential relations, such as "we" , which appears frequently in ACE2005, we tag each "we" with a different label in a sentence such as "we_1" , "we_2" , ... to distinguish them from each other. For simplicity of comparison, we use the results reproduced by (Yan et al., 2021b) on the dataset with different subtasks. Moreover, since we conducted the experiments on the subtoken-level, we only kept the experimental results of BPE in (Yan et al., 2021b). As can be seen from Tables 1 to 3, our model achieves similar or even better results on all three subtasks than the model in (Yan et al., 2021b). This may be caused by the fact that we use a different pre-trained model and not use pointer mechanism. Compared with other non-generative models, same as (Yan et al., 2021b), our method achieves comparable results with models focusing on only one subtask of NER on most of datasets, for exceptional cases, (Akbik et al., 2019) in Table 1 tags tokens at token-level; (Wang et al., 2020a) in Table 2 classifies candidate span, which integrates information of all subtokens in span, and is based on span-level; while our model only focuses on subtoken, which is based on subtoken-level. In comparing the results of Without-De and Intra-De in Table 1-3, we can see that when intra-entity deconfounding are performed, the model has different degrees of improvement in all datasets. It is worth noting that the selection method we used to do augmentation differs slightly from dataset to dataset. Specifically, in each dataset we select entities considering on occurrence frequency, nesting status and character length of entity, in particular, we kick out some special entities that have referential relationships with others. In comparing the results of Without-De and Inter-De in Table 1-3, we can see that when inter-entity deconfounding are performed, the model also have different degrees of improvement in all datasets. Here, it is worth noting that when selecting the sample for inter-entity deconfounding, we select samples based on factors with which the order confounder is most likely to have impact, such as the minimum order of last entity in the whole training dataset and whether it is easy to perform permutation such as the number of target entities. Besides, we have not select all samples for augmentation, and the results in Table 1-3 may not be the best. To verify the effectiveness of the two data augmentation methods we designed for de-confounding, we conducted robustness testing experiments on CoNLL03, ACE04 and CADEC, respectively. The pre-context confounder introduce error bias into the model by incorrectly relying on prefix sequences during entity sequence generation in the training phase. To verify the effectiveness of our Intra-entity Deconfounding Data Augmentation method in eliminating the pre-context confounder, we designed robustness testing experiments. In decoding, we randomly sample several words as pre-context sequences, and then require the model to continue decoding the entities. The experimental results are shown in Table 4. We can observe that the performance of both the baseline model and the Intra-entity Deconfounding model have different degrees of degradation after the attack of random fixed pre-context. However, the relative performance degradation of the Intra-entity Deconfounding model is less, and the  F1 on ACE04, CADEC and CoNLL are improved by+1.44%, +1.21% and +0.43% relative to the baseline model. This indicates that after Intra-entity Deconfounding Data Augmentation, the model can eliminate the pre-context confounder to some extent. We also verify the robustness of the Inter-entity Deconfounding Data Augmentation method against the entity-order confounder. We first randomly sample k entities as the prefix of the decoding sequence, and then let the model continue to generate entities. For convenience, we choose a sample of the test set with the number of entities greater than k for evaluation, and we do not consider the k randomly sampled correct entities in our evaluation. In our experiments, k = 4 . From Table 5, we can observe that the performance of both models decreases after the attack of random entity order. However, after deconfounding the entity sequences by the Inter-entity Deconfounding Data Augmentation method, the model degradation is reduced, and the  F1 on ACE04, CADEC, and CoNLL are improved by +0.49%, +0.71%, and +0.19% relative to the baseline model. This indicates that the Inter-entity Deconfounding Data Augmentation method we designed can enhance the robustness of the model to cope with random entity order when generating entity sequences, i.e., the entity-order confounder are eliminated to some extent. The existing models can be basically divided into sequence labeling formulation, span-based formulation and generative-based formulation. Among them, the sequence labeling formulation was earlier applied to solve the NER problem (McCal-lum and Li, 2003; Collobert et al., 2011; Huang et al., 2015; Chiu and Nichols, 2016; Lample et al., 2016; Strakov et al., 2019; Yan et al., 2019; Li et al., 2020a). After Nested NER and Discontinuous NER were discovered and raised, inspired by the successful application of sequence labeling formulation on Flat NER subtask, Metke-Jimenez and Karimi (2016); Muis and Lu (2017) attempted to extend this approach to the new subtasks. Others chose a different path, based on the characteristics of Nested NER, Xu et al. (2017); Wang and Lu (2019); Yu et al. (2020) try to traverse all possible spans and do classification at the span-level. Shen et al. (2021a) try to reduce the number of candidate spans and Tan et al. (2021) make the left and right boundaries of the candidate spans completely unfastened. In addition, in order to apply span-based formulation to the Discontinuous NER, the concept of hypergraph was introduced to efficiently represent spans (Lu and Roth, 2015; Katiyar and Cardie, 2018; Muis and Lu, 2016). Although sequence labeling formulation and span-based formulation can be applied to different subtasks separately, these formulations are difficult to be applied to them simultaneously. Among them, sequence labelling formulation needs to design different tagging schema for different NER subtasks (Ratinov and Roth, 2009; Metke-Jimenez and Karimi, 2016; Strakov et al., 2019; Dai et al., 2020), while span-based formulation needs to sacrifice a certain degree of performance. For example, span-based methods need to set a maximum span length to avoid the number of candidate spans to be traversed (Xu et al., 2017; Luan et al., 2019b; Wang and Lu, 2018), since it is impossible to enumerate all possible spans, which is quadratic to the length of the sentence and fragment numbers of discontinuous entity. Contrary to sequence-labeling and span-based formulation, generative-based formulation can be used to model these subtasks in a unified manner because it can generate variable-length sequences (Yan et al., 2021b). However, since the generative model uses autoregressive generation, its optimiza-814 tion objective differs significantly from the extraction objective of the NER task, which results in the model being influenced by some confounders and thus reduces the performance of model. Causal inference is a science that studies the relationship between correlation and causality. It is not only an explanatory framework, but also a way to provide solutions to achieve desired goals by pursuing causal effects (Pearl et al., 2016; Fenton et al., 2020). So far, it has been achieved greatly success in various domains such as psychology, politics and epidemiology for years (Mackinnon et al., 2007; Luke, 2015; Alves et al., 2014). Recently, causal inference has also attracted increasing attention in nature language process for improving model's performance in various ways. For example, Gardner et al. (2020) constructs counterfactual samples by manually rewriting the rules, and Garg et al. (2019) frames counterfactual samples by heuristically replace some keywords. Compared to them, our method offers a fundamental way to remove the confounder in training phase for generative models which is applied to various tasks of essentially non-sequential problem. In this paper, we analyze two kinds of confounder that generative models arised when applied to NER and use backdoor adjustment methods in causal inference to perform deconfounding. Specifically, for pre-context confounder and entity-order confounder, we respectively design Intra-entity and Inter-entity De-confounding Data Augmentation methods. Experiments show that the performance of the model improves on all datasets after deconfounding. In the future, we will continue to explore the application of causal inference to other tasks. This work is supported by the Key Research and Development Program of Zhejiang Province, China (No. 2021C01013), the Chinese Knowledge Center of Engineering Science and Technology (CKCEST) and MOE Engineering Research Center of Digital Library.
abstain abstain abstain abstain method method result abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain method abstain abstain abstain abstain abstain abstain method objective method objective abstain other abstain abstain abstain other other abstain abstain abstain abstain method abstain abstain abstain abstain abstain method abstain method method abstain abstain abstain abstain abstain abstain abstain abstain method method abstain abstain abstain method method abstain method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain result result result abstain result result abstain abstain result abstain abstain result result abstain abstain result abstain abstain result abstain abstain result abstain abstain abstain result abstain abstain other other other other other other other other other other other other other other other other method method method abstain objective other