Information extraction suffers from its varying targets, heterogeneous structures, and demand-specific schemas. In this paper, we propose a unified text-to-structure generation framework, namely UIE, which can universally model different IE tasks, adaptively generate targeted structures, and collaboratively learn general IE abilities from different knowledge sources. Specifically, UIE uniformly encodes different extraction structures via a structured extraction language, adaptively generates target extractions via a schema-based prompt mechanism  structural schema instructor, and captures the common IE abilities via a large-scale pre-trained text-to-structure model. Experiments show that UIE achieved the state-of-the-art performance on 4 IE tasks, 13 datasets, and on all supervised, low-resource, and few-shot settings for a wide range of entity, relation, event and sentiment extraction tasks and their unifi-cation. These results verified the effectiveness, universality, and transferability of UIE 1 . Information extraction (IE) aims to identify and structure user-specified information from unstructured texts (Andersen et al., 1992; Grishman, 2019). IE tasks are highly diversified due to its varying targets (entity, relation, event, sentiment, etc.), heterogeneous structures (spans, triplets, records, etc.), and demand-specific schemas (Grishman and Sund-heim, 1996; Mitchell et al., 2005; Ji and Grishman, 2011). edge sources for different IE task. These task-specialized solutions greatly hinder the rapid architecture development, effective knowledge sharing, and quick cross-domain adaptation of IE systems. First, it is very complicated to develop dedicated architectures for a large amount of IE tasks/settings/scenarios. Second, learning isolated models severely restricts the knowledge sharing between related tasks and settings. Finally, it is costly and time-consuming to construct data sets and knowledge sources specialized for different IE tasks. Therefore, it will be of great benefit to develop a universal IE architecture that can uniformly model different IE tasks, adaptively predict heterogeneous structures and effectively learn from various resources, which we referred to as Universal IE . Fundamentally, all IE tasks can be modeled as text-to-structure transformations, with different 5755 tasks correspond to different structures. For example, as shown in Figure 1, an entity is a named span structure, an event is a schema-defined record structure. These text-to-structure transformations in IE can be further decomposed into several atomic transformation operations: 1) Spotting , which locates the desirable spans concerning to given specific semantic types (Kripke and Munitz, 1971; Chen and Yuille, 2004). For example, locating span Steve as a Person entity and locating ex-cited as a sentiment expression. 2) Associating , which connects spans by assigning them with semantic roles in pre-defined schemas (Onyshkevych, 1994; Milward and Thomas, 2000). For example, associating Steve and Apple by assigning them as the Arg1 and the Arg2 of a Work-for relation. In this way, different IE tasks can be decomposed into a sequence of atomic text-to-structure transformations, and all IE models share the same underlying spotting and associating abilities. For example, entity extraction can be viewed as spotting mention spans of corresponding entity types, while event detection can be reformulated as spotting triggers spans with event types. And the spotting abilities can be shared between these two tasks. Based on the above observations, we propose UIE, a unified text-to-structure generation architecture that can universally model different IE tasks, adaptively generate targeted structures, and collaboratively learn general IE abilities from different knowledge sources. Specifically, to model heterogeneous IE structures, we design a structural extraction language (SEL) that can effectively encode different IE structures into a uniform representation, so that various IE tasks can be universally modeled in the same text-to-structure generation framework. To adaptively generate targeted structures for different IE tasks, we propose structural schema instructor (SSI), a schema-based prompt mechanism which controls what to spot, what to associate, and what to generate in UIE. To learn common IE abilities for UIE, we pre-train UIE on large-scale, heterogeneous datasets mined from easily accessible web sources. The large-scale pre-trained UIE model provides a solid foundation for knowledge sharing and quick adaptation to new IE settings, and significantly boosts the IE performance in all supervised, low-resource, and few-shot settings. extraction and their unification), and supervised, low-resource, and few-shot settings. Experiment results show that UIE achieves significant improvements in all settings. On supervised settings, UIE achieved 1.42% F1 scores improvements over the state-of-the-art, task-specialized architectures on all datasets. On few-shot and low-resource settings, UIE exhibits strong on-demand adaptation ability: it outperforms baselines dramatically by a large margin. These results verified the effectiveness, universality, and transferability of UIE across different IE tasks, settings, and scenarios. The main contributions of this paper are: 1) We propose UIE, a unified text-to-structure generation architecture that can universally model different IE tasks, adaptively generate targeted structures, and collaboratively learn general IE abilities from different knowledge sources. 2) We design a unified structure generation network, which encodes heterogeneous IE structures into a uniform representation via a structural extraction language, and controls the UIE model which to spot, which to associate, and which to generate via structural schema instructor mechanism. 3) We pre-train a large-scale text-to-structure generation model via a unified pre-training algorithm. To the best of our knowledge, this is the first text-to-structure pre-trained extraction model, which can benefit future IE studies. Information extraction tasks can be formulated as text-to-structure problems, where different IE tasks correspond to different structures. This paper aims to uniformly model the text-to-structure transformations of different IE tasks via a single framework, i.e., different structure transformations will share the same underlying operations and different transformation abilities in a universal model. Formally, given a specific pre-defined schema s and texts x , a universal IE model needs to generate a structure that contains the desirable structural information in the text x indicated by the schema s . Generally, there are two main challenges here. Firstly, due to the diversity of IE tasks, there are many different target structures to extract, e.g., entity, relation, event, etc. Secondly, IE tasks are often demand-specific which are defined using different schemas, therefore we need to adaptively control the extraction process. (person: Steve (work for: Apple) )(start-position: became (employee: Steve) (employer: Apple) (time: 1997) )(organization: Apple) (time: 1997) ) (b) The SEL representation of the extraction structure of Steve became CEO of Apple in 1997., where the relation structure is marked blue, the event structure is marked red, and the rest are entities. In this section, we describe how to jointly formulate, learn, and conduct various IE tasks in a unified text-to-structure generation architecture, named UIE . Specifically, we first design structured extraction language (SEL) to uniformly encode heterogeneous extraction structures, i.e., encode entity, relation, event into a unified representation. Then we describe structural schema instructor (SSI), a schema-based prompt mechanism that controls the UIE model which to spot, which to associate, and which to generate for different extraction settings. The details are as follows. This section describes how to encode heterogeneous IE structures into a uniform representation. Based on the above discussions, IE structure generation can be decomposed into two atomic operations: 1. Spotting indicates locating target information pieces from the sentence, e.g., the entity and the trigger word in the event. 2. Associating indicates connecting different information pieces based on the desirable associations, e.g., the relation between entity pair or the role between event and its argument. Then different IE structures can be represented as a combination of atomic structure generation operations. extraction language (SEL), which encodes different IE structures via the spotting-associating structure. As shown in Figure 2a, each SEL expression contains three types of semantic units: 1) SPOTNAME represents there is a specific information piece with the type of spot name existing in the source text; 2) ASSONAME indicates there exists a specific information piece in the source text that is with the AssoName association to its upper-level Spotted information in the structure; 3) INFOSPAN represents the text span corresponding to the specific spotting or associating information piece in the source text. Furthermore, : in the SEL indicates the mapping from InfoSpan to its spotting or associating names, and the two structure indicators ( and ) are used to form the hierarchical structure between the extracted information. Using SEL, Figure 2b shows how to represent entity, relation, and event structures. There are three entities and each entity is represented as a spotting structure such as person:Steve, organiza-tion:Apple, and time:1997; one relation which is represented as an association structure between Steve and Apple with association name work for; and one event which is represented as an association structure, where the trigger is a spotting structure start-position:became, and its arguments are associated with the trigger: Steve as employee, Apple as employer, 1997 as time. We can see that, SEL have the advantages that: 1) uniformly encodes varying IE structures, therefore different IE tasks can be modeled as the same text-to-structure generation process; 2) efficiently represents all extraction results of a sentence in the same structure, thus can perform joint extraction naturally; 3) the output structure of generation is very compact, which greatly reduce the complexity of decoding. For example, the two different tasks entity recognition and event detection can be revisited using the same (SpotName: InfoSpan) grammar. While both relation extraction and event extraction can be formulated using the grammar (SpotName: InfoSpan (AssoName: InfoSpan), ...), even they are with totally different binary entity-relation-entity and N-ary event-arguments structures. Such a unified structured extraction language enables UIE to learn from and adapt to different IE tasks without designing task-specialized architectures, because these IE tasks are all universally formulated as the transformation from texts to SEL representations. Using SEL, UIE can uniformly generate different IE structures. However, because different IE tasks have different schemas, one challenge here is how to adaptively control which information we want to generate during extraction. For example, given a sentence Steve became CEO of Apple in 1997., an entity recognition system will generate ((person: Steve) (organization: Apple) (Time: 1997)), and an event extraction system will generate (start position: became (employee: Steve) (employer: Apple)). To this end, we propose structural schema instructor (SSI), a schema-based prompt mechanism that controls which kinds of information need to be spotted and associated. Figure 3 shows the overall framework of UIE. Formally, UIE takes the given structural schema instructor ( s ) and the text sequence ( x ) as input, and generates the linearized SEL ( y ) which contains the extracted information from x based on schema s : y = UIE ( s  x ) (1) where x = [ x 1 , ..., x | x | ] is the text sequence, s = [ s 1 , ..., s | s | ] is the structural schema instructor, and y = [ y 1 , ..., y | y | ] is a SEL sequence that can be easily converted into the extracted information record. To describe the extraction target of a task, the structural schema instructor constructs a schema-based prompt and uses it as a prefix during generation. Specifically, corresponding to the spotting-association structure, the structural schema instructor contains three types of token segments: 1) SPOTNAME : the targeted spotting name in the specific information extraction task, such as person in the NER task; 2) ASSONAME : the targeted association name, such as work for in the relation extraction task; 3) Special Symbols ([spot], [asso], [text]) which are added before each SPOTNAME , ASSONAME , and input text sequence. All tokens in SSI are concatenated and put before the original text sequences. As shown in Figure 3, the entire input for UIE is in the form of: s  x = [ s 1 , s 2 , ..., s | s | , x 1 , x 2 , ..., x | x | ] = [ [spot] , ... [spot] ..., [asso] , ..., [asso] ..., [text] , x 1 , x 2 , ..., x | x | ] (2) For example, the SSI [spot] person [spot] company [asso] work for [text] indicates extracting records of the relation schema the person works for the company from the sentence. Given the SSI s , UIE first encodes the text x , then generates the target record y in linearized SEL using an encoder-decoder-style architecture. We found that the schema-based prompt can: 1) effectively guide the SEL generation of UIE, so that the general IE ability can be transferred to new IE tasks; 2) adaptively control which to spot, which to associate, and which to generate, so that semantic knowledge across different labels and tasks can be better shared. Given SSI s and text x as input, UIE extracts targeted information by generating a linearized SEL. We formulate this text-to-SEL generation process using an encoder-decoder-style architecture. Given the raw text sequence x and the schema instructor s , UIE first compute the hidden representation H = [ s 1 , ..., s | s | , x 1 , ..., x | x | ] of each token: H = Encoder ( s 1 , ..., s | s | , x 1 , ..., x | x | ) (3) where Encoder (  ) is a Transformer encoder. Then UIE will decode the input text into a linearized SEL in an auto-regressive style. At the step i of decoding, UIE generates the i -th token y i in the SEL 5758 sequence and the decoder state h di as following: y i , h di = Decoder ([ H ; h d 1 , ..., h di  1 ]) (4) Decoder (  ) is a transformer decoder, which predicts the conditional probability p ( y i | y <i , x, s ) of token y i . Finally, Decoder (  ) finishes prediction when outputting the end symbol <eos>, then we convert the predicted SEL expression into the extracted information record. Compared with previous IE studies which treat labels as specific symbols, the text-to-structure generation paradigm treats labels as natural language tokens. By verbalizing and generating labels and structures, our method can effectively transfer knowledge from pre-trained language models such as BART (Lewis et al., 2020), T5 (Raffel et al., 2020), and related tasks can easily share knowledge because their labels have similar semantics (e.g., location and place ) and share common label-text associations (e.g., victim for different event types). In this section, we describe: 1) how to pre-train a large-scale UIE model which captures common IE abilities for different IE tasks; 2) how to adapt UIE to different IE tasks in different settings via quick fine-tuning. Specifically, we first collect several large-scale datasets from the Web, including structured (e.g., knowledge bases), unstructured (e.g., raw texts), and parallel (e.g., Wikipedia-Wikidata links) data, then we uniformly pre-train our UIE model on these heterogeneous datasets. Finally, we adapt the pre-trained UIE model to the specific downstream IE tasks via on-demand fine-tuning. We found that the pre-trained UIE model provides a solid foundation for capturing, sharing, and transferring knowledge between different IE tasks, and new IE tasks can be effectively solved because UIE learns general IE ability. UIE needs to encode the text, map text to structure, and decode valid structure. Therefore, we collect a large-scale pre-training corpus from easily accessible web data sources (more details are in the appendix): D pair is the text-structure parallel data, where each instance is a parallel pair (token sequence x , structured record y ). We collect large-scale parallel text-structure pairs by aligning Wikidata with English Wikipedia. text-to-structure transformation ability of UIE. D record is the structure dataset where each instance is structured record y . We collect structured records from ConceptNet (Speer et al., 2017) and Wikidata. D record is used to pre-train the structure decoding ability of UIE. Text-to-Structure Pre-training using D pair . To capture the fundamental text-to-structure mapping ability, we pre-train UIE using D pair = { ( x, y ) } . Specifically, for each parallel pair ( x , y ), we extract the spot type s s + and the associating type s a + in the record y as the positive schema s + = s s+  s a+ . However, we found that if we only feed UIE with a positive schema, it will only simply remember the triplet in the pre-training data. To learn general mapping ability, we also automatically construct negative schemas for each pair, i.e., we first sample negative spots s sand negative association set s a, then concatenate meta-schema s meta = s +  s s s a, and construct the final extraction target. For example, person and work for is the positive schema in the record ((person: Steve (work for: Apple))), and we sample vehicle and located in as the negative schema to construct meta-schema. Finally, the objective of text-to-structure pre-training is: L Pair = (cid:88) ( x,y ) D pair  log p ( y | x, s meta ;  e ,  d ) (5) where  e and  d are the parameter of encoder and decoder, respectively. Structure Generation Pre-training with D record . To pre-train the ability of generating valid structures defined by SEL and schemas, we pre-train UIE on D record . We pre-train UIE decoder as an structured language model, where each record in D record is an expression of SEL: L Record = (cid:88) y D record  log p ( y i | y <i ;  d ) (6) By pre-training for structure generation, the decoder can capture the regularity of SEL and the interactions between different labels. Retrofitting Semantic Representation using D text . During text-to-structure pre-training, we continually pre-train UIE also with the masked language model tasks (Raffel et al., 2020) on D text to retrofit semantic representations of UIE. Specifi-cally, we add span corruption based mask language modeling objective in the pre-training stage: L Text = (cid:88) x D text  log p ( x (cid:48)(cid:48) | x (cid:48) ;  e ,  d ) (7) where x (cid:48) is the corrupted source text and x (cid:48)(cid:48) is corrupted target spans. We found this pre-training can effectively alleviate the catastrophic forgetting of token semantics especially on SPOTNAME and ASSONAME tokens. Final Pre-training Criteria. We initialize UIE-base and UIE-large with T5-v1.1-base and T5-v1.1-large (Raffel et al., 2020), and the model architectures are shown in Table 7. The final objective is the combine of the above tasks: L = L Pair + L Record + L Text (8) For implementation, we uniformly represent all pre-training data as triplets. For text data ( x ) in D text , we build a triplet (None, x (cid:48) , x (cid:48)(cid:48) ) where x (cid:48) is the corrupted source text and x (cid:48)(cid:48) is corrupted spans. For text-record data ( x , y ) in D pair , we construct ( s , x , y ) by sampling the meta-schema s for each text-record pair. For record data ( y ) in D record , we take (None, None, y ) as the input triplet. We randomly pack instances for different tasks in one batch, and details are shown in Algorithm 1 in the appendix. Given the pre-trained UIE model, we can quickly adapt it to different IE tasks and settings through model fine-tuning. Given a labeled corpus D task = { ( s, x, y ) } , we fine-tune the UIE model using teacher-forcing cross-entropy loss: LFT = (cid:88) ( s,x,y ) D Task  log p ( y | x, s ;  e ,  d ) (9) To alleviate the exposure bias (Ranzato et al., 2016; Zhang et al., 2020) of the auto-regressive model during decoding, we also design a Rejection Mechanism for effective fine-tuning. Specifically, given an instance ( s , x , y ), we first encode y using SEL language, then we randomly insert several [ NULL ] unit with negative SPOTNAME and ASSONAME : (SPOTNAME , [ NULL ]) and (ASSONAME , [ NULL ]) into the ground-truth SEL with the SSI <spot> person ... <spot> facility <asso> ... <text> Text Steve became CEO of Apple in 1997. probability of p (cid:15) . For example, in Table 1, facility is the negative spot in the schema prompt, i.e., there is no facility entity in the sentence Steve became CEO of Apple in 1997. Therefore, we randomly inject the noise of (facility: [ NULL ]) into the target record during model learning. In this way, the UIE can effectively learn to reject misleading generation by generating [ NULL ] token. To verify the effectiveness of UIE, we conducted experiments on different IE tasks and settings. Datasets. We conduct experiments on 13 IE benchmarks across 4 well-representative IE tasks (including entity extraction, relation extraction, event extraction, structured sentiment extraction) and their combinations (e.g., joint entity-relation extraction). The used datasets includes ACE04 (Mitchell et al., 2005), ACE05 (Walker et al., 2006); CoNLL03 (Tjong Kim Sang and De Meul-der, 2003), CoNLL04 (Roth and Yih, 2004), SciERC (Luan et al., 2018), NYT (Riedel et al., 2010), CASIE (Satyapanich et al., 2020), SemEval-14 (Pontiki et al., 2014), SemEval-15 (Pontiki et al., 2015), SemEval-16 (Pontiki et al., 2016), see Table 8 for detail. We employ the end-to-end setting for all extraction tasks, which takes the raw text as input and directly generates the target structure. Evaluation. We use the same evaluation metrics as all previous methods, and details of metrics are shown in the appendix. For each fine-tuning experiment, we report the average performance on 3 random seeds. Because UIE only generates text spans, we map spans to offsets by finding the first matched offsets that are not already matched in the same SEL hierarchical level (details in appendix). We found this simple heuristic rule is very effective (<0.5% error offsets) and more complicated mapping approaches (such as attention-weight guided span mapping) are left as the future work. UIE provides a universal backbone for IE tasks. This section assesses the UIE performance in supervised settings. We compare UIE with the state-of-the-art, task-specific supervised models. For a fair comparison, we only compare the state-of-the-art without leveraging additional dataset-specific knowledge or larger-scale contexts. These extensions are good complementary of UIE, and can be left for further improvement. Table 2 shows the performance of UIE on the 13 IE datasets across 4 tasks. We can observe that: 1) By modeling IE as text-to-structure generation and encoding with an effective SEL language, UIE provides an effective universal architecture for IE. The UIE model achieves state-of-the-art performance on nearly all datasets and tasks, even without pre-training (SEL). 2) The large-scale pre-trained model provides a solid foundation for universal IE. Compared with baselines, the pre-trained model achieves the performance of the state-of-the-art in most datasets and improves 1.42% F1 on average. 3) By universally modeling IE tasks and pre-training using large-scale datasets, UIE can effectively capture, share, and transfer IE abilities. Pre-training improves all tasks at the same time, especially events and sentiment knowledge rarely appear in the pretrain dataset. It proves that SEL is a unified and cross-task transferable structured representation for IE, which allows UIE to share learned capabilities and information across different and various information extraction tasks. To verify the quick adaptation ability of UIE, we conducted low-resource experiments on six different partitions of the original training sets (1/5/10-shot, 1/5/10% ratio) across 4 tasks. For the few-shot experiments, we sample 1/5/10 sentences for each entity/relation/event/sentiment type in the training set. To avoid the influence of random sampling, we repeated each experiment 10 times with different samples and reported their averaged results as previous works (Huang et al., 2021). We compare UIE with the following pre-trained model: 1) T5-v1.1-base is an initial model of UIE-base; 2) Fine-tuned T5-base is fine-tuned with sequence generation tasks such as summarization, which have been shown effective in many low-resource NLP tasks (Paolini et al., 2021); 3) UIE-base w/o SSI is the distant supervised version of UIE without SSI in the pre-training stage, which is used to verify the necessity of SSI when adapting UIE in low-resource settings. Table 3 shows the performance of 4 IE tasks under 6 low-resource settings. We observe that: 1) By guiding the generation using schema-based prompts, SSI is an effective way for adaptively controlling which to ex-5761 Model 1-Shot 5-Shot 10-Shot AVE-S 1% 5% 10% AVE-R Entity ( CoNLL03 ) Ent-F1 T5-v1.1-base 12.73 30.17 58.89 33.93 75.74 85.71 87.70 83.05 Fine-tuned T5-base 24.93 54.85 65.31 48.36 78.51 87.67 88.91 85.03 UIE-base w/o SSI 43.52 64.76 72.47 60.25 81.91 88.41 89.84 86.72 UIE-base 46.43 67.09 73.90 62.47 82.84 88.34 89.63 86.94 Relation ( CoNLL04 ) Rel-S F1 T5-v1.1-base 2.35 7.99 25.98 12.11 6.08 32.38 41.87 26.78 Fine-tuned T5-base 4.24 28.16 41.44 24.61 12.89 37.75 49.95 33.53 UIE-base w/o SSI 13.21 40.35 49.47 34.34 24.21 48.70 56.59 43.17 UIE-base 22.05 45.41 52.39 39.95 30.77 51.72 59.18 47.22 Event Trigger ( ACE05-Evt ) Evt Tri F1 T5-v1.1-base 19.40 43.35 50.57 37.77 25.59 49.47 57.18 44.08 Fine-tuned T5-base 30.18 48.31 51.27 43.25 31.08 51.16 57.76 46.67 UIE-base w/o SSI 32.07 48.11 51.00 43.73 32.71 53.20 59.26 48.39 UIE-base 38.14 51.21 53.23 47.53 41.53 55.70 60.29 52.51 Event Argument ( ACE05-Evt ) Evt Arg F1 T5-v1.1-base 2.75 20.21 27.53 16.83 3.59 21.53 30.90 18.67 Fine-tuned T5-base 6.96 25.07 30.96 21.00 7.39 24.97 33.90 22.09 UIE-base w/o SSI 9.31 23.99 30.31 21.20 9.57 27.25 34.18 23.67 UIE-base 11.88 27.44 33.64 24.32 12.80 30.43 36.28 26.50 Sentiment ( 16res ) Rel-S F1 T5-v1.1-base 0.04 2.11 12.66 4.94 3.50 27.08 45.97 25.52 Fine-tuned T5-base 6.55 21.06 29.92 19.18 18.72 39.63 51.65 36.67 UIE-base w/o SSI 7.79 17.77 32.07 19.21 19.14 42.76 53.44 38.45 UIE-base 10.50 26.24 39.11 25.28 24.24 49.31 57.61 43.72 Table 3: Low-resource results on end-to-end IE tasks, where AVE-S (hot) and AVE-R (atio) are the averaged performance across 3 few-shot settings and 3 low-resource settings respectively. tract. Compared with the UIE model w/o SSI, UIE equipped with SSI achieves improvements of 4.16 and 3.30 on average for n-shot and n-ratio experiments. 2) Our pre-training algorithms can learn general IE ability rather than capture task-specific information. Even the pre-training of UIE didn't include event and sentiment knowledge, UIE still achieved significantly better performance on these tasks compared to the baseline with only a small number of samples. To investigate the effect of different pre-training tasks, Table 4 shows ablation experiment results of UIE-base on four downstream tasks. We can  P P R F UIE-base +11.41 79.54 72.63 75.91 w/o rejection 68.13 67.85 66.13 UIE-base w/o SSI +9.41 78.96 70.50 74.49 w/o rejection 69.55 63.69 66.44 T5-base +17.95 74.12 61.72 67.33 w/o rejection 56.17 56.00 55.94 T5-v11 +13.88 71.88 51.23 59.67 w/o rejection 58.00 45.04 50.38 Table 5: Experiment results of 10-shot setting on the CoNLL 03 development set. see that: (1) The pre-training of SEL ( L Record ) and sequence-to-structure mapping ( L Pair ) is crucial for UIE, and such a structure generation pre-training is especially useful for small-scale datasets. In small datasets CoNLL04 and 16res, adding structure generation pre-training (from T5-v1.1-base to UIE-base w/o L Text ), the performance significantly increases from 72.12 to 75.70 and 72.03 to 74.28. (2) Retrofitting semantic using the mask language model task ( L Text ) is more important for the complex extraction task. In the tasks with more semantic types such as event extraction (33 types), the performance drops significantly after removing the L Text task, e.g., 72.63  70.89 and 5762 57.27  54.16. (3) The mapping pre-training with L Pair enables the model to learn the ability of extraction. After ablating L Pair , the extraction ability of UIE is significantly decreased, i.e., the performance on the relation (-0.90), event (-1.43/-1.48), and sentiment (-0.46) tasks all see large decline. This section investigates the effect of the proposed rejection noise. Table 5 shows the results of the different pre-trained models on the development set of CoNLL 03 under the 10-shot setting. The mis-generated label has a negative influence on the precision of the proposed generation method leading to a large number of error extraction results. The proposed rejection noise is useful for the generation method, which leads to improvements of 13.16 precision (P) on average. Building and pre-training universal models of NLP tasks has attracted a lot of attention in recent years, e.g., contextualized representation (Devlin et al., 2019; Liu et al., 2019), text generation (Lewis et al., 2020; Raffel et al., 2020), multi-modal (Li et al., 2021b; Cho et al., 2021), and multi-lingual (Con-neau et al., 2020; Xue et al., 2021). This paper proposes and pre-trains the first universal model for information extraction. IE is a long-researched area and many classical neural architectures have been proposed, such as sequence tagging (Lample et al., 2016; Zheng et al., 2017; Lin et al., 2019), span classification (Sohrab and Miwa, 2018; Lin et al., 2018; Wadden et al., 2019), and MRC (Levy et al., 2017; Li et al., 2020; Du and Cardie, 2020). And several task-specific pre-training techniques are proposed on these architectures (Mengge et al., 2020; Wang et al., 2021b; Qin et al., 2021). More relevant to our work are generation-based IE methods, which generate text spans via tagging (Strakov et al., 2019; Ma et al., 2019), index pointer (Ren et al., 2021; Yan et al., 2021b) or copy mechanism (Zeng et al., 2018), and these methods usually employ specific classifiers to represent labels. The generation can be enhanced using label templates (Li et al., 2021a; Liu et al., 2021; Cui et al., 2021), schema (Lu et al., 2021; Ahmad et al., 2021), and augmented language methods (Paolini et al., 2021). models, this paper aims to universally model various IE tasks in an unified text-to-structure framework, which can greatly benefit the rapid development, effective knowledge sharing, and quick adaptation of IE systems. In this paper, we propose a unified text-to-structure generation framework  UIE, which can universally model different IE tasks, adaptively generate targeted structures, and unfiedly learn general IE abilities from different knowledge sources. Experimental results show that UIE achieves very competitive performance in both supervised and low-resource settings, which verified its universality, effectiveness, and transferability. A large-scale pre-trained text-to-structure model is also released, which will benefit future studies. For future work, we want to extend UIE to KB-aware IE tasks such as entity linking (Cao et al., 2021), and document-aware IE tasks such as co-reference (Lee et al., 2017; Lu et al., 2022). We sincerely thank the reviewers for their insightful comments and valuable suggestions. This research work is supported by the National Natural Science Foundation of China under Grants no. U1936207, 62122077 and 62106251, the Project of the Chinese Language Committee under Grant no. YB2003C002.
abstain objective abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain objective abstain abstain abstain abstain abstain abstain abstain abstain abstain objective method objective abstain abstain abstain abstain abstain abstain abstain objective method method objective abstain abstain abstain abstain abstain method abstain abstain method abstain method abstain abstain abstain abstain abstain abstain abstain other abstain abstain other abstain abstain abstain abstain abstain method abstain abstain abstain other abstain other abstain other abstain abstain abstain method other abstain other method abstain method method abstain method abstain abstain method method abstain abstain method abstain abstain method method abstain abstain method abstain abstain method abstain abstain abstain abstain abstain abstain abstain abstain method method method abstain method method other abstain abstain abstain abstain result abstain abstain other abstain abstain result abstain abstain result abstain abstain abstain abstain abstain abstain result abstain abstain abstain abstain abstain abstain result abstain result result abstain result abstain abstain abstain abstain abstain result abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain other objective other other method other objective objective abstain abstain abstain other other other other