Recent entity and relation extraction works focus on investigating how to obtain a better span representation from the pre-trained encoder. However, a major limitation of existing works is that they ignore the interrelation between spans (pairs). In this work, we propose a novel span representation approach, named Packed Levitated Markers (PL-Marker), to consider the interrelation between the spans (pairs) by strategically packing the markers in the encoder. In particular, we propose a neighborhood-oriented packing strategy, which considers the neighbor spans integrally to better model the entity boundary information. Furthermore, for those more complicated span pair classification tasks, we design a subject-oriented packing strategy, which packs each subject and all its objects to model the interrelation between the same-subject span pairs. The experimental results show that, with the enhanced marker feature, our model advances baselines on six NER benchmarks, and obtains a 4.1%-4.3% strict relation F1 improvement with higher speed over previous state-of-the-art models on ACE04 and ACE05. Our code and models are publicly available at https://github.com/ thunlp/PL-Marker . Recently, pre-trained language models (PLMs) (De-vlin et al., 2019; Liu et al., 2019) have achieved significant improvements in Named Entity Recognition (NER, Luo et al. (2020); Fu et al. (2021)) and Relation Extraction (RE, Wadden et al. (2019); Zhou and Chen (2021)), two key sub-tasks of information extraction. Recent works (Wang et al., 2021c; Zhong and Chen, 2021) regard these two tasks as span classification or span pair classification, and thus focus on extracting better span representations from the PLMs. [O]David[/O] assaulted a pair of restaurant workers during a night out with national squad teammates in [S]Copenhagen[/S]. David assaulted a pair of restaurant [O]workers[/O] during a night out with national squad teammates in [S]Copenhagen[/S]. David assaulted a pair of restaurant workers during a night out with national squad [O]teammates[/O] in [S]Copenhagen[/S]. Three span representation extraction methods are widely used: (1) T-Concat (Lee et al., 2017; Jiang et al., 2020) concatenates the representation of the span's boundary (start and end) tokens to obtain the span representation. It collects information at the token level but ignores the connection between boundary tokens of a span when they pass through the network; (2) Solid Marker (Soares et al., 2019; Xiao et al., 2020) explicitly insert two solid markers before and after the span to highlight the span in the input text. And it inserts two pair of markers to locate the subject and object of a span pair. However, the method cannot handle multiple span pairs at the same time because of its weakness in specifying the solid markers of a span pair from more than two pairs of markers in the sequence. (3) 4904 Levitated Marker (Zhong and Chen, 2021) first sets a pair of levitated markers to share the same position with the span's boundary tokens and then ties a pair of markers by a directional attention. To be specific, the markers within a pair are set to be visible to each other in the attention mask matrix, but not to the text token and other pairs of markers. Existing work (Zhong and Chen, 2021) simply replaces solid markers with levitated markers for an efficient batch computation, but sacrifices the model performance. As the RE example shown in Figure 1, to correctly identify that David, workers and teammates are located_in Copenhagen, it is important to separate out that David attacked the restaurant workers and he had social relation with his teammates. However, prior works with markers (Zhong and Chen, 2021) independently processes the span pairs with different insertions of markers in the training phrase, and thus ignore interrelation between spans (pairs) (Sorokin and Gurevych, 2017; Luan et al., 2019; Wadden et al., 2019). In this work, we introduce Packed Levitated Marker (PL-Marker), to model the interrelation between spans (pairs) by strategically packing levitated markers in the encoding phase. A key challenge of packing levitated markers together for span classification tasks is that the increasing number of inserted levitated markers would exacerbate the complexity of PLMs quadratically (Ye et al., 2021). Thus, we have to divide spans into several groups to control the length of each input sequence for a higher speed and feasibility. In this case, it is necessary to consider the neighbor spans integrally, which could help the model compare neighbor spans, e.g. the span with the same start token, to acquire a more precise entity boundary. Hence, we propose a neighborhood-oriented packing strategy, which packs the spans with the same start token into a training instance as much as possible to better distinguish the entity boundary. For the more complicated span pair classification tasks, an ideal packing scheme is to pack all the span pairs together with multiple pairs of levitated markers, to model all the span pairs integrally. However, since each pair of levitated markers is already tied by directional attention, if we continue to apply directional attention to bind two pairs of markers, the levitated marker will not be able to identify its partner marker of the same span. Hence, we adopt a fusion of solid markers and levitated markers, and use a subject-oriented packing strategy to model the subject with all its related objects integrally. To be specific, we emphasize the subject span with solid markers and pack all its candidate object spans with levitated markers. Moreover, we apply an object-oriented packing strategy for an intact bidirectional modeling (Wu et al., 2020). We examine the effect of PL-Marker on two typical span (pair) classification tasks, NER and end-to-end RE. The experimental results indicate that PL-Marker with neighborhood-oriented packing scheme performs much better than the model with random packing scheme on NER, which shows the necessity of considering the neighbor spans integrally. And our model also advances the T-Concat model on six NER benchmarks, which demonstrates the effectiveness of the feature obtained by span marker. Moreover, compared with the previous state-of-the-art RE model, our model gains a 4.1%-4.3% strict relation F1 improvement with higher speed on ACE04 and ACE05 and also achieves better performance on SciERC, which shows the importance of considering the interrelation between the subject-oriented span pairs. In recent years, span representation has attracted great attention from academia, which facilitates various NLP applications, such as named entity recognition (Ouchi et al., 2020), relation and event extraction (Luan et al., 2019), coreference resolution (Lee et al., 2017), semantic role labeling (He et al., 2018) and question answering (Lee et al., 2016). Existing methods to enhance span representation can be roughly grouped into three categories: Span Pre-training The span pre-training approaches enhance the span representation for PLMs via span-level pre-training tasks. Sun et al. (2019); Lewis et al. (2020); Raffel et al. (2020) mask and learn to recover random contiguous spans rather than random tokens. Joshi et al. (2020) further learns to store the span information in its boundary tokens for downstream tasks. Knowledge Infusion This series of methods focuses on infusing external knowledge into their models. Zhang et al. (2019); Peters et al. (2019); Wang et al. (2021a) learn to use the external entity embedding from the knowledge graph or the synonym net to acquire knowledge. Soares et al. (2019); Xiong et al. (2020); Wang et al. (2021b); 4905 Yamada et al. (2020) conduct specific entity-related pre-training to incorporate knowledge into their models with the help of Wikipeidia anchor texts. Structural Extension The structural extension methods add reasoning modules to the existing models, such as biaffine attention (Wang et al., 2021d), graph propagation (Wadden et al., 2019) and memory flow (Shen et al., 2021). With the support of modern pre-training encoders ( e.g. BERT), the simple model with solid markers could achieve state-of-art results in RE (Zhou and Chen, 2021; Zhong and Chen, 2021). However, it is hard to specify the solid markers of a span pair from more than two pairs of markers in the sequence. Hence, previous work (Zhong and Chen, 2021) has to process span pairs independently, which is time-consuming and ignores the interrelation between the span pairs. In this work, we introduce the neighborhood-oriented and the subject-oriented packing strategies to take advantage of the levitated markers to provide an integral modeling on spans (pairs). To our best knowledge, we are the first to apply the levitated markers on the NER. On the RE, the closest work to ours is the PURE (Approx.) (Zhong and Chen, 2021), which independently encodes each span pair with two pairs of levitated markers in the training phase and batches multiple pairs of markers to accelerate the inference process. Compared to their work, our model adopts a fusion subject-oriented packing scheme and thus handle multiple span pairs well in both the training and inference process. We detail the differences between our work and PURE in Section 4.4.2 and explain why our model performs better. In this section, we first introduce the architecture of the levitated marker. Then, we present how we pack the levitated marker to obtain the span representation and span pair representation. Levitated marker is used as an approximation of solid markers, which allows models to classify multiple pairs of entities simultaneously to accelerate the inference process (Zhong and Chen, 2021). A pair of levitated markers, associated with a span, consists of a start token marker and an end token marker. These two markers share the same position embedding with the start and end tokens of the corresponding span, while keeping the position id of original text tokens unchanged. In order to specify multiple pairs of levitated markers in parallel, a directional attention mask matrix is applied. Specifically, each levitated marker is visible to its partner marker within pair in the attention mask matrix, but not to the text tokens and other levitated markers. In the meantime, the levitated markers are able to attend to the text tokens to aggregate information for their associated spans. Benefiting from the parallelism of levitated markers, we can flexibly pack a series of related spans into a training instance. In practice, we append multiple associated levitated markers to an input sequence to conduct a comprehensive modeling on each span. However, even though the entity length is restricted, some of the span classification tasks still contain a large number of candidate spans. Hence, we have to group the markers into several batches to equip the model with higher speed and feasibility in practice. To better model the connection between spans with the same start tokens, we adopt a neighborhood-oriented packing scheme. As shown in Figure 2, we first sort the pairs of levitated markers by taking the position of start marker as the first keyword and the position of end marker as the second keyword. After that, we split them into groups of size up to K and thus gather adjacent spans into the same group. We packs each groups of markers and dispersedly process them in multiple runs. Formally, given a sequence of N text tokens, X = { x 1 , . . . , x N } and a maximum span length L , we define the candidate spans set as S ( X ) = { (1 , 1) , .., (1 , L ) , ..., ( N, N  L ) , .., ( N, N )) } . We first divide S ( X ) into multiple groups up to the size of K in order. For example, we cluster K spans, { (1 , 1) , (1 , 2) , ..., ( (cid:100) KL (cid:101) , K  (cid:98) K  1 L (cid:99)  L ) } , into a group S 1 . We associate a pair of levitated markers to each span in S 1 . Then, we provide the combined sequence of the text token and the inserted levitated markers to the PLM ( e.g. BERT) to obtain the contextualized representations of the start token marker H ( s ) = { h ( s ) i } and that of the end token marker H ( e ) = { h ( e ) i } . Here, h ( s ) a and h ( e ) b are associated with the span s i = ( a, b ) , for which we obtain the span representations:  ( s i ) = [ h ( s ) a ; h ( e ) b ] (1) where [ A ; B ] denotes the concatenation operation 4906 David Green and are doctors in [S] [/S] wife [O2] [/O2] [O3] [/O3] Dallas PER-SOC PHYS [O1] [/O1] his NA Bank of China is open 1 2 3 4 5 [O1] [O2] [O3] [O4] [O5] [/O1] [/O2] [/O3] [/O4] [/O5] 1 1 1 1 1 1 2 3 4 5 [O6] [O7] [O8] [O9] [O10] [/O6] [/O7] [/O8] [/O9] [/O10] 2 2 2 2 2 3 3 4 5 3   [O11] [O15] [/O11] [/O15] 3 4 5 5 Neighborhood-oriented Packing for Span Subject-oriented Packing for Span Pair GPE PER (his) PER (wife) GPE (Dallas) Position ID: PER Figure 2: An overview of our neighborhood-oriented packing and subject-oriented packing strategies. For instance, we apply the levitated marker to a typical overlapping span classification task, NER, which aims to assign an entity type or a non-entity type to each possible span in a sentence. We obtain the span representation from the PLM via the packed levitated markers and then combine the features of PL-Marker and T-Concat to better predict the entity type of the cadidate span. To obtain a span pair representation, a feasible method is to adopt levitated markers to emphasize a series of the subject and object spans simultaneously. Commonly, each pair of levitated markers is tied by the directional attention. But if we continue to apply directional attention to bind two pairs of markers, the levitated marker will not be able to identify its partner marker of the same span. Hence, as shown in Figure 2, our span pair model adopts a fusion subject-oriented packing scheme to offer an integral modeling for the same-subject spans. Formally, given an input sequence X , a subject span, s i = ( a, b ) and its candidate object spans ( c 1 , d 1 ) , ( c 2 , d 2 ) , ... ( c m , d m ) , We insert a pair of solid markers [S] and [/S] before and after the subject span. Then, we apply levitated markers [O] and [/O] to all candidate object spans, and pack them into an instance. Let  X denotes this modified sequence with inserted markers:  X = ... [S] , x a , ..., x b , [/S] , ..., x c 1  [O1] , ..., x d 1  [/O1] , ..., x c 2  [O2] , ..., x d 2  [/O2] ..., where the tokens jointed by the symbol  share the same position embedding. We apply a pretrained encoder on  X and finally obtain the span pair representation for s i = ( a, b ) and s j = ( c, d ) :  ( s i , s j ) = [ h a  1 ; h b +1 ; h ( s ) c ; h ( e ) d ] (2) where [ ; ] denotes the concatenation operation. h a  1 and h b +1 denote the contextualized representation of the inserted solid markers for s i ; h ( s ) c and h ( e ) d are the contextualized representation of the inserted levitated markers for s j . Compared to the method that applies two pairs of solid markers on the subject and object respectively (Zhong and Chen, 2021), our fusion marker scheme replaces the solid markers with the levitated markers for the object span, which would impair the emphasis on the object span to some extent. To provide the supplemental information, we introduce an inverse relation from the object to the subject for a bidirectional prediction (Wu et al., 2020). For instance, we evaluate our model on a typical span pair classification task, end-to-end RE, which concentrates on identifying whether all span pairs are related and their relation types. Following Zhong and Chen (2021), we first use a NER model to filter candidate entity spans, and then acquire the 4907 span pair representation of the filtered entity span pairs to predict the relation between them. Moreover, to build the connection between entity type and relation type, we add an auxiliary loss for predicting the type of object entity (Zhou and Chen, 2021; Han et al., 2021). Dominated by the large feed-forward network, the computation of PLM rises almost linearly with the increase in small sequence length (Dai et al., 2020; Ye et al., 2021). Gradually, as the sequence length continues to grow, the computation dilates quadratically due to the Self-Attention module (Vaswani et al., 2017). Obviously, the insertion of levitated markers extends the length of input sequence. For the span pair classification tasks, the number of candidate spans is relatively small , thus the increased computation is limited. For the span classification tasks, we group the markers into several batches, which can control the sequence length within the interval in which the complexity increases nearly linearly. For the NER, we enumerate candidate spans in a small-length sentence and then use its context words to expand the sentence to 512 tokens, for which the number of candidate spans in a sentence is usually less than the context length in practice. Hence, with a small number of packing groups, the complexity of PL-Marker is still near-linearly to the complexity of previous models. Moreover, to further alleviate the inference cost, we adopt PL-Marker as a post-processing module of a two-stage model, in which it is used to identify entities from a small number of candidate entities proposed by a simpler and faster model. For the NER task, we conduct experiments on both flat and nested benchmarks. Firstly, on the flat NER, we adopt CoNLL03 (Sang and Meulder, 2003), OntoNotes 5.0 (Pradhan et al., 2013) and Few-NERD (Ding et al., 2021). Then, on the nested NER, we use ACE04 (Doddington et al., 2004), ACE05 (Walker et al., 2006) and SciERC (Luan et al., 2018). The three nested NER datasets are also used to evaluate the end-to-end RE. We follow Luan et al. (2019) to split ACE04 into 5 folds and split ACE05 into train, development, and test Dataset #Sents #Ents (#Types) #Rels (#Types) CoNLL03 22.1k 35.1k (4) OntoNotes 5.0 103.8k 161.8k (18) Few-NERD 188.2k 491.7k (66) ACE05 14.5k 38.3k (7) 7.1k (6) ACE04 8.7k 22.7k (7) 4.1k (6) SciERC 2.7k 8.1k (6) 4.6k (7) Table 1: The statistics of the adopted datasets. sets. For other datasets, we adopt the official split. Table 1 shows the statistics of each dataset. For NER task, we follow a span-level evaluation setting, where the entity boundary and entity type are required to correctly predicted. For the end-to-end RE, we report two evaluation metrics: (1) Boundaries evaluation (Rel) requires the model to correctly predict the boundaries of the subject entity and the object entity, and the entity relation; (2) Strict evaluation (Rel+) further requires the model to predict the entity types on the basis of the requirement of the boundary prediction. Moreover, following Wang et al. (2021d), we regard each symmetric relational instance as two directed relational instances. We adopt bert-base-uncased (Devlin et al., 2019) and albert-xxlarge-v1 (Lan et al., 2020) encoders for ACE04 and ACE05. For SciERC, we use the in-domain scibert-scivocab-uncased (Beltagy et al., 2019) encoder. For flat NER, we adopt roberta-large encoder. We also leverage the cross-sentence information (Luan et al., 2019; Luoma and Pyysalo, 2020), which extends each sentence by its context and ensures that the original sentence is located in the middle of the expanded sentence as much as possible. As discussed in Section 4.4.1, for the packing scheme on NER, we set the group size to 256 to improve efficiency. We run all experiments with 5 different seeds and report the average score. See the appendix for the standard deviations and the detailed training configuration. Our packing scheme allows the model to apply the levitated markers to process massive span pairs and to our best knowledge, we are the first to apply the levitated markers on the NER task. We compare our neighborhood-oriented packing scheme with 4908 Model CoNLL03 OntoN5 F-NERD Ma and Hovy (2016) 91.0 86.3 Devlin et al. (2019) 92.8 89.2 68.9 Li et al. (2020) 93.0 91.1 Yu et al. (2020) 93.5 91.3 Yan et al. (2021) 93.2 90.4 SeqTagger (Our impl.) 93.6 91.2 69.0 T-Concat (Our impl.) 93.0 91.7 70.6 Random Packing 93.9 91.8 61.5 PL-Marker (Our model) 94.0 91.9 70.9 Table 2: Micro F1 on the test set for the flat NER. the Random Packing , which randomly packs the candidate spans into groups. We adopt two common NER models: (1) SeqTagger (Devlin et al., 2019) regards NER as a sequence tagging task and applies a token-level classifier to distinguish the IOB2 tags for each word (Sang and Veenstra, 1999). (2) T-Concat (Jiang et al., 2020; Zhong and Chen, 2021) assigns an entity type or a non-entity type to each span based on its T-Concat span representation. Note that solid markers cannot deal with the overlapping spans simultaneously, thus it is too inefficient to apply solid markers independently on the NER task. We show the flat NER results in the Table 2 and the nested NER results in the Ent column of Table 3, where PURE (Zhong and Chen, 2021) applies the T-Concat feature on its NER module. As follow, some observations are summarized from the experimental results: (1) The model with our neighborhood-oriented packing strategy outperforms the model with random packing strategy on all three flat NER datsets, especially obtaining a 9.4% improvement on Few-NERD. Few-NERD contains longer sentences and thus includes 325 candidate spans on average, while CoNLL03 and OntoNotes 5.0 only contain 90 and 174 respectively. It shows that the neighborhood-oriented packing strategy can well handle the dataset with longer sentences and more groups of markers, to better model the interrelation among neighbor spans. (2) With the same large pre-trained encoder, PL-Marker achieves an absolute F1 improvement of +0.1%-1.1% over T-Concat on all six NER benchmarks, which shows the advantage of levitated markers in aggregating span-wise representation for the entity type prediction; (3) PL-Marker outperforms SeqTagger by an absolute F1 of +0.4%, +0.7%, +1.9% in CoNLL03, OntoNote 5.0 and Few-NERD respectively, where CoNLL03, OntoNote 5.0 and Few-NERD contain 4, 18 and 66 entity types respectively. Such improvements prove the effectiveness of PL-Marker in handling diverse interrelation between entities of diverse types. For the end-to-end RE, we compare our model, PL-Marker, with a series of state-of-the-art models. Here, we introduce two of the most representative works with T-Concat and Solid Markers span representation: (1) DyGIE++ (Wadden et al., 2019) first acquires the T-Concat span representation, and then iteratively propagates coreference and relation type confidences through a span graph to refine the representation; (2) PURE (Zhong and Chen, 2021) adopts independent NER and RE models, where the RE model processes each possible entity pair in one pass. In their work, PURE (Full) adopts two pairs of solid markers to emphasize a span pair and the PURE (Approx) employs two pairs of levitated markers to underline the span pair. As shown in Table 3, with the same BERTBASE encoder, our approach outperforms previous methods by strict F1 of +1.7% on ACE05 and +2.5% on ACE04. With the SciBERT encoder, our approach also achieves the best performance on SciERC. Using a larger encoder, ALBERTXXLARGE , both of our NER and RE models are further improved. Compared to the previous state-of-the-art model, PURE (Full), our model gains a substantially +4.1% and +4.3% strict relation F1 improvement on ACE05 and ACE04 respectively. Such improvements over PURE indicate the effectiveness of modeling the interrelation between the same-subject or the same-object entity pairs in the training process. In this section, we compare the models' inference speed on an A100 GPU with a batch size of 32. We use the BASE size encoder for ACE05 and SciERC in the experiments and the LARGE size encoder for flat NER models. PL-Marker with different group size K on CoNLL03 and Few-NERD. We also evaluate a cascade Two-stage 4909 Model Encoder Rep ACE05 ACE04 SciERC Type Ent Rel Rel+ Ent Rel Rel+ Ent Rel Rel+ Li and Ji (2014) -80.8 52.1 49.5 79.7 48.3 45.3 -SPtree (Miwa and Bansal, 2016) LSTM T 83.4 -55.6 81.8 -48.4 -DYGIE (Luan et al., 2019) (cid:51) ELMo T 88.4 63.2 -87.4 59.7 -65.2 41.6 -Multi-turn QA (Li et al., 2019) BERTL -84.8 -60.2 83.6 -49.4 -OneIE (Lin et al., 2020) T 88.8 67.5 ----DYGIE++ (Wadden et al., 2019) (cid:51) BERTB / SciBERT T 88.6 63.4 ---TriMF (Shen et al., 2021) (cid:51) T 87.6 66.5 62.8 --70.2 52.4 UniRE (Wang et al., 2021d) (cid:51) T 88.8 -64.3 87.7 -60.0 68.4 -36.9 PURE-F (Zhong and Chen, 2021) (cid:51) S 90.1 67.7 64.8 89.2 63.9 60.1 68.9 50.1 36.8 PURE-A (Zhong and Chen, 2021) (cid:51) L -66.5 ---48.1 -PL-Marker (Our Model) (cid:51) S&L 89.8 69.0 66.5 88.8 66.7 62.6 69.9 53.2 41.6 TableSeq (Wang and Lu, 2020) ALBXXLT 89.5 67.6 64.3 88.6 63.3 59.6 -UniRE (Wang et al., 2021d) (cid:51) T 90.2 -66.0 89.5 -63.0 --PURE-F (Zhong and Chen, 2021) (cid:51) S 90.9 69.4 67.0 90.3 66.1 62.2 --PL-Marker (Our Model) (cid:51) S&L 91.1 73.0 71.1 90.4 69.7 66.5 -Table 3: Overall entity and relation F1 scores on the test sets of ACE04, ACE05 and SciERC. model, which uses a fast BASE -size T-Concat model to filter candidate spans for our model. As shown in Table 4, PL-Marker achieves a 0.4 F1 improvement on CoNLL03 but sacrifices 60% speed compared to the SeqTagger model. And we observe that our proposed Two-stage model achieves similar performance to PL-Marker with 3.1x speedup on Few-NERD, which shows it is more efficient to use PL-Marker as a post-processing module to elaborate the coarse prediction from a simple model. In addition, when the group size grows to 512, PL-Marker slows down due to the increased complexity of the Transformer. Hence, we choose a group size of 256 in practice. We apply the subject-oriented and the object-oriented packing strategies on levitated markers for RE. Here, we compare our model with the other two marker-based models. Firstly, PURE (Full) (Zhong and Chen, 2021) applies solid markers to process each entity pair independently. Secondly, PURE (Approx.) packs the levitated markers of all entity pairs into an instance for batch computation. Since the performance and the running time of the above methods rely on the quality and the number of predicted entities, for a fair comparison, we adopt the same entity input from the entity model of PURE on all the RE models. Table 5 shows the relation F1 scores and the inference speed of the above three methods. On both datasets, our RE model, PL-Marker, achieves the best performance and PURE (Approx.) has highest efficiency 4910 Named Entity Recognition Text : This is the Cross Strait program on CCTV International Channel. in the inference process. Compared to the PURE (Full), our model obtains a 2.2x-2.8x speedup and better performance on ACE05 and SciERC. Compared to PURE (Approx.), our model achieves a 2.8%-4.0% relation F1 (boundaries) improvement on ACE05 and SciERC, which again demonstrates the effectiveness of our fusion markers and packing strategy. Overall, our model, with a novel subject-oriented packing strategy for markers, has been proven effective in practice, with satisfactory accuracy and affordable cost. We show several cases to compare our span model with T-Concat and to compare our span pair model with PURE (Full). As shown in Table 6, our span model could collect contextual information, such as Taiwan and mainland , for underlined span, Cross Strait , assisting in predicting its type as organization rather than work of art. Our span model learns to integrally consider the interrelation between the same-object relational facts in training phase, so as to successfully obtain the fact that both Liana and her parents are located in Manhattan . In this section, we conduct ablation studies to investigate the contribution of different components to our RE model, where we apply BASE size encoder in the experiments. Two pairs of Levitated Markers We evaluate the w/o solid marker baseline, which applies two pairs of levitated markers on the subject and object respectively and packs all the span pairs into an instance. As shown in Table 7, compared to PL-Model ACE05 SciERC gold e2e gold e2e PL-Marker 74.0 69.0 72.5 53.2 w/o. Marker, the model without solid markers drops a huge 2.0%-3.8% F1 on ACE05 and SciERC when the golden entities are given. The result demonstrates that it is sub-optimal to continue to apply directional attention to bind two pairs of levitated markers, since a pair of levitated marker is already tied by the directional attention. Inverse Relation We establish an inverse relation for each asymmetric relation for a bidirectional prediction. We evaluate the model without inverse relation, which replaces the constructed inverse relation with a non-relation type and adopts a unidirectional prediction. As shown in Table 7, the model without inverse relation drops 0.9%-1.1% F1 on both datasets with the gold entities given, indicating the significance of modeling the information from the object entity to the subject entity in our asymmetric framework. Entity Type We add an auxiliary entity type loss to RE model to introduce the entity type information. As shown in Table 7, when the gold entities are given, the model without entity type loss drops 0.4%-0.7% F1 on both datasets, which shows the importance of entity type information in RE. Moreover, we try to apply the type markers (Zhong and Chen, 2021), such as [Subject:PER] and [Ob-ject:GPE] , to inject entity type information predicted by the NER model into the RE model. We find the RE model with type marker performs slightly worse than the model with entity type loss in the end-to-end setting. It shows that the entity type prediction error from the NER model may be propagated to the RE model if we adopt the type markers as input features. Finally, we discuss when to use the entity type prediction from the RE model to refine the NER prediction in the Appendix and we finally refine entity type for ACE04 and ACE05 except SciERC according to their dataset statistic. In this work, we present a novel packed levitated markers, with a neighborhood-oriented packing strategy and a subject-oriented packing strategy, to obtain the span (pair) representation. Considering the interrelation between spans and span pairs, our model achieves the state-of-the-art F1 scores and a promising efficiency on both NER and RE tasks across six standard benchmarks. In future, we will further investigate how to generalize the marker-based span representation to more NLP tasks. This work is supported by the National Key R&D Program of China (No. 2020AAA0106502), Institute Guo Qiang at Tsinghua University, and International Innovation Center of Tsinghua University, Shanghai, China. We thank Chaojun Xiao and other members of THUNLP for their helpful discussion and feedback. Deming Ye conducted the experiments. Deming Ye, Yankai Lin, Xiaojun Xie and Peng Li wrote the paper. Maosong Sun provided valuable advices to the research.
abstain abstain objective objective method result other other abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain method abstain objective abstain method method objective method abstain abstain objective result other other other other other other other other other other other method objective other method method method method other other other other other other method method abstain method method method method method method abstain abstain method method method method method abstain abstain abstain abstain method method method method other method abstain method method method method method abstain abstain abstain abstain method method abstain method abstain abstain abstain abstain other abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain result abstain abstain abstain other abstain abstain abstain abstain result abstain abstain abstain other abstain abstain other abstain result result result abstain abstain abstain abstain abstain method method abstain abstain abstain method abstain abstain abstain abstain abstain abstain result abstain result abstain abstain result result result abstain abstain abstain abstain abstain abstain abstain result abstain abstain abstain result result abstain objective result objective other other other other other