Neural named entity recognition (NER) models may easily encounter the over-confidence issue, which degrades the performance and calibration. Inspired by label smoothing and driven by the ambiguity of boundary annotation in NER engineering, we propose boundary smoothing as a regularization technique for span-based neural NER models. It re-assigns entity probabilities from annotated spans to the surrounding ones. Built on a simple but strong baseline, our model achieves results better than or competitive with previous state-of-the-art systems on eight well-known NER benchmarks. 1 Further empirical analysis suggests that boundary smoothing effectively mitigates over-confidence, improves model calibration, and brings flatter neural minima and more smoothed loss landscapes. Named entity recognition (NER) is one of the fundamental natural language processing (NLP) tasks with extensive investigations. As a common setting, an entity is regarded as correctly recognized only if its type and two boundaries exactly match the ground truth. The annotation of boundaries is more ambiguous, error-prone, and raises more inconsistencies than entity types. For example, the CoNLL 2003 task contains four entity types (i.e., person, location, organization, miscellaneous), which are easy to distinguish between. However, the boundaries of a entity mention could be ambiguous, because of the boundary words (e.g., articles or modi-fiers). Considerable efforts are required to specify the gold standard practice case by case. Table 1 presents some examples from CoNLL 2003 AnCorresponding author. notation Guidelines. 2 In addition, some studies have also reported that incorrect boundary is a ma-jor source of entity recognition error (Wang et al., 2019; Eberts and Ulges, 2020). Recently, span-based models have gained much popularity in NER studies, and achieved state-of-the-art (SOTA) results (Eberts and Ulges, 2020; Yu et al., 2020; Li et al., 2021). This approach typically enumerates all candidate spans and classi-fies them into entity types (including a non-entity type); the annotated spans are scarce and assigned with full probability to be an entity, whereas all other spans are assigned with zero probability. This creates noticeable sharpness between the classification targets of adjacent spans, and may thus plague the trainability of neural networks. In addition, empirical evidence shows that these models easily encounter the over-confidence issue, i.e., the confidence of a predicted entity is much higher than its correctness probability. This is a manifestation of miscalibration (Guo et al., 2017). Inspired by label smoothing (Szegedy et al., 2016; Mller et al., 2019), we propose boundary smoothing as a regularization technique for span-based neural NER models. By explicitly reallocating entity probabilities from annotated spans 2 https://www-nlpir.nist.gov/related_ projects/muc/proceedings/ne_task.html . to the surrounding ones, boundary smoothing can effectively mitigate over-confidence, and result in consistently better performance. Specifically, our baseline employs the contextualized embeddings from a pretrained Transformer of a base size (768 hidden size, 12 layers), and the biaffine decoder proposed by Yu et al. (2020). With boundary smoothing, our model outperforms previous SOTA on four English NER datasets (CoNLL 2003, OntoNotes 5, ACE 2004 and ACE 2005) and two Chinese datasets (Weibo NER and Resume NER), and achieves competitive results on other two Chinese datasets (OntoNotes 4 and MSRA). Such extensive experiments support the effectiveness and robustness of our proposed technique. In addition, we show that boundary smoothing can help the trained NER models to preserve calibration, such that the produced confidences can better represent the precision rate of a predicted entity. This corresponds to the effect of label smoothing on the image classification task (Mller et al., 2019). Further, visualization results qualitatively suggest that boundary smoothing can lead to flatter solutions and more smoothed loss landscapes, which are typically associated with better generalization and trainability (Hochreiter and Schmidhu-ber, 1997; Li et al., 2018). Named Entity Recognition The mainstream NER systems are designed to recognize flat entities and based on a sequence tagging framework. Col-lobert et al. (2011) introduced the linear-chain conditional random field (CRF) into neural network-based sequence tagging models, which can explicitly encode the transition likelihoods between adjacent tags. Many researchers followed this work, and employed LSTM as the encoder. In addition, character-level representations are typically used for English tasks (Huang et al., 2015; Lample et al., 2016; Ma and Hovy, 2016; Chiu and Nichols, 2016), whereas lexicon information is helpful for Chinese NER (Zhang and Yang, 2018; Ma et al., 2020; Li et al., 2020a). Nested NER allows a token to belong to multiple entities, which conflicts with the plain sequence tagging framework. Ju et al. (2018) proposed to use stacked LSTM-CRFs to predict from inner to outer entities. Strakov et al. (2019) concatenated the BILOU tags for each token inside the nested entities, which allows the LSTM-CRF to work as for flat entities. Li et al. (2020b) reformulated nested NER as a machine reading comprehension task. Shen et al. (2021) proposed to recognize nested entities by the two-stage object detection method widely used in computer vision. Recent years, a body of literature emerged on span-based models, which were compatible with both flat and nested entities, and achieved SOTA performance (Eberts and Ulges, 2020; Yu et al., 2020; Li et al., 2021). These models typically enumerate all possible candidate text spans and then classify each span into entity types. In this work, the biaffine model (Yu et al., 2020) is chosen and re-implemented with slight modifications as our baseline, because of its high performance and compatibility with boundary smoothing. In addition, pretrained language models, also known as contextualized embeddings, were also widely introduced to NER models, and significantly boosted the model performance (Peters et al., 2018; Devlin et al., 2019). They are used in our baseline by default. Label Smoothing Szegedy et al. (2016) proposed the label smoothing as a regularization technique to improve the accuracy of the Inception networks on ImageNet. By explicitly assigning a small probability to non-ground-truth labels, label smoothing can prevent the models from becoming too confident about the predictions, and thus improve generalization. It turned out to be a useful alternative to the standard cross entropy loss, and has been widely adopted to fight against the over-confidence (Zoph et al., 2018; Chorowski and Jaitly, 2017; Vaswani et al., 2017), improve the model calibration (Mller et al., 2019), and de-noise incorrect labels (Lukasik et al., 2020). Our proposed boundary smoothing applies the smoothing technique to entity boundaries, rather than labels. This is driven by the observation that entity boundaries are more ambiguous and inconsistent to annotate in NER engineering. 3 To the best of our knowledge, this study is the first that focuses on the effect of smoothing regularization on NER models. 3 We note that Shen et al. (2021) also allocate a weight to the non-entity but partially matched spans; however, boundary smoothing additionally regularizes the weight of entity spans, which is intuitively crucial for mitigating over-confidence. A neural network-based NER model typically encodes the input tokens to a sequence of representations x = x 1 , x 2 , . . . , x T of length T , and then decodes these representations to task outputs, i.e., a list of entities specified by types and boundaries. We follow Yu et al. (2020) and use the biaffine decoder. Specifically, the representations x are separately affined by two feedforward networks, resulting in two representations h s  RT  d and h e  RT  d , which correspond to the start and end positions of spans. For c entity types (a non-entity type included), given a span starting at the i -th token and ending at the j -th token, a scoring vector r ij  R c can be computed as: r ij = ( h si ) T Uh ej + W ( h si  h ej  w j  i ) + b, (1) where w j  i  R d w is the ( j  i ) -th width embedding from a dedicated learnable matrix; U  R d  c  d , W  R c  (2 d + d w ) and b  R c are learnable parameters. r ij is then fed into a softmax layer:  y ij = softmax( r ij ) , (2) which yields the predicted probabilities over all entity types. The ground truth y ij  R c is an one-hot encoded vector, with value being 1 if the index corresponds with the annotated entity type, and 0 otherwise. Thus, the model can be optimized by the standard cross entropy loss for all candidate spans: LCE =  (cid:88) 0  i  j<T y T ij log( y ij ) . In the inference time, the spans predicted to be non-entity are first discarded, and the remaining ones are ranked by their predictive confidences. Spans with lower confidences would also be discarded if they clash with the boundaries of spans with higher confidences. Refer to Yu et al. (2020) for more details. Figure 1a visualizes the ground truth y ij for an example sentence with two annotated entities. The valid candidate spans cover the upper triangular area of the matrix. In existing NER models, the annotated boundaries are considered to be absolutely reliable. Hence, each annotated span is assigned Start End 0 0 1 1 2 2 3 3 4 4 5 5 6 6 7 7 8 8 9 9 (b) Smoothed boundary Figure 1: An example of hard and smoothed boundaries. with the full probability to be an entity, whereas all unannotated spans are assigned with zero probability. We refer to this probability allocation as hard boundary , which is, however, probably not the best choice. As aforementioned, the entity boundaries may be ambiguous and inconsistent, so the spans surrounding an annotated one deserve a small probability to be an entity. Figure 1b visualizes  y ij , the boundary smoothing version of y ij . Specifically, given an annotated entity, a portion of probability (cid:15) is assigned to its surrounding spans, and the remaining probability 1  (cid:15) is assigned to the originally annotated span. With smoothing size D , all the spans with Manhattan distance d ( d  D ) to the annotated entity equally share probability (cid:15)/D . After 7098 such entity probability re-allocation, any remaining probability of a span is assigned to be non-entity. We refer to this as smoothed boundary . Thus, the biaffine model can be optimized by the boundary-smoothing regularized cross entropy loss: LBS =  (cid:88) 0  i  j<T  y T ij log( y ij ) . Empirically, the positive samples (i.e., ground-truth entities) are sparsely distributed over the candidate spans. For example, the CoNLL 2003 dataset has about 35 thousand entities, which represent only 0.93% in the 3.78 million candidate spans. By explicitly assigning probability to surrounding spans, boundary smoothing prevents the model from concentrating all probability mass on the scarce positive samples. This intuitively helps alleviate over-confidence. In addition, hard boundary presents noticeable sharpness between the classification targets of positive spans and surrounding ones, although they share similar contextualized representations. Smoothed boundary provides more continuous targets across spans, which are conceptually more compatible with the inductive bias of neural networks that prefers continuous solutions (Hornik et al., 1989). Datasets We use four English NER datasets: CoNLL 2003 (Tjong Kim Sang and Veenstra, 1999), OntoNotes 5 4 , ACE 2004 5 and ACE 2005 6 ; and four Chinese NER datasets: OntoNotes 4 7 , MSRA (Levow, 2006), Weibo NER (Peng and Dredze, 2015) and Resume NER (Zhang and Yang, 2018). Among them, ACE 2004 and ACE 2005 are nested NER tasks, and the others are flat tasks. Hyperparameters For English corpora, we use RoBERTa (Liu et al., 2019) followed by a BiLSTM layer to produce the contextualized representations. For Chinese, we choose the BERT pretrained with whole word masking (Cui et al., 2019). 4 https://catalog.ldc.upenn.edu/ LDC2013T19 ; Data splits follow Pradhan et al. (2013). 5 https://catalog.ldc.upenn.edu/ LDC2005T09 ; Data splits follow Lu and Roth (2015). 6 https://catalog.ldc.upenn.edu/ LDC2006T06 ; Data splits follow Lu and Roth (2015). 7 https://catalog.ldc.upenn.edu/ LDC2011T03 ; Data splits follow Che et al. (2013). The BiLSTM has one layer and 200 hidden size with dropout rate of 0.5. The biaffine decoder follows Yu et al. (2020), with the affine layers of hidden size 150 and dropout rate 0.2. We additionally introduce a span width embedding of size 25. Note that the pretrained language models are all of the base size (768 hidden size, 12 layers), and the model is free of any additional auxiliary embeddings; this configuration is relatively simple, compared with those in related work. The boundary smoothing parameter (cid:15) is selected in { 0 . 1 , 0 . 2 , 0 . 3 } ; smoothing size D is selected in { 1 , 2 } . All the models are trained by the AdamW optimizer (Loshchilov and Hutter, 2018) with a gradient clipping at L2-norm of 5.0 (Pascanu et al., 2013). The models are trained for 50 epochs with batch size of 48. The learning rate is searched between 1e-3 and 3e-3 on the randomly initialized weights, and between 8e-6 and 3e-5 on the pretrained weights; a scheduler of linear warmup in the first 20% steps followed by linear decay is applied. Evaluation A predicted entity is considered correct if its type and boundaries exactly match the ground truth. Hyperparameters are tuned according to the F 1 scores on the development set, and the evaluation metrics (precision, recall, F 1 score) are reported on the testing set. Table 2 presents the evaluation results on four English datasets, in which CoNLL 2003 and OntoNotes 5 are flat NER corpora, whereas ACE 2004 and ACE 2005 contains a high proportion of nested entities. Compared with previous SOTA systems, our simple baseline (RoBERTa-base + BiLSTM + Biaffine) achieves on-par or slightly inferior performance. Provided the strong baseline, our experiments show that boundary smoothing can effectively and consistently boost the F 1 score of entity recognition across different datasets. With the help of boundary smoothing, our model outperforms the best of the previous SOTA systems by a magnitude from 0.2 to 0.5 percentages. Table 3 presents the results on four Chinese datasets, which are all flat NER corpora. Again, boundary smoothing consistently improves model performance against the baseline (BERT-base-wwm + BiLSTM + Biaffine) across all datasets. In addition, our model outperforms previous SOTA 7099 CoNLL 2003 Model Prec. by 2.16 and 0.55 percentages on Weibo and Resume NER datasets, and achieves comparable F 1 scores on OntoNotes 4 and MSRA. Note that almost all previous systems solve these tasks within a sequence tagging framework; this work adds to the literature by introducing a span-based approach and establishing SOTA results on multiple Chinese NER benchmarks. In five out of the above eight datasets, integrating boundary smoothing significantly increases the precision rate with a slight drop in the recall, resulting in a better overall F 1 score. This is consistent with our expectation, because boundary smoothing OntoNotes 4 Model Prec. discourages over-confidence when recognizing entities, which implicitly leads the model to establish a more critical threshold to admit entities. Given the use of well pretrained language models, most of the performance gains are relatively marginal. However, boundary smoothing can work effectively and consistently for different languages and datasets. In addition, it is easy to implement and integrate into any span-based neural NER models, with almost no side effects. flat/nested and English/Chinese datasets), to evaluate the effects of boundary smoothing parameter (cid:15) and D , as well as other components of our NER system. Boundary Smoothing Parameters We train the model with (cid:15) in { 0 . 1 , 0 . 2 , 0 . 3 } and D in { 1 , 2 } ; the corresponding results are reported in Table 4. Most combinations of the two hyperparameters can achieve higher F 1 scores than the baseline, which suggests the robustness of boundary smoothing. On the other hand, the best smoothing parameters are different across datasets, which are probably related to the languages/domains of the text, the entity types, and the annotation scheme (e.g., flat or nested NER). Hence, if the best performance is desired for a new NER task in practice, hyperpa-rameter tuning would be necessary. Label Smoothing We replace boundary smoothing with label smoothing in the span classifier. Label smoothing cannot improve, or may even impair the performance of the model, compared with the baseline (see Table 4). As aforementioned, we hypothesize that the semantic differences between the typical entity types are quite clear, so it is ineffective to smooth between them. Pretrained Language Models We test if the performance gain by boundary smoothing is robust to different baselines. For English datasets, we use BERT (Devlin et al., 2019) of the base and large sizes, and RoBERTa (Liu et al., 2019) of the large size (1024 hidden size, 24 layers). It shows that boundary smoothing can consistently increase the F 1 scores by 0.10.2 and 0.40.6 percentages for CoNLL 2003 and ACE 2005, respectively. For Chinese, we use MacBERT (Cui et al., CoNLL ACE Resume 2003 2005 NER Baseline 93.48 86.56 96.34 + BS 93.65 87.15 96.66 Baseline w/ BERT-base 91.84 84.51 + BS 92.05 84.95 Baseline w/ BERT-large 92.92 85.83 + BS 93.08 86.33 Baseline w/ RoBERTa-large 93.66 87.82 + BS 93.77 88.02 Baseline w/ MacBERT-base 96.41 + BS 96.75 Baseline w/ MacBERT-large 96.46 + BS 96.75 Baseline w/o BiLSTM 93.13 86.22 96.24 + BS 93.30 86.58 96.56 Table 5: Ablation studies of model structure. F 1 scores are reported. BS means boundary smoothing. 2020) of the base and large sizes, and boundary smoothing still performs positively and consistently, with an improvement of 0.20.3 percentage F 1 scores on Resume NER (see Table 5). It is noteworthy that boundary smoothing achieves performance gains roughly comparable to the gains by switching the pretrained language model from the base size to the large size. This suggests that the effect of boundary smoothing is quite considerable, although the performance improvements seem small in magnitude. In addition, our results show that RoBERTa substantially outperforms the original BERT on English NER. This is probably because that (1) RoBERTa is trained on much more data; and (2) RoBERTa focuses on the token-level task (i.e., masked language modeling) by removing the sequence-level objective (i.e., next sentence predic-tion), hence, it is particularly suitable for within-sequence downstream tasks, e.g., NER. This is also the reason why we choose RoBERTa for our baseline. BiLSTM Layer We remove the BiLSTM layer, directly feeding the output of pretrained language model into the biaffine decoder. The results show that this does not change the positive effect of boundary smoothing (see Table 5). In addition, absence of the BiLSTM layer will result in drops of the F 1 scores by about 0.3, 0.5 and 0.1 percentages on the three datasets. The model performance (evaluated by, e.g., accuracy or F 1 score) is certainly important. However, the confidences of model predictions are also of interest in many applications. For example, when it requires the predicted entities to be highly reliable (i.e., precision is of more priority than recall), we may filter out the entities with confidences lower than a specific threshold. However, Guo et al. (2017) have indicated that modern neural networks are poorly calibrated, and typically over-confident with their predictions. By calibration, they mean the extent to which the prediction confidences produced by a model can represent the true correctness probability. We find neural NER models also easy to become miscali-brated and over-confident. We observe that, with the standard cross entropy loss, both the development loss and F 1 score increase in the later training stage, which goes against the common perception that the loss and F 1 score should change in the opposite directions. This phenomenon is similar to the disconnect between negative likelihood and accuracy in image classification described by Guo et al. (2017). We suppose that the model becomes over-confident with its predictions, including the incorrect ones, which contributes to the increase of loss (see Appendix A for more details). To formally investigate the over-confidence issue, we plot the reliability diagrams and calculate expected calibration error (ECE). In brief, for an NER model, we group all the predicted entities by the associated confidences into ten bins, and then calculate the precision rate for each bin. If the model is well calibrated, the precision rate should be close to the confidence level for each bin (see Appendix B for more details). Figure 2 compares the reliability diagrams and ECEs between models with different smoothness (cid:15) on CoNLL 2003 and OntoNotes 5. For the baseline model ( (cid:15) = 0), the precision rates are much lower than corresponding confidence levels, suggesting significant over-confidence. By introducing boundary smoothing and increasing the smoothness (cid:15) , the over-confidence is gradually mitigated, and shifted to under-confidence ( (cid:15) = 0.3). In general, the model presents best reliability diagrams when (cid:15) is 0.1 or 0.2. In addition, the ECEs of the baseline model are 0.072 and 0.063 on CoNLL 2003 and OntoNotes 5, respectively; with (cid:15) of 0.1, the ECEs are reduced       &RQILGHQFH       3 U HF L V L RQ  (&(   (&(   (&(   (&(  In conclusion, boundary smoothing can prevent the model from becoming over-confident with the predicted entities, and result in better calibration. In addition, as mentioned previously, spans with lower confidences are discarded if they clash with those of higher confidences when decoding. With the better calibration, the model can obtain a very marginal but consistent increase in the F 1 score. How does boundary smoothing improve the model performance? We originally conjectured that boundary smoothing can de-noise the inconsistently annotated entity boundaries (Lukasik et al., 2020), but failed to find enough evidence  the 7102            performance improvement did not significantly increase when we injected boundary noises into the training data. 8 As aforementioned, positive samples are very sparse among the candidate spans. Without boundary smoothing, the annotated spans are regarded to be entities with full probability, whereas all other spans are assigned with zero probability. This creates noticeable sharpness between the targets of the annotated spans and surrounding ones, although their neural representations are similar. Boundary smoothing re-allocates the entity probabilities across contiguous spans, which mitigates the sharpness and results in more continuous targets. Conceptually, such targets are more compatible with the inductive bias of neural networks that prefers continuous solutions (Hornik et al., 1989). Li et al. (2018) have shown that residual connections and well-tuned hyperparameters (e.g., learning rate, batch size) can produce flatter minima and less chaotic loss landscapes, which account for the better generalization and trainability. Their findings provide important insights into the geometric 8 On the other hand, this cannot rule out the de-noising effect of boundary smoothing, because the synthesized boundary noises are distributed differently from the real noises. properties of non-convex neural loss functions. Figure 3 visualizes the loss landscapes for models with different smoothness (cid:15) on CoNLL 2003 and OntoNotes 5, following Li et al. (2018). In short, for a trained model, a direction of the parameters is randomly sampled, normalized and fixed, and the loss landscape is computed by sampling over this direction (refer to Appendix C for more details). The visualization results qualitatively show that, the solutions found by the standard cross entropy are relatively sharp, whereas boundary smoothing can help arrive at flatter minima. As many theoretical studies regard the flatness as a promising predictor for model generalization (Hochreiter and Schmidhuber, 1997; Jiang et al., 2019), this result may explain why boundary smoothing can improve the model performance. In addition, boundary smoothing is associated with more smoothed landscapes  the surrounding local minima are small, shallow, and thus easy for the optimizer to escape. Intuitively, such geometric property suggests that the underlying loss functions are easier to train (Li et al., 2018). and chaotic loss landscape. Boundary smoothing can effectively mitigate the sharpness, and result in loss landscapes of better generalization and trainability. In this study, we propose boundary smoothing as a regularization technique for span-based neural NER models. Boundary smoothing re-assigns entity probabilities from annotated spans to the surrounding ones. It can be easily integrated into any span-based neural NER systems, but consistently bring improved performance. Built on a simple but strong baseline (a base -sized pretrained language model followed by a BiLSTM layer, and the biaffine decoder), our model achieves SOTA results on eight well-known NER benchmarks, covering English and Chinese, flat and nested NER tasks. In addition, experimental results show that boundary smoothing leads to less over-confidence, better model calibration, flatter neural minima and more smoothed loss landscapes. These properties plausibly explain the performance improvement. Our findings shed light on the effects of smoothing regularization technique in the NER task. As discussed, boundary smoothing typically increases the overall F 1 score at the risk of a slight drop in the recall rate; hence, one may be careful to use it for recall-sensitive applications. Future work will apply boundary smoothing to more variants of span-based NER models, and investigate its effect in a broader range of information extraction tasks. We thank Yiyang Liu for his efforts in data processing, and the anonymous reviewers for their insightful comments and feedback. This work is supported by the National Natural Science Foundation of China (No. 62106248), Ningbo Science and Technology Service Industry Demonstration Project (No. 2020F041), and Ningbo Public Service Technology Foundation (No. 2021S152).
abstain objective abstain result abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain other abstain abstain abstain result objective result abstain abstain other other method other other other other other other other other method other method other other other objective other objective method abstain method abstain other abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain method abstain abstain abstain abstain abstain method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain other other other other abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain result result result abstain abstain result abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain result abstain abstain result abstain abstain abstain abstain abstain abstain abstain method abstain abstain abstain abstain abstain abstain abstain method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain objective abstain abstain result abstain abstain abstain abstain abstain other other