Exploiting sentence-level labels, which are easy to obtain, is one of the plausible methods to improve low-resource named entity recognition (NER), where token-level labels are costly to annotate. Current models for jointly learning sentence and token labeling are limited to binary classification. We present a joint model that supports multi-class classification and introduce a simple variant of self-attention that allows the model to learn scaling factors. Our model produces 3.78%, 4.20%, 2.08% improvements in F1 over the BiLSTM-CRF baseline on e-commerce product titles in three different low-resource languages: Vietnamese, Thai, and Indonesian, respectively. Neural named entity recognition (NER) has become a mainstream approach due to its superior performance (Huang et al., 2015; Lample et al., 2016; Ma and Hovy, 2016; Chiu and Nichols, 2016; Akbik et al., 2018). However, neural NER typically requires a large amount of manually labeled training data, which are not always available in low-resource languages. Training neural NER with limited labeled data can be very challenging. In this paper, we consider bridging multi-task learning (MTL) (Caruana, 1993; Ruder, 2017) and pretraining (Peters et al., 2018; Devlin et al., 2019) to leverage training signals of an auxiliary task that has a sufficiently large number of labeled data. Researchers have investigated a wide variety of auxiliary tasks and resources to boost the performance of neural NER, e.g., training coarse-grained NER (Aguilar et al., 2017), fine-tuning bilingual word embeddings (Wang et al., 2017), applying language models (Rei, 2017), integrating part-of-speech (POS) tagging (Lin et al., 2018), using cross-lingual knowledge (Feng et al., 2018), and learning paraphrases (Watanabe et al., 2019). While most of the previous studies have exploited token-level information from auxiliary tasks, a few of them have tried to use sentence-level information (Rei and S  gaard, 2018; Devlin et al., 2019). Our work is closely related to the joint labeling framework in Rei and S  gaard (2019). However, they only focused on binary classification, while we attempt to handle multi-class classification on both sentence and token levels. In this work, we focus on improving low-resource NER by exploiting large data, only having sentence-level labels. Figure 1 shows examples of product titles on an e-commerce website in Vietnamese. While the product titles with NER annotation done by our annotators are limited, those with product categories (e.g., ELECTRONICS ) labeled by sellers are abundant, which can be used to train a sentence-level classifier. 1 A key challenge is to pass useful training signals from the sentence-level classification to the token-level NER. Our contributions are as follows. We present the joint sentence and token labeling framework that enables multi-class classification equipped with a pre-training strategy (  2.1). We show that the current attention mechanisms can produce suboptimal 1 The sellers are required to assign a category when uploading the product, but such input could be noisy as well. results and propose a simple approach that allows the model to learn scaling factors to obtain a proper attention distribution (  2.2). Results on product title texts indicate that the proposed method is effective for low-resource NER across three different languages: Vietnamese, Thai, and Indonesian. Figure 2 shows the architecture of our joint sentence and token labeling model. Our model is based on hard parameter sharing (Ruder, 2017) in which the hidden layers are shared between two tasks. The task-specific layers include a conditional random field (CRF) layer for NER and a linear layer for sentence classification. 2 Unlike the standard MTL, which trains multiple tasks at once and expects the model to perform well on all tasks (Hashimoto et al., 2017; Rei and S  gaard, 2019), the goal of our work is to improve the performance of the main task (NER) using the auxiliary task (sentence classification) for creating pre-trained representations and as a regularizer. Shared layers Let w 1 , . . . , w T be an input token sequence, where w t denotes the t -th token in the sequence. We represent each w t using a pre-trained word embedding e t  R d e , where d e is the dimensionality of word embeddings. We do not fine-tune word embeddings but project them into a new space 2 We use the term sentence to conform with the literature, although our data are not always complete sentences. using x t = W 1 e t , where W 1  R d e  d e is a trainable weight matrix. We then feed the projected embedding sequence X = [ x 1 , . . . , x T ]  RT  d e to a bidirectional long short-term memory (BiLSTM) layer to obtain a forward hidden state sequence  H = [  h 1 , . . . ,  h T ]  RT  dh 2 and a backward hidden state sequence  H = [  h 1 , . . . ,  h T ]  RT  dh 2 , where d h is the number of hidden units. We concatenate the hidden states of both directions to obtain the final hidden representation H = [ h 1 , . . . , h T ]  RT  d h , where h t = concat(  h t ,  h t )  R d h . We can either use H for both the sentence classification and NER tasks directly or apply an attention mechanism on it to help the model focus on particular tokens (detailed in  2.2). Sentence classification We create a fixed size vector by applying max-pooling (Collobert et al., 2011; Conneau et al., 2017) over H , which encourages the model to capture the most useful local features encoded in the hidden states. We feed the fixed size global feature vector to a linear layer to obtain the unnormalized predicted scores for each class. Let K be the number of target classes, s k be the k -th normalized predicted score after applying a softmax function, and t  RK be the one-hot encoded true label. To train the sentence classification model, we minimize the multi-class cross-entropy loss: LC =  1 NN (cid:88) i =1 K (cid:88) k =1 t ( i ) k log( s ( i ) k ) , (1) where i denotes the sentence index, and N is the number of training examples. We not only train the sentence classification and NER models jointly but also pre-train the sentence classification model using a sufficiently large number of training examples with sentence-level labels only. We expect that pre-trained hidden representations would help the model generalize better on our main task, as described below. NER Following Huang et al. (2015); Lample et al. (2016), we feed H to a CRF layer to obtain the probability of a label sequence y . To train the NER model, we minimize the negative log-likelihood of the correct label sequences over the training set: LNER =  1 NN (cid:88) i =1 log p ( y ( i ) | H ( i ) ) . Joint labeling objective Combining Eqs. (1) and (2), we obtain: LJOINT = LNER +  LC , (3) where  is the balancing parameter. The LC acts as a regularization term, which helps in reducing the risk of overfitting on our main task. We first consider a soft-attention mechanism (Shen and Lee, 2016), which is used in Rei and S  gaard (2018, 2019). This method is computationally efficient because the attention distribution a  RT over tokens in a sentence is computed from the final hidden representation without considering relationships between hidden states. Specifically, the new final representation H (cid:48)  RT  d h can be derived as follows: H (cid:48) = H + H  a , a =  a (cid:80) Tj =1  a j ,  a =  ( w 2 g + b 2 ) , g = tanh( W 3 H (cid:62) + b 3 ) , (4) where w 2  R d h , b 2  R , W 3  R d h  d h , b 3  R d h are trainable parameters, and  denotes the column-wise matrix-vector multiplication. We use a residual connection (He et al., 2016) between the input hidden representation and the attention output as shown in Figure 2. H (cid:48) can be fed to NER and sentence classification. We further explore attention mechanisms that take into account the relationships between hidden states. In particular, we apply the multi-head self-attention mechanism in Transformer (Vaswani et al., 2017), which has shown promising results in many applications (Radford et al., 2018; Devlin et al., 2019). We replace Eq. (4) with: H (cid:48) = H + concat(head 1 , . . . , head n ) WO , head j = attention( Q j , K j , V j ) , Q j , K j , V j = HW Qj , HW Kj , HW Vj , (5) where W Qj , W Kj , W Vj  R d h  dhn ; WO  R d h  d h are trainable parameters, and n is the number of parallel heads. The attention function can be computed by: attention( Q , K , V ) = softmax( QK (cid:62)  ) V . We drop the head index j for simplicity and introduce the scaling factor   R . When setting  = (cid:112) d h /n , Eq. (6) falls back to the standard scaled dot-product attention in Transformer. Yan et al. (2019) observed that the scaled dot-product attention produces poor results for NER and proposed the un-scaled dot-product attention, where  = 1 . In this work, we consider  as the softmax temperature (Hinton et al., 2015) that allows adjusting the probability distribution of a softmax output. Using a higher temperature yields a softer attention distribution. However, a sharper attention distribution might be more suitable for NER because only a few tokens in the sentence are named entities. Instead of setting  to 1 or (cid:112) d h /n , we propose to learn the scaling factors   RT for each token. We modify Eq. (6) with: attention( Q , K , V ) = softmax( QK (cid:62)  ) V ,  = min(ReLU( w 4 H (cid:62) + b 4 ) , (cid:112) d h /n ) + 1 , (7) where w 4  R d h , b 4  R are the trainable parameters. Since the ReLU activation function produces output values in the range [0 ,  ) , the t -th element of  is bounded in the range [1 , 1 + (cid:112) d h /n ] . This allows the model to dynamically adapt  without increasing much computational cost. The data used in our experiments are product titles obtained from major e-commerce websites in Southeast Asian countries during May-June, 2019. They cover three languages, including Vietnamese (VI), Thai (TH), and Indonesian (ID). A product title is a brief, information-rich description (less than 200 characters) written by the sellers. We hired annotators and linguists for each language to annotate the product titles based on our definitions and annotation guidelines. After the annotation process, we obtained 2,000 product titles per language labeled with 6 product attribute NER tags, including PRODUCT , BRAND , CONSUMER _G ROUP , MATERIAL , PATTERN , and COLOR . For each language, we split the data into 1,000/500/500  training/development/test sets. 3 The statistics of NER tags can be found in Table 3 (see Appendix A). For some NER tags, especially PRODUCT , the number of tags is much larger than the number of examples used. One reason is that the sellers writing a product title tend to include multiple different expressions referring to the same entity (near-synonyms), with the likely intention of acquiring more hits from potential customers. Using English to illustrate:  Genuine Leather Sling Bag Crossbody Bag Messenger bag for Men Women Office Laptop , the underlined elements are 3 PRODUCT and 2 CONSUMER _G ROUP entities. The other reason is that in one product title, it is common to find repeated identical expressions in the same language, as well as the same entity words appearing in English. Using a VI example to illustrate:  T-Shirt o thun in phn quang Ao thun Nam Ao thun n o thun phong cch Nam N , the underlined elements refer to the same product ( t-shirt ), appearing multiple times in VI and in English. We implement our model on top of the Flair framework (Akbik et al., 2019), which has recently achieved state-of-the-art results in various sequence labeling tasks. Following Lample et al. (2016), we use the IOBES tagging scheme. We use the pre-trained word embeddings of fastText 4 (Bojanowski et al., 2016) with d e = 300 dimensions for each language and a single-layer BiLSTM with d h = 512 hidden units. We apply a locked dropout (Merity et al., 2018) with the probability of 0.5 before and after the BiLSTM layer and to the attention output before the residual connection. For the multi-head self-attention layer, we adapt the implementation of The Annotated Transformer (Rush, 2018) 5 and use its default hyperparameters. We train all models using Adam (Kingma and Ba, 2015) with the batch size of 32, the learning rate of 1e-3, and the gradient clipping of 5. We initialize all model parameters by sampling from U (  0 . 1 , 0 . 1) . We set  in Eq. (3) to 1. We use the same parameter setting for all languages. We apply early stopping in which the learning rate decays by 3 For TH, 941 training examples remain after removing annotation errors. 0.5 if the F1 score on the NER development set does not improve 3 times. We train until the learning rate drops below 1e-5, or the training epochs reach 100. We collect unannotated product titles for each language and group them into 6 main categories, including FASHION , HEALTH _B EAUTY , ELECTRONICS , HOME _F URNITURE , MOTORS , and OTHER . Since the number of product titles is different from one language to another, we can create 360k/30k, 1.2M/60k, 864k/60k  train-ing/development sets for VI, TH, and ID, respectively. Since product titles are not segmented in TH, we segment them using a character cluster-based method simplified from the hybrid model of Kruengkrai et al. (2009). We implement our word segmenter based on CRFsuite (Okazaki, 2007) and train the model using the BEST corpus (Kosawat et al., 2009). We pre-train the classification models for each language. Since our batch size is relatively small compared to the training data size, we find it suffices to train for 2 epochs. The F1 scores on the development sets are 90.08%, 89.79%, and 91.91% for VI, TH, and ID, respectively. The pre-trained model parameters are used to initialize the projection and BiLSTM layers. We run each experiment 10 times using different random seeds and report the average F1 score. All experiments are run on NVIDIA Tesla P100 GPUs. Table 1 shows the results of various models on the test sets. The Joint models consistently show improvements over the NER-only models, while the Joint + Pre-trained models further boost the F1 scores. These results suggest that the proposed framework is effective for all three languages. The Joint + Pre-trained model with the Self + Learned attention mechanism achieves the best F1 scores at 62.16%, 61.54%, and 76.10% (i.e., 3.78%, 4.20%, and 2.08% improvements over the NER-only baselines) for VI, TH, and ID, respectively. In addition, we experiment using simple data augmentation. The +10k and +50k rows in Table 1 indicate the number of additional training examples automatically labeled using a dictionary created from the training set. We do not observe any improvement in both the development and test Model Attention VI TH ID NER-only (+10k)  53.47 52.47 74.22 NER-only (+50k)  51.12 50.35 71.60 NER-only  58.38 57.34 74.02 Soft 58.18 57.49 74.20 Self + Scaled 58.82 57.80 74.55 Self + Un-scaled 59.68 58.53 75.24 Self + Learned 60.18 58.63 74.83 Joint  59.47 58.81 74.67 Soft 59.50 58.82 74.88 Self + Scaled 59.34 58.46 75.03 Self + Un-scaled 60.58 59.56 75.66 Self + Learned 60.25 59.35 75.18 Joint + Pre-trained  61.26 60.27 75.86 Soft 61.05 60.50 75.80 Self + Scaled 61.80 61.32 75.90 Self + Un-scaled 62.09 61.45 76.01 Self + Learned 62.16 61.54 76.10 Table 1: F1 scores on the test sets. Table 2 shows the model ablations for our best configuration, the Joint + Pre-trained model with the Self + Learned attention mechanism. Feeding the attention output to the CRF layer without the residual connection leads to a consistent drop in the F1 scores, although it shows a less pronounced effect on TH. The results indicate that the residual connection is a useful component in our architecture. Adding the attention output to the hidden representation without applying the locked dropout (i.e., setting the dropout probability to 0) hurts the F1 scores on VI and TH but shows an improvement on ID, suggesting that fine-tuning the dropout rate could help boost the F1 scores. Our Self + Learned scaling approach shows the competitive results for the NER-only model and achieves the best results when training in tandem with the Joint + Pre-trained model. The Soft attention mechanism (Shen and Lee, 2016; Rei and S  gaard, 2019) shows slight or no improvements, suggesting that considering relationships between hidden states when computing the attention distribution is crucial for the NER task. The Self + Unscaled approach (Yan et al., 2019) yields better F1 scores than the Self + Scaled approach (Vaswani et al., 2017) for all configurations, suggesting that a sharper attention distribution is helpful for the NER task. Although VI, TH, and ID are used in Southeast Asia, they do not belong to the same language fam-ily and have different writing systems and scripts (i.e., VI = Austroasiatic; TH = Kra-Dai; ID = Aus-tronesian). Handling these three languages without much engineering effort reflects the generalizability of our method. Furthermore, we examine whether our method still provides improvements, even if the NER training data size increases. We create an additional set of 2k labeled examples for VI and add them to the training set (3k in to-tal). The baseline NER-only produces 66.81% F1, while Joint + Pre-trained with Self + Learned achieves 69.26% F1 (i.e., 2.45% improvement). We have shown that the proposed joint sentence and token labeling model is remarkably effective for low-resource NER in three different languages: Vietnamese, Thai, and Indonesian. Our model supports multi-class classification where the sentence and token labels can be weakly related, which indicates the potential of our model for many other real-world applications. Using a larger amount of general domain texts to build pre-trained representations (Peters et al., 2018; Radford et al., 2018; Devlin et al., 2019; Clark et al., 2020) can complement with our model and is one of the directions that we plan to take in future work. We thank the anonymous reviewers for their constructive comments. Kruengkrai is grateful for support from National Institute of Informatics, Japan.
abstain abstain method method abstain abstain abstain method other abstain abstain method result abstain method abstain objective method result abstain abstain method method abstain method abstain method method abstain method method method method method abstain method method method method method abstain method method method abstain other method abstain method method method other abstain method abstain abstain abstain method abstain abstain method method abstain abstain abstain result abstain abstain abstain result abstain abstain abstain abstain abstain abstain abstain result abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain result abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain result result abstain result abstain result abstain abstain abstain abstain abstain abstain abstain objective abstain method other other