We investigate subword information for Chinese word segmentation, by integrating sub word embeddings trained using byte-pair encoding into a Lattice LSTM (LaLSTM) network over a character sequence. Experiments on standard benchmark show that subword information brings significant gains over strong character-based segmentation models. To our knowledge, this is the first research on the effectiveness of subwords on neural word segmentation. Chinese word segmentation (CWS) is a traditional NLP task (Sproat et al., 1996), the features for which have been a central research topic. Statistical methods consider characters (Xue et al., 2003), subwords (Zhang et al., 2006), and words (Zhang and Clark, 2007) as input features. Among these, both characters (Chen et al., 2015a) and words (Zhang et al., 2016; Cai and Zhao, 2016; Yang et al., 2017) have also shown useful in recent neural models. However, how to utilize the subword features in neural networks has not been investigated yet. In this paper, we fill this gap by proposing a subword-based neural word segmentor, by integrating two strands of works: the byte pair encoding (BPE) algorithm (Gage, 1994) and the lattice LSTM structure (Zhang and Yang, 2018). The BPE algorithm constructs a subword list from raw data and lattice LSTM introduces subwords into character LSTM representation. In particular, our baseline is a BiLSTM-CRF segmentor (Chen et al., 2015b) and we replace LSTM with lattice LSTM using subwords to encode character composition information. Our code 1 is based on NCRF++ (Yang and Zhang, 2018). Compared with character-based neural segmentors, our model can utilize abundant character combination (subword) information, which is effective to disambiguate characters. For example, in Figure 1, the subword    (Academy) ensures that the character    means Academy(noun) rather than study(verb). Compared with the word-based neural models (Zhang et al., 2016; Cai and Zhao, 2016), ambiguous subwords in a context can provide additional information for disambiguation. For instance, the subword     (Academy of Sciences) and    (Academy) can be useful in determining the correct segmentation, which is     /(Academy of Sciences/). To our knowledge, we are the first to use subwords in a neural network segmentor. We investigate the contributions of subword lexicons and their pretrained embeddings through controlled experiments. Results on four benchmarks show that the proposed model can give comparable results with state-of-the-art models. State-of-the-art statistical segmentors use either sequence labeling methods e.g. CRF (Lafferty et al., 2001) with character features (Peng et al., 2004; Zhao et al., 2006) or the transition-based models with word features (Zhang and Clark, 2007; Sun, 2010). Neural segmentors (Chen et al., LSTM   BLSTM   ELSTM   BLSTM   MLSTM   ELSTM   BLSTM  </E>  E Cell  Cell  Cell  Cell  Cell  CRF Layer LSTM Layer Unichar emb Bichar emb Word emb Figure 2: Models. Only forward LSTM is illustrated here. 2015a; Cai and Zhao, 2016) generally take the same framework except using neural networks as automatic feature extractor. Lattice LSTM was proposed by Zhang and Yang (2018) for Chinese named entity recognition (NER). It integrates the character sequence features and all lexicon word embeddings that match a character subsequence in the input into a sequence labeling model. Zhu et al. (2016) proposed a DAG-structured LSTM structure which is similar to the lattice LSTM model but binarizing the paths in the merging process. Chen et al. (2017) also built a DAG-LSTM structure for word segmentation but without memory cells. Our model consistently gives better performance. BPE is a data compression algorithm (Gage, 1994) which has been used in neural machine translation (NMT) by capturing the most frequent subwords instead of words (Sennrich et al., 2016). Here we use it for collecting subwords in Chinese, similar to the use in Chinese NMT. We take the state-of-the-art LSTM-CRF framework as our baseline. For an input sentence with m characters s = c 1 , c 2 , . . . , c m , where c i denotes the i th character, the segmentor is to assign each character c i with a label l i . Figure 2 shows the segmentor framework on input character sequence        (Fellow of the Chinese Academy of Sciences), where the black part represents the baseline LSTM-CRF model and the red part shows the lattice structure. As shown in Figure 2, for each input character c i the corresponding character unigram embeddings and character bigram embeddings are represented as e c i and e c i c i +1 , respectively. The character representation is calculated as following: x i = e c i  e c i c i +1 , (1) where  represents concatenate operation. Unlike Zhang et al. (2016) which uses a window to strengthen the local features, or Zhou et al. (2017) which adds a non-linear layer before the LSTM layer, we feed { x 1 , x 2 , . . . , x m } into a bidirectional LSTM:  h 1 ,  h 2 , . . . ,  h m =  LST M ( x 1 , x 2 , . . . , x m )  h 1 ,  h 2 , . . . ,  h m =  LST M ( x 1 , x 2 , . . . , x m ) , (2) where  LST M and  LST M represent the forward and backward LSTM, respectively. The detailed equations are listed in Appendix . The hidden vector of character c i is h i =  h i   h i (3) 3.2 Lattice LSTM The lattice LSTM adds shortcut paths (red part in Figure 2) to LSTM. The input of the lattice LSTM model is character sequence and all subsequences which are matched words in a lexicon D , collected from BPE. Following Zhang and Yang (2018), we use w b,e to represent the subsequence that has a start character index b and a end character index e , and the embeddings of the subsequence is represented as e w b,e . During the forward lattice LSTM calculation, the cell in Figure 2 of a subsequence w b,e takes the hidden vector of the start character h b and the subsequence embeddings e w b,e as input, an extra Parameter Value Parameter Value char emb size 50 bigram emb size 50 word emb size 50 subword emb size 50 char dropout 0.5 lattice dropout 0.5 LSTM layer 1 LSTM hidden 200 learning rate lr 0.01 lr decay 0.05 Table 1: Hyper-parameter values. LSTM cell is applied to calculate the memory vector of the sequence c w b,e : c w b,e = LST MCell ( h b , e w b,e ) , (4) where the LST MCell is a simplified LSTM unit which calculate the memory only. The output memory vector c w b,e links to the end character c e to calculate its hidden vector  h e . For character with multiple memory cell inputs 2 , we assign a gate for each subsequence input to control its contribution. The detailed equations are listed in Appendix . The final output  h i includes both the character sequence history information and all the matched subsequence information. We use a standard CRF layer for inference (de-tails in Appendix ). Viterbi (1967) is used to find the highest scored label sequence over the input. During training, we choose sentence-level log-likelihood as the loss function. where y i is the gold labels of sentence s i . Data . We evaluate our model on four standard Chinese word segmentation datasets: CTB6, PKU, MSR, and Weibo. PKU and MSR are taken from the SIGHAN 2005 bake-off (Emerson, 2005) and Weibo dataset is the NLPCC 2016 shared task (Qiu et al., 2016), standard split are used. We take CTB6 as the main dataset and split the train/dev/test following Zhang et al. (2016). The statistics of the datasets are listed in Appendix . Hyperparameters . We keep the hyperparameters the same among all datasets. Standard gradient descent (SGD) with a learning rate decay is used as the optimizer. The embedding sizes of character unigram/bigram and subword are all 50. Dropout (Srivastava et al., 2014) is used on both the character input and the subword input to prevent over-fitting. Details are listed in Table 1. Embeddings . We take the same character unigram and bigram embeddings as Zhang et al. (2016), who pretrain embeddings using word2vec (Mikolov et al., 2013) on Chinese Gigaword 3 . The vocabulary of subword is constructed with 200000 merge operations and the subword embeddings are also trained using word2vec (Heinzerling and Strube, 2018). Trie (Fredkin, 1960) is used to accelerate lattice building. All the embeddings are fine-tuned during training. We perform experiments on the CTB6 development dataset to investigate the contribution of character bigram information and the subword information. Figure 3 shows the iteration curve of F-scores against different numbers of training iterations with different character representations. Bigram represents the model using both character unigram and bigram information (embed-ding concatenation). Character bigram information can improve the baseline significantly. When the LaLSTM+Subword structure is added, the model performance is further improved. This shows that subword information has a great ability to disambiguate the characters. Zhang and Yang (2018) observed that character bigram information has a negative effect in lattice LSTM on Chinese NER task, while we find a different result on Chinese word segmentation where character bigram information gives significant improvements in the lattice LSTM. This is likely because character bigrams are informative but ambiguous. They can provide more useful character disambiguation evidence in segmentation than in NER where lattice LSTM works well in disambiguating characters. Table 2 shows the main results and the recent state-of-the-art neural CWS models. Zhang et al. (2016) integrated both discrete features and neural features in a transition-based framework. Xu and Sun (2016) proposed the dependency-based gated recursive neural network to utilize long distance dependencies. Yang et al. (2017)  utilized pretrained character representations from multitasks. We examine their non-pretrained model performance for fair comparison. Ma et al. (2018) built a bidirectional LSTM model with carefully hyperparame-ter selection. These methods are orthogonal to and can be integrated into our lattice structure. As shown in Table 2, the subword lattice LSTM gives significant improvements on all evaluated datasets. In the PKU dataset, our model is slightly behind Xu and Sun (2016) which preprocesses the dataset by replacing all the Chinese idioms, lead-10< 30 50 70 90 >90 Sentence length 0.960 0.965 0.970 F 1 v a l u e LaLSTM+Subword Baseline Figure 4: F1-value against the sentence length. ing the comparison not entirely fair. Our model gives the best performance on MSR and Weibo datasets, which demonstrates that subword encoding can help the lattice LSTM model gives comparable performance to the state-of-the-art word segmentation models. Lexicon and Embeddings . To distinguish the contribution of subword lexicon and their pretrained embeddings, we conduct a set of experiments by using the same subword lexicon with randomly initialized embeddings 4 on CTB6 data. As shown in Table 3, the contribution of the error reduction by the lexicon is 4 . 5% . While 6 . 9% error reduction comes from both lexicon and pretrained embeddings. We can estimate that the contribution of pretraining is (6 . 9%  4 . 5%) = 2 . 4% . This roughly shows that both lexicon and pretraining are useful to lattice LSTM, and the former contributes more than the latter. OOV Analysis . Table 3 also shows the recall of in-vocabulary ( RIV ) and out-of-vocabulary ( ROOV ) words, respectively. As shown in the table, the ROOV can be largely improved with the lattice structure ( 2 . 43% absolute improvement). Sentence Length . We compare the baseline model with our proposed model on the sentence length distribution in Figure 4. The performance of the baseline has a valley in around 30-character length and decreases when the sentence length over 90. This phenomenon has also been observed in transition-based neural segmentor Yang et al. (2017). While LaLSTM+Subword gives a more stable performance along sentence length. Subword Coverage in lexicon . Table 4 5 shows the subword coverage rate in four datasets. Subword level coverage is consistently higher than the entity level coverage in Zhang and Yang (2018). We can see that higher subword coverage (PKU/MSR, > 90% ) gives better error reduction rate. Weibo dataset gets the minimum improvement due to the low subword coverage. Case Study . Figure 5 shows an example of CTB6 test dataset. In this example, there are two matched subwords       (BiologicalDiversity) and     (Diversity) which can guide the segmentor to get the right split of      (DiversityDay), which is segmented incorrectly by the baseline. We examined the effectiveness of subwords for neural CWS. Subwords are deduced using BPE, and then integrated into a character-based neural segmentor through lattice LSTM. Results on four benchmarks show that subword brings significant improvements over a character baseline, and our proposed model gives comparable performances to the best systems on all datasets. Our experiments also showed that the matched subwords contribute more than embedding pertaining, which 5 #Word is the word number in the corresponding dataset, #Match is the matched words number between the dataset and subword lexicon, #Ratio = #Match #Word represents the subword coverage rate. #ER is the error reduction compared with baseline model. indicates that the lattice LSTM structure with do-main lexicons can be useful for cross-domain segmentation training. We thank the anonymous reviewers for their insightful comments. Yue Zhang is the corresponding author.
objective abstain objective abstain abstain abstain abstain abstain abstain method abstain method abstain abstain abstain objective objective abstain other other other other other other abstain other method method abstain abstain abstain abstain abstain method abstain abstain abstain abstain method other abstain abstain abstain abstain abstain method abstain method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain result abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain result abstain abstain abstain abstain abstain abstain objective objective abstain abstain other other