We propose a new type of representation learning method that models words, phrases and sentences seamlessly. Our method does not depend on word segmentation and any human-annotated resources (e.g., word dictionaries), yet it is very effective for noisy corpora written in unsegmented languages such as Chinese and Japanese. The main idea of our method is to ignore word boundaries completely (i.e., segmentation-free ), and construct representations for all character n -grams in a raw corpus with embeddings of compositional subn grams. Although the idea is simple, our experiments on various benchmarks and real-world datasets show the efficacy of our proposal. Most existing word embedding models (Mikolov et al., 2013; Pennington et al., 2014; Bojanowski et al., 2017) take a sequence of words as their input. Therefore, the conventional models are dependent on word segmentation (Yang et al., 2017; Shao et al., 2018), which is a process of converting a raw corpus (i.e., a sequence of characters) into a sequence of segmented character n -grams. After the segmentation, the segmented character n -grams are assumed to be words, and each word's representation is constructed from distribution of neighbour words that co-occur together across the estimated word boundaries. However, in practice, this kind of approach has several problems. First, word segmentation is difficult especially when texts in a corpus are noisy or unsegmented (Saito et al., 2014; Kim et al., 2018). For example, word segmentation on social network service (SNS) corpora, such as Twitter, is a challenging task since it tends to include many misspellings, informal words, neologisms, and even emoticons. This problem becomes more severe in unsegmented languages, such as Chinese and Japanese, whose word boundaries are not explicitly indicated. Second, word segmentation has ambiguities (Luo et al., 2002; Li et al., 2003). For example, a compound word      (linear algebra) can be seen as a single word or sequence of words, such as   |    (linear | algebra). Word segmentation errors negatively influence subsequent processes (Xu et al., 2004). For example, we may lose some words in training corpora, leading to a larger Out-Of-Vocabulary (OOV) rate (Sun et al., 2005). Moreover, segmentation errors, such as segmenting  (yesterday) as  |  (tree | brain), produce false co-occurrence information. This problem is crucial for most existing word embedding methods as they are based on distributional hypothesis (Harris, 1954), which can be summarized as:  a word is characterized by the company it keeps  (Firth, 1957). To enhance word segmentation, some recent works (Junyi, 2013; Sato, 2015; Jeon, 2016) made rich resources publicly available. However, maintaining them up-to-date is difficult and it is infeasible for them to cover all types of words. To avoid the negative impacts of word segmentation errors, Oshikiri (2017) proposed a word embedding method called segmentation-free word embedding ( sembei ). The key idea of sembei is to directly embed frequent character n -grams from a raw corpus without conducting word segmentation. However, most of the frequent n -grams are non-words (Kim et al., 2018), and hence sembei still suffers from the OOV problems. The fundamental problem also lies in its extension (Kim et al., 2018), although it uses external resources to reduce the number of OOV. To handle OOV problems, Bojanowski et al. (2017) proposed a novel compositional word embedding method with subword modeling, called subword-information skipgram ( sisg ). The key idea of sisg is to extend the notion of vocabulary to include subwords,          linear algebra learn study -ing ! number learn algebra linear algebra endeavor learn study mathmatics algebra linear algebra studying Corpus namely, substrings of words, for enriching the representations of words by the embeddings of its subwords. In sisg , the embeddings of OOV (or unseen) words are computed from the embedings of their subwords. However, sisg requires word segmentation as a prepossessing step, and the way of collecting co-occurrence information is dependent on the results of explicit word segmentation. For solving the issues of word segmentation and OOV, we propose a simple but effective unsupervised representation learning method for words, phrases and sentences, called segmentation-free compositional n -gram embedding ( scne ). The key idea of scne is to train embeddings of character n -grams to compose representations of all character n -grams in a raw corpus, and it enables treating all words, phrases and sentences seamlessly (see Figure 1 for an illustrative explanation). Our experimental results on a range of datasets suggest that scne can compute high-quality representations for words and sentences although it does not consider any word boundaries and is not dependent on any human annotated resources. Our method scne successfully combines a subword model (Zhang et al., 2015; Wieting et al., 2016; Bojanowski et al., 2017; Zhao et al., 2018) with an idea of character n -gram embedding (Os-x hikiri, 2017; Kim et al., 2018). In scne , the vector representation of a target character n -gram is defined as follows. Let x 1 x 2    x N be a raw unsegmented corpus of N characters. For a range i, i + 1 , . . . , j specified by index t = ( i, j ) , 1  i  j  N , we denote the substring x i x i +1    x j as x ( i,j ) or x t . In a training phase, scne first counts frequency of character n -grams in the raw corpus to construct n -gram set V by collecting M most frequent n -grams with n  n max , where M and n max are hyperparameters. For any target character n -gram x ( i,j ) = x i x i +1    x j in the corpus, scne constructs its representation v x ( i,j )  R d by summing the embeddings of its subn grams as follows: v x ( i,j ) = (cid:88) s  S ( x ( i,j ) ) z s , where S ( x ( i,j ) ) = { x ( i (cid:48) ,j (cid:48) )  V | i  i (cid:48)  j (cid:48)  j } consists of all subn -grams of target x ( i,j ) , and the embeddings of subn -grams z s  R d , s  V are model parameters to be learned. The objective of scne is similar to that of Mikolov et al. (2013), (cid:88) t D   (cid:88) c C ( t ) log  (cid:16) v (cid:62) x t u x c (cid:17) + k (cid:88)  s  P neg log  (cid:16)  v (cid:62) x t u  s (cid:17)  , where  ( x ) = 1 1+exp(  x ) , D = { ( i, j ) | 1  i  j  N, j  i + 1  n target } , and C (( i, j )) = { ( i (cid:48) , j (cid:48) ) | x ( i (cid:48) ,j (cid:48) )  V, j (cid:48) = i  1 or i (cid:48) = j + 1 } . D is the set of indexes of all possible target n -grams in the raw corpus with n  n target , where n target is a hyperparameter. C ( t ) is the set of indexes of contexts of the target x t , that is, all character n grams in V that are adjacent to the target (see Figures 1 and 2). The negative sampling distribution P neg of  s  V is proportional to its frequency in the corpus. The model parameters z s , u  s  R d , s,  s  V , are learned by maximizing the objective. We set n target = n max in our experiments. Although we examine frequent n -grams for simplicity, incorporating supervised word boundary information or byte pair encoding into the construction of compositional n -gram set would be an interesting future work (Kim et al., 2018; Sennrich et al., 2016; Heinzerling and Strube, 2018). To avoid the problems of word segmentation, Oshikiri (2017) proposed segmentation-free word embedding ( sembei ) (Oshikiri, 2017) that considers the M -most frequent character n -grams as individual words. Then, a frequent n -gram lattice is constructed, which is similar to a word lattice used in morphological analysis (see Figure 3). Finally, the pairs of adjacent n -grams in the lattice are considered as target-context pairs and they are fed to existing word embedding methods, e.g., skipgram (Mikolov et al., 2013). Although sembei is simple, the frequent n -gram vocabulary tends to include a vast amount of nonwords (Kim et al., 2018). Furthermore, its vocabulary size is limited to M , hence, sembei can not avoid the undesirable issue of OOV. The proposed scne avoids these problems by taking all possible character n -grams as embedding targets. Note that the target-context pairs of sembei are fully contained in those of scne (see Figure 1). To overcome the problem of OOV in sembei , Kim et al. (2018) proposed an extension of sembei called word-like n -gram embedding ( wne ). In wne , the n -gram vocabulary is fil-tered to have more vaild words by taking advantage of a supervised probabilistic word segmenter. Although wne reduce the number of non-words, there is still the problem of OOV since its vocabulary size is limited. In addition, wne is dependent on word segmenter while scne does not. To deal with OOV words as well as rare words, Bojanowski et al. (2017) proposed subword information skip-gram ( sisg ) that enriches word embeddings with the representations of its subwords, i.e., sub-character n -grams of words. In sisg , a vector representation of a target word is encoded as the sum of the embeddings of its subwords. For instance, subwords of length n = 3 of the word where are extracted as <wh, whe, her, ere, re> , where  < , >  are special symbols added to the original word to represent its left and right word boundaries. Then, a vector representation of where is encoded as the sum of the embeddings of these subwords and that of the special sequence <where> , which corresponds to the original word itself. Although sisg is powerful, it requires the information of word boundaries as its input, that is, semantic units need to be specified when encoding targets. Therefore, it cannot be directly applied to unsegmented languages. Unlike sisg , scne does not require such information. The proposed scne is much simpler, but due to its simpleness, the embedding target of scne should contains many non-words, which seems to be a problem (see Figure 1). However, our experimental results show that scne successfully captures the semantics of words and even sentences for unsegmented languages without using any knowledge of word boundaries (see Section 3). In this section, we perform two intrinsic and two extrinsic tasks at both word and sentence level, focusing on unsegmented languages. The implementation of our method is available on GitHub 1 . Baselines : We use skipgram (Mikolov et al., 2013), sisg (Bojanowski et al., 2017) and sembei (Oshikiri, 2017) as word embedding baselines. For sentence embedding, we first test simple baselines obtained by averaging the word vectors over a word-segmented sentence. In addition, we examine several recent successful sentence embedding methods, pv-dbow , pv-dm (Le and Mikolov, 2014) and sent2vec (Pagliardini et al., 2018) in an extrinsic task. Note that both scne and sembei have embeddings of frequent character n -grams as their model parameters, but 1 www.github.com/kdrl/SCNE 10 50 100 200 300 Corpus size (MB) 20 30 40 50 60 70 Sp e a r m a n r a n k c o rr e l a t i o n skipgram rich sembei-sum sisg rich scne sembei 10 50 100 200 300 Corpus size (MB) 70 75 80 Figure 4: Word (left) and sentence (right) similarity tasks on portions of Chinese Wikipedia corpus. the differences come from training strategies, such as embedding targets and the way of collecting co-occurrence information (see Section 2.1 for more details). For contrasting scne with sembei , we also propose a variant of sembei (denoted by sembei-sum ) as one of baselines, which composes word and sentence embeddings by simply summing up the embeddings of their subn -grams which are learned by sembei . Hyperparameters Tuning : To see the effect of rich resources for the segmentation-dependent baselines, we employ widely-used word segmenter with two settings: Using only a basic dictionary ( basic ) or using a rich dictionary together ( rich ). The dimension of embeddings is 200 , the number of epochs is 10 and the number of negative samples is 10 for all the methods. The n -gram vocabulary size M = 2  10 6 is used for sisg , sembei and scne . The other hyperparameters, such as learning rate and n max , are carefully adjusted via a grid search in the validation set. In the word similarity task, 2 -fold cross validation is used for evaluation. In the sentence similarity task, we use the provided validation set. In the downstream tasks, vector representations are combined with a supervised logistic regression classifier. We repeat training and testing of the classifier 10 times, while the prepared dataset is randomly split into train ( 60% ) and test ( 40% ) sets at each time, and the hyperparameters are tuned by 3 -fold cross validation in the train set. We adopt mean accuracy as the evaluation metric. See Appendix A.1 for more experimental details. We measure the ability of models to capture semantic similarity for words and sentences in Chinese; see Appendix A.2 for the experiment in Japanese. Given a set of word pairs, or sentence pairs, and their human annotated similarity scores, we calculated Spearman's rank correlation between the cosine similarities of the embeddings and the scores. We use the dataset of Jin and Wu (2012) and Wang et al. (2017) for Chinese word and sentence similarity respectively. Note that the conventional models, such as skipgram , cannot provide the embeddings for OOV words, while the compositional models, such as sisg and scne , can compute the embeddings by using their subword modeling. In order to show comparable results, we use the null vector for these OOV words following Bojanowski et al. (2017). Results : To see the effect of training corpus size, we train all models on portions of Wikipedia 2 . The results are shown in Figure 4. As it can be seen, the proposed scne is competitive with or outperforms the baselines for both word and sentence similarity tasks. Moreover, it is worth noting that scne provides high-quality representations even when the size of training corpus is small, which is crucial for practical real-world settings where rich data is not available. For a next experiment to see the effect of noisiness of training corpus, we test both noisy SNS corpus and the Wikipedia corpus 3 of the same size. The results are reported in Table 1. As it can be seen, the performance of segmentation-dependent methods ( skipgram , sisg ) are decreased greatly by the noisiness of the corpus, while scne degrades only marginally. The other two segmentation-free methods ( sembei , sembei-sum ) performed poorly. This shows the efficacy of our method in the noisy texts. On the other hand, in preliminary experiments on English (not shown), scne did not get better results than our segmentation-dependent baselines and it will be a future work to incorporate easily obtainable word boundary information into scne for segmented languages. As a word-level downstream task, we conduct a noun category prediction on Chinese, Japanese and Korean 4 . Most settings are the same as those of Oshikiri (2017). Noun words and their semantic categories are extracted from Wikidata (Vrandecic and Krotzsch, 2014) with a predetermined semantic category set 5 , and the classifier is trained to predict the semantic category of words from the learned word representations, where unseen words are skipped in training and treated as errors in testing. To see the effect of the noisiness of corpora, both noisy SNS corpus and Wikipedia corpus of the same size are examined as training corpora 6 . Results : The results are reported in Table 2. Since the set of covered nouns (i.e., non-OOV words) depends on the methods, we calculate accuracies in two ways for a fair comparison: Using all the nouns and using the intersection of the covered nouns. scne achieved the highest accuracies in all the settings when using all the nouns, and also 4 Although Korean has spacing, word boundaries are not obviously determined by space. 5 { food, song, music band name, manga, fictional character name, television series, drama, chemical compound, disease, taxon, city, island, country, year, business enterprise, public company, profession, university, language, book } 6 For each language, we use 100MB of Wikipedia and SNS data as training corpora. For the SNS data, we use Sina Weibo for Chinese and Twitter for the rest. performed well when using the intersection of the covered nouns, especially for the noisy corpora. As a sentence-level evaluation, we perform sentiment analysis on movie review data. We use 101k, 56k and 200k movie reviews and their scores respectively from Chinese, Japanese and Korean movie review websites (see Appendix A.1.6 for more details). Each review is labeled as positive or negative by its rating score. Sentence embedding models are trained using the whole movie reviews as training corpus. Among the reviews, 5k positive and 5k negative reviews are randomly selected, and the selected reviews are used to train and test the classifiers as explained in Section 3.1. Results : The results are reported in Table 3. The accuracies show that scne is also very effective in the sentence-level application. In this experiment, we observe that the larger n max contributes to the performance improvement in sentence-level application by allowing our model to capture composed representations for longer phrases or sentences. We proposed a simple yet effective unsupervised method to acquire general-purpose vector representations of words, phrases and sentences seamlessly, which is especially useful for languages whose word boundaries are not obvious, i.e., unsegmented languages. Although our method does not rely on any manually annotated resources or word segmenter, our extensive experiments show that our method outperforms the conventional approaches that depend on such resources. We would like to thank anonymous reviewers for their helpful advice. This work was partially supported by JSPS KAKENHI grant 16H02789 to HS and 18J15053 to KF.
objective abstain method result abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain objective abstain result method abstain abstain abstain abstain abstain other other abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain other abstain abstain result abstain abstain abstain abstain abstain abstain abstain abstain abstain result abstain abstain abstain abstain result result abstain abstain abstain result abstain abstain abstain result result abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain result abstain abstain abstain abstain abstain abstain objective result other other