Chinese is a logographic writing system, and the shape of Chinese characters contain rich syntactic and semantic information. In this paper, we propose a model to learn Chinese word embeddings via three-level composition: (1) a convolutional neural network to extract the intra-character compositionality from the visual shape of a character; (2) a recurrent neural network with self-attention to compose character representation into word embeddings; (3) the Skip-Gram framework to capture non-compositionality directly from the contextual information. Evaluations demonstrate the superior performance of our model on four tasks: word similarity, sentiment analysis, named entity recognition and part-of-speech tagging. 1 1 Introduction Distributed representations of words, namely word embeddings, encode both semantic and syntactic information into a dense vector. Currently, word embeddings have been playing a pivotal role in many natural language processing (NLP) tasks. Most of these NLP tasks also benefit from the pre-trained word embeddings, such as word2vec (Mikolov et al., 2013a) and GloVe (Pen-nington et al., 2014), which are based on the distributional hypothesis (Harris, 1954): words that occur in the same contexts tend to have similar meanings. Earlier word embeddings often take a word as a basic unit, and they ignore compositionality of its sub-word information such as morphemes and character n-grams, and cannot competently handle the rare words. To improve the performance of word embeddings, sub-word information has been employed (Luong et al., 2013; Qiu  Corresponding author. 1 The source codes are available at https://github. com/HSLCY/VCWE et al., 2014; Cao and Rei, 2016; Sun et al., 2016a; Wieting et al., 2016; Bojanowski et al., 2016). Compositionality is more critical for Chinese, since Chinese is a logographic writing system. In Chinese, each word typically consists of fewer characters and each character also contains richer semantic information. For example, Chinese character    (rest) is composed of the characters for    (person) and    (tree), with the intended idea of someone leaning against a tree, i.e., resting. Based on the linguistic features of Chinese, re-cent methods have used the character information to improve Chinese word embeddings. These methods can be categorized into two kinds: 1) One kind of methods learn word embeddings with its constituent character (Chen et al., 2015), radical 2 (Shi et al., 2015; Yin et al., 2016; Yu et al., 2017) or strokes 3 (Cao et al., 2018). However, these methods usually use simple operations, such as averaging and n-gram, to model the inherent compositionality within a word, which is not enough to handle the complicated linguistic compositionality. 2) The other kind of methods learns word embeddings with the visual information of the character. Liu et al. (2017) learn character embedding based on its visual characteristics in the text classification task. Su and Lee (2017) also introduce a pixel-based model that learns character features from font images. However, their model is not shown to be better than word2vec model because it has little flexibility and fixed character features. Besides, most of these methods pay less attention to the non-compositionality. For example, the 2 the graphical component of Chinese, referring to https://en.wikipedia.org/wiki/Radical_(Chinese_characters) 3 the basic pattern of Chinese characters, referring to https://en.wikipedia.org/wiki/Stroke_(CJKV_character) semantic of Chinese word     (sofa) cannot be composed by its contained characters    (sand) and    (hair). In this paper, we fully consider the compositionality and non-compositionality of Chinese words and propose a visual character-enhanced word embedding model (VCWE) to learn Chinese word embeddings. VCWE learns Chinese word embeddings via three-level composition:  The first level is to learn the intra-character composition, which gains the representation of each character from its visual appearance via a convolutional neural network;  The second level is to learn the inter-character composition, where a bidirectional long short-term neural network (Bi-LSTM) (Hochreiter and Schmidhuber, 1997) with self-attention to compose character representation into word embeddings;  The third level is to learn the non-compositionality, we can learn the contextual information because the overall framework of our model is based on the skip-gram. Evaluations demonstrate the superior performance of our model on four tasks such as word similarity, sentiment analysis, named entity recognition and part-of-speech tagging. In the past decade, there has been much research on word embeddings. Bengio et al. (2003) use a feedforward neural network language model to predict the next word given its history. Later methods (Mikolov et al., 2010) replace feedforward neural network with the recurrent neural network for further exploration. The most popular word embedding system is word2vec , which uses continuous-bag-of-words and Skip-gram models, in conjunction with negative sampling for efficient conditional probability estimation (Mikolov et al., 2013a). A different way to learn word embeddings is through factorization of word co-occurrence matrices such as GloVe embeddings (Pennington et al., 2014), which have been shown to be intrinsically linked to Skip-gram and negative sampling (Levy and Goldberg, 2014). The models mentioned above are popular and useful, but they regard individual words as atomic tokens, and the potentially useful internal structured information of words is ignored. To improve the performance of word embedding, sub-word information has been employed (Luong et al., 2013; Qiu et al., 2014; Cao and Rei, 2016; Sun et al., 2016a; Wieting et al., 2016; Bojanowski et al., 2016). These methods focus on alphabetic writing systems, but they are not directly applicable to logographic writing systems. For the alphabetic writing systems, research on Chinese word embedding has gradually emerged. These methods focus on the discovery of making full use of sub-word information. Chen et al. (2015) design a CWE model for jointly learning Chinese characters and word embeddings. Based on the CWE model, Yin et al. (2016) present a multi-granularity embedding (MGE) model, additionally using the embeddings associated with radicals detected in the target word. Xu et al. (2016) propose a similarity-based character-enhanced word embedding (SCWE) model, exploiting the similarity between a word and its component characters with the semantic knowledge obtained from other languages. Shi et al. (2015) utilize radical information to improve Chinese word embeddings. Yu et al. (2017) introduce a joint learning word embedding (JWE) model and Cao et al. (2018) represent Chinese words as sequences of strokes and learn word embeddings with stroke n-grams information. From another perspective, Liu et al. (2017) provide a new way to automatically extract character-level features, creating an image for the character and running it through a convolutional neural network to produce a visual character embedding. Su and Lee (2017) also introduce a pixel-based model that learns character features from its image. Chinese word embeddings have recently begun to be explored, and have so far shown great promise. In this paper, we propose a visual character-enhanced word embedding (VCWE) model that can learn Chinese word embeddings from corpus and images of characters. The model combines the semantic information of the context with the image features of the character, with superior performance in several benchmarks. In this section, we introduce the visual character-enhanced word embedding (VCWE) model for Chinese word representation. Given a Chinese word w consisting of n characters c 1 ,    , c n , its semantic may come from either its contained characters or its contexts. Therefore, we use the two-level hierarchical composition to compose the word embedding, which further learned according to its context. The overall architecture of our approach is on Figure 1. We first use a convolutional neural network (CNN) to model the intra-character compositionality of character from its visual shape information. We use the output of CNN as the embeddings of the character. Then the character embeddings are used as the input of the bidirectional LSTM network to model the inter-character compositionality. After a self-attention layer, we can get the representation of the word. Finally, based on the Skip-Gram framework, we learn the word embeddings with the visual character-enhanced embedding of the context. Since the shape of a Chinese character provides rich syntactic and semantic information, the representation of a character can be composed by its intrinsic visual components. Following the success of the convolutional neural network (CNN) (LeCun et al., 1995) in computer vision, we use CNN to directly model the natural composition of a character from its image. We first convert each character into an image of size 40  40 , a deep CNN is used to fuse its visual information fully. The specific structure of the CNN is shown in Figure 1(a), which consists of two convolution layers and one linear layer. Each convolution layer is followed by a max pooling layer and a batch normalization layer. The lower layers aim to capture the stroke-level information, and the higher layers aim to capture the radical-level and component-level information. The output of CNN can be regarded as the representation of the character. The character representation by its visual information can fully capture its intrinsic syntactic and semantic information with the intra-character compositionality. The parameters of CNN are learned through backpropagation in end-to-end fashion. After obtaining the representation of characters, we combine them into word embedding. The word embedding need to capture the character-level compositionality fully. Here, we use the bidirectional LSTM (Bi-LSTM) (Hochreiter and Schmidhuber, 1997) with self-attention to fuse the inter-character information of a word. self-attention is shown in Figure 1(b). Given a word w consisting of n characters c 1 ,    , c n , we use e 1 ,    , e n denote is the character representations, which are the output of the CNN rather than randomly initialized. h Fi = LSTM( h Fi  1 , e i ) , (1) h Bi = LSTM( h Bi +1 , e i ) , (2) h i = [ h Fi ; h Bi ] , (3) H = [ h 1 , h 2 , ..., h n ] , (4) where h i is the hidden state of the i -th character in w . Then we use self-attention to obtain the inter-character compositionality. Following the self-attention proposed by (Lin et al., 2017), we compute an attention vector  :  = softmax( v tanh( U h T i )) , (5) where v and U are learnable weight parameters. Since the Bi-LSTM's hidden state of each character is different according to its contexts, we believe the hidden state can capture both the compositional and non-compositional relations of the characters within a word. After obtaining the word representation, Skip-Gram (Mikolov et al., 2013a) is used to learn the word embedding with its context information. Skip-Gram is a useful framework for learning word vectors, which aims to predict context words given a target word in a sentence. Given a pair of words ( w, c ) , we denote p ( c | w ) as the probability that the word c is observed in the context of the target word w . With the negative-sampling approach, skip-gram formulates the probability p ( c | w ) as follows: Given a pair of words ( w, c ) , the probability that the word c is observed in the context of the target word w is given by p ( D = 1 | w, c ) =  ( w T c ) , (7) where w and c are embedding vectors of w and c respectively,  is the sigmoid function. The probability of not observing word c in the context of w is given by: p ( D = 0 | w, c ) = 1   ( w T c ) . Given the target word w , its context word c and k negative words  c 1 , ...,  c k . The word w is a word selected from a sentence in the corpus, and the context c is a nearby word within a window size l . The negative sample  c i is a word that is randomly sampled at a certain frequency in the vocabulary. The loss function of VCWE model is as follows: L = L 1 + L 2 , (9) L 1 = log  ( w T c )+ k (cid:88) i =1 log  (  w T  c i ) , (10) L 2 = log  ( w T m c )+ k (cid:88) i =1 log  (  w T  m i ) , (11) where w is the lookup embedding of target word; c and  c i are the lookup embeddings of the context and negative words respectively; m c and  m i are visual enhanced word embeddings of the context and negative words respectively. Here, we use the visually enhanced word embedding as the representation of context word instead of the target word. The final embedding of the target word is indirectly affected by the visual information. Thus, the final word embedding can have an advantage of fully utilizing intra-character compositionality from CNN, inter-character compositionality from LSTM, and context information from Skip-gram. We use a word sampling scheme similar to the implementation in word2vec (Mikolov et al., 2013a,b) to balance the importance of frequent words and rare words. Frequent words such as   (of),   (is),   (this) are not as meaningful as relatively less frequent words such as   (cat),    (like),    (fruit). To improve the performance of word embeddings, we use sub-sampling(Mikolov et al., 2013b) to discard the word w with the probability of P ( w ) = 1  (cid:113) t f ( w ) when generating the batch, where f ( w ) is the frequency of word w and t is a chosen threshold, typically around 10  5 . To generate negative context words, we sample each word w according to distribution P ( w )  U ( w ) 34 , where U ( w ) is the unigram distribution, which is the frequency of single words appearing in the corpus. This method also plays a role in reducing the frequency of occurrence of high-frequency words. We download Chinese Wikipedia dump 4 on May 20, 2018, which consists of 278K Chinese Wikipedia articles. We use the WikiExtractor toolkit 5 to convert data from XML into text format. We find that the corpus consists of both simplified and traditional Chinese characters. Hence we utilize the opencc toolkit 6 to normalize all characters as simplified Chinese. We remove non-Chinese characters such as punctuation marks by retaining the characters whose Unicode falls into the range between 0x4E00 and 0x9FA5. We use THULAC 7 (Sun et al., 2016b) for word segmentation. We discard words that appeared less than 100 times and obtain a vocabulary of size 66,856. We count the frequency of occurrence of each word to prepare for the subsampling work. In all 66,856 words, we extract 5030 unique characters. We use a Chinese character image generation software to generate the images of these Chinese characters. We subtract a mean image from each input image to center it before feeding it into the CNN. The pre-processed Chinese character images are shown in Figure 2. Figure 2: The pre-processed Chinese character images. 4 https://dumps.wikimedia.org/zhwiki/20180520/ 5 https://github.com/attardi/wikiextractor/blob/master/Wiki Extractor.py 6 https://github.com/BYVoid/OpenCC 7 https://github.com/thunlp/THULAC-Python 5.2 Hyperparameters Models used for evaluation have dimension D = 100 and use context window l = 5 unless stated otherwise. We use the threshold t = 10  5 for subsampling, which is the recommended value for word2vec Skip-gram (Mikolov et al., 2013a) on large datasets. The number of negative samples per word is 5. We use mini-batch asynchronous gradient descent with Adam (Kingma and Ba, 2014). The initial learning rate is 0.001. word2vec 8 (Mikolov et al., 2013a) is arguably the most popular word embedding, which uses continuous-bag-of-words (CBOW) and Skip-gram models. We train word2vec with both Skip-gram and CBOW models. We did not train Glove(Pennington et al., 2014) because it did not perform well in many previous Chinese word embedding papers. CWE 9 (Chen et al., 2015) is character-enhanced word embeddings which introduce internal character information into word embedding methods to alleviate excessive re-liance on the external information. GWE 10 (Su and Lee, 2017) is a pixel-based Chinese word embedding model, which exploits character features from font images by convolutional autoencoders. JWE 11 (Yu et al., 2017) is a model to jointly learn the embeddings of Chinese words, characters, and sub character components. For a fair comparison between different algorithms, we use the same corpus and the same hyperparameters mentioned in previous subsections. We evaluate our embeddings on the Chinese word similarity datasets wordsim-240 and wordsim-296 provided by (Chen et al., 2015). Besides, we translate two English word similarity datasets MC-30 8 https://code.google.com/archive/p/word2vec/ 9 https://github.com/Leonard-Xu/CWE 10 https://github.com/ray1007/gwe 11 https://github.com/hkust-knowcomp/jwe Model WS-240 WS-296 MC-30 RG-65 avg  Skip-gram 50.23 56.94 69.66 59.86 59.17 CBOW 51.49 61.01 68.97 63.85 61.33 +2.16 CWE 52.63 58.98 68.82 59.60 60.01 +0.84 GWE 52.74 58.22 68.23 60.74 59.98 +0.81 JWE 51.92 59.84 70.27 62.83 61.22 +2.05 VCWE 57.81 61.29 72.77 70.62 65.62 +6.45 -CNN 55.82 59.60 66.87 68.53 62.71 +3.54 -LSTM 58.13 60.85 68.03 69.78 64.20 +5.03 Table 1: Spearman correlation for word similarity datasets, -CNN represents replacing the CNN and image information with randomly initialized character embedding, -LSTM represents replacing Bi-LSTM network and self-attention with the averaging operation. (Miller and Charles, 1991) and RG-65 (Ruben-stein and Goodenough, 1965) to Chinese 12 . Each dataset contains a list of word pairs with a human score of how related or similar the two words are. We calculate the Spearman correlation (Spear-man, 1904) between the labels and our scores generated by the embeddings. The Spearman correlation is a rank-based correlation measure that assesses how well the scores describe the true labels. The evaluation results of our model and baseline methods on word similarity datasets are shown in Table 1. From the results, we can see that VCWE outperforms other baseline models. The effect of CBOW is much better than Skip-gram. The impact of GWE and CWE are relatively close. The JWE model works better than other benchmark models. In the VCWE model, when we remove the CNN and the image information, the result falls by 2.91. When we replace Bi-LSTM network and self-attention with the averaging operation, the re-12 https://github.com/FudanNLP/VCWE sult drops by 1.42. In the last subsection, we will qualitatively analyze the results of word similarity for different models. To evaluate the quality of our vectors regarding semantics, we use datasets 13 collected by (Peng et al., 2018), which contain Chinese reviews in four domains: notebook, car, camera, and phone. They manually labeled the sentiment polarity towards each aspect target as either positive or negative. It is a binary classification task. Similar to how we process the training data, we remove non-Chinese characters and use THULAC for performing Chinese word segmentation. We build classifiers with the bidirectional LSTM (Hochre-iter and Schmidhuber, 1997) network with self-attention (Lin et al., 2017). We use the standard training/dev/test split and report accuracy using different embeddings generated by different meth-13 http://sentic.net/chinese-review-datasets.zip ods in Table 2. As shown in Table 2, Skip-gram performs well on the combination of the four groups, but it does not perform well in the works of a particular group. JWE outstrips other baseline methods by around 1.1 points. The VCWE model has achieved outstanding results in the car, camera and phone category, with an accuracy rate of at least 3 points higher than other models, indicating that this method of training word embeddings with visual character-level features can achieve better results on downstream tasks. We evaluate our model on the named entity recognition task. We use an open source Chinese NER model to test our word embeddings on MSRA dataset 14 . MSRA is a dataset for simplified Chinese NER. It comes from SIGHAN 2006 shared task for Chinese NER (Levow, 2006). We pretrain word embeddings from different models and feed them into the input layer as features. The key to the task is to extract named entities and their associated types. Better word embeddings could get a higher F1 score of NER. The results in Table 3 show that our model also outperforms baseline models in this task. The performance of CWE and GWE models are similar, both slightly lower than Skip-gram and CBOW models. The F1 score of the JWE model exceeds that of other baseline models and is similar to our model. When removing the CNN and image information, our LSTM with the self-attention model can also achieve the best results on this task, indicating that the learned inter-character composition is practical. The evaluation is performed on the PKU's Peo-ple's Daily 15 (PPD) (Yu et al., 2001) with the standard training/dev/test split. The model is trained with the bidirectional LSTM model using the same hyper-parameters. Results on the POS accuracy on the test set are reported in Table 3. The gap between the usage of different embeddings is not significant, and our model has achieved the best results with a slight advantage. Explicitly, we present the top 10 words that are most similar to our target word. The similar words are retrieved based on the cosine similarity calculated using the learned embeddings. The first example word we consider is    (Tang poetry). It refers to poetry written in or around the time of or in the characteristic style of China's Tang Dynasty. All the top-ranked words identified by GWE contain the character   (Tang) and   (poetry), but in addition to the Tang Dynasty,   (Tang) also has other meanings such as surnames. GWE yields several words such as    (Don Juan),    (Tang Yin),    (Monk Tang) and    (Tang Ku), which do not appear to be semantically close to the target word. In Skip-gram and JWE, certain words such as   (anonymity) and    (ancient and modern) do not appear to be semantically very closely related to the target word. In our VCWE model, all the top-ranked words are semantically related to the target word, including the genre of poetry, poets of the Tang Dynasty, and so on. We choose the    (sofa) as the second target word. Like the first two words, GWE only pays attention to the character   (sand). Skip-gram and JWE have some irrelevant words such as     (telephone box) and     (billboard). VCWE pays more attention to Targets Skip-gram GWE JWE VCWE   (Tang poetry)   (Qu-Poetry)   (Song poetry)  (notes on poetry)  (notes on poetry)   (music score)   (indite)   (ancient and modern)   (Song poetry)  (anonymity)   (rhyme)  (anonymity)   (jueju)   (Song Ci Poetry)   (Chinese poetry)   (Yuefu)   (Song Ci Poetry)    (Bai Juyi)   (Don Juan)   (music score)  (chant)   (jueju)   (recite poems)   (compile)   (Yuefu)   (record)   (Tang Yin)   (carving copy)   (seven-character)   (Chu Songs)   (Monk Tang)   (be handed down)    (Li Shangyin)   (Yuefu)   (Tang Ku)   (ancient poetry)   (ancient poetry)   (compile)  (poetry)   (Qu-Poetry)   (poetic prose)   (sofa)    (bureau)   (hourglass)   (desk)   (wardrobe)   (bedroom)   (sand)   (wardrobe)   (bedroom)   (chair)   (sandbag)   (washcloth)   (bathtub)   (upstairs)   (sandbox)   (secretaire)   (living room)   (living room)   (raucity)   (quilt)   (curtain)   (bathtub)   (sandspit)   (bench)   (chair)   (downstairs)   (satay)   (curtain)   (fireplace)   (raincoat)   (sandbag)   (bathtub)   (door)   (bloodstain)   (Saori)   (door)   (bench)    (telephone box)   (nereid)    (billboard)   (desk) Table 4: Case study for qualitative analysis. Limited to the width of the table, we do not show the results of CWE model. The results of the GWE model are not much different from the CWE model, indicating that the image features obtained by pre-training of GWE may not play a decisive role. However, our model does not pre-train image information, but jointly trains and dynamically updates image feature information and it works better. JWE model is similar to Skip-gram model in that they pay more attention to contextual information, but sometimes the model gets some irrelevant words. Unlike phonograms, logograms have word and phrase meanings singularly. The images of Chinese characters contain rich semantic information. Since logographic languages are more closely associated with images than alphabet languages, it makes sense to mine the characteristics of these images. Liu et al. (2017) provide a new way to automatically extract character-level features, creating an image for the character and running it through a convolutional neural network to produce a visual character embedding. However, this method does not utilize the rich semantic information of contextual words. Our model extracts both image features and contextual semantic information. Su and Lee (2017) introduce a pixel-based model that learns character features from font images. However, they use convolutional auto-encoder(convAE) to extract image features in advance, and then add these features to the CWE (Chen et al., 2015) model. In the end, the effect of the model is not much different from CWE. Our model is an end-to-end model. We update the im-age's feature parameters in real time during training, and our model achieves better results than the GWE model. Our research focuses on simplified Chinese word embeddings, and the idea can also be applied to other languages that share a similar writing system, such as traditional Chinese, Japanese, and so on. In this paper, we proposed a pixel-based model to learn Chinese word embeddings with character embeddings that are compositional in the components of the characters. We utilized the visual features of Chinese characters to enhance the word embedding. We showed that our model outperforms the baseline model in the word similarity, sentiment analysis, named entity recognition and part-of-speech tagging tasks. end-to-end and make full use of the contextual information. In the future, we hope to apply our model to other downstream tasks and other logographic writing systems. We would like to thank the anonymous reviewers for their valuable comments. The research work is supported by Shanghai Municipal Science and Technology Commission (No. 17JC1404100 and 16JC1420401), National Key Research and Development Program of China (No. 2017YFB1002104), and National Natural Science Foundation of China (No. 61672162 and 61751201).
abstain objective objective abstain abstain abstain abstain other abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain other objective other objective other other other other other other other other other other other other other other other other other other objective other method abstain method abstain method abstain abstain abstain abstain method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain method abstain abstain method abstain abstain abstain abstain method abstain abstain abstain other abstain abstain abstain abstain other method abstain abstain method abstain abstain method abstain abstain abstain result abstain abstain abstain result abstain abstain abstain abstain abstain other abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain other abstain abstain abstain abstain result abstain abstain abstain result other result abstain abstain abstain abstain abstain other abstain abstain abstain abstain abstain abstain abstain abstain abstain result abstain result result abstain abstain result abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain other result abstain abstain abstain abstain abstain abstain abstain abstain method abstain abstain abstain abstain result method objective method result abstain method other other