{
  "name" : "1708.04755.pdf",
  "metadata" : {
    "source" : "CRF",
    "title" : "Learning Chinese Word Representations From Glyphs Of Characters",
    "authors" : [ "Tzu-Ray Su", "Hung-Yi Lee" ],
    "emails" : [ "b01901007@ntu.edu.tw", "hungyilee@ntu.edu.tw" ],
    "sections" : [ {
      "heading" : "1 Introduction",
      "text" : "No matter which target language it is, high quality word representations (also known as word “embeddings”) are keys to many natural language processing tasks, for example, sentence classification (Kim, 2014), question answering (Zhou et al., 2015), machine translation (Sutskever et al., 2014), etc. Besides, word-level representations are building blocks in producing phrase-level (Cho et al., 2014) and sentence-level (Kiros et al., 2015) representations.\nIn this paper, we focus on learning Chinese word representations. A Chinese word is composed of characters which contain rich semantics. The meaning of a Chinese word is often related to the meaning of its compositional characters. Therefore, Chinese word embedding can be enhanced by its compositional character embeddings (Chen et al., 2015; Xu et al., 2016). Further-\nmore, a Chinese character is composed of several graphical components. Characters with the same component share similar semantic or pronunciation. When a Chinese user encounters a previously unseen character, it is instinctive to guess the meaning (and pronunciation) from its graphical components, so understanding the graphical components and associating them with semantics help people learning Chinese. Radicals1 are the graphical components used to index Chinese characters in a dictionary. By identifying the radical of a character, one obtains a rough meaning of that character, so it is used in learning Chinese word embedding (Yin et al., 2016) and character embedding (Sun et al., 2014; Li et al., 2015). However, other components in addition to radicals may contain potentially useful information in word representation learning.\nOur research begins with a question: Can machines learn Chinese word representations from glyphs of characters? By exploiting the glyphs of characters as images in word representation learning, all the graphical components in a character are considered, not limited to radicals. In our proposed methods, we render character glyphs to fixed-size grayscale images which are referred to as “character bitmaps”, as illustrated in Fig.1. A similar idea was also used in (Liu et al., 2017) to help classifying wikipedia article titles into 12 categories. We use a convAE to extract character features from the bitmap to represent the glyphs. It is also possible to represent the glyph of a character by the graphical components in it. We do not choose this way because there is no unique way to decompose a character, and directly learning representation from bitmaps is more straightforward. Then we use the models parallel to Skipgram (Mikolov et al., 2013a) or GloVe (Penning-\n1https://en.wikipedia.org/wiki/ Radical_(Chinese_characters)\nar X\niv :1\n70 8.\n04 75\n5v 1\n[ cs\n.C L\n] 1\n6 A\nug 2\n01 7\nton et al., 2014) to learn word representations from the character glyph features. Although we only consider traditional Chinese characters in this paper, and the examples given below are based on the traditional characters, the same ideas and methods can be applied on the simplified characters."
    }, {
      "heading" : "2 Background Knowledge and Related Works",
      "text" : "To give a clear illustration of our own work, we briefly introduce the representative methods of word representation learning in Section 2.1. In Section 2.2, we will introduce some of the linguistic properties of Chinese, and then introduce the methods that utilize these properties to improve word representations."
    }, {
      "heading" : "2.1 Word Representation Learning",
      "text" : "Mainstream research of word representation is built upon the distributional hypothesis, that is, words with similar contexts share similar meanings. Usually a large-scale corpus is used, and word representations are produced from the cooccurrence information of a word and its context. Existing methods of producing word representations could be separated into two families (Levy et al., 2015): count-based family (Turney and Pantel, 2010; Bullinaria and Levy, 2007), and prediction-based family. Word representations can be obtained by training a neural-networkbased models (Bengio et al., 2003; Collobert et al., 2011). The representative methods are briefly introduced below."
    }, {
      "heading" : "2.1.1 CBOW and Skipgram",
      "text" : "Both continuous bag-of-words (CBOW) model and Skipgram model train with words and contexts in a sliding local context window (Mikolov\net al., 2013a). Both of them assign each word wi with an embedding ~wi. CBOW predicts the word given its context embeddings, while Skipgram predicts contexts given the word embedding. Predicting the occurrence of word/context in CBOW and Skipgram models could be viewed as learning a multi-class classification neural network (the number of classes is the size of vocabulary). In (Mikolov et al., 2013b), the authors introduced several techniques to improve the performance. Negative sampling is introduced to speed up learning, and subsampling frequent words is introduced to randomly discard training examples with frequent words (such as “the”, “a”, “of”), and has an effect similar to the removal of stop words."
    }, {
      "heading" : "2.1.2 GloVe",
      "text" : "Instead of using local context windows, (Pennington et al., 2014) proposed GloVe model. Training GloVe word representations begins with creating a co-occurrence matrix X from a corpus, where each matrix entry Xij represents the counts that word wj appears in the context of word wi. In (Pennington et al., 2014), the authors used a harmonic weighting function for co-occurrence count, that is, word-context pairs with distance d contributes 1d to the global co-occurrence count.\nLet ~wi be the word representation of word wi, and ~̃wj be the word representation of word wj as context, GloVe model minimizes the loss:∑ i,j∈ non−zeroentries of X f(Xij)(~w T i ~̃wj+bi+b̃j−log(Xij)),\nwhere bi is the bias for word wi, and b̃j is the bias for context wj . A weighting function f(Xij) is introduced because the authors consider rare cooccurrence word-context pairs carry less information than frequent ones, and their contributions to the total loss should be decreased. The weighting function f(Xij) is defined as below. It depends on the co-occurrence count, and the authors set parameters xmax = 100, α = 0.75.\nf(Xij) =\n{ ( Xij xmax\n)α if Xij < xmax 1 otherwise\nIn the GloVe model, each word has 2 representations ~w and ~̃w. The authors suggest using ~w+ ~̃w as the word representation, and reported improvements over using ~w only."
    }, {
      "heading" : "2.2 Improving Chinese Word Representation Learning",
      "text" : ""
    }, {
      "heading" : "2.2.1 The Chinese Language",
      "text" : "A Chinese word is composed of a sequence of characters. The meanings of some Chinese words are related to the composition of the meanings of their characters. For example, “戰艦” (battleship), is composed of two characters, “戰” (war) and “艦” (ship). More examples are given in Fig. 2. To improve Chinese word representations with sub-word information, character-enhanced word embedding (CWE) (Chen et al., 2015) in Section 2.2.2 is proposed.\nA Chinese character is composed of several graphical components. Characters with the same component share similar semantic or phonetic properties. In a Chinese dictionary characters with similar coarse semantics are grouped into categories for the ease of searching. The common graphical component which relates to the common semantic is chosen to index the category, known\nas a radical. Examples are given in Fig. 3. There are three radicals in row (A), and their semantic meanings are in row (B). In each column, there are five characters containing each radical. It is easy to find that the characters having the same radical have meanings related to the radical in some aspect. A radical can be put in different positions in a character. For example, in rows (C-1) to (C-4), the radicals are at the left hand side of a character, but in row (C-5), the radicals are at the bottom. The shape of a radical can be different in different positions. For example, the third radical which represents “water” or “liquid” has different forms when it is at the left hand side or the bottom of a character. Because radicals serve as a strong semantic indicator of a character, multigranularity embedding (MGE) (Yin et al., 2016) in Section 2.2.3 incorporates radical embeddings in learning word representation.\nUsually the components other than radicals determine the pronunciation of the characters, but in some cases they also influence the meaning of a character. Two examples are given in Fig. 42. Both characters in Fig. 4 have the same radical “亻” (means humans) at the left hand side, but the graphical components at the right hand side also have semantic meanings related to the characters. Considering the left character “伐” (means attack). Its right component “戈” means “weapon”, and the meaning of the character “伐” is the composition of the meaning of its two components (a human with a weapon). None of the previous word embedding approach considers all the components of Chinese characters in our best knowledge.\n2The two example characters here have the same glyphs in the traditional and simplified Chinese characters."
    }, {
      "heading" : "2.2.2 Character-enhanced Word Embedding (CWE)",
      "text" : "The main idea of CWE is that word embedding is enhanced by its compositional character embeddings. CWE predicts the word from both word and character embeddings of contexts, as illustrated in Fig. 5 (a). For wordwi, the CWE word embedding ~wcwei has the following form:\n~wcwei = ~wi + 1 |C(i)| ∑\ncj∈C(i)\n~cj\nwhere ~wi is the word embedding, ~cj is the embedding of the j-th character in wi, and C(i) is the set of compositional characters of word wi. Mean value of CWE word embeddings of contexts are then used to predict the word wi.\nSometimes one character has several different meanings, this is known as the ambiguity problem. To deal with this, each character is assigned with a bag of embeddings. During training, one of the embeddings is picked to form the modified word embedding. The authors proposed three methods to decide which embedding is picked: positionbased, cluster-based, and non-parametric clusterbased character embeddings."
    }, {
      "heading" : "2.2.3 Multi-granularity Embedding (MGE)",
      "text" : "Based on CBOW and CWE, (Yin et al., 2016) proposed MGE, which predicts target word with its radical embeddings and modified word embeddings of context in CWE, as shown in Fig.5 (b).\nThere is no ambiguity of radicals, so each radical is assigned with one embedding ~r. We denote\n~rk as the radical embedding of character ck. MGE predicts the target word wi with the following hidden vector:\n~hi = 1 |C(i)| ∑\nck∈C(i)\n~rk + 1 |W (i)| ∑\nwj∈W (i)\n~wcwej\n, where W(i) is the set of contexts words of wi, ~wcwej is the CWE word embedding of wj . MGE picks character embeddings with the positionbased method in CWE, and picks radical embeddings according to a character-radical index built from a dictionary during training. When noncompositional word is encountered, only the word embedding is used to form ~hi."
    }, {
      "heading" : "3 Model",
      "text" : "We first extract glyph features from bitmaps with the convAE in Section 3.1. The glyph features are used to enhance the existing word representation learning models in Section 3.2. In Section 3.3, we try to learn word representations directly from the glyph features."
    }, {
      "heading" : "3.1 Character Bitmap Feature Extraction",
      "text" : "A convAE (Masci et al., 2011) is used to reduce the dimensions of rendered character bitmaps and capture high-level features. The architecture of the convAE is shown in Fig. 6. The convAE is composed of 5 convolutional layers in both encoder and decoder. The stride larger than one is used instead of pooling layers. Convolutional and deconvolutional layers on the same level share the same kernel. The input image is a 60×60 8-bit grayscale bitmap, and the encoder extracts 512-dimensional feature. The feature of character ck from the encoder is refer to as character glyph feature ~gk in the paper."
    }, {
      "heading" : "3.2 Glyph-Enhanced Word Embedding (GWE)",
      "text" : ""
    }, {
      "heading" : "3.2.1 Enhanced by Context Word Glyphs",
      "text" : "We modify CWE model based on CBOW in Section 2.2.2 to incorporate context character glyph features (ctxG). This modified word embedding ~wctxGi of word wi has the form:\n~wctxGi = ~wi + 1 |C(i)| ∑\ncj∈C(i)\n(~cj + ~gj),\nwhere C(i) is the compositional characters of wi and ~gj is the glyph feature of cj . The model predicts target word wi from ctxG word embeddings of contexts, as shown in Fig.7. The parameters in the convAE are pre-trained, thus not jointly learned with embeddings ~w and ~c, so character glyph features ~g are fixed during training."
    }, {
      "heading" : "3.2.2 Enhanced by Target Word Glyphs",
      "text" : "Here we propose another variant. In this model, the model structure is the same as in Fig.7. The difference lies in the hidden vector used to predict the target word. Instead of adding mean value of character glyph features of the contexts, it adds mean value of glyph feature of the target word (tarG), as shown in Fig.8. As in Section 3.2.1, convAE is not jointly learned."
    }, {
      "heading" : "3.3 Directly Learn From Character Glyph Features",
      "text" : ""
    }, {
      "heading" : "3.3.1 RNN-Skipgram",
      "text" : "We learn word representation ~wi directly from the sequence of character glyph features {~gk, ck ∈ C(i)} of word wi, with the objective of Skipgram. As in Fig.9, a 2-layer Gated Recurrent Units (GRU) (Cho et al., 2014) network followed by 2 fully connected ELU (Clevert et al., 2015) layers produces word representation ~wi from input sequence {~gk} of word wi. ~wi is then used to predict the contexts of wi. In the training we use negative sampling and subsampling on frequent words from (Mikolov et al., 2013b)."
    }, {
      "heading" : "3.3.2 RNN-GloVe",
      "text" : "We modify GloVe model to directly learn from character glyph features as in Fig.10. We feed character glyph feature sequence {~gk, ck ∈ C(i)}, {~gk′ , ck′ ∈ C(j)} of word wi and context wj to a shared GRU network. Outputs of GRU are then fed to two different fully connected ELU layers to produce word representations ~wi and ~̃wj . The inner product of ~wi and ~̃wj is the prediction of log co-occurrence log(Xij). We apply the same loss function with weights in GloVe. We follow (Pennington et al., 2014) and use ~wi+ ~̃wi for evaluations of word representation."
    }, {
      "heading" : "4 Experimental Setup",
      "text" : ""
    }, {
      "heading" : "4.1 Preprocessing",
      "text" : "We learned word representations with traditional Chinese texts from Central News Agency daily newspapers from 1991 to 2002 (Chinese Giga-\nword, LDC2003T09). All foreign words, numerical words, and punctuations were removed. Word segmentation was performed using open source python package jieba3. In all 316,960,386 segmented words, we extracted 8780 unique characters, and used a true type font (BiauKai) to render each character glyph to a 60×60 8-bit grayscale bitmap. Furthermore, We removed words whose frequency <= 25, leaving 158,565 unique words as the vocabulary set."
    }, {
      "heading" : "4.2 Extracting Visual Features of Character Bitmap",
      "text" : "Inspired by (Zeiler et al., 2011), layer-wise training was applied to our convAE. From lower level to higher, the kernel of each layer is trained individually, with other kernels frozen for 100 epochs. Loss function is the Euclidean distance between input and reconstructed bitmap, and we added l1 regularization to the activations of convolution layers. We chose Adagrad as the optimizing algorithm, and set batch size = 20 and learning rate = 0.001.\n3https://github.com/fxsjy/jieba\nThe comparison between the input bitmaps and their reconstructions is shown in Fig 11. The input bitmaps are in the upper row, while the reconstructions are in the lower row. We further visualized the extracted character glyph features with tSNE (Maaten and Hinton, 2008). Part of the visualization result is shown in Fig. 12. From Fig. 12, we found that the characters with the same components are clustered. The result shows that the features extracted by the convAE are capable of expressing the graphical information in the bitmaps."
    }, {
      "heading" : "4.3 Training Details of Word Representations",
      "text" : "We used CWE code4 to implement both CBOW and Skipgram, along with the CWE. The number of multi-embedding was set to 3. We modified the CWE code to produce GWE representations. For CBOW, Skipgram, CWE, GWE and RNN-Skipgram, we used the following hyperparameters. Context window was set to 5 to both sides of a word. We used 10 negative samples, and threshold t of subsampling was set to 10−5.\nSince Yin at al. did not publish their code, we followed their paper and reproduced the MGE model. We created the mapping between characters and radicals from the Unihan database5. Each character corresponds to one of the 214 radicals in this dataset, and the same hyperparameters were used in training as above. Note that we did not separate non-compositional words during training as the original CWE and MGE did.\nWe used the GloVe code6 to train the baseline GloVe vectors. In construction of co-occurrence matrix for GloVe and RNN-GloVe, we followed the parameter settings of xmax = 100 and α = 0.75 in (Pennington et al., 2014). Context window was 5 words to the both sides of a word, and harmonic weighting was used on co-occurrence counts. For the RNN-GloVe model, we removed entries whose value < 0.5 to speed up training.\nRNN-Skipgram and RNN-GloVe generated 200-dimensional word embeddings, while other models generated 512-dimensional word embeddings.\nTo encourage further research, we published our convAE and embedding models on github7. Evaluation datasets were also uploaded, whose details will be explained in Section 5.\n4https://github.com/Leonard-Xu/CWE 5http://unicode.org/charts/unihan.html 6https://github.com/stanfordnlp/GloVe 7https://github.com/ray1007/GWE"
    }, {
      "heading" : "5 Evaluation",
      "text" : ""
    }, {
      "heading" : "5.1 Word Similarity",
      "text" : "A word similarity test contains multiple word pairs and their human annotated similarity scores. Word representations are considered good if the calculated similarity and human annotated scores have a high rank correlation. We computed the Spearman’s correlation between human annotated scores and cosine similarity of word representations.\nSince there is little resource for traditional Chinese, we translated WordSim-240 and WordSim296 datasets provided by (Chen et al., 2015). Note that this translation is non-trivial. Some frequent words are considered out-of-vocabulary (OOV) due to the different usage between the simplified and traditional. For example, “butter” is translated to “黃油” in simplified, but “奶油” in traditional. Besides, we manually translated SimLex-999 (Hill et al., 2016) to traditional Chinese, and used it as the third testing dataset. We also made these datasets public along with our code.\nWhen calculating similarities, word pairs containing OOVs were removed. In Table 1, there are only 237, 284 and 979 word pairs left in WordSim240, WordSim-296 and SimLex-999, respectively. The results are presented in Table 1. The results of ordinary CBOW and Skipgram are shown in the table. CBOW/Skipgram+CWE represents CWE trained as CBOW or Skipgram. For CWE, we\nonly show the results of position-based character embeddings here because the results of clusterbased character embeddings are worse in the experiments. We found that CWE only consistently improved the performance on SimLex-999 for both CBOW and Skipgram probably because SimLex-999 contains more words that could be understood from their compositional characters. On SimLex-999, we observed that CWE was better with CBOW than Skipgram. We think the reason is that CBOW+CWE predicts the target word with the mean value of all character embeddings in the context, thus has a less noisy feature; however Skipgram+CWE uses character embeddings of an individual word. This noisy feature could cause\nnegative effects on predicting the target word. The GWEs were learned based on CWE in two ways. “ctxG” represents using glyph features of context words, while “tarG” represents using glyph features of target words. The glyph features improved CWE on WordSim-240 and SimLex-999, but not WordSim-296.\nAs for MGE results, we were not able to reproduce the performance in (Yin et al., 2016). We list possible reasons as below: we did not separate non-compositional word during training (character and radical embeddings are not used for these words), and the we created character-radical index from different data source. We conjecture that the first to be the most crucial factor in reproducing MGE.\nThe results of RNN-Skipgram and RNN-GloVe are also in Table 1. Their results are not comparable with CBOW and Skipgram. From the results, we conclude that it is not easy to produce word representations directly from glyphs. We think the reason is that RNN representations are dependent on each other. Updating model parameters for word wi would also change the word representation of word wj . As a result it is much more difficult to train such models.\nWe further inspect the impact of glyph features by doing significance test8 between proposed methods and existing ones. The p-values of the tests are given in Table 2. We found only “tarG” method has a p-value less than 0.05 over CWE."
    }, {
      "heading" : "5.2 Word Analogy",
      "text" : "An analogy problem has the following form: “king”:“queen” = “man”:“?”, and “woman” is answer to “?”. By answering the question correctly, the model is considered capable of expressing semantic relationships. Furthermore, the analogy relation could be expressed by vector arithmetic of word representations as shown in (Mikolov et al., 2013b). For the above problem, we find word wi such that wi = argmax\nw cos(~w, ~wqueen − ~wking + ~wman).\n8We followed the method described in https:// stats.stackexchange.com/questions/17696/\nAs in the previous subsection, we translated the word analogy dataset in (Chen et al., 2015) to traditional. The dataset contains 3 groups of analogy problems: capitals of countries, (China) states/provinces of cities, and family relations. Considering that most capital and city names do not relate to the meaning of their compositional characters, and that we did not separate noncompositional word in our experiments, we proposed a new analogy dataset composed of jobs and places (job&place). Nonetheless, there might be multiple corresponding places for a single job. For instance, A “doctor” could be in a “hospital” or “clinic”. In this job&place dataset, we provide a set of places for each job. The model is considered to answer correctly as long as the predicted word is in this set.\nWe take the mean of all word representations of places (mean(~wplaces1)) for the first job (job1), and find the place for another job (job2) by calculating wi such that wi = argmax\nw cos(~w,mean(~wplaces1)− ~wjob1+ ~wjob2). The results are shown in Table 3. we observed CWE only improved accuracy only for the family group. The results are not surprising. The words of family relations are compositional in Chinese, however capital and city names are usually not. We observed that GWE further improved CWE for words in the family group. From Table 3, we found that glyph features are helpful when the characters can enhance word representations. This is very reasonable because glyph features are fruitful representations of characters. If character information does not play a role in learning word representations, character glyphs may not be useful. The same phenomenon is observed in Table 1.\nIn our job&place, we still observed that GWE improving CWE, however both CWE and GWE were slightly worse than CBOW. We also observed that Skipgram-based methods became worse than CBOW-based methods, while in all previous evaluation Skipgram-based methods are consistently better.\nThe results of RNN-Skipgram and RNN-GloVe are still poor. We observe that the word representations learned from RNN can no longer be expressed by vector arithmetic. The reason is still under investigation."
    }, {
      "heading" : "5.3 Case Study",
      "text" : "To further probe the effect of glyph features, we show the following word pairs in SimLex-999 whose calculated cosine similarities are higher based on GWE models than CWE. The pairs may not look alike, but their components share related semantics. For example, in “伶俐” (clever), the component “利”(sharp) is compositional to the meaning of “俐”(acute), describing someone with a sharp mind. Other examples show the ability to associate semantics with radicals.\nWe also provide several counter-examples. Below are some word pairs which are not similar, however GWE methods produces higher similarity than CBOW or CWE. Take “山峰” (mountain) and “蜂蜜” (honey) as example. Since they share no\ncommon characters, the only thing in common is the component “夆”, and we assume this to be the reason for the higher similarity. Also note that in the pair “無趣” (boring) and “好笑” (funny), the CWE similarity is also higher. We conclude that the character “無” (none) is not strong enough, so the character “趣” (fun) overrides the word “無 趣” (boring), thus a higher score was mistakenly assigned."
    }, {
      "heading" : "6 Conclusions",
      "text" : "This work is a pioneer in enhancing Chinese word representations with character glyphs. The character glyph features are directly learned from the bitmaps of characters by convAE. We then proposed 2 methods in learning Chinese word representations: the first is to use character glyph features as enhancement; the other is to directly learn word representation from sequences of glyph features. In experiments, we found the latter totally infeasible. Training word representations with RNN without word and character information is challenging. Nonetheless, the glyph features improved the character-enhanced Chinese word representations, especially on the word analogy task related to family.\nThe results of exploiting character glyph features in word representation learning was ordinary. Perhaps the co-occurrence information in the corpus plays a bigger role than glyph features. Nonetheless, the idea to treat each Chinese character as image is innovative. As more character-level models(Zheng et al., 2013; Kim, 2014; Zhang et al., 2015) are proposed in the NLP field, we believe glyph features could serve as an enhancement, and we will further examine the effect of glyph features on other tasks, such as word segmentation, POS tagging, dependency parsing, or downstream tasks such as text classification, or document retrieval."
    } ],
    "references" : [ {
      "title" : "A neural probabilistic language model",
      "author" : [ "Yoshua Bengio", "Réjean Ducharme", "Pascal Vincent", "Christian Jauvin." ],
      "venue" : "Journal of machine learning research, 3(Feb):1137–1155.",
      "citeRegEx" : "Bengio et al\\.,? 2003",
      "shortCiteRegEx" : "Bengio et al\\.",
      "year" : 2003
    }, {
      "title" : "Extracting semantic representations from word co-occurrence statistics: A computational study",
      "author" : [ "John A Bullinaria", "Joseph P Levy." ],
      "venue" : "Behavior research methods, 39(3):510–526.",
      "citeRegEx" : "Bullinaria and Levy.,? 2007",
      "shortCiteRegEx" : "Bullinaria and Levy.",
      "year" : 2007
    }, {
      "title" : "Joint learning of character and word embeddings",
      "author" : [ "Xinxiong Chen", "Lei Xu", "Zhiyuan Liu", "Maosong Sun", "Huan-Bo Luan." ],
      "venue" : "Proceedings of the Twenty-Fourth International Joint Conference on Artificial Intelligence, IJCAI 2015, Buenos Aires,",
      "citeRegEx" : "Chen et al\\.,? 2015",
      "shortCiteRegEx" : "Chen et al\\.",
      "year" : 2015
    }, {
      "title" : "Learning phrase representations using RNN encoder-decoder for statistical machine translation",
      "author" : [ "Kyunghyun Cho", "Bart Van Merriënboer", "Caglar Gulcehre", "Dzmitry Bahdanau", "Fethi Bougares", "Holger Schwenk", "Yoshua Bengio." ],
      "venue" : "arXiv preprint",
      "citeRegEx" : "Cho et al\\.,? 2014",
      "shortCiteRegEx" : "Cho et al\\.",
      "year" : 2014
    }, {
      "title" : "Fast and accurate deep network learning by exponential linear units (elus)",
      "author" : [ "Djork-Arné Clevert", "Thomas Unterthiner", "Sepp Hochreiter." ],
      "venue" : "arXiv preprint arXiv:1511.07289.",
      "citeRegEx" : "Clevert et al\\.,? 2015",
      "shortCiteRegEx" : "Clevert et al\\.",
      "year" : 2015
    }, {
      "title" : "Natural language processing (almost) from scratch",
      "author" : [ "Ronan Collobert", "Jason Weston", "Léon Bottou", "Michael Karlen", "Koray Kavukcuoglu", "Pavel Kuksa." ],
      "venue" : "Journal of Machine Learning Research, 12(Aug):2493–2537.",
      "citeRegEx" : "Collobert et al\\.,? 2011",
      "shortCiteRegEx" : "Collobert et al\\.",
      "year" : 2011
    }, {
      "title" : "Simlex-999: Evaluating semantic models with (genuine) similarity estimation",
      "author" : [ "Felix Hill", "Roi Reichart", "Anna Korhonen." ],
      "venue" : "Computational Linguistics.",
      "citeRegEx" : "Hill et al\\.,? 2016",
      "shortCiteRegEx" : "Hill et al\\.",
      "year" : 2016
    }, {
      "title" : "Convolutional neural networks for sentence classification",
      "author" : [ "Yoon Kim." ],
      "venue" : "Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing, EMNLP 2014, October 25-29, 2014, Doha, Qatar, A meeting of SIGDAT, a Special",
      "citeRegEx" : "Kim.,? 2014",
      "shortCiteRegEx" : "Kim.",
      "year" : 2014
    }, {
      "title" : "Skip-thought vectors",
      "author" : [ "Ryan Kiros", "Yukun Zhu", "Ruslan R Salakhutdinov", "Richard Zemel", "Raquel Urtasun", "Antonio Torralba", "Sanja Fidler." ],
      "venue" : "Advances in neural information processing systems, pages 3294–3302.",
      "citeRegEx" : "Kiros et al\\.,? 2015",
      "shortCiteRegEx" : "Kiros et al\\.",
      "year" : 2015
    }, {
      "title" : "Improving distributional similarity with lessons learned from word embeddings",
      "author" : [ "Omer Levy", "Yoav Goldberg", "Ido Dagan." ],
      "venue" : "Transactions of the Association for Computational Linguistics, 3:211–225.",
      "citeRegEx" : "Levy et al\\.,? 2015",
      "shortCiteRegEx" : "Levy et al\\.",
      "year" : 2015
    }, {
      "title" : "Component-enhanced chinese character embeddings",
      "author" : [ "Yanran Li", "Wenjie Li", "Fei Sun", "Sujian Li." ],
      "venue" : "Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pages 829–834, Lisbon, Portugal. Association",
      "citeRegEx" : "Li et al\\.,? 2015",
      "shortCiteRegEx" : "Li et al\\.",
      "year" : 2015
    }, {
      "title" : "Learning character-level compositionality with visual features",
      "author" : [ "Frederick Liu", "Han Lu", "Chieh Lo", "Graham Neubig." ],
      "venue" : "CoRR, abs/1704.04859.",
      "citeRegEx" : "Liu et al\\.,? 2017",
      "shortCiteRegEx" : "Liu et al\\.",
      "year" : 2017
    }, {
      "title" : "Visualizing data using t-SNE",
      "author" : [ "Laurens van der Maaten", "Geoffrey Hinton." ],
      "venue" : "Journal of Machine Learning Research, 9(Nov):2579–2605.",
      "citeRegEx" : "Maaten and Hinton.,? 2008",
      "shortCiteRegEx" : "Maaten and Hinton.",
      "year" : 2008
    }, {
      "title" : "Stacked convolutional autoencoders for hierarchical feature extraction",
      "author" : [ "Jonathan Masci", "Ueli Meier", "Dan Cireşan", "Jürgen Schmidhuber." ],
      "venue" : "In",
      "citeRegEx" : "Masci et al\\.,? 2011",
      "shortCiteRegEx" : "Masci et al\\.",
      "year" : 2011
    }, {
      "title" : "Efficient estimation of word representations in vector space",
      "author" : [ "Tomas Mikolov", "Kai Chen", "Greg Corrado", "Jeffrey Dean." ],
      "venue" : "Proceedings of the International Conference on Learning Representations (ICLR).",
      "citeRegEx" : "Mikolov et al\\.,? 2013a",
      "shortCiteRegEx" : "Mikolov et al\\.",
      "year" : 2013
    }, {
      "title" : "Distributed representations of words and phrases and their compositionality",
      "author" : [ "Tomas Mikolov", "Ilya Sutskever", "Kai Chen", "Greg S Corrado", "Jeff Dean." ],
      "venue" : "Advances in neural information processing systems, pages 3111–3119.",
      "citeRegEx" : "Mikolov et al\\.,? 2013b",
      "shortCiteRegEx" : "Mikolov et al\\.",
      "year" : 2013
    }, {
      "title" : "Glove: Global vectors for word representation",
      "author" : [ "Jeffrey Pennington", "Richard Socher", "Christopher D Manning." ],
      "venue" : "EMNLP, volume 14, pages 1532– 1543.",
      "citeRegEx" : "Pennington et al\\.,? 2014",
      "shortCiteRegEx" : "Pennington et al\\.",
      "year" : 2014
    }, {
      "title" : "Radical-enhanced chinese character embedding",
      "author" : [ "Yaming Sun", "Lei Lin", "Nan Yang", "Zhenzhou Ji", "Xiaolong Wang." ],
      "venue" : "International Conference on Neural Information Processing, pages 279–286. Springer.",
      "citeRegEx" : "Sun et al\\.,? 2014",
      "shortCiteRegEx" : "Sun et al\\.",
      "year" : 2014
    }, {
      "title" : "Sequence to sequence learning with neural networks",
      "author" : [ "Ilya Sutskever", "Oriol Vinyals", "Quoc V Le." ],
      "venue" : "Advances in neural information processing systems, pages 3104–3112.",
      "citeRegEx" : "Sutskever et al\\.,? 2014",
      "shortCiteRegEx" : "Sutskever et al\\.",
      "year" : 2014
    }, {
      "title" : "From frequency to meaning: Vector space models of semantics",
      "author" : [ "Peter D Turney", "Patrick Pantel." ],
      "venue" : "Journal of artificial intelligence research, 37:141–188.",
      "citeRegEx" : "Turney and Pantel.,? 2010",
      "shortCiteRegEx" : "Turney and Pantel.",
      "year" : 2010
    }, {
      "title" : "Improve chinese word embeddings by exploiting internal structure",
      "author" : [ "Jian Xu", "Jiawei Liu", "Liangang Zhang", "Zhengyu Li", "Huanhuan Chen." ],
      "venue" : "Proceedings of NAACL-HLT, pages 1041–1050.",
      "citeRegEx" : "Xu et al\\.,? 2016",
      "shortCiteRegEx" : "Xu et al\\.",
      "year" : 2016
    }, {
      "title" : "Multi-granularity chinese word embedding",
      "author" : [ "Rongchao Yin", "Quan Wang", "Peng Li", "Rui Li", "Bin Wang." ],
      "venue" : "Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, EMNLP 2016, Austin, Texas, USA, November",
      "citeRegEx" : "Yin et al\\.,? 2016",
      "shortCiteRegEx" : "Yin et al\\.",
      "year" : 2016
    }, {
      "title" : "Adaptive deconvolutional networks for mid and high level feature learning",
      "author" : [ "Matthew D Zeiler", "Graham W Taylor", "Rob Fergus." ],
      "venue" : "Computer Vision (ICCV), 2011 IEEE International Conference on, pages 2018–2025. IEEE.",
      "citeRegEx" : "Zeiler et al\\.,? 2011",
      "shortCiteRegEx" : "Zeiler et al\\.",
      "year" : 2011
    }, {
      "title" : "Character-level convolutional networks for text classification",
      "author" : [ "Xiang Zhang", "Junbo Zhao", "Yann LeCun." ],
      "venue" : "Advances in neural information processing systems, pages 649–657.",
      "citeRegEx" : "Zhang et al\\.,? 2015",
      "shortCiteRegEx" : "Zhang et al\\.",
      "year" : 2015
    }, {
      "title" : "Deep learning for chinese word segmentation and pos tagging",
      "author" : [ "Xiaoqing Zheng", "Hanyang Chen", "Tianyu Xu." ],
      "venue" : "EMNLP, pages 647–657.",
      "citeRegEx" : "Zheng et al\\.,? 2013",
      "shortCiteRegEx" : "Zheng et al\\.",
      "year" : 2013
    }, {
      "title" : "Learning continuous word embedding with metadata for question retrieval in community question answering",
      "author" : [ "Guangyou Zhou", "Tingting He", "Jun Zhao", "Po Hu." ],
      "venue" : "ACL (1), pages 250–259.",
      "citeRegEx" : "Zhou et al\\.,? 2015",
      "shortCiteRegEx" : "Zhou et al\\.",
      "year" : 2015
    } ],
    "referenceMentions" : [ {
      "referenceID" : 7,
      "context" : "No matter which target language it is, high quality word representations (also known as word “embeddings”) are keys to many natural language processing tasks, for example, sentence classification (Kim, 2014), question answering (Zhou et al.",
      "startOffset" : 196,
      "endOffset" : 207
    }, {
      "referenceID" : 25,
      "context" : "No matter which target language it is, high quality word representations (also known as word “embeddings”) are keys to many natural language processing tasks, for example, sentence classification (Kim, 2014), question answering (Zhou et al., 2015), machine translation (Sutskever et al.",
      "startOffset" : 228,
      "endOffset" : 247
    }, {
      "referenceID" : 18,
      "context" : ", 2015), machine translation (Sutskever et al., 2014), etc.",
      "startOffset" : 29,
      "endOffset" : 53
    }, {
      "referenceID" : 3,
      "context" : "Besides, word-level representations are building blocks in producing phrase-level (Cho et al., 2014) and sentence-level (Kiros et al.",
      "startOffset" : 82,
      "endOffset" : 100
    }, {
      "referenceID" : 8,
      "context" : ", 2014) and sentence-level (Kiros et al., 2015) representations.",
      "startOffset" : 27,
      "endOffset" : 47
    }, {
      "referenceID" : 2,
      "context" : "Therefore, Chinese word embedding can be enhanced by its compositional character embeddings (Chen et al., 2015; Xu et al., 2016).",
      "startOffset" : 92,
      "endOffset" : 128
    }, {
      "referenceID" : 20,
      "context" : "Therefore, Chinese word embedding can be enhanced by its compositional character embeddings (Chen et al., 2015; Xu et al., 2016).",
      "startOffset" : 92,
      "endOffset" : 128
    }, {
      "referenceID" : 21,
      "context" : "By identifying the radical of a character, one obtains a rough meaning of that character, so it is used in learning Chinese word embedding (Yin et al., 2016) and character embedding (Sun et al.",
      "startOffset" : 139,
      "endOffset" : 157
    }, {
      "referenceID" : 17,
      "context" : ", 2016) and character embedding (Sun et al., 2014; Li et al., 2015).",
      "startOffset" : 32,
      "endOffset" : 67
    }, {
      "referenceID" : 10,
      "context" : ", 2016) and character embedding (Sun et al., 2014; Li et al., 2015).",
      "startOffset" : 32,
      "endOffset" : 67
    }, {
      "referenceID" : 11,
      "context" : "A similar idea was also used in (Liu et al., 2017) to help classifying wikipedia article titles into 12 categories.",
      "startOffset" : 32,
      "endOffset" : 50
    }, {
      "referenceID" : 14,
      "context" : "Then we use the models parallel to Skipgram (Mikolov et al., 2013a) or GloVe (Penning-",
      "startOffset" : 44,
      "endOffset" : 67
    }, {
      "referenceID" : 9,
      "context" : "Existing methods of producing word representations could be separated into two families (Levy et al., 2015): count-based family (Turney and Pantel, 2010; Bullinaria and Levy, 2007), and prediction-based family.",
      "startOffset" : 88,
      "endOffset" : 107
    }, {
      "referenceID" : 19,
      "context" : ", 2015): count-based family (Turney and Pantel, 2010; Bullinaria and Levy, 2007), and prediction-based family.",
      "startOffset" : 28,
      "endOffset" : 80
    }, {
      "referenceID" : 1,
      "context" : ", 2015): count-based family (Turney and Pantel, 2010; Bullinaria and Levy, 2007), and prediction-based family.",
      "startOffset" : 28,
      "endOffset" : 80
    }, {
      "referenceID" : 0,
      "context" : "Word representations can be obtained by training a neural-networkbased models (Bengio et al., 2003; Collobert et al., 2011).",
      "startOffset" : 78,
      "endOffset" : 123
    }, {
      "referenceID" : 5,
      "context" : "Word representations can be obtained by training a neural-networkbased models (Bengio et al., 2003; Collobert et al., 2011).",
      "startOffset" : 78,
      "endOffset" : 123
    }, {
      "referenceID" : 14,
      "context" : "Both continuous bag-of-words (CBOW) model and Skipgram model train with words and contexts in a sliding local context window (Mikolov et al., 2013a).",
      "startOffset" : 125,
      "endOffset" : 148
    }, {
      "referenceID" : 15,
      "context" : "In (Mikolov et al., 2013b), the authors introduced several techniques to improve the performance.",
      "startOffset" : 3,
      "endOffset" : 26
    }, {
      "referenceID" : 16,
      "context" : "Instead of using local context windows, (Pennington et al., 2014) proposed GloVe model.",
      "startOffset" : 40,
      "endOffset" : 65
    }, {
      "referenceID" : 16,
      "context" : "In (Pennington et al., 2014), the authors used a harmonic weighting function for co-occurrence count, that is, word-context pairs with distance d contributes 1 d to the global co-occurrence count.",
      "startOffset" : 3,
      "endOffset" : 28
    }, {
      "referenceID" : 2,
      "context" : "To improve Chinese word representations with sub-word information, character-enhanced word embedding (CWE) (Chen et al., 2015) in Section 2.",
      "startOffset" : 107,
      "endOffset" : 126
    }, {
      "referenceID" : 21,
      "context" : "Because radicals serve as a strong semantic indicator of a character, multigranularity embedding (MGE) (Yin et al., 2016) in Section 2.",
      "startOffset" : 103,
      "endOffset" : 121
    }, {
      "referenceID" : 21,
      "context" : "Based on CBOW and CWE, (Yin et al., 2016) proposed MGE, which predicts target word with its radical embeddings and modified word embeddings of context in CWE, as shown in Fig.",
      "startOffset" : 23,
      "endOffset" : 41
    }, {
      "referenceID" : 13,
      "context" : "A convAE (Masci et al., 2011) is used to reduce the dimensions of rendered character bitmaps and capture high-level features.",
      "startOffset" : 9,
      "endOffset" : 29
    }, {
      "referenceID" : 3,
      "context" : "9, a 2-layer Gated Recurrent Units (GRU) (Cho et al., 2014) network followed by 2 fully connected ELU (Clevert et al.",
      "startOffset" : 41,
      "endOffset" : 59
    }, {
      "referenceID" : 4,
      "context" : ", 2014) network followed by 2 fully connected ELU (Clevert et al., 2015) layers produces word representation ~ wi from input sequence {~gk} of word wi.",
      "startOffset" : 50,
      "endOffset" : 72
    }, {
      "referenceID" : 15,
      "context" : "In the training we use negative sampling and subsampling on frequent words from (Mikolov et al., 2013b).",
      "startOffset" : 80,
      "endOffset" : 103
    }, {
      "referenceID" : 16,
      "context" : "We follow (Pennington et al., 2014) and use ~ wi+ ~̃ wi for evaluations of word representation.",
      "startOffset" : 10,
      "endOffset" : 35
    }, {
      "referenceID" : 22,
      "context" : "Inspired by (Zeiler et al., 2011), layer-wise training was applied to our convAE.",
      "startOffset" : 12,
      "endOffset" : 33
    }, {
      "referenceID" : 12,
      "context" : "We further visualized the extracted character glyph features with tSNE (Maaten and Hinton, 2008).",
      "startOffset" : 71,
      "endOffset" : 96
    }, {
      "referenceID" : 16,
      "context" : "75 in (Pennington et al., 2014).",
      "startOffset" : 6,
      "endOffset" : 31
    }, {
      "referenceID" : 2,
      "context" : "Since there is little resource for traditional Chinese, we translated WordSim-240 and WordSim296 datasets provided by (Chen et al., 2015).",
      "startOffset" : 118,
      "endOffset" : 137
    }, {
      "referenceID" : 6,
      "context" : "Besides, we manually translated SimLex-999 (Hill et al., 2016) to traditional Chinese, and used it as the third testing dataset.",
      "startOffset" : 43,
      "endOffset" : 62
    }, {
      "referenceID" : 21,
      "context" : "As for MGE results, we were not able to reproduce the performance in (Yin et al., 2016).",
      "startOffset" : 69,
      "endOffset" : 87
    }, {
      "referenceID" : 15,
      "context" : "Furthermore, the analogy relation could be expressed by vector arithmetic of word representations as shown in (Mikolov et al., 2013b).",
      "startOffset" : 110,
      "endOffset" : 133
    }, {
      "referenceID" : 2,
      "context" : "As in the previous subsection, we translated the word analogy dataset in (Chen et al., 2015) to traditional.",
      "startOffset" : 73,
      "endOffset" : 92
    }, {
      "referenceID" : 24,
      "context" : "As more character-level models(Zheng et al., 2013; Kim, 2014; Zhang et al., 2015) are proposed in the NLP field, we believe glyph features could serve as an enhancement, and we will further examine the effect of glyph features on other tasks, such as word segmentation, POS tagging, dependency parsing, or downstream tasks such as text classification, or document retrieval.",
      "startOffset" : 30,
      "endOffset" : 81
    }, {
      "referenceID" : 7,
      "context" : "As more character-level models(Zheng et al., 2013; Kim, 2014; Zhang et al., 2015) are proposed in the NLP field, we believe glyph features could serve as an enhancement, and we will further examine the effect of glyph features on other tasks, such as word segmentation, POS tagging, dependency parsing, or downstream tasks such as text classification, or document retrieval.",
      "startOffset" : 30,
      "endOffset" : 81
    }, {
      "referenceID" : 23,
      "context" : "As more character-level models(Zheng et al., 2013; Kim, 2014; Zhang et al., 2015) are proposed in the NLP field, we believe glyph features could serve as an enhancement, and we will further examine the effect of glyph features on other tasks, such as word segmentation, POS tagging, dependency parsing, or downstream tasks such as text classification, or document retrieval.",
      "startOffset" : 30,
      "endOffset" : 81
    } ],
    "year" : 2017,
    "abstractText" : "In this paper, we propose new methods to learn Chinese word representations. Chinese characters are composed of graphical components, which carry rich semantics. It is common for a Chinese learner to comprehend the meaning of a word from these graphical components. As a result, we propose models that enhance word representations by character glyphs. The character glyph features are directly learned from the bitmaps of characters by convolutional auto-encoder(convAE), and the glyph features improve Chinese word representations which are already enhanced by character embeddings. Another contribution in this paper is that we created several evaluation datasets in traditional Chinese and made them public.",
    "creator" : "TeX"
  }
}