Chinese NLP applications that rely on large text often contain huge amounts of vocabulary which are sparse in corpus. We show that characters' written form, Glyphs , in ideographic languages could carry rich semantics. We present a multi-modal model, Glyph2Vec , to tackle Chinese out-of-vocabulary word embedding problem. Glyph2Vec extracts visual features from word glyphs to expand current word embedding space for out-of-vocabulary word embedding, without the need of accessing any corpus, which is useful for improving Chinese NLP systems, especially for low-resource scenarios. Experiments across different applications show the significant effectiveness of our model. Word embedding encoded semantic and syntactic information (Mikolov et al., 2013a,b) in low-dimensional space have served as useful features for various NLP applications but often require large-scale corpus with billions of tokens to train. A natural constraint of word embedding is that it is not practical to collect the entire vocabulary of any language with large enough frequency to train the embedding for every word, since some new words may appear in downstream tasks. A typical solution is to simply assign a specific UNK embedding to all out-of-vocabulary (OOV) words that do not appear in the training data. Current solutions such as using subwords (e.g., characters) are mainly considering alphabetic languages (e.g., English and French) that are composed of small amount of characters. Such techniques may not be sufficient for ideographic lanEqually contribution. guages (e.g., Chinese and Japanese) in which a word is often composed with characters of a large amounts. An example is that traditional Chinese includes about 17k distinct tokens. Therefore, it could be expected to suffer from underfitting not only word embedding but also character embedding. Even worse, words in ideographic languages are often composed of 2-3 characters only, unlike words in alphabetic languages are longer but with smaller types of characters. Figure 1 provides the statistics in Chinese Sinica Corpus. acter is made up of several graphical components. Figure 2 shows some examples that components in characters represent similar semantic or pronunciation. In addition to glyphs, we propose to use the high-quality features provided by Cangjie input method to represent each character. Cangjie is a popular Chinese input method. Similar to radicals, characters are composed of 24 basic graphical units. Each unit is mapped to a corresponded letter key on a standard QWERTY keyboard. Building beyond character glyphs, one can intuitively guess the semantic of a word. Recent work (Chen et al., 2015; Xu et al., 2016; Yin et al., 2016; Liu et al., 2017; Su and Lee, 2017) have shown ben-efits of the compositionality at character level or visual feature of Chinese glyphs for some tasks. In this work, we suggest that in the OOV scenario glyphs can be particularly useful. A key observation for solving OOV problem matches the intuition of human generalization in Chinese. When a Chinese user reads an unseen word or a character, by decomposing the structure, graphical components such as radicals for a character often help Chinese users understand the meaning and sometimes pronunciation of the character. We study a novel application that recovers Chinese OOV word embeddings from glyphs. Our work is to answer a question : given the pretrained word embeddings, can we directly learn a mapping from word glyphs to their word embedding and generalize the mapping for the purpose of generating the embedding of OOV words? We formulate it as a visual-to-text transfer learning problem and show that the visual structure of Chinese characters is helpful in learning Chinese OOV embeddings. Exploiting Structure of Chinese Characters Recent work have explored the use of Chinese character structure in different settings (E and Xi-ang, 2017; Liu et al., 2017; Dai and Cai, 2017). Several work aim to use character-level feature to enhance standard word embedding learning models (e.g., Word2Vec or GloVe). CWE (Chen et al., 2015) propose to use character-level formulation for words in training word embeddings; SCWE (Xu et al., 2016) and Li et al. (2015) extends to consider the relations of characters compositionally. MGE (Yin et al., 2016) and Shi et al. (2015) further includes radical information associated to characters. Yu et al. (2017) jointly embed Chinese words, characters, and radicals. GWE (Su and Lee, 2017) proposes to extract feature from character bitmaps as the inputs of Word2Vec and GloVe. Our work is different from all of them, since we emphasize on generating the OOV word embeddings, which is not handled by them. Learning Embedding for OOVs To handle OOV words, an approach is operating character level embeddings, then averages them into word embeddings (Kim et al., 2016; Wieting et al., 2016). Morphology-based approaches take advantage of meaningful linguistic substructures (Botha and Blunsom, 2014; Luong et al., 2013; Bhatia et al., 2016). Morphology-based approaches often struggle with those vocabularies lacking linguistic substructures such as names and transliterations of foreign language, which often appears as OOV words. In all the models above, just like Word2Vec (Mikolov et al., 2013c)), the embeddings meed to learned by training over a large corpus. The most similar work is Mimick model (Pin-ter et al., 2017). By learning a character language generating model, guided by minimizing the distance between the output embedding of LSTMs and pre-trained word embeddings, Mimick shows feasibility of generating OOV word embedding from character compositions. However, Mimick is mainly from the view of alphabetic languages that does not consider glyphs. Chinese words often consist of short sequences composed of many kinds of tokens that are difficult for language model approaches to handle (see Figure 1) and could suffer from under-fitting. We formulate the task of learning OOV embeddings as a transfer learning problem. Formally, given a Chinese vocabulary set V of size |V| , and a pre-trained embeddings matrix E  R |V| d where each word w i is associated with a vector e i of dimension d as training set { w i , e i } |V| i =1 . We aim to learn a mapping F : w  R d , where F projects the input word to the d dimension embedding space such that F ( w i )  e i . In testing, a word w t may be out of V , while the model is still obliged to predict the embedding e t with F ( w t ) . Given the glyphs for a word x = [ c j ] | x | 1 as a sequence of character 2D bitmaps c provided according to V , we can considering a function g : x  R k that transforms glyphs into vi-Figure 3: Complete network architecture of our Glyph2Vec. White boxes annotate the feature dimension of each character . Different features are combined by concatenating. GRU takes sequence of character feature as inputs. sual features of k dimension. Another function f : g ( x )  R d later maps the visual space to the word embedding space. The final embedding can be obtained with e i = F ( x i ) = f ( g ( x i )) , where input is glyph x i . The overall framework is illustrated in Figure 3. 3.1 Visual Feature Extractor We consider two implementations of visual feature extractor g . ConvAE We adopt the convolutional autoencoder ConvAE (Masci et al., 2011) to capture the structure of characters bitmaps c . The architecture of the ConvAE follows Figure 6 in (Su and Lee, 2017). Eventually, the well-trained encoder is fixed as extractor that extracts 512-dimensional feature for every character c . The input bitmaps are 60  60 8-bit images in grayscale. Cangjie Composition We propose to use Cangjie input codes as high-level annotations of characters, which can be easily collected from the input method dictionary. We construct a Bag-of-Root (BoR) vector for each character according to the Cangjie dictionary. Each BoR binary vector of 24 dimensions representing the roots that a character possesses. After the visual features of every character in a word are extracted, we still need to compose them to word level. A compositional model f takes a sequence of characters' visual feature and projects them onto the word embedding space. The right portion of Figure 3 shows the architecture of f . We construct a bi-directional RNN network with GRU cells (Cho et al., 2014) to compute the expected word embedding over the character feature sequence. Finally, the 300D word embeddings are predicted. To calculate the loss for backpropa-gation, we adopt squared Euclidean distance between the prediction F = f ( g ( x )) and the gold word embedding w : (cid:107) F ( x )  w (cid:107) 2 . Unlike alphabetical languages, each Chinese character carries its own meaning. State-of-the-art Chinese word embedding models (Chen et al., 2015; Xu et al., 2016; Yin et al., 2016) often consider learning character embedding jointly. We demonstrate how to incorporate pre-trained character embedding to further improve the performance. The character embeddings are concatenated with the glyph features and the BoR Cangjie vectors as inputs. Character embedding is a huge embedding matrix. In Table 1, we summarized the required #parameters. We note that Glyph2Vec can infer OOV embedding directly from glyphs without character embedding. We adopt the Word2Vec traditional Chinese 300d word embedding pre-trained on public-available Sinica Corpus 4.0 which includes about 10M tokens. For optimization, we train 100 epochs with RMSProp optimizer with learning rate 4e-4 with batch-size 128. We note the models compared in the following experiments here. M is for Mimick baseline (Pinter et al., 2017) based on the authors' code. For the proposed feature, we test several combinations. C is for using Cangjie BoR vector; V is for using glyph visual feature; Char is for appending pre-trained character embedding. We uti-http://asbc.iis.sinica.edu.tw/ Figure 4: Principal component analysis visualization of the produced word embedding. lize the embeddings from Polyglot (Al-Rfou et al., 2013). As a sanity check, in Fig. 4 we visualize the embedding of seen and OOV words. One could observe meaningful clusters that have similar visual structure. For example,   (roast chicken) could be mapped with   (roast duck) because  (chicken) and  (duck) have different glyphs both about bird. Some cooking verbs that have the radical  (fire) like  (roast) and   (roast) are also mapped closely. Some unseen characters (or words with only one character) can also be predicted reasonably. We qualitatively analyze Glyph2Vec with nearest neighbor (NN) sanity check. Table 2 shows the results of retrieved nearest neighbors with OOV word queries for Mimick and our Glyph2Vec embeddings (using V ), respectively. We observe Glyph2Vec is able to model visual semantic by associating those characters that share related visual features since Glyph2Vec learns from the images of characters. For example,  (eel) in   (snake-eel) shares the radicals of  (fish) with   (Haemulidae, fish name). (Rh) and  (Cl) in     (RhCl3) associate some visual features relate to chemicals like  in  (Ce),  in  (F),  in  (acid), and more. On the other hand, we observe some properties including composition (e.g., numbers) and character semantic that both Glyph2Vec and Mimick can provide. (1) Composition: composing characters that have very different mean-https://sites.google.com/site/rmyeid/projects/polyglot ing after splitting them. For instance,     is a transliteration of Seleznev (Russian name), for which every character is meaningless alone but a meaningful transliteration when combined. With character-level compositional model in Glyph2Vec, it could be retrieved given     (Claudio, western name). Moreover, Glyph2Vec preserves correct meaning of a character when attaching with the other characters. For example,  (abrupt)  (decrease) can retrieve   (cut back) and   (reduce) properly when  (subtract) is associated to different characters. (2) character semantic: associating different characters with similar meaning. For example,  (street) is related to  (lane) or  (alley) and they are retrieved by our model given     (Xuefu 2nd Street) as the OOV word even though the characters look completely different. We follow the experiment protocols of parts-of-speech tagging and morphosyntactic attributes tagging stated in Mimick (Pinter et al., 2017) for this experiment. There are two parts-of-speech tagging tasks based on the Chinese thread of Universal Dependencies (UD) scheme (De Marneffe et al., 2014). To avoid tuning towards those OOV words, we consider the similar evaluation protocols of generalized zero-shot learning (Chao et al., 2016; Xian et al., 2017) that the embedding of not only unseen but also seen words need to be generated. Both word-level LSTM and character LSTM are reported (Table 3). With visual feature available, Glyph2Vec consistently outperforms Mimick. On the other hand, we observe using pretrained character embedding only helps on accuracy of seen words but not OOV words, which suggests that it is necessary for a module like Mimick or Glyph2Vec to learn to compose characters for OOV words. As we introduced in Sec. 1, in real-world scenario Chinese systems could suffer from severe OOV problem. An example is Wikipedia encyclopedia. It contains lots of rarewords that easily become OOV words such as terminologies, scien-tific names, geography locations, ... etc. We utilize Wikipedia Title dataset (Liu et al., 2017) to https://github.com/frederick0329/Wikipedia-TitleDataset Query Word Top 5 Nearest Neighbors     (numbers)    (city)    (numbers)    (county)            (numbers)   (drama)   (dream)   (lividly)    (name)     (Juliette Binoche)    (name)   (slump)    (water resource)   (shrimp)    (dosage)      (proportion)   (pond)   (provided job)   (shrimp)    (office)    (name)    (name)    (office)   (snakebird)    (euryhaline)   (fish)   (fish)    (gull)   (worm)     (Claudio)   (Cha)    (Selig)   (Same)   (Okana)    (Ladur)     (RhCl3)  (stew)   (viscous)    (medicine)    (name)     (bacteria)     (idiom)    (goggles)   (catch crab)   (riverside)    (time)    (riverside)     (street)    (name)   (shrimp)   (person)    (name)    (name)     (numbers)    (numbers)    (numbers)    (numbers)    (numbers)    (numbers)   (drama)  (unit for drama)   (naked play)   (dance drama)    (musical drama)   (drama)   (slump)  (slump)   (slump)   (year by year)   (dramatically)   (slump)   (provided job)   (job)   (job)   (military service)   (hire)   (hire)   (snakebird)   (fish)    (euryhaline)   (fish)    (fish)   (fish)     (Claudio)    (Puchkov)    (Chimet)   (Tsev)     (Seleznev)    (Itkine)     (RhCl3)    (inorganic acid)  (Ce)    (FCl2)    (chemical Eq.)    (anode plate)     (idiom)     (idiom)     (idiom)     (idiom)     (idiom)     (idiom)     (street)     (street)   (street)     (street)       (street)    (street) Table 2: Nearest neighbors examples retrieved by Mimick (upper) and Glyph2Vec (lower). study the problem. The dataset is a collection of 593K Chinese articles from Wikipedia and categorizing them into 12 classes based on their titles. We preprocessed the data by removing punctuation, special characters, and other non-Chinese instances, and turning Arabic numbers into Chinese text. We use opensource Jieba toolkit to segment each title into words. 52.5% are OOV based on Sinica Corpus, and we generate their embeddings by Glyph2Vec. We construct a neural network classifier with the generated word embedding as input to evaluate our method. The classifier is consist of 3 fully-connected (FC) layers on top of the averaged word embedding of titles. Results are shown in Table 4. With glyph feature and Cangie BoR feature provided, the performance could be improved significantly compared to neglecting OOV (as UNK) in such challenging setting. In this work, we propose a multi-modal framework that expand pre-trained embedding space to include OOV words using character visual features such as Cangjie feature and Chinese character glyphs. We have demonstrated the effectiveness of Glyph2Vec on traditional Chinese, and we believe Glyph2Vec can also be applied to other ideographic languages to handle OOV words as well. We note that the accuracy cannot be compared with the report in (Liu et al., 2017) since they did not consider OOV and char/word embeddings. Here we only use the dataset to examine the performance of OOV embedding. For simplified Chinese, we suggest users to first translate into traditional Chinese since traditional characters have richer structures and probably more semantics can be extracted through Glyph2Vec.
abstain result method abstain result abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain objective abstain abstain abstain abstain abstain abstain abstain abstain objective objective result other other other other other other objective other other other other other other other other method abstain abstain abstain method abstain abstain abstain abstain abstain abstain abstain method method abstain abstain abstain abstain method abstain method abstain abstain method abstain method abstain abstain abstain abstain abstain abstain method method method method abstain method abstain other abstain method abstain abstain abstain abstain method abstain abstain abstain abstain abstain other abstain abstain abstain abstain abstain method method abstain abstain abstain abstain abstain abstain abstain abstain other abstain abstain method method method method abstain abstain abstain objective objective method method objective