Chinese Spelling Check (CSC) is a task to detect and correct spelling errors in Chinese natural language. Existing methods have made attempts to incorporate the similarity knowledge between Chinese characters. However, they take the similarity knowledge as either an external input resource or just heuristic rules. This paper proposes to incorporate phonological and visual similarity knowledge into language models for CSC via a specialized graph convolutional network (SpellGCN). The model builds a graph over the characters, and SpellGCN is learned to map this graph into a set of inter-dependent character classifiers. These classifiers are applied to the representations extracted by another network, such as BERT, enabling the whole network to be end-to-end trainable. Experiments 1 are conducted on three human-annotated datasets. Our method achieves superior performance against previous models by a large margin. Spelling errors are common in our daily life, caused typically by human writing, automatic speech recognition, and optical character recognition sys-tems. Among these errors, misspelling a character frequently occurs due to the similarity between characters. In Chinese, many characters are phonologically and visually similar, but semantically very different. According to Liu et al. (2010), about 83% of errors are related to phonological similarity and 48% are related to visual similarity. The Chinese Spelling Check (CSC) task aims to detect and correct such misuse of the Chinese language. Despite recent development, CSC remains a challenging task. Notably, the spelling checking on Chinese is very different from English, due to its language  Equal contribution. nature. Chinese is a language consisting of many pictographic characters without word delimiters. And the meaning of each character changes dramatically when the context changes. Therefore, a CSC system needs to recognize the semantics and aggregate the surrounding information for necessary modifications. Previous methods followed the line of generative models. They used either language models (Liu et al., 2013, 2010; Yu and Li, 2014) or sequence-to-sequence models (Wang et al., 2019). To fuse the external knowledge of the similarity between characters, some of them leveraged a confusion set, which contains a set of similar character pairs. For instance, Yu and Li (2014) proposed to produce several candidates by retrieving the confusion set and then filter them via language models. Wang et al. (2019) used a pointer network to copy a similar character from the confusion set. These methods attempted to utilize the similarity information to confine the candidates, rather than modeling the relationship between characters explicitly. captures the pronunciation/shape similarity and explore the prior dependencies between characters. Specifically, two similarity graphs are constructed for the pronunciation and shape relationship correspondingly. SpellGCN takes the graphs as input and generates for each character a vector representation after the interaction between similar characters. These representations are then constructed into a character classifier for the semantic representation extracted from another backbone module. We use BERT (Devlin et al., 2019) due to its powerful semantic capacity. Combining the graph representations with BERT, SpellGCN can leverage the similarity knowledge and generate the right corrections accordingly. Regarding the example as in Table 1, SpellGCN is able to modify the sentence correctly within the pronunciation constraint. Experiments were conducted on three open benchmarks. The results demonstrate that SpellGCN improves BERT evidently, outperforming all competitor models by a large margin. In summary, our contributions are as follows:  We propose a novel end-to-end trainable SpellGCN to integrate the pronunciation and shape similarities into the semantic space. Its essential components such as the specialized graph convolution and attentive combination operations are carefully investigated. We investigate the performance of SpellGCN both quantitatively and qualitatively. Experimental results indicate that our method achieves the best results on three benchmark datasets. The CSC task is a long-standing problem and has attracted much attention from the community. The research emerges in recent years (Jia et al., 2013; Xin et al., 2014; Yu and Li, 2014; Tseng et al., 2015; Fung et al., 2017; Wang et al., 2019; Hong et al., 2019), together with other topics, e.g., grammar error correction (GEC) (Rao et al., 2018; Ji et al., 2017; Chollampatt et al., 2016; Ge et al., 2018). CSC focuses on detecting and correcting character errors, while GEC also includes errors that need deletion and insertion. Previous work handles CSC using unsupervised language models (Liu et al., 2013; Yu and Li, 2014). The errors are detected/corrected by evaluating the perplexity of sentences/phrases. However, these models were unable to condition the correction on the input sentence. To circumvent this problem, several discriminative sequence tagging methods were adopted for CSC (Wang et al., 2018). For more flexibility and better performance, several sequence-to-sequence models were also employed (Wang et al., 2019; Ji et al., 2017; Chollampatt et al., 2016; Ge et al., 2018), as well as BERT (Hong et al., 2019). Recent attention was paid to utilizing the external knowledge of character similarity. The similarity knowledge can be gathered into a dictionary, i.e., confusion set, where similar pairs are stored. Yu and Li (2014) first used the dictionary to retrieve similar candidates for potential errors. Wang et al. (2019) incorporated a copy mechanism into a recurrent neural model. When given similar characters as input, their model uses the copy mechanism to directly copy the character to the target sentence. In a sense, these models face difficulty in modeling the relationship between similar characters as the similarity information is solely used for candidate selection. To capture the pronunciation/shape similarity and explore the prior dependencies between characters, we propose to use graph convolution network (GCN) (Kipf and Welling, 2017) to model character inter-dependence, which is combined with the pre-training of BERT (Devlin et al., 2019; Cheng et al., 2019) for the CSC task. GCN has been applied to model the relationship on several tasks. Yan et al. (2019) equipped it into the relation extraction task where relations construct a hierarchical tree. Li et al. (2018); Cheng et al. (2018) use it to model spatial-temporal to predict traffic flow. GCN was also used to model the relationship between labels in a multi-label task (Chen et al., 2019). In this paper, it is the first time that GCN is applied successfully into the CSC task. The relationship in CSC is much different from those tasks where objects in the graph are semantically related. By contrast, the similar characters are semantically distinct in CSC. Therefore, we deeply investigate the effect of our SpellGCN and propose several essential techniques. In this section, we elaborate on our method for CSC. Firstly, the problem formulation is presented. Then, we introduce the motivations for SpellGCN, followed by its detailed description. At last, we present its application in the CSC task. The Chinese Spelling Check task aims to detect and correct the errors in the Chinese language. When given a text sequence X = { x 1 , x 2 , ..., x n } consisting of n characters, the model takes X as input and output a target character sequence Y = { y 1 , y 2 , ..., y n } . We formulate the task as a conditional generation problem by modeling and maximizing the conditional probability p ( Y | X ) . The framework of the proposed method is depicted in Figure 1. It consists of two components, i.e., a character representation extractor and a SpellGCN. The extractor derives a representation vector for each character. Above the extractor, SpellGCN is used to model the inter-dependence between characters. It outputs target vectors containing the information of similar characters after interactions. As illustrated in Table 1, a vanilla language model is able to provide feasible corrections in semantic meaning but faces the difficulty in meeting the pronunciation constraint. Although the correction       is semantically plausible, its phonics differs much from       and      . This indicates that the similarity information between characters is necessary so that the model can learn to generate related answers. Previous methods have taken the similarity into consideration. However, they typically regarded similar characters as potential candidates, neglecting their inter-relationship in terms of pronunciation and shape. This work makes a preliminary attempt to handle this issue, trying to fuse both the symbolic space (phonological and visual similarity knowledge) and the semantic space (language semantic knowledge) into one model. To achieve this, we leverage the power of graph neural network (GNN) to infuse the similarity knowledge directly. The essential idea is to update the representations by aggregating the information between similar characters. Intuitively, a model is likely to have a sense of similar symbols when equipped with our method. Among various GNN models, we use GCN in our implementation. Since there are up to 5K Chinese characters in the graph, the light-weight GCN is more suitable for our problem. The proposed SpellGCN is depicted as follows in detail. SpellGCN requires two similarity graphs A p , A s for pronunciation and shape similarities correspondingly, which are derived from an open-sourced confusion set (Wu et al., 2013). For simplicity, the superscript will be omitted if unnecessary and A denotes one of these two similarity graphs. Each similarity graph is a binary adjacent matrix of size RN  N , constructed from N characters in the confusion set. The edge A i,j  { 0 , 1 } between i -th character and j -th character denotes whether the ( i, j ) pair exists in the confusion set. The goal of SpellGCN is to learn a map function to map the input node embedding H l  RN  D of l -th layer (where D is the dimensionality of character embedding) to a new representation H l +1 via convolutional operation defined by A . This map function has two main sub-components: a graph convolution operation and an attentive graph combination operation. Graph Convolution Operation The graph convolution operation is to absorb the information from neighboring characters in the graph. In each layer, the light-weight convolution layer in GCN (Kipf and Welling, 2017) is adopted: f ( A , H l ) =  AH l W lg , (1) where W lg  RD  D is a trainable matrix and  A  RN  N is the normalized version of the adjacent matrix A . For the definition of  A , we direct you to the original paper (Kipf and Welling, 2017). Note that we use the character embedding of BERT as the initial node features H 0 , and we omit the non-linearity function after convolution. Since we adopted BERT as our extractor, which has its own learned semantic space, we remove the activation function from the equation to keep the derived representation identical with original space, rather than a completely different space. During our experiments, using non-linearity activation such as ReLU is ineffective, resulting in a performance drop. Attentive Graph Combination Operation The graph convolution operation handles the similarity of a single graph. To combine the pronunciation and shape similarity graphs, the attention mechanism (Bahdanau et al., 2015) is adopted. For each character, we represent the combination operation SpellGCN BERT Extractor   $  $     $        :Word Embedding Layer-1 Layer-2 PronunciationSimilarity Graph ShapeSimilarity Graph A tt e n t i v e C o m b i n a t i o n            : GeneratedClassifier  Layer-3                                 DotProduct O u t o ft he C on f u s i on S e t Figure 1: The framework of the proposed SpellGCN. where C l  RN  D and f k ( A k , H l ) i is the i -th row of convolved representation of graph k ,  i,k is a scalar for i -th character denoting the weight of graph k . The weight  i,k is computed by  i,k = exp( w a f k ( A k , H l ) i / ) (cid:80) k (cid:48) exp( w a f k (cid:48) ( A k (cid:48) , H l ) i / ) , (3) where w a  RD is a learnable vector shared across the layers and  is a hyper-parameter which controls the smoothness of attention weights. We found  essential for the attention mechanism. Accumulated Output After graph convolution and attentive combination operations, we obtain a representation C l for l -th layer. To maintain the original semantic of the extractor, all outputs of previous layers are accumulated as the output: H l +1 = C l + l (cid:88) i =0 H i . In this way, SpellGCN is able to focus on capturing the knowledge of character similarity, leaving the responsibility of semantic reasoning to the extractor. Hopefully, each layer can learn to aggregate the information for the specific hop. During the experiments, the model failed when excluding H 0 . Similarity Graphs from Confusion Set The similarity graphs used in SpellGCN are constructed from the confusion set provided in (Wu et al., 2013). It is a pre-defined set consisting of similar characters for most of (  95%) the Chinese characters and these characters are categorized into five categories, i.e., (1) similar shape, (2) same pronunciation and same tone, (3) same pronunciation and different tone, (4) similar pronunciation and same tone, (5) similar pronunciation and different tone. Since the pronunciation similarity is more fine-grained compared with the shape similarity category, we combine the pronunciation similarities into one graph. Consequently, we construct two graphs corresponding to pronunciation and shape similarities. Character Representation by Extractor The representation of characters used for final classification is given by an extractor. We can use any model that is able to output representation vectors V = { v 1 , v 2 , ..., v n } (where v i  RD ) for n characters X = { x 1 , x 2 , ..., x n } . In our experiment, we adopt BERT as the backbone model. It takes X as input and uses the output of the last layer as V . We conduct the experiment using the base version, which has 12 layers, 12 self-attention heads with a hidden size of 768 2 . SpellGCN as Character Classifier When given the representation vector v i of a character x i , the model needs to predict a target character through a fully-connected layer whose weight W  RM  D is configured by the output of SpellGCN ( M is the size of the extractor vocabulary): p ( y i | X ) = softmax ( Wv i ) . Concretely, the output vectors of SpellGCN plays the role of the classifier in our task. We use the output of the last layer of SpellGCN HL (where L is the number of layers) to classify the characters in the confusion set. And since the confusion set only covers a subset of vocabulary, we use the word embedding of the extractor as the classifier for those excluded by the confusion set. In this way, denoting u i  { 1 , ..., N } is the index of confusion set for the i -th character in the extractor vocabulary, W is presented by: W i = (cid:40) H Lu i , if i -th character  confusion set E i , otherwise , (6) where E  RM  D is the embedding matrix of extractor. In brief, we use the embedding from SpellGCN if the character is in the confusion set. Otherwise, the embedding vectors are used as in BERT. Instead of modeling a large compact graph containing all characters in the extractor vocabulary, we chose this implementation for computational efficiency, since there are around 5K characters in the confusion set and more than 20K characters in the extractor vocabulary. Overall, the objective is to maximize the log likelihood of target characters: L = (cid:88) X , Y (cid:88) i log p ( y i = y i | X ) . (7) 3.5 Prediction Inference The CSC task consists of two sub-tasks in evaluation, i.e., detection and correction. Some previous work (Yu and Li, 2014; Liu et al., 2013) used two models for these sub-tasks separately. In this work, we simply use the character with the highest probability arg max  y i p ( y i | X ) as the prediction for the correction task. And the detection is achieved by checking whether the prediction matches the target character y i . as well as the evaluation metrics. Then we introduce our main results for SpellGCN. After that, the ablation studies are made to analyze the effect of the proposed components, followed by a case study. Finally, quantitative results are provided. Training Data The training data is composed of three training datasets (Wu et al., 2013; Yu et al., 2014; Tseng et al., 2015), which has 10K data samples in total. Following (Wang et al., 2019), we also include additional 271K samples as the training data, which are generated by an automatic method (Wang et al., 2018) 3 . Test Data To evaluate the performance of the proposed method, we used three test datasets from the SIGHAN 2013, SIGHAN 2014, SIGHAN 2015 benchmarks (Wu et al., 2013; Yu et al., 2014; Tseng et al., 2015) as in (Wang et al., 2019). We also follow the same data pre-processing procedure, i.e., the characters in these datasets are converted to simplified Chinese using OpenCC 4 . The statistic of the data is listed in Table 2. Baseline Models We compare our method with five typical baselines. LMC (Xie et al., 2015): This method utilizes the confusion set to replace the characters and then evaluates the modified sentence via a N-gram L anguage M odel. is adopted for detection. The incorrect characters are marked as 1 (0 otherwise). PN (Wang et al., 2019): This method incorporates a P ointer N etwork to consider the extra candidates from the confusion set. FASpell (Hong et al., 2019): This model utilizes a specialized candidate selection method based on the similarity metric. This metric is measured using some empirical methods, e.g., edit distance, rather than a pre-defined confusion set. BERT (Devlin et al., 2019): The word embedding is used as the softmax layer on the top of BERT for the CSC task. We trained this model using the same setting, i.e., the comparable model w/o SpellGCN. Evaluation Metrics The precision, recall and F1 scores are reported as the evaluation metrics, which are commonly used in the CSC tasks. These metrics are provided for the detection and correction sub-tasks. Besides the evaluation on the character level, we also report the sentence-level metrics on the detection and correction sub-tasks, which is more appealing for real-world applications. On the sentence level, we consider a sentence to be correctly annotated only if all errors in the sentence are corrected as in (Hong et al., 2019) 5 . On the character level, we calculate the metrics using the evaluation script from (Wang et al., 2019) 6 . We also evaluated BERT and SpellGCN by the official evaluation metrics tools 7 , which gives False Positive Rate (FTR), Accuracy and Precision/Recall/F1. Our code is based on the repository of BERT 8 . We fine-tune the models using AdamW (Loshchilov and Hutter, 2018) optimizer for 6 epochs with a batch size of 32 and a learning rate of 5e-5. The number of the layer in SpellGCN is 2, and the attentive combination operation with factor 3 is used. All experiments were conducted for 4 runs and the averaged metric is reported. The code and trained models will be released publicly after review (cur-rently, the code is attached in the supplementary files). Table 3 shows the performance of the proposed method on the three CSC datasets, compared with five typical CSC systems. When using SpellGCN, the model achieves better results in all test sets against vanilla BERT, which verifies its effectiveness. The improvement is considerable with such a large amount of training data (cf. the comparison in Figure 2). This indicates the similarity knowledge is essential for CSC and it can hardly be learned by simply increasing the data amount. In terms of sentence-level F1score metric in the correction subtask, i.e., C-F score in the last column, the improvements against previous best results (FASPell) are 9.2%, 9.7% and 13.3% respectively. Nevertheless, it should be noted that FASpell was trained on different training data while this paper follows the setting mentioned in the PN paper (Wang et al., 2019). Ideally, our method is compatible with FASpell and better results can be achieved when FASpell is employed. FASpell used their own metrics, which are different from the sentence-level false postive and false negtivate counting strategy of the official evaluation toolkit. We used the scripts by PGNet and FASpell to compute their metrics for fair comparison. We further add the official evaluation results of BERT and SpellGCN in Table 4. Actually, SpellGCN consistently improves the performance when evaluated by the PGNet/FASpell scripts and the official evaluation toolkit. We will add the FPR results in our revision. The FPR scores are 14.1% (SpellGCN) v.s. 15.3% (BERT) on SIGHAN 14, and 13.2% (SpellGCN) v.s. 13.6% (BERT) on SIGHAN 15. FPR on SIGHAN 13 is statistically meaningless since almost all the tested sentences have the spelling errors. In this subsection, we analyze the effect of several components, including the number of layers and the attention mechanism. The ablation experiments were performed using 10K training data. Effect of the Number of Layers Generally, the performance of a GCN varies with the number of layers. We investigate how the number of SpellGCN layers influence the performance in CSC. In this comparison, the number of layers changes from 1 to 4, and the results are illustrated in Figure 3. For clarity, we report the character-level C-F on the three test datasets. The results indicate that SpellGCN is able to make use of multiple layers. With multiple layers, SpellGCN can aggregate the information in more hops and therefore, achieve better performance. However, the F1score drops when the number of layers is larger than 3. This is reasonable due to the over-smooth problem noted in (Yan et al., 2019). When the number of GCN layers increases, the representations of neighboring characters in the similarity graph will get more and more similar since they all are calculated via those of their neighbors in the similarity graph. Effect of Attention Mechanism We investigate how to better combine the graphs in the SpellGCN layer. Here, we compare the attention mechanism against sum-pooling and mean-pooling, with different hyper-parameter  mentioned in Section 3.3. The experiments are conducted based on the 2-layer SpellGCN on SIGHAN 2013 test set. The results presented in Table 5 show that the sum pooling fails in the CSC task. We suggest that the sum pooling is inconsistent with the normalization of GCN and fails to combine the information from different channels (i.e., graphs). The mean pooling is feasible but is surpassed by the attention mechanism. This indicates that the adaptive combination for each character node is beneficial. We incorporate a hyper-parameter  into the attention operation since the dot products may grow large in magnitude, pushing the softmax function into regions where it has extremely small gradients. With these results, we chose the attention mechanism with a  of 3 in SpellGCN. We show several correction results to demonstrate the properties of SpellGCN. In addition to the sample illustrated in Table 1, several prediction results are given in Table 6. From these cases, we can tell that our SpellGCN is capable of revising the incorrect characters into correct ones with the pronunciation and shape constraint. For instance, in the first case,    (fang) is detected as errors and modified into    (fan). Without pronunciation similarity constraint,    (m`u) becomes the most probable answer. And surprisingly, in the second case, our SpellGCN successfully modifies the character reasonable in the context. The meaning of input sentence       is watch the audio recorder, and our method corrects it into      which means watch the video recorder. We suggest that SpellGCN injects a prior similarity between    and    in the representation space so that the model derives a higher posterior probability of   . In the last case, we show a correction result under the shape constraint. In the confusion set,    is similar to    and therefore, using SpellGCN is able to retrieve the correct result. Previous experiments have explored the performance of SpellGCN quantitatively in detail. To qualitatively study whether SpellGCN learns meaningful representations, we dive into the target embedding space W derived from SpellGCN. In Figure 4, the embedding of characters with phonics chang and s` is presented using t-SNE (Maaten and Hinton, 2008). The embedding              learned by BERT captures the semantic similarity but fails to model the similarity in terms of pronunciation for the CSC task. This is reasonable as this similarity knowledge is absent in the modeling. In contrast, our SpellGCN successfully infuses this prior knowledge into the embedding and the resulting embedding exhibits cluster patterns. The embedding of characters with these two different pronunciations forms two clusters, corresponding to chang and s` respectively. Due to this property, the model tends to recognize similar characters and hence is able to retrieve the answers under pronunciation constraint. Figure 5 shows the same situation for the shape similarity, where two sets of characters with the shape similar to    and    are scattered. This verifies the ability of SpellGCN in modeling shape similarity. We proposed SpellGCN for CSC to incorporate both phonological and visual similarities into language models. The empirical comparison and the results of analytical experiments verify its effectiveness. Beyond CSC, SpellGCN can be generalized to other situations where specific prior knowledge is available, and to other languages by leveraging specific similarity graphs analogously. Our method can also be adapted to grammar error correction, which needs insertion and deletion, by utilizing more flexible extractors such as Levenshtein Transformer (Gu et al., 2019). We leave this direction to future work.
abstain abstain abstain objective abstain abstain abstain result abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain method abstain abstain abstain abstain objective objective objective result other other other other other other other other other other other other other other abstain other other other other objective other other objective method abstain method method abstain abstain method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain method method abstain method method method abstain abstain abstain abstain abstain abstain abstain abstain abstain method method method method abstain abstain method abstain other method method abstain abstain abstain abstain abstain abstain method method abstain method method abstain method abstain method method method abstain method abstain method abstain abstain abstain method abstain abstain result abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain result abstain abstain result abstain result abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain result abstain abstain abstain abstain abstain abstain abstain abstain result abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain objective abstain abstain method abstain