While GPT has become the de-facto method for text generation tasks, its application to pinyin input method remains unexplored. In this work, we make the first exploration to leverage Chinese GPT for pinyin input method. We find that a frozen GPT achieves state-of-the-art performance on perfect pinyin. However, the performance drops dramatically when the input includes abbreviated pinyin. A reason is that an abbreviated pinyin can be mapped to many perfect pinyin, which links to even larger number of Chinese characters. We mitigate this issue with two strategies, including enriching the context with pinyin and optimizing the training process to help distinguish homophones. To further facilitate the evaluation of pinyin input method, we create a dataset consisting of 270K instances from fifteen domains. Results show that our approach improves the performance on abbreviated pinyin across all domains. Model analysis demonstrates that both strategies contribute to the performance boost. GPT (Radford et al., 2018, 2019) is a Transformer-based (Vaswani et al., 2017) language model that predicts tokens in an autoregressive manner. With a generic model architecture and the availability of vast web text data, GPT has been successfully developed for English, Chinese (Du, 2019; Zhang et al., 2021b), and many other languages. It shows extraordinary ability to generate fluent sentences and has been successfully applied to a wide range of natural language generation tasks. However, it remains unexplored to what extent GPT handles Chinese pinyin input method 1 , which is used by  Work done during internship at Tencent AI Lab. Pinyin input method allows users to enter Chinese characters based on their pronunciations. Given a pinyin 2 as the input, pinyin input method returns a list of Chinese characters pronounced with that pinyin. Fundamental elements of pinyin include initials (   ) and finals (   ). In most cases, a Chinese character is spelled with one initial followed by one final. For example, as shown in Table 1, the initial and final for the Chinese character   (me) are w and o , respectively. People may enter perfect pinyin (e.g.,  wo men  for    ), where initials and finals of all Chinese characters are entered. There are about 420 perfect pinyin in common use. Sometimes, especially when multiple Chinese characters are entered at once, people may use abbreviated pinyin by only entering the initials of characters (e.g.,  w m  for    ). This work, to the best of our knowledge, is the first one to explore the use of Chinese GPT for pinyin input method. We start by testing the performance of a frozen GPT. In this setting, we fix the parameters of GPT and predict Chinese characters from left to right in an autoregressive manner. At each time step, only characters pronounced with the same pinyin are legitimate candidates to be predicted. We find that, when the input is perfect pinyin, a frozen GPT performs comparably to state-of-the-art systems on the benchmark dataset (Yang et al., 2012). However, when the input is abbreviated pinyin with only initials of characters, the 2 https://en.wikipedia.org/wiki/Pinyin 1899 Id Context of Characters Input Pinyin Target Pinyin Type s1          li bai yi you dian shi       Perfect s2          l b y y d s       Abbreviated s3           l b y y d s       Abbreviated Table 2: Illustrative examples of the task of pinyin input method with perfect pinyin and abbreviated pinyin. performance of GPT has a drastic drop. A major reason is that an abbreviated pinyin maps to many perfect pinyin. For example, the initial  w  can be the abbreviation for  wo ,  wei ,  wang ,  wai ,  wu , etc. This would lead to exponentially larger number of legitimate candidates of Chinese characters. We mitigate this problem by incorporating pinyin information from two directions. One is to enrich the input by adding pinyin as additional context. The other is learning over pinyin-constrained vocabulary, which enhances the model's ability to distinguish between Chinese characters pronounced with the same pinyin. To further facilitate the research on pinyin input method, we construct a new dataset based on the WuDaoCorpora (Yuan et al., 2021). Our dataset includes 270K instances from 15 commonly used news domains. 3 To evaluate towards multiple facets, the dataset covers instances with different numbers of context characters and pinyin. From our experiment results, we have these key findings: 1. On perfect pinyin, frozen GPT achieves state-of-the-art results. 2. On abbreviated pinyin, the performance of frozen GPT drops drastically. Context enrichment with pinyin and pinyin-constrained training both improve the performance. 3. The performance of GPT-based models increases as the context of Chinese characters becomes longer. The input of pinyin input method includes a sequence of Chinese characters C = { w 1 , . . . , w n } as the context and a sequence of pinyin P = { p n +1 , . . . , p n + k } , where w i  V w , p n + j  V p , and V w and V p are the vocabularies of words and pinyin, respectively. The output is a sequence of Chinese characters O = { w n +1 , . . . , w n + k } , where w n + i  V w . The number of output characters is the same as the number of pinyin (i.e., k ) and each character should be pronounced with the corresponding pinyin. The output sequence is desired to follow the context of Chinese characters to form a coherent sentence. As mentioned earlier in the introduction section, the input pinyin might be perfect (e.g.,  wo men ) or abbreviated (e.g.,  w m ). Examples of the task are given in Table 2. 4 In our definition, one situation is that the context of characters is empty, which corresponds to the scenario that people are entering pinyin at the beginning of a sentence. The other situation is that the context includes real words, which stands for the scenario that people are entering pinyin in the middle of a written sentence. In this paper, we assume that the oracle pinyin segmentation results are provided. Sometimes, a raw pinyin sequence can be mapped to different segmentation results. For example, the raw pinyin input  jianshi  can be segmented as  ji an shi  (    , a city in the southwestern part of Jilin province, China) or  jian shi  (   , which is translated as experience in English). Pinyin segmentation is a subtask (Zhao et al., 2006; Zhou et al., 2007) of pinyin input method, which is well solved with the accuracy of 98% (Zhang et al., 2017). We leave the integration of pinyin segmentation as future work. In this section, we first introduce standard text-based GPT models adopted in this work (Sec-tion 3.1). Afterwards, we introduce how to extend GPT models for pinyin input method with enriched pinyin context (Section 3.2) and pinyin-constrained 4 People may also input pinyin like  l b y you dian shi , we leave this as a future work. training (Section 3.3), respectively. In this work, we use character-level Chinese GPT as the backbone. We describe character-level GPT models in this subsection. We start with a publicly available character-level GPT (Du, 2019) 5 , which we call GPT (public) . The model has the same configuration as the standard 12-layer GPT 6 . It is trained on the CLUECor-pusSmall dataset of 14GB (Xu et al., 2020), which consists of Chinese news, Wikipedia, online forum message, and consumer comments. We have tried another well known Chinese pretrained language model called CPM (Zhang et al., 2021b), which is trained on 100GB data. The vocabulary of CPM contains both Chinese characters and words. 7 We built a baseline with the CPM model of 12 layers 8 and forced the generated token to be a Chinese character. However, this baseline does not work well on pinyin input method, partly because our character-level decoding is inconsistent with the way how CPM is trained. It is promising to leverage the advantage of CPM on word-level decoding, and we leave this as a future work. To build a stronger Chinese GPT baseline, we use GPT (public) as the starting point and further pretrain on a 800GB data crawled by us that is composed of news, Wikipedia, and novel texts. The model is trained with a batch size of 2,560 on 32x Tesla V100 GPUs. We adopt the Adam optimizer (Kingma and Ba, 2015) and set the learning rate to 1e-5 with a linear warmup scheduler. We run the warmup process for 10k steps and train 100k steps in total. We call this 12-layer GPT model as GPT (ours) . To apply GPT (public) and GPT (ours) to pinyin input method, we use the traditional decoding pipeline of GPT to generate the sequence of Chinese characters in an autoregressive way. After encoding all the context of characters, the model predicts a Chinese character at each time step conditioned on the pinyin. Only Chinese characters pronounced with the same pinyin are legitimate 5 https://github.com/Morizeyao/ GPT2-Chinese 6 https://huggingface.co/gpt2 7 A Chinese word may consist of multiple Chinese characters. We explore two simple ways to incorporate pinyin information and build two models correspondingly. The first model uses pinyin information horizontally by concatenating pinyin input to the context of characters. The second model incorporates pinyin information vertically by adding a pinyin embedding layer at the bottom of GPT. PinyinGPT-Concat In this model, we append a pinyin sequence to the context of Chinese characters. In the inference stage, the input has the form of x = [ w 1 , . . . , w n , [SEP] , p n +1 , . . . , p n + k , [SEP] ] , where [SEP] is a special token to separate text and pinyin. The model largely follows the architecture of the standard GPT. Since there is one-one relationship between pinyin tokens and generated Chinese characters (i.e., the pronunciation of w n + j is p n + j ), we adjust the absolute positions of the characters to be generated. We assign the position of p n + j to w n + j , expecting the model to learn the alignments between pinyin and target characters. 9 We further expand the vocabulary of the word embedding layer by adding pinyin tokens. In the training stage, given an training instance of [ w 1 , . . . , w n , [SEP] , p n +1 , . . . , p n + k , [SEP] , w n +1 , . . . , w n + k ] , the model is trained to minimize the following loss function, where w <n + j stands for the characters before w n + j and p = [ p n +1 , . . . , p n + k ] . PinyinGPT-Embed The original GPT model includes a word embedding layer and a position embedding layer. In this model, we add a pinyin embedding layer. The basic idea is to provide the model with the pinyin of the character to be generated next. Specifically, the embedding of each character is the sum of the token embedding of the current character, the position embedding of the current character and the pinyin embedding of the next character. When a word (e.g., numbers, punctuations and symbols) has no corresponding pinyin, we use a special token [unk] to represent it instead. The training process is similar with 9 On abbreviated pinyin, this strategy could bring 0.3 points in terms of P@5. 1. The loss function is given as follows. In the inference stage, we transform the input sequence to the same format. We describe training details in this subsection. In standard GPT, the loss function is computed over the whole vocabulary. However, this is suboptimal for pinyin input method because the major challenge in the inference stage is how to select the best one from characters pronounced with the same pinyin (as described in the end of Section 3.1). This leads to inconsistency between training and inference stages. Therefore, in the training stage, the probability of a character is calculated over characters pronounced with the same pinyin, which is formulated as follows. where V p i is the set of Chinese characters whose pinyin is p i and g is the logit before the softmax layer. In this section, we show the results on pinyin input method over the two settings (i.e., perfect pinyin and abbreviated pinyin). We describe the two datasets used in the following experiments and the evaluation metric. PD Dataset PD dataset (Yang et al., 2012) is a commonly used benchmark dataset for the evaluation of pinyin input method (Jia and Zhao, 2014; Zhang et al., 2017; Huang et al., 2018; Zhang et al., 2019). The texts in PD are extracted from the Peo-ple's Daily 10 from 1992 to 1998. It contains 5.04 million segments of consecutive Chinese characters (or Maximum Input Unit in some literature) for training and 2,000 segments for testing. For each test case, the input pinyin are all perfect pinyin and the context is null. WD Dataset Since the PD data includes out-of-date news from 20 years ago and does not support us to study the scenario where the context includes real words, we construct a new dataset called WD. We use the WuDaoCorpora (Yuan et al., 2021) that contains 3TB Chinese corpus collected from 822 million Web pages. Currently, 200GB of the corpus 10 http://www.people.com.cn/ 1902 has been made publicly available 11 . We randomly select 15 domains from WuDaoCorpora. For each domain, we first use an off-the-shelf Chinese segmentation toolkit (Zhang et al., 2020) to segment the documents into sentences. Then we automatically obtain the perfect pinyin and abbreviated pinyin of characters with pinyin converting tools. For each sentence, we randomly choose a context with a range from 0-3, 4-9 and 10+ words. Consecutively, we choose the target to be 1-3, 4-9 or 10+ words, respectively. It's further required that the target should be continuous characters that each has its own pinyin. We call each context-target length tuple like (4-9, 10+) as an evaluation configuration. For each configuration, we sample 2,000 test cases. In total, there are 9 configurations of 18,000 cases for each domain. The whole dataset consists of 270,000 examples. We investigate extremely long target lengths for the purpose of research that these configurations may not appear in real cases. All the instances in the WD dataset are only used for evaluation. Evaluation Metric We use precision at top-K (P@K) as the evaluation metric, which is widely adopted in the literature (Jia and Zhao, 2014; Zhang et al., 2017, 2019). It measures if the ground truth exists in the topK generated results. Some existing works also use keystroke-based metrics (Jia and Zhao, 2013; Huang et al., 2015) and human evaluation, which we don't use in this work because the evaluation process is more complex and time-consuming. Other Settings We train both PinyinGPT models with the training data of GPT (ours). To preprocess the corpus, we use a public library pypinyin 12 to get the pinyin of Chinese characters. 13 We initialize both PinyinGPT models with GPT (ours). Both models are trained for 100k steps on 32 GPUs of NVIDIA V100 Tensor Core with a bach size of 25,000. The learning rate is 5e-5. We maintain a maximum of 128 tokens for every training example. We use a probability of 50% to sample a target sequence with less than 5 words, otherwise we randomly sample a target sequence with 6 to 25 words. This rate is empirically selected as it's less practical for users to type very long sequences. During inference stage, we use beam search with a beam size of 16 for text generation. We report results on the PD dataset (Yang et al., 2012). We use pinyin-constraint training in all configurations and train PinyinGPT models with different pinyin vocabularies for perfect pinyin and abbreviated pinyin, respectively. We compare with the following baselines. Google IME is a commercial Chinese IME which provides a debuggable API. On-OMWA (Zhang et al., 2017) is an online model for word acquisition which adaptively learns new words for Chinese IME. On-P2C (Zhang et al., 2019) is a neural pinyin-to-Chinese character conversion model, which is augmented by an online updated vocabulary to support open vocabulary learning. In Table 3, the first group (top) shows the results of the aforementioned baselines, which are directly extracted from On-P2C (Zhang et al., 2019). The bottom group shows the performance of GPT (pub-lic) and GPT (ours) with frozen parameters. We can find that GPT (public) achieves comparative performance with existing systems in terms of P@5 and P@10. After being trained with a larger corpus, GPT (ours) surpasses all the baseline models in terms of all metrics. It is worth noting that existing baselines are supervised models that are fine-tuned on training instances. The results demonstrate the effectiveness of GPT models pretrained on vast amount of texts. In this section, we report results for both perfect pinyin and abbreviated pinyin on WD. In Table 4, we list the overall experiment results of two GPT baselines as well as our PinyinGPT models. We have several findings based on the results. First, from each row, we can see that there is a drastic performance drop for all models. The reason is that each abbreviated pinyin can be mapped to a large amount of candidate characters, so that the problem is more challenging compared to perfect pinyin. We also believe that the evaluation metric of P@1 might be too strict for abbreviated pinyin because sometimes the top predictions might be correct (as reflected in Figure 3) even though they may be different from the ground truth. Second, adding pinyin information to GPT obtains limited improvement on perfect pinyin, but boosts the abbreviated setting by 5 points on P@5 and 7 points on P@10, respectively. Third, concatenating pinyin context horizontally is better than adding pinyin embedding vertically. Last, fine-tuning all the parameters performs better than keeping the parameters of GPT fixed. In this section, we conduct experiments to understand the importance of pinyin context and pinyin-constrained training. Results are given in Figure 2. The baseline model is GPT (ours). The model + Pinyin Context means that we concatenate pinyin context (i.e., PinyinGPT-Concat) and learn over the whole vocabulary. The model + Pinyin Context + PC-LOSS means that we use both pinyin context and pinyin-constrained training. The figure shows that taking pinyin as extra context works well to improve results in terms of P@5 and P@10. When the two components are adopted, the performance is further improved. To analyze how context length and target length affect performance, we aggregate experiment results to form a matrix of accuracy for each configuration in Table 5. Each score is averaged over all the domains. From each column, we can see that longer context benefits both GPT and our model in pinyin input method, which verifies the power of context understanding ability of GPT models. An interesting finding is that, when the context is long enough (e.g., 10+), adding pinyin does not help improve the P@1. We list three cases in Figure 3 to compare model outputs produced by GPT (ours) and PinyinGPT-Concat. The first case shows that, given perfect pinyin as the input, both GPT (ours) and PinyinGPT-Concat make the correct predictions. In the second case, abbreviated pinyin is given as the input. PinyinGPT-Concat makes the correct prediction while the prediction of GPT (ours) does not fit to the context well. In Case 3, even if PinyinGPT-Concat ranks the ground truth as the second best, the top 1 prediction still makes much sense and fit well with the context. In all cases, GPT (ours) usually generate predictions which are grammatically sound but semantically inappropriate. In this subsection, we analyze how performance differs with respect to domains. We sample six domains for illustration in Table 6. 14 The table shows that PinyinGPT-Concat achieves consistent improvement over GPT on all domains. We also 14 The table of all 15 domains is attached in the Appendix. find that the absolute scores vary a lot across domains. This reflects different predictability for texts on different domains. For example, the P@10 score of the Culture domain is 16 points lower than the Medical domain. In the Medical domain, the texts contain plenty of descriptions of symptoms and instructions of medicines, which are somehow canonically used. While in the Culture domain, the texts are less constrained and have more variations. Considering pinyin input method requires both accuracy and efficiency, we further train a 6-layer GPT to investigate the trade-off. Our 6-layer GPT is directly truncated and initialized from the 12-layer GPT and is continually trained for 50k steps with the same configuration of 12-layer GPT. The evaluation is conducted over the 9 configurations of context-target length and averaged across all domains. Specifically, each configuration is inferred using a data center GPU NVIDIA V100 Tensor Core, and the GPU is fully occupied by one model. The beam size is set to be 16. We report the average inference time in millisecond as well as accuracy in terms of P@ K of PinyinGPT-Concat. Table 7 is the result for the configuration (4-9, 4-9). The table shows that the inference time of the model with 6-layer transformer is almost 30% faster than that with 12-layer. However, the performance of the 6-layer model drops consistently in the abbreviated setting. 15 15 We also list the experiment results for all configurations 1905 Model Games Culture Sports P@1 P@5 P@10 P@1 P@5 P@10 P@1 P@5 P@10 GPT (ours) 24.04 32.78 34.23 21.86 29.33 30.94 28.54 37.13 38.69 PinyinGPT-Concat 25.78 38.26 41.89 22.10 33.33 36.72 29.81 43.56 46.95 Real Estate Medical Finance P@1 P@5 P@10 P@1 P@5 P@10 P@1 P@5 P@10 GPT (ours) 26.53 35.27 36.74 33.59 43.54 44.93 29.00 37.24 38.47 PinyinGPT-Concat 27.28 40.16 43.86 34.76 49.28 52.56 29.17 42.17 45.52 Table 6: Performance of six sample domains over WD using abbreviated pinyin. Pinyin Input Method We describe existing works based on whether the input pinyin is perfect or abbreviated. A majority of existing works focus on perfect pinyin. Traditional models are typically based on statistical language models (Chen and Lee, 2000) and statistical machine translation (Yang et al., 2012). Recent works are usually built with neural network. For example, Moon IME (Huang et al., 2018) integrates attention-based neural network and an information retrieval module. Zhang et al. (2019) improves an LSTM-based encoder-decoder model with online vocabulary adaptation. For abbreviated pinyin, Co-CAT (Huang et al., 2015) uses machine translation technology to reduce the number of the typing letters. Huang and Zhao (2018) propose an LSTM-based encoder-decoder approach with the concatenation of context words and abbreviated pinyin as input. Our work differs from existing works in that we are the first one to exploit GPT and verify the pros and cons of GPT in different situations. In addition, there are some works handling pinyin with typing errors. Chen and Lee (2000) investigate a typing model which handles spelling correction in sentence-based pinyin input method. CHIME (Zheng et al., 2011) is a error-tolerant Chi-in the Appendix. nese pinyin input method. It finds similar pinyin which will be further ranked with Chinese specific features. Jia and Zhao (2014) propose a joint graph model to globally optimize the tasks of pinyin input method and typo correction. We leave error-tolerant pinyin input method as a future work. Pinyin-enhanced Pretrained Models Our methodology also relates to pretrained models that use pinyin information. Sun et al. (2021) propose a general-purpose Chinese BERT with new embedding layers to inject pinyin and glyph information of characters. There are also task-specific BERT models, especially for the task of grammatical error correction since an important type of error is caused by characters pronounced with the same pinyin. Zhang et al. (2021a) add a pinyin embedding layer and learns to predict characters from similarly pronounced candidates. PLOME (Liu et al., 2021) add two embedding layers implemented with two GRU networks to inject both pinyin and shape of characters, respectively. Xu et al. (2021) add a hierarchical encoder to inject the pinyin letters at character and sentence levels, and add a ResNet encoder to use graphic features of character image. In this paper, we explore how to adapt pretrained Chinese GPT to pinyin input method. To begin with, we find that a frozen GPT with decoding conditioned on pinyin can reach state-of-the-art performance on perfect pinyin. However, in abbreviated setting, the performance drops by a large gap. Through our experiments, we find that both context enrichment with pinyin and pinyin-constrained training improve the performance. In the future, we would like to investigate more challenging settings including error-tolerant pinyin input method and mixtures of perfect pinyin and abbreviated pinyin.
abstain objective result abstain abstain method method result abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain objective abstain method abstain result other abstain abstain abstain abstain method abstain abstain abstain method abstain result objective objective objective objective abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain method abstain abstain abstain abstain method abstain method method abstain abstain abstain method method abstain method abstain other abstain abstain abstain abstain abstain abstain method abstain method other abstain method abstain abstain method abstain abstain method abstain abstain abstain abstain abstain abstain result abstain abstain abstain abstain abstain abstain abstain other abstain abstain result abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain result abstain abstain abstain abstain abstain abstain abstain result abstain abstain abstain result result result abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain result abstain abstain result abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain result abstain other other other other other other other objective other other other other other other abstain method other other other other other objective result abstain result objective