  Unlike alphabetic languages, Chinese spelling and pronunciation are
different. Both characters and pinyin take an important role in Chinese
language understanding. In Chinese NLP tasks, we almost adopt characters or
words as model input, and few works study how to use pinyin. However, pinyin is
essential in many scenarios, such as error correction and fault tolerance for
ASR-introduced errors. Most of these errors are caused by the same or similar
pronunciation words, and we refer to this type of error as SSP(the same or
similar pronunciation) errors for short. In this work, we explore various ways
of using pinyin in pretraining models and propose a new pretraining method
called PmBERT. Our method uses characters and pinyin in parallel for
pretraining. Through delicate pretraining tasks, the characters and pinyin
representation are fused, which can enhance the error tolerance for SSP errors.
We do comprehensive experiments and ablation tests to explore what makes a
robust phonetic enhanced Chinese language model. The experimental results on
both the constructed noise-added dataset and the public error-correction
dataset demonstrate that our model is more robust compared to SOTA models.

Computation and Language