Neural Grammatical Error Correction Systems with Unsupervised Pre training on Synthetic Data
Considerable effort has been made to address the data sparsity problem in neural grammatical error correction. In this work, we propose a simple and surprisingly effective unsupervised synthetic error generation method based on confusion sets extracted from a spellchecker to increase the amount of training data. Synthetic data is used to pre train a Transformer sequence to sequence model, which not only improves over a strong baseline trained on authentic error annotated data, but also enables the development of a practical GEC system in a scenario where little genuine error annotated data is available. The developed systems placed first in the BEA19 shared task, achieving 69.47 and 64.24 F_{0.5} in the restricted and low resource tracks respectively, both on the W&I+LOCNESS test set. On the popular CoNLL 2014 test set, we report state of the art results of 64.16 M2 for the submitted system, and 61.30 M2 for the constrained system trained on the NUCLE and Lang 8 data.
BEA@ACL