In this paper, we investigate improvements to the GEC sequence tagging architecture with a focus on ensembling of recent cutting-edge Transformer-based encoders in Large configurations. We encourage ensembling models by majority votes on span-level edits because this approach is tolerant to the model architecture and vocabulary size. Our best ensemble achieves a new SOTA result with an F 0 . 5 score of 76.05 on BEA-2019 (test), even without pretraining on synthetic datasets. In addition, we perform knowledge distillation with a trained ensemble to generate new synthetic training datasets, "Troy-Blogs" and "Troy-1BW". Our best single sequence tagging model that is pretrained on the generated Troydatasets in combination with the publicly available synthetic PIE dataset achieves a near-SOTA 1 result with an F 0 . 5 score of 73.21 on BEA-2019 (test). The code, datasets, and trained models are publicly available. 2 1 Introduction The purpose of the Grammatical Error Correction (GEC) task is to correct grammatical errors in natural texts. This includes correcting errors in spelling, punctuation, grammar, morphology, word choice, and others. An intelligent GEC system receives text containing mistakes and produces its corrected version. The GEC task is complicated and challenging: the accuracy of edits, inference speed, and memory limitations are topics of intensive research. Currently, Machine Translation (MT) is the mainstream approach for GEC. In this setting, errorful sentences correspond to the source language, and error-free sentences correspond to the  This research was performed during Maksym Tar-navskyi's work on Ms.Sc. thesis at Ukrainian Catholic University (Tarnavskyi, 2021). target language. Early GEC-MT methods leveraged phrase-based statistical machine translation (PBSMT) (Yuan and Felice, 2013). Then this approach rapidly evolved to seq2seq Neural Machine Translation (NMT) based on gated recurrent neural networks (Yuan and Briscoe, 2016) and recent powerful Transformer-based seq2seq models. Transformer-based models autoregressively capture the full dependency among output tokens; however, inference can be slow due to sequential decoding. Grundkiewicz et al. (2019) leveraged a Transformer model (Vaswani et al., 2017) that was pre-trained on synthetic GEC data and right-to-left re-ranking for ensemble. Kaneko et al. (2020) adopted several strategies of BERT (Devlin et al., 2018) usage for GEC. Recently, Rothe et al. (2021) built their system on top of T5 (Xue et al., 2021), a xxl version of the T5 Transformer encoder-decoder model and reached new state-of-the-art results (11B parameters). While still not as widespread as MT, the sequence tagging approach for GEC, which generates a sequence of text edit operations encoded by tags for errorful input text is becoming more common. LaserTagger (Malmi et al., 2019) is a sequence tagging model that casts text generation as a text editing task. Corrected texts are reconstructed from the inputs using three main edit operations: keeping a token, deleting it, and adding a phrase before the token. LaserTagger combines a BERT encoder with an autoregressive Transformer decoder, which predicts edit operations. The Parallel Iterative Edit (PIE) model (Awasthi et al., 2019) does parallel decoding, achieving quality that is competitive with the seq2seq models. 3 It predicts edits instead of tokens and iteratively refines predictions to capture dependencies. A similar approach is presented in (Omelianchuk et al., 2020). The GECToR system achieves competitive results using various Trans-3 http://nlpprogress.com/english/ grammatical_error_correction 3842 formers as an encoder; and linear layers with soft-max for tag prediction and error detection. By replacing an autoregressive decoder with linear output layers, it's also potentially several times faster than seq2seq systems. Today, the generation of synthetic data is becoming significant for most GEC models. Natural languages are rich, and their grammars contain many rules and exceptions; therefore, professional linguists are often utilized to annotate high-quality corpora for further training ML-based systems mostly in a supervised manner (Dahlmeier et al., 2013), (Bryant et al., 2019). However, human annotation is expensive, so researchers are working on methods for augmentation of training data, synthetic data generation, and strategies for its efficient usage (Lichtarge et al., 2019), (Kiyono et al., 2019), (Stahlberg and Kumar, 2021). The majority of GEC systems today use synthetic data to pre-train Transformer-based components of their models. In this work, we are focusing on exploring sequence tagging models and their ensembles. Although most of our developments may eventually be applied to other languages, we work with English only in this study. Being a resource-rich language, English is a highly competitive area for the GEC task 3 . Our tagging models are inherited from GECToR (Omelianchuk et al., 2020). To date, GECToR shows near-SOTA results on CoNLL-2014 and BEA-2019 benchmarks. 3 It is based on AllenNLP (Gardner et al., 2017) and HuggingFace Transformers (Wolf et al., 2019), and its source code is freely available. 4 GECToR is a sequence tagging model that contains a Transformer-based encoder stacked with two output linear layers that are responsible for error detection and error correction. The model is trained with a cross-entropy loss function to produce tags that encode token-level edits. Then iterative postprocessing is performed. GECToR predicts the tag-encoded transformations for each token in the input sequence; it can then apply these transformations to get the modified output sequence. Since some corrections in a sentence may depend on others, applying the GEC sequence tagger only once may not be enough to correct the sentence entirely. Therefore, GECToR uses an iterative correction approach, modifying the sentence by repeatedly running it through the model (up to four times) (Fig. 1). As in GECToR, our primary edit operations are encoded by the following tags: $KEEP (leave the current token unchanged), $DELETE (delete the current token), $APPEND _ t 1 (append the token t 1 after the current token), $REPLACE _ t 2 (replace the current token with the token t 2 ). GECToR also has special edit operations, such as changing the case of a token, changing the verb form to express a different number or tense, or converting singular nouns to plural, and other. We refer to (Omelianchuk et al., 2020) for the details of edit transformations. 1. We empirically investigate and improve the GECToR sequence tagging system (Omelianchuk et al., 2020) by upgrading the Transformer encoders to Large configurations, leveraging an advanced tokenizer, performing additional filtering of edits-free sentences, and increasing the vocabulary size. 2. We show that the ensembling of sequence taggers by majority votes on output edit spans provides better performance compared to ensembling by averaging output tag probabilities while staying tolerant to the models' architecture and vocabulary sizes. quence taggers. When trained on the distilled data, single GEC tagging models show competitive performance. 4. We make the code, datasets, and trained models publicly available. For training single models and ensembles, we use parallel annotated data from the Lang-8 Corpus of Learner English (Lang-8) 5 (Tajiri et al., 2012), the National University of Singapore Corpus of Learner English (NUCLE) 6 (Dahlmeier et al., 2013), the First Certificate in English dataset (FCE) 7 (Yannakoudakis et al., 2011), and the Write & Improve (W&I) Corpus (Bryant et al., 2019). 8 Please, see Table 1 for details. For knowledge distillation from the ensemble, we use parts of two monolingual datasets: the One Billion Word Benchmark (1BW) 9 (Chelba et al., 2013) and the Blog Authorship Corpus (Blogs) 10 (Schler 5 https://sites.google.com/site/ naistlang8corpora 6 https://www.comp.nus.edu.sg/~nlp/ corpora.html 7 https://ilexir.co.uk/datasets/index. html 8 https://www.cl.cam.ac.uk/research/nl/ bea2019st/data/wi+locness_v2.1.bea19.tar.gz 9 http://statmt.org/wmt11/ training-monolingual.tgz 10 https://www.kaggle.com/rtatman/ blog-authorship-corpus et al., 2005). Corresponding distilled datasets have prefixes "Troy-"; see more details about their generation in Section 6. After knowledge distillation for the final training of the student model we also use parallel sentences with synthetically generated grammatical errors from the PIE dataset (Awasthi et al., 2019). 11 3.4 Evaluation We report F 0 . 5 , P recision , and Recall metrics computed by ERRANT scorer (Bryant et al., 2017) on dev and test datasets from the W&I + LOCNESS Corpus from the BEA-2019 GEC Shared Task (Bryant et al., 2019). In the original GECToR system, the Byte-Pair Encoding (BPE) tokenizer (Sennrich et al., 2016) uses a custom implementation. 12 This was chosen because the out-off-the-box AllenNLP tokenizer was too slow, and HuggingFace Transformers' tokenizers did not provide a BPE-to-words mapping. Our work is fully implemented with Transformers from the HuggingFace Transformers library. In particular, we moved to the recently released fast tokenizers from HuggingFace. Now, our encoders have the same tokenizers for fine-tuning as they had for initial pretraining, which leads to better quality after fine-tuning. Our encoder is loaded with its default pretrained weights; the linear layers' weights are initialized with random numbers. Our models are trained by Adam optimizer (Kingma and Ba, 2015) with default hyperparameters. We use a multi-class categorical cross-entropy loss function. The early stopping technique is used: Stopping criteria is 3 epochs without improving the loss function on the dev set, which is a random 2% sample from the same source as training data and is different for each stage. Model training is performed in several stages (Ta-ble 2). In Stage I, the model is pretrained on synthetic datasets; this stage is optional. Then, in Stage II, we carry out warm-up training on the Joint Train Dataset , which contains the Lang-8, NUCLE, FCE, and W&I datasets (Table 1). Thus, we perform coarse fine-tuning on a large amount of diverse GEC data. Datasets are used sequentially with no shuffling. In order not to adversely impact the out-of-box pretrained weights of the encoder, during the first two epochs we train only the linear layers (so-called "cold epochs"); later, we make all model's weights trainable. In Stage III, we continue fine-tuning on the W&I Train dataset, which contains only the highest-quality data. Another difference between Stages II and III is the share of edit-free sentences in the training data. We observed that too many sentences in training data without edits lead to reducing the appearance rate of the tagger and deteriorating the overall quality. Therefore, we filter out edit-free sentences from the Joint Train Dataset, which is used in Stage II. In Stage III, we fine-tune the model on the unfiltered version of the W&I Train dataset. The final stage is inference tweaks (Omelianchuk et al., 2020) for balancing between the model's precision and recall. This is done by introducing additional hyperparameters: additional confidence (AC) to the probability for the $KEEP tag and minimum error probability (MEP) for corrections tags. These hyperparameters are found via a random search on the BEA-2019 dev set. In the GECToR paper (Omelianchuk et al., 2020), authors investigated encoders from ALBERT (Lan et al., 2020), BERT (Devlin et al., 2018), GPT-2 (Radford et al., 2018), RoBERTa (Liu et al., 2019), and XLNet (Yang et al., 2019) Transformers in their Base configurations. Most likely, Base configurations were chosen due to the better inference speed/quality ratio. They found that XLNet, RoBERTa, and BERT show the best performance. We reproduce experiments for these encoders, but now we explore Large configurations as well. We additionally explore encoders from DeBERTa (He et al., 2020) (Table 3). We observe that all models that are equipped with Large encoders have higher precision, recall, and F 0 . 5 values than those equipped with their Base versions. The price of this performance is 2.32.5 times slower inference for Large configurations (Table 4). The single model with RoBERTa encoder shows the best performance for Large configurations, whereas DeBERTa slightly outperforms RoBERTa for Base configurations. RoBERTa is the fastest in both configurations. Most of the tag-encoded edits are token-specific, e.g., $APPEND_it , $REPLACE_the , and so on. Thus, the tag vocabulary size matters, and should be a tradeoff between coverage and model quality. We create the tag vocabulary by taking the most frequent edit tags generated from the Joint Train Dataset (Table 1). To find the optimal tag vocabulary sizes, we experiment with {5K, 10K} vocabulary sizes (Table 5). We observe that increasing the vocabulary size to 10K for Large encoders may improve the quality, e.g. for models with RoBERTa and DeBERTa. Nevertheless, we also see an example of quality deterioration for the model with XLNet. Ensembling is a proven quality-boosting method for models sets that have diverse outputs. Most of the recent GEC solutions achieved their best results by ensembling single models (Stahlberg and Kumar, 2021), (Omelianchuk et al., 2020), (Awasthi et al., 2019). In this section we consider two ensembling methods for our GEC tagging models: averaging of output tag probabilities and majority votes on output edit spans (Fig. 2). First, we reproduce the ensembling approach from (Omelianchuk et al., 2020). We add DeBERTa and carry out experiments with varying Base and Large configurations of encoders (Table 6). We observe that ensembling by averaging of output tag probabilities improves the quality of corrections; the more models we combine, the better results we obtain. More surprisingly, combining Ensemble P R F 0 . the same encoders' architectures in Base and Large configurations may provide slightly better results than we get for the Base and Large models separately (see RoBERTa ( B ) + RoBERTa ( L ) in Table 6). Although the ensemble RoBERTa ( L ) + BERT ( L ) + DeBERTa ( L ) + XLNet ( L ) shows the best performance, we select ensemble the RoBERTa ( L ) + DeBERTa ( L ) + XLNet ( L ) for further experiments. It has higher recall, making it possible to trade recall for precision later during inference tweaks. This aggregation method combines single models' outputs in the post-processing step (Fig. 2). We take span-level edits and retain only those which have most of the votes from the ensemble. A similar approach is used in (Liang et al., 2020), where the authors combined sequence tagging and seq2seq models for the Chinese language. The advantage of this ensembling method is that we can combine the results of models with different output dimensions and even different architectures. In our work, it allows us to combine models with different tag vocabulary sizes. We leave ensembling with seq2seq GEC systems for future work. First, we compare ensembling by averaging of output tag probabilities + and by majority votes on output edit spans  for the selected ensemble after training on the Joint Train Dataset (Stage II), finetuning on the W&I dataset (Stage III) and optimization of hyperparameters (inference tweaks) (Table 7). We observe that ensembles based on 3846 majority votes on output edit spans show better results because of better precision. However, after inference tweaks, the two ensembling types achieve close F 0 . 5 scores. To additionally improve the precision of ensembling by majority votes we introduce the "majority quorum" hyperparameter N min . Majority quorum N min denotes minumum number of votes for triggering the edit , here 1  N min  N single _ models . Increasing N min boosts precision by the cost of recall because it filters out more edits where single models disagree (Table 8). Setting N min = 1 is a poor strategy because we can't rely on a majority when resolving conflicting edits, so the resulting text might contain controversial and incoherent edits. Increasing the number of systems in the ensemble leads to higher quality, but requires adapting the N min parameter (Table 8). Based on this limited analysis we observe that N min = N single _ models  1 achieves the best results. For our pool of models there is no gain over using more than 4 models, but we want to explore adding more diverse seq2seq models to such an ensemble in future works. Next, since the majority votes on output edit spans is capable of combining any models, we test the ensemble of the best models that we already have trained (Table 9). Finally, we evaluate our best ensemble DeBERTa ( L ) 10 K  RoBERTa ( L ) 10 K  XLNet ( L ) 5 K on the BEA-2019 (test) dataset and achieve F 0 . 5 score of 76 . 05 . This is a significant improvement over F 0 . 5 = 73 . 70 for the best ensemble from (Omelianchuk et al., 2020) and to the best of our knowledge is a new state-of-the-art (SOTA) result for ensembles on the BEA-2019 (test) benchmark . obtained without pre-training on synthetic data. Knowledge distillation is the method for transferring knowledge from a large model ("teacher") to a smaller one ("student") (Hinton et al., 2015), (Kim and Rush, 2016). It has strong practical applications because large models usually have expensive inference costs and are inconvenient for deployment. In our case, the teacher model is an ensemble of trained sequence taggers, whereas the student model is a single sequence tagger. The ensemble receives errorful texts and generates their corrected versions. Later these input-output pairs of sentences are used for training single models. Like any synthetic annotation method, knowledge-distilled data contains a certain share of systematic errors that deteriorates the student model's quality. In this work, we use two monolingual corpora to generate our distilled datasets: the One Billion Words Benchmark ("1BW"), which mostly contains news texts, and the Blog Authorship Corpus ("Blogs"), which contains blog texts on various topics (Table 1). Being real-world natural texts, these datasets contain a certain share of grammatical errors, which are corrected by our system. For text pre-processing, we use the tokenizer from Spacy. 13 As a teacher, we use the ensemble of the sequence taggers containing Large encoders with a 5K vocabulary: DeBERTa ( L ) 5 K + RoBERTa ( L ) 5 K + XLNet ( L ) 5 K (Table 7). The ensemble corrects 5% of processed sentences in 1BW and 28% of sentences in Blogs. Distilled versions of the datasets have the prefix "Troy-" in their names (Table 1). Considering our past experience, we use only edited sentence pairs in our distilled datasets, and we limit their number to 1.2M. We also reduce the synthetic PIE dataset from (Awasthi et al., 2019) to 1.2M sentence pairs for better comparability in the experiments. We leave exploring other ensembles in the role of a teacher model for future research. First, we reproduce the training scheme from (Omelianchuk et al., 2020) for a single model, RoBERTa ( L ) 5 K where PIE synthetic data is used for 13 https://spacy.io/ pre-training (Stage I), then the model is trained on the Joint Train Dataset (Stage II), fine-tuned on the high-quality W&I dataset (Stage III), and finally, hyperparameters are applied to balance precision and recall (inteference tweaks). We observe that the sequence tagger with a RoBERTa-Large encoder shows slightly better performance than RoBERTa-Base from (Omelianchuk et al., 2020), where RoBERTa-Base had an 8x larger training dataset in Stage I (Fig. 3). Next, we replace the synthetic PIE dataset with our distilled datasets, Troy-1BW and Troy-Blogs. We observe that in Stage I, training on purely synthetic data leads to a dramatic boost in recall. When we start training in Stage II, a sharp deterioration in both precision and recall occurs. It seems that the student model does not receive new information compared to Stage I. This is more noticeable for models trained on the Troy-Blogs dataset, where recall significantly drops after training. However, the F 0 . 5 in Stage II is higher for models pretrained on distilled Troydatasets. Finally, after training on Stage III and performing inference tweaks, single models pretrained on both datasets show very similar performance, but the model with RoBERTa ( L ) 5 K trained on Troy-Figure 3: Pre-training of single tagging models on synthetic and distilled datasets with a tag vocabulary size of 5K. Benchmark is BEA-2019 (dev). 1BW is slightly higher-performing. This single model reaches F 0 . 5 = 73 . 21 on BEA-2019 (test), a significant improvement on the results from (Omelianchuk et al., 2020) for single models F 0 . 5 = 71 . 5 for RoBERTa ( B ) 5 K and F 0 . 5 = 72 . 4 for XLNet ( B ) 5 K . We observed that models pretrained on the Troy-Blogs dataset show good results on Stage I, but lose their advantage after training on Stage II. Thus, we decided to try a one-stage training approach with a RoBERTa ( L ) 5 K encoder. For our training dataset, we concatenated Troy-Blogs with high-quality W&I dataset that we usually reserve for Stage III. As a result, we achieved F 0 . 5 = 55 . 81 on BEA-2019 (dev) and F 0 . 5 = 72 . 69 on BEA-2019 (test) (Table 10). These results are obtained much more easily than with our best single model: just one-stage training with out-of-the-box RoBERTa , no pre-training on synthetic GEC data or multi-stage training. Our research investigates the impact of encoder configurations, ensembling methods, and knowledge distillation on the GECToR system. We found that Replacing Base encoders in GECToR (Omelianchuk et al., 2020) with their Large configurations does improve the quality by several F0.5 points, at the cost of 2.32.5 times slower inference. Our best ensemble achieves a new SOTA result with F 0 . 5 = 76 . 05 on BEA-2019 (test). Ensembling sequence taggers by majority votes on output edit spans provides better performance than averaging output tag probabilities because it lets us combine a variety of modeling approaches and vocabulary sizes. Single models in the ensemble were not pre-trained on synthetic GEC datasets, providing room for improvement in future work. to an ensemble of sequence taggers and produce the annotated Troy-Blogs and Troy-1BW datasets. After training on these datasets, single GEC sequence tagging models show near-SOTA results: F 0 . 5 = 73 . 21 / 72 . 69 on BEA-2019 (test) for multi-stage/one-stage training. To our knowledge, our best single model is outperformed only by the much more compute-intensive T5 XXL model (Rothe et al., 2021), which is 30 times larger with 11B parameters (Table 10). We make the code, datasets, and trained models publicly available. 14 8 Acknowledgements We express our gratitude to Oleksii Molchanovskyi, Dmytro Lider, Viktor Zamaruiev, Paige Schwartz, the Ukrainian Catholic University, and Grammarly for providing support and computational resources. We also thank anonymous reviewers for their contributions. To our communities: While we are writing this, our homeland Ukraine continues to resist the unprovoked Russian invasion. We are grateful to everyone who defends Ukraine, declares support to the people of Ukraine, and is sending aid. Thank you!
objective method objective abstain objective result abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain other abstain abstain abstain abstain abstain objective method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain method abstain abstain abstain abstain abstain abstain method method abstain method other abstain abstain method abstain abstain abstain abstain method abstain abstain method method method abstain abstain abstain abstain method abstain abstain abstain abstain abstain method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain method abstain abstain abstain abstain abstain method abstain method abstain abstain abstain abstain abstain abstain method abstain abstain method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain method abstain abstain abstain abstain abstain abstain abstain abstain abstain method abstain abstain abstain method abstain method method abstain abstain method method abstain other abstain abstain method abstain method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain objective result objective abstain abstain method abstain abstain abstain abstain abstain abstain result method method objective abstain abstain abstain