This study explores the necessity of performing cross-corpora evaluation for grammatical error correction (GEC) models. GEC models have been previously evaluated based on a single commonly applied corpus: the CoNLL-2014 benchmark. However, the evaluation remains incomplete because the task difficulty varies depending on the test corpus and conditions such as the proficiency levels of the writers and essay topics. To overcome this limitation, we evaluate the performance of several GEC models, including NMT-based (LSTM, CNN, and transformer) and an SMT-based model, against various learner corpora (CoNLL-2013, CoNLL-2014, FCE, JFLEG, ICNALE, and KJ). Evaluation results reveal that the models' rankings considerably vary depending on the corpus, indicating that single-corpus evaluation is insufficient for GEC models. Grammatical error correction (GEC) is the task of correcting various grammatical errors in a given text, which is typically written by non-native speakers. Previous studies focused on typical errors such as those in the use of articles (Han et al., 2006), prepositions (Felice and Pulman, 2008), and noun numbers (Nagata et al., 2006). Machine translation approaches are being presently applied for GEC (Junczys-Dowmunt et al., 2018; Chollampatt and Ng, 2018; Ge et al., 2018; Junczys-Dowmunt and Grundkiewicz, 2016). In these approaches, GEC is treated as a translation problem from the erroneous text to the correct text (Mizumoto et al., 2012; Felice et al., 2014; Junczys-Dowmunt and Grundkiewicz, 2014). However, the evaluation of GEC performance is unfortunately not complete because researchers tend to evaluate their models on a single corpus. 2014) has been recently used for such evaluation. Single-corpus evaluation may be insufficient in cases wherein a GEC model generally aims to robustly correct grammatical errors in any written text partly because the task difficulty varies depending on proficiency levels and essay topics. Although a model outperforms a baseline in one corpus, the model in another corpus may perform better, leading to different conclusions from what we know. This study explores the necessity of performing cross-corpora evaluation for GEC models. The performance of four recent models, namely three neural machine translation (NMT)-based models (LSTM, CNN, and transformer) and a statistical machine translation (SMT)-based model is evaluated against six learner corpora (CoNLL-2014, CoNLL-2013 (Ng et al., 2013), FCE (Yannakoudakis et al., 2011), JFLEG ( Napoles et al., 2017), KJ (Nagata et al., 2011), and ICNLAE (Ishikawa, 2013)). Evaluation results show that the models' rankings considerably vary depending on the corpus. Empirical results reveal that models must be evaluated using multiple corpora from different perspectives. (cid:15) We first explore the necessity of performing cross-corpora evaluation for GEC models. (cid:15) We empirically show that the single-corpus evaluation may be unreliable. (cid:15) Our source code is published for cross-corpora evaluation so that researchers in the community can adequately and easily evaluate their models based on multiple corpora. 1 2 Related Work We are motivated by the issue of robustness in the parsing community. This field pre-1 https://github.com/tomo-wb/GEC_CCE 1310 viously focused on improving parsing accuracy on Penn Treebank (Marcus et al., 1993). However, robustness was largely improved by evaluation using multiple corpora including Ontonotes (Hovy et al., 2006) and Google Web Treebank (Petrov and McDonald, 2012). A situation similar to this might also occur in GEC. In other words, evaluation in GEC has relied heavily on the CoNLL-2014 benchmark, which implies that the field is overdeveloping on this dataset. Other corpora are used for evaluation, such as KJ (Mizumoto et al., 2012) and JFLEG (Sakaguchi et al., 2017; Junczys-Dowmunt et al., 2018; Chollampatt and Ng, 2018; Ge et al., 2018; Xie et al., 2018). However, these corpora still depend on one or at most two corpora. Cross-corpora evaluation is discussed herein using six corpora, namely CoNLL-2014, CoNLL-2013, FCE, JFLEG, KJ, and ICNALE. The following conditions were considered when selecting corpora: (cid:15) The corpus must be used at least once in the GEC community. (cid:15) Based on the hypothesis that writers' proficiency affects the error distribution of any given text, we add a corpus with relatively low proficiency (KJ) compared to CoNLL-2014. CoNLL-2014 (Ng et al., 2014) , the official dataset of CoNLL-2014 shared task, is a collection of essays written by students at the National University of Singapore and is commonly used as test data for the CoNLL-2014 benchmark. This dataset contains only two essay topics. CoNLL-2013 (Ng et al., 2013) , the official dataset of CoNLL-2013 shared tasks, is commonly used as the development data for the CoNLL-2014 benchmark and contains only two essay topics. Cambridge ESOL First Certificate in English (FCE) (Yannakoudakis et al., 2011) is a dataset containing 1,244 examination scripts of the Cambridge FCE examination. Topics and first languages (L1s) in the dataset are diversified because it contains essays for 10 topics written by nonnative speakers from various countries. JHU FLuency-Extended GUG Corpus (JF-LEG) (Napoles et al., 2017) contains approximately 1,500 sentences from an English proficiency test. It contains sentences written by learners of the English language having various L1s and proficiency levels. Konan-JIEM Learner Corpus (KJ) (Nagata et al., 2011) contains 233 essays written on 10 topics by students of a Japanese college, which are manually error-tagged and shallow-parsed. International Corpus Network of Asian Learners of English, Written Essays (IC-NALE) (Ishikawa, 2013) contains essays written by college and graduate students from ten Asian countries/regions (China, Hong Kong, Indonesia, Japan, Korea, Pakistan, the Philippines, Singapore, Taiwan, and Thailand). The original ICNALE is not error annotated. Therefore, we sampled a total number of 1,736 sentences, which are manually annotated with grammatical errors based on KJ 's annotation scheme. Table 1 summarizes the properties of these corpora. Let N and M denote the total number of source words and sentences in a corpus, respectively. Word error rate (WER) is defined as follows: WER =  Mm =1 d ( X m ; Y m )  Mm =1 N m where X m denotes each source sentence, Y m denotes each corrected sentence, and d ( X m ; Y m ) denotes the edit distance between X m and Y m using dynamic programming. The following conclusions are derived: (1) CoNLL-2014 has narrow coverage of topics, proficiency and L1s compared with other corporas such as JFLEG and FCE. (2) Several learner corpora are available for the evaluation of GEC models. These corpora can help investigate the performance of GEC models under different conditions. The following factors are considered while selecting our model. the aforementioned factors: LSTM : We use a bi-directional LSTM in the encoder and an LSTM with an attention mechanism in the decoder. Both the encoder and the decoder comprise two layers. The LSTM hidden state and word embedding sizes are set to be 500. CNN : We follow the previous study ( Chollampatt and Ng, 2018), namely a fully convolutional encoderdecoder architecture with seven convolutional layers. The hyperparam-eters used in a previous study are used herein (Chollampatt and Ng, 2018). Transformer : Transformer is the self-attention-based model proposed by Vaswani et al. (2017). Six layers are used for both the encoder and decoder along with eight attention heads. The word embedding size is set to 1024 dimensions, and the size of position-wise feed-forward networks is set to 4096 dimensions at each inner layer. SMT : We essentially follow the idea used in a previous study ( Junczys-Dowmunt and Grundkiewicz, 2016), with some key differences. Specifically, we only use English Wikipedia for language model training and only the NUS Corpus of Learner English (NUCLE) and the Lang-8 Learner Corpora (Lang-8) for translation model training to make the experimental settings equal in all models. We use two public datasets, namely Lang-8 (Mizumoto et al., 2011) and NUCLE (Dahlmeier et al., 2013), for training. Our pre-processing and experimental setup is similar to that reported previously (Chollampatt and Ng, 2018). In particular, a subset of NUCLE (5.4K) is utilized as the development data for selecting the model; the remaining subset (1.3M) is utilized as the training data. All the models are trained, tuned, and tested in the same way. The models are tested on each test data shown in Table 1. As an evaluation metric, we use F 0 : 5 score computed by applying the MaxMatch scorer (Dahlmeier and Ng, 2012) and GLEU (Napoles et al., 2015). We determine the average F 0 : 5 and average GLEU scores of the four models, which are trained with different random initializations, following a previously reported approach ( Chollampatt and Ng, 2018). Figure 1 shows the performance of each model sorted from best to worst based on their F 0 : 5 score, revealing that the performance substantially varies depending on the corpus. For example, the performance of the transformer ranges from the score of F 0 : 5 , which is as low as 36.20 on CoNLL-2013, to as high as 60.06 on JFLEG. Notably, their rankings also considerably vary. Transformer performs best on CoNLL-2014. However, it exhibits third-best performance among FCE, KJ, and ICNALE; LSTM outperforms the other models by a large margin of up to 5.3 F 0 : 5 points. Some examples of the model outputs are presented in Table 2 and Table 3. Some situations are successfully corrected using transformer (Table 2), whereas it failed to perform in other situations (Table 3). The reason for difference in the model rankings cannot be generally stated because it is influenced by various factors such as the learner's proficiency, essay topic, and L1. The experimental results show, however, that discussions based on the performance on CoNLL-2014 may only hold under certain conditions. rankings on FCE show different trends in Figure 1 and Figure 2. This is partly because F 0 : 5 and GLEU evaluate different perspectives of the models. Furthermore, evaluation data and metric must be appropriately set depending on the factors that need to be evaluated in the model. Experimental results indicate that the benchmark single-corpus evaluation is not robust; however, more diverse corpora remain undetermined. Both JFLEG and FCE can be diverse corpora because they contain examination scripts written by language learners from all over the world. JFLEG is particularly designed to contain more diverse corpus for developing and evaluating GEC models ( Napoles et al., 2017). If a diverse single-corpus evaluation suffices, the rankings of the models will remain the same. However, experimental results have shown that the model rankings on both JFLEG and FCE are different (Figure 1). Thus, single-corpus evaluation is deemed weak regardless of its diversity. This study discusses the importance of evaluating GEC models from various perspectives using multiple corpora. Multi-perspective evaluation does not necessarily mean using multiple corpora. Many aspects in a corpus can be used for analysis, such as the proficiency of the writers, essay topics, and the writer 's native language. As a case study, we evaluate and analyze the models regarding the essay WER. Table 4 shows the performance (in precision, recall, and F 0 : 5 ) of all the models when WER is the lowest (7.64 % for ICNALE) and the highest (20.86 % for JFLEG). Transformer and LSTM outperform all the other models in the highest and the lowest error-rated corpora, respectively. Experimental results show that LSTM and transformer may be more precision-oriented and recall-oriented, respectively. Further, precision-oriented models have an advantage over recall-oriented models when a given text contains several errors, and vice versa. This knowledge enables choosing a model based on the task that has to be completed. This study explored the necessity of performing cross-corpora evaluation for GEC models, for which the performance of several GEC models was investigated against various learner corpora. Empirical evaluation results revealed that the model performance and rankings considerably vary depending on the corpus, suggesting that a single-corpus evaluation can be unreliable. Therefore, cross-corpora evaluation should be applied to GEC models. We also published our source code for the cross-corpora evaluation framework so that researchers in the community can adequately and easily evaluate their models based on multiple corpora. Our future study will further examine the robustness of several existing evaluation metrics and explore new metrics appropriate for cross-corpora and/or cross-domain evaluation. We are grateful to the members of the Tohoku University Natural Language Processing Laboratory as well as the anonymous reviewers for their insightful comments and suggestions.
objective abstain abstain method abstain abstain abstain abstain abstain abstain abstain abstain result objective other abstain abstain objective objective result objective method objective other abstain abstain abstain abstain abstain abstain abstain abstain method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain method abstain abstain abstain abstain abstain abstain method method abstain abstain method abstain abstain abstain abstain method method method method abstain abstain abstain method method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain method abstain abstain method abstain abstain abstain abstain abstain objective abstain abstain method objective other