Here’s My Point: Argumentation Mining with Pointer Networks
Argumentation mining seeks to uncover the argument structure present in argumentative text. In order to determine this structure, one must understand how different individual components of the overall argument are linked. General consensus in this field dictates that the argument components form a hierarchy of persuasion, which manifests itself in a tree structure. This work provides the first neural network-based approach to argumentation mining, focusing on the dual tasks of extracting links between argument components, and classifying types of argument components. We propose to use a joint model based on a Pointer Network architecture to simultaneously solve these tasks. In doing so, we construct a joint model that simultaneously attempts to learn the type of argument component, as well as continuing to predict links between argument components. The proposed joint model achieves state-of-the-art results on two separate evaluation corpora, achieving far superior performance than a regular Pointer Network model. Our results show that optimizing for both tasks is crucial for high performance.
1 Introduction
Computational approaches to argument mining/understanding have become very popular (Persing and Ng, 2016; Cano-Basave and He, 2016; Wei et al., 2016; Ghosh et al., 2016; Palau and Moens, 2009; Habernal and Gurevych, 2016). An important avenue in this work is to understand the structure in argumentative text (Persing and Ng, 2016; Peldszus and Stede, 2015; Stab
and Gurevych, 2016; Nguyen and Litman, 2016). One fundamental assumption when working with argumentative text is the presence of Arguments Components (ACs). The types of ACs are generally characterized as a claim or a premise (Govier, 2013), with premises acting as support (or possibly attack) units for claims. To model more complex structures of arguments, some annotation schemes also include a major claim AC type (Stab and Gurevych, 2016, 2014b).
Generally, the task of processing argument structure encapsulates four distinct subtasks: 1) Given a sequence of tokens that represents an entire argumentative text, determine the token subsequences that constitute non-intersecting ACs; 2) Given an AC, determine the type of AC (claim, premise, etc.); 3) Given a set/list of ACs, determine which ACs have directed links that encapsulate overall argument structure; 4) Given two linked ACs, determine whether the link is a supporting or attacking relation. In this work, we focus on subtasks 2 and 3.
There are two key assumptions our work makes going forward. First, we assume subtask 1 has been completed, i.e. ACs have already been identified. Second, we follow previous work that assumes a tree structure for the linking of ACs (Palau and Moens, 2009; Cohen, 1987; Peldszus and Stede, 2015; Stab and Gurevych, 2016). Specifically, a given AC can only have a single outgoing link, but can have numerous incoming links. Furthermore, there is a ‘head’ component that has no outgoing link (the top of the tree). However, we note that in a broader sense, depending on the corpus (see Section 4), an argument structure can be either a single tree or a forest, consisting of multiple trees. Figure 1 shows an example that we will use throughout the paper to concretely explain how our approach works. First, the left side of the figure presents the raw text of a paragraph in a per-
2
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
First, [ ::::::: cloning ::::: will ::: be :::::::::: beneficial :::: for ::::: many ::::::: people :::: who :::: are ::: in ::::: need ::: of ::::: organ ::::::::: transplants]AC1. In addition, [it shortens the healing process]AC2. Usually, [it is very rare to find an appropriate organ donor]AC3 and [by using cloning in order to raise required organs the waiting time can be shortened tremendously]AC4.
AC1 Claim
AC2 Premise
AC4 Premise AC3 Premise
Figure 1: An example of argument structure with four ACs. The left side shows raw text that has been annotated for the presence of ACs. Squiggly and straight underlining means an AC is a claim or premise, respectively. The ACs in the text have also been annotated for links to other ACs, which is show in the right figure. ACs 3 and 4 are premises that link to another premise, AC2. Finally, AC2 links to a claim, AC1. AC1 therefore acts as the central argumentative component.
suasive essay (Stab and Gurevych, 2016), with the ACs contained in square brackets. Squiggly verse straight underlining differentiates between claims and premises, respectively. The ACs have been annotated as to how they are linked, and the right side of the figure reflects this structure. The argument structure with four ACs forms a tree, where AC2 has two incoming links, and AC1 acts as the head, with no outgoing links. We also specify the type of AC, with the head AC marked as a claim and the remaining ACs marked as premises. Lastly, we note that the order of arguments components can be a strong indicator of how components should related. Linking to the first argument component can provide a competitive baseline heuristic (Peldszus and Stede, 2015; Stab and Gurevych, 2016).
Given the task at hand, we propose a modification of a Pointer Network (PN) (Vinyals et al., 2015b). A PN is a sequence-to-sequence model that outputs a distribution over the encoding indices at each decoding timestep. The PN is a promising model for link extraction in argumentative text because it inherently possesses three important characteristics: 1) it is able to model the sequential nature of ACs; 2) it constrains ACs to have a single outgoing link, thus partly enforcing the tree structure; 3) the hidden representations learned by the model can be used for jointly predicting multiple subtasks. We also note that since a PN is a type of sequence-to-sequence model (Sutskever et al., 2014), it allows the entire sequence to be seen before making prediction. This is important because if the problem were to be approached as standard sequence modeling (Graves and Schmidhuber, 2009; Robinson, 1994), mak-
ing predictions at each forward timestep, it would only allow links to ACs that have already been seen. This is equivalent to only allowing backward links. We note that we do test a simplified model that only uses hidden states from an encoding network to make predictions, as opposed to the sequence-to-sequence architecture present in the PN (see Section 5).
PNs were originally proposed to allow a variable length decoding sequence (Vinyals et al., 2015b). Alternatively, the PN we implement differs from the original model in that we decode for the same number of timesteps as there are input components. We also propose a joint PN for both extracting links between ACs and predicting the type of AC. The model uses the hidden representation of ACs produced during the encoding step (see Section 3.4). Aside from the partial assumption of tree structure in the argumentative text, our models do not make any additional assumptions about the AC types or connectivity, unlike the work of (Peldszus, 2014). Lastly, in respect to the broad task of parsing, our model is flexible because it can easily handle non-projective, multiroot dependencies. We evaluate our models on the corpora of (Stab and Gurevych, 2016) and (Peldszus, 2014), and compare our results with the results of the aformentioned authors.
2 Related Work
In this section, we discuss previous argumentation mining work that deals with the subtask pipeline we discussed in the previous section. This can be labeled as a ‘micro’ approach to argumentation mining (Stab and Gurevych, 2016) (in con-
3
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
trast, there has been a number of efforts to identify argument structure at a higher level (Boltuzic and Šnajder, 2014; Ghosh et al., 2014; Cabrio and Villata, 2012)). Palau and Moens (2009) is an early work in this area, using a hand-crafted ContextFree Grammar to determine the structure of ACs in a corpus of legal texts. Lawrence et al. (2014) leverage a topic modeling-based AC similarity to uncover tree-structured arguments in philosophical texts. Unlike these methodologies, recent work offers data-driven approaches to the task of predicting links between ACs. Stab and Gurevych (2014b) approach the task as a binary classification problem. The authors train an SVM with various semantic and structural features. Peldszus and Stede have also used classification models for predicting the presence of links (2015).
Various authors have also proposed to jointly model link extraction with other subtasks from the argumentation mining pipeline, using either a Integer Linear Programming (ILP) framework (Persing and Ng, 2016; Stab and Gurevych, 2016) or directly feeding previous subtask predictions into a tree-based parser. The former joint approaches are evaluated on annotated corpora of persuasive essays (Stab and Gurevych, 2014a, 2016), and the latter on a corpus of microtexts (Peldszus, 2014). The ILP framework is effective in enforcing a tree structure between ACs when predictions are made from otherwise naive base classifiers.
Unrelated to argumentation mining specifically, recurrent neural networks have previously been proposed to model tree/graph structures in a linear manner. (Vinyals et al., 2015c) use a sequenceto-sequence model for the task of syntactic parsing. The authors linearize input parse graphs using a depth-first search, allowing it to be consumed as a sequence, achieving state-of-the-art results on several syntactic parsing datasets. (Bowman et al., 2015) experiment on an artificial entailment dataset that is specifically engineered to capture recursive logic (Bowman et al., 2014). The text is annotated with brackets, in an original attempt to provide easy input into a recursive neural network. However, standard recurrent neural networks can take in complete sentence sequences, brackets included, and perform competitively with a recursive neural network.
3 Pointer Network for Link Extraction
In this section we will describe how we use a PN for the problem of extracting links between ACs. We begin by giving a general description of the PN model.
3.1 Pointer Network
A PN is a sequence-to-sequence model (Sutskever et al., 2014) with attention (Bahdanau et al., 2014) that was proposed to handle decoding sequences over the encoding inputs, and can be extended to arbitrary sets (Vinyals et al., 2015a). The original motivation for a pointer network was to allow networks to learn solutions to algorithmic problems, such as the traveling salesperson and convex hull, where the solution is a sequence over candidate points. The PN model is trained on input/output sequence pairs (E,D), where E is the source and D is the target (our choice of E,D is meant to represent the encoding, decoding steps of the sequence-to-sequence model). Given model parameters Θ, we apply the chain rule to determine the probability of a single training example:
p(D|E; Θ) = m(E)∏ i=1 p(Di|D1, ..., Di−1, E; Θ)
(1) where the function m signifies that the number of decoding timesteps is a function of each individual training example. We will discuss shortly why we need to modify the original definition of m for our application. By taking the log-likelihood of Equation 1, we arrive at the optimization objective:
Θ∗ = arg max Θ ∑ E,D log p(D|E; Θ) (2)
which is the sum over all training example pairs. The PN uses Long Short-Term Memory (LSTM) (Hochreiter and Schmidhuber, 1997) for sequential modeling, which produces a hidden layer h at each encoding/decoding timestep. In practice, the PN has two separate LSTMs, one for encoding and one for decoding. Thus, we refer to encoding hidden layers as e, and decoding hidden layers as d.
The PN uses a form of content-based attention (Bahdanau et al., 2014) to allow the model to produce a distribution over input elements. This can also be thought of as a distribution over input indices, wherein a decoding
4
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
Component 1 Component 2 Component 3 Component 4
LSTM D1
LSTM D2
LSTM D3
LSTM D4 LSTM E1 LSTM E2 LSTM E3 LSTM E4
Figure 2: Applying a Pointer Network to the example paragraph in Figure 1 with LSTMs unrolled over time.
step ‘points’ to the input. Formally, given encoding hidden states (e1, ..., en), the model calculates p(Di|D1, ..., Di−1, E) as follows:
uij = v T tanh(W1ej +W2di) (3)
p(Di|D1, ..., Dj−1, E) = softmax(ui) (4)
where matrices W1, W2 and vector v are parameters of the model (along with the LSTM parameters used for encoding and decoding). In Equation 3, prior to taking the dot product with v, the resulting transformation can be thought of as creating a joint, hidden representation of inputs i and j. Vector ui in equation 4 is of length n, and index j corresponds to input element j. Therefore, by taking the softmax of ui, we are able to create a distribution over the input.
3.2 Link Extraction as Sequence Modeling
A given piece of text has a set of ACs, which occur in a specific order in the text, (C1, ..., Cn). Therefore, at encoding timestep i, the model is fed a representation of Ci. Since the representation is large and sparse (see Section 3.3 for details on how we represent ACs), we add a fully-connected layer before the LSTM input. Given a representation Ri for AC Ci the LSTM input Ai becomes:
Ai = σ(WrepRi + brep) (5)
where Wrep, brep in turn become model parameters, and σ is the sigmoid function1. Similarly, the decoding network applies a fully-connected layer with sigmoid activation to its inputs, see Figure 3. At encoding step i, the encoding LSTM produces hidden layer ei, which can be thought of as a hidden representation of AC Ci.
1We also experimented with relu and elu activations, but found sigmoid to yield the best performance.
In order to make the PN applicable to the problem of link extraction, we explicitly set the number of decoding timesteps to be equal to the number of input components. Using notation from Equation 1, the decoding sequence length for an encoding sequence E is simply m(E) = |{C1, ..., Cn}|, which is trivially equal to n. By constructing the decoding sequence in this manner, we can associate decoding timestep i with AC Ci.
From Equation 4, decoding timestep Di will output a distribution over input indices. The result of this distribution will indicate to which AC component Ci links. Recall there is a possibility that an AC has no outgoing link, such as if it’s the root of the tree. In this case, we state that if ACCi does not have an outgoing link, decoding step Di will output index i. Conversely, if Di outputs index j, such that j is not equal to i, this implies thatCi has an outgoing link to Cj . For the argument structure in Figure 1, the corresponding decoding sequence is (1, 1, 2, 2). The topology of this decoding sequence is illustrated in Figure 2. Observe how C1 points to itself since it has no outgoing link.
Finally, we note that we modify the PN structure to have a Bidirectional LSTM as the encoder. Thus, ei is the concatenation of forward and backward hidden states −→e i and←−e n−i+1, produced by two separate LSTMs. The decoder remains a standard forward LSTM.
3.3 Representing Argument Components
At each timestep of the decoder, the network takes in the representation of an AC. Each AC is itself a sequence of tokens, similar to the recently proposed Question-Answering dataset (Weston et al., 2015). We follow the work of (Stab and Gurevych, 2016) and focus on three different types of features to represent our ACs: 1) Bag-of-Words of the AC; 2) Embedding representation based on GloVe embeddings (Pennington et al., 2014); 3) Structural features: Whether or not the AC is the first AC in
5
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
465
466
467
468
469
470
471
472
473
474
475
476
477
478
479
480
481
482
483
484
485
486
487
488
489
490
491
492
493
494
495
496
497
498
499
Bi-LSTM E1
FC1 FC1 FC1 FC1
Bidirectional LSTM Encoder
Component 1 Component 2 Component 3 Component 4
FC2 FC2 FC2 FC2
Claim Premise Premise Premise
LSTM D1
LSTM D2
LSTM D3
Bi-LSTM E2 Bi-LSTM E3 Bi-LSTM E4
FC3 FC3 FC3
a paragraph, and Whether the AC is in an opening, body, or closing paragraph. See Section 6 for an ablation study of the proposed features.
3.4 Joint Neural Model
Up to this point, we focused on the task of extracting links between ACs. However, recent work has shown that joint models that simultaneously try to complete multiple aspects of the subtask pipeline outperform models that focus on a single subtask (Persing and Ng, 2016; Stab and Gurevych, 2014b; Peldszus and Stede, 2015). Therefore, we will modify the architecture we proposed in Section 3 so that it would allow us to perform AC classification (Kwon et al., 2007; Rooney et al., 2012) together with link prediction. Knowledge of an individual subtask’s predictions can aid in other subtasks. For example, claims do not have an outgoing link, so knowing the type of AC can aid in the link prediction task. This can be seen as a way of regularizing the hidden representations from the encoding component (Che et al., 2015).
Predicting AC type is a straightforward classification task: given AC Ci, we need to predict whether it is a claim or premise. Some annotation schemes also include the class major claim (Stab and Gurevych, 2014a), which means this can be a multi-class classification task. For encoding timestep i, the model creates hidden representation ei. This can be thought of as a representation of AC Ci. Therefore, our joint model will simply pass this representation through a fully connected layer as follows:
zi = Wclsei + bcls (6)
where Wcls, bcls become elements of the model
parameters, Θ. The dimensionality of Wcls, bcls is determined by the number of classes. Lastly, we use softmax to form a distribution over the possible classes.
Consequently, the probability of predicting component type at timestep i is defined as:
p(Ci) = p(Ei| −→ E i, ←− E i; Θ) (7)
p(Ei| −→ E i, ←− E i; Θ) = softmax(zi) (8)
Finally, combining this new prediction task with Equation 2, we arrive at the new training objective:
Θ∗ = arg max Θ α ∑ E,D log p(D|E; Θ)
+(1− α) ∑ E log p(E|Θ) (9)
which simply sums the costs of the individual prediction tasks, and the second summation is the cost for the new task of predicting AC type. α ∈ [0, 1] is a hyperparameter that specifies how we weight the two prediction tasks in our cost function. The architecture of the joint model, applied to our ongoing example, is illustrated in Figure 3.
4 Experimental Design
As we have previously mentioned, our work assumes that ACs have already been identified. That is, the token sequence that comprises a given AC is already known. The order of ACs corresponds directly to the order in which the ACs appear in the text. Since ACs are non-overlapping, there is no
6
501
502
503
504
505
506
507
508
509
510
511
512
513
514
515
516
517
518
519
520
521
522
523
524
525
526
527
528
529
530
531
532
533
534
535
536
537
538
539
540
541
542
543
544
545
546
547
548
549
550
551
552
553
554
555
556
557
558
559
560
561
562
563
564
565
566
567
568
569
570
571
572
573
574
575
576
577
578
579
580
581
582
583
584
585
586
587
588
589
590
591
592
593
594
595
596
597
598
599
Type prediction Link prediction Model Macro f1 MC f1 Cl f1 Pr f1 Macro f1 Link f1 No Link f1 Base Classifier .794 .891 .611 .879 .717 .508 .917 ILP Joint Model .826 .891 .682 .903 .751 .585 .918 BLSTM .810 .830 .688 .912 .754 .589 .919 PN - - - - .709 .511 .906 Joint Model No FC Input .791 .826 .642 .906 .708 .514 .901 Joint Model .849 .894 .732 .921 .767 .608 .925
Table 1: Results on the Persuasive Essay corpus.
Type prediction Link prediction Model Macro f1 Cl f1 Pr f1 Macro f1 Link f1 No Link f1 Simple .817 - - .663 .478 .848 Best EG .869 - - .693 .502 .884 MP+p .831 - - .720 .546 .894 Base Classifier .830 .712 .937 .650 .446 .841 ILP Joint Model .857 .770 .943 .683 .486 .881 PN .813 .692 .934 .740 .577 .903
Table 2: Results on the Microtext corpus.
ambiguity in this ordering. We test the effectiveness of our proposed model on a dataset of persuasive essays (Stab and Gurevych, 2016), as well as a dataset of microtexts (Peldszus, 2014). The feature space for the persuasive essay corpus has roughly 3,000 dimensions, and the microtext corpus feature space has between 2,500 and 3,000 dimensions, depending on the data split (see below).
The persuasive essay corpus contains a total of 402 essays, with a frozen set of 80 essays held out for testing. There are three AC types in this corpus: major claim, claim, and premise. Also, in this corpus individual structures can be either trees or forests. We follow the creators of the corpus and only evaluate ACs within a given paragraph. That is, each training/test example is a sequence of ACs from a paragraph. This results in a 1,405/144 training/test split.
The microtext corpus contains 112 short texts. Unlike the persuasive essay corpus, each text in this corpus is itself a complete example, as well as a single tree. Since the dataset is small, the authors have created 10 sets of 5-fold cross-validation, reporting the the average across all splits for final model evaluation. This corpus contains only two types of ACs: claim and premise. The annotation of argument structure of the microtext corpus varies from the persuasive essay corpus; ACs can be linked to other links, as opposed to ACs. Therefore, if AC Ci is annotated to be linked to link l,
we create a link to the source AC of l. On average, this corpus has 5.14 ACs per text. Lastly, we note that predicting the presence of links is directional (ordered): predicting a link between the pair Ci, Cj(i 6= j) is different than Cj , Ci.
We implement our models in TensorFlow (Abadi et al., 2015). Our model has the following parameters: hidden input dimension size 512, hidden layer size 256 for the bidirectional LSTMs, hidden layer size 512 for the LSTM decoder, α equal to 0.5, and dropout (Srivastava et al., 2014) of 0.9. We believe the need for such high dropout is due to the small amounts of training data (Zarrella and Marsh, 2016), particularly in the Microtext corpus. All models are trained with Adam optimizer (Kingma and Ba, 2014) with a batch size of 16. For a given training set, we randomly select 10% to become the validation set. Training occurs for 4,000 epochs. Once training is completed, we select the model with the highest validation accuracy (on the link prediction task) and evaluate it on the held-out test set. At test time, we take a greedy approach and select the index of the probability distribution (whether link or type prediction) with the highest value.
5 Results
The results of our experiments are presented in Tables 1 and 2. For each corpus, we present f1 scores
7
601
602
603
604
605
606
607
608
609
610
611
612
613
614
615
616
617
618
619
620
621
622
623
624
625
626
627
628
629
630
631
632
633
634
635
636
637
638
639
640
641
642
643
644
645
646
647
648
649
650
651
652
653
654
655
656
657
658
659
660
661
662
663
664
665
666
667
668
669
670
671
672
673
674
675
676
677
678
679
680
681
682
683
684
685
686
687
688
689
690
691
692
693
694
695
696
697
698
699
Type prediction Link prediction Model Macro f1 MC f1 Cl f1 Pr f1 Macro f1 Link f1 No Link f1 No structural .808 .824 .694 .907 .760 .598 .922 No BOW .796 .833 .652 .902 .728 .543 .912 No Embeddings .827 .874 .695 .911 .750 .581 .918 Only Avg Emb* .832 .873 .717 .917 .751 .583 .918 Only Max Emb* .843 .874 .732 .923 .766 .608 .924 Only Min Emb* .838 .878 .719 .918 .763 .602 .924 All features .849 .894 .732 .921 .767 .608 .925
Table 3: Feature ablation study. * indicates that both BOW and Structural are present, as well as the stated embedding type.
Type prediction Link prediction Bin Macro f1 MC f1 Cl f1 Pr f1 Macro f1 Link f1 No Link f1
1 ≤ len < 4 .863 .902 .798 .889 .918 .866 .969 4 ≤ len < 8 .680 .444 .675 .920 .749 .586 .912 8 ≤ len < 12 .862* .000* .762 .961 .742 .542 .941
Table 4: Results of binning test data by length of AC sequence. * indicates that this bin does not contain any major claim labels, and this average only applies to claim and premise classes. However, we do not disable the model from predicting this class: the model was able to avoid predicting this class on its own.
for the AC type classification experiment, with a macro-averaged score of the individual class f1 scores. We also present the f1 scores for predicting the presence/absence of links between ACs, as well as the associated macro-average between these two values.
We implement and compare four types of neural models: 1) The previously described PN-based model depicted in Figure 3 (called Joint Model in the tables); 2) The same as 1), but without the fully-connected input layers (called Joint Model No FC Input in the table); 3) The same as 1), but the model only predicts the link task, and is therefore not optimized for type prediction (called PN in the tables); 4) A non-sequence-to-sequence model that uses the hidden layers produced by the BLSTM encoder with the same type of attention as the PN (called BLSTM in the table). That is, di in Equation 3 is replaced by ei.
In both corpora we compare against the following previously proposed models: Base Classifier (Stab and Gurevych, 2016) is a feature-rich, taskspecific (AC type or link extraction) SVM classifier. Neither of these classifiers enforce structural or global constraints. Conversely, the ILP Joint Model (Stab and Gurevych, 2016) provides constraints by sharing prediction information between the base classifiers. For example, the model attempts to enforce a tree structure among ACs
within a given paragraph, as well as using incoming link predictions to better predict the type class claim. For the microtext corpus only, we have the following comparative models: Simple (Peldszus and Stede, 2015) is a feature-rich logistic regression classifier. Best EG (Peldszus and Stede, 2015) creates an Evidence Graph (EG) from the predictions of a set of base classifiers. The EG models the potential argument structure, and offers a global optimization objective that the base classifiers attempt to optimize by adjusting their individual weights. Lastly, MP+p (Peldszus and Stede, 2015) combines predictions from base classifiers with a Minimum Spanning Tree Parser (MSTParser).
6 Discussion
First, we point out that the Joint Model achieves state-of-the-art on 10 of the 13 metrics in Tables 1 and 2, including the highest results in all metrics on the Persuasive Essay corpus, as well as link prediction on the Microtext corpus. The performance on the Microtext corpus is very encouraging for several reasons. First, the fact that the model can perform so well with only a hundred training examples is rather remarkable. Second, although we motivate the use of a PN due to the fact that it partially enforces the tree structure in argumentation, other models we compare against explic-
8
701
702
703
704
705
706
707
708
709
710
711
712
713
714
715
716
717
718
719
720
721
722
723
724
725
726
727
728
729
730
731
732
733
734
735
736
737
738
739
740
741
742
743
744
745
746
747
748
749
750
751
752
753
754
755
756
757
758
759
760
761
762
763
764
765
766
767
768
769
770
771
772
773
774
775
776
777
778
779
780
781
782
783
784
785
786
787
788
789
790
791
792
793
794
795
796
797
798
799
itly contain further constraints. For example, only premises can have outgoing links, and there can be only one claim in an AC. Moreover, the MP+p model directly enforces the tree constraint unique to the microtext corpus. Even though the Joint Model does not have the tree constraint directly encoded, it able to learn the structure effectively from the training examples so that it can outperform the Mp+p model for link prediction. As for the other neural models, the BLSTM model performs competitively with the ILP Joint Model on the persuasive essay corpus, but trails the performance of the Joint Model. We believe this is because the Joint Model is able to create two different representations for each AC, one each in the encoding/decoding state, which benefits performance in the dual tasks. Conversely, the BLSTM model must encode information relating to type as well as link prediction in a single hidden representation. On one hand, the BLSTM model outperforms the ILP model on link prediction, yet it is not able to match the ILP Joint Model’s performance on type prediction, primarily due to the BLSTM’s poor performance on predicting the major claim class. Another interesting outcome is the importance of the fully-connected layer before the LSTM input. The results show that this extra layer of depth is crucial for good performance on this task. Without it, the Joint Model is only able to perform competitively with the Base Classifier. The results dictate that even a simple fullyconnected layer with sigmoid activation can provide a useful dimensionality reduction for feature representation. Finally, the PN model, only optimized for link prediction, suffers a large drop in performance, conveying that the dual optimization of the Joint Model is crucial for high performance in the link prediction task.
Table 3 shows the results of an ablation study for AC feature representation. Regarding link prediction, BOW features are clearly the most important, as their absence results in the highest drop in performance. Conversely, the presence of structural features provides the smallest boost in performance, as the model is still able to record stateof-the-art results compared to the ILP Joint Model. This shows that the Joint Model is able to capture structural cues through sequence modeling and semantics. When considering type prediction, both BOW and structural features are important, and it is the embedding features that provide the least
benefit. The Ablation results also provide an interesting insight into the effectiveness of different ‘pooling’ strategies for using individual token embeddings to create a multi-word embedding. The popular method of averaging embeddings (which is used by (Stab and Gurevych, 2016) in their system) is in fact the worst method, although its performance is still competitive with the previous state-of-the-art. Conversely, max pooling results are on par with the Joint Model results from Table 1.
Table 4 shows the results on the Persuasive Essay test set with the test examples binned by sequence length. First, it is not surprising to see that the model performs best when the sequences are the shortest. As the sequence length increases, the accuracy on link prediction drops. This is possibly due to the fact that as the length increases, a given AC has more possibilities as to which other AC it can link to, making the task more difficult. Conversely, there is actually a rise in no link prediction accuracy from the second to third row. This is likely due to the fact that since the model predicts at most one outgoing link, it indirectly predicts no link for the remaining ACs in the sequence. Since the chance probability is low for having a link between a given AC in a long sequence, the no link performance is actually better in longer sequences.
7 Conclusion
In this paper we have proposed how to use a modified PN (Vinyals et al., 2015b) to extract links between ACs in argumentative text. We evaluate our models on two corpora: a corpus of persuasive essays (Stab and Gurevych, 2016), and a corpus of microtexts (Peldszus, 2014). The Joint Model records state-of-the-art results on the persuasive essay corpus, as well as achieving state-of-the-art results for link prediction on the microtext corpusThe results show that jointly modeling the two prediction tasks is crucial for high performance. Future work can attempt to learn the AC representations themselves, such as in (Kumar et al., 2015). Lastly, future work can integrate subtasks 1 and 4 into the model. The representations produced by Equation 3 could potentially be used to predict the type of link connecting ACs, i.e. supporting or attacking; this is the fourth subtask in the pipeline. In addition, a segmenting technique, such as the one proposed by (Weston et al., 2014), can accomplish subtask 1.
9
801
802
803
804
805
806
807
808
809
810
811
812
813
814
815
816
817
818
819
820
821
822
823
824
825
826
827
828
829
830
831
832
833
834
835
836
837
838
839
840
841
842
843
844
845
846
847
848
849
850
851
852
853
854
855
856
857
858
859
860
861
862
863
864
865
866
867
868
869
870
871
872
873
874
875
876
877
878
879
880
881
882
883
884
885
886
887
888
889
890
891
892
893
894
895
896
897
898
899

The paper presents an application of Pointer Networks, a recurrent neural network model original used for solving algorithmic tasks, to two subtasks of Argumentation Mining: determining the types of Argument Components, and finding the links between them. The model achieves state-of-the-art results.  Strengths:  - Thorough review of prior art in the specific formulation of argument mining handled in this paper. - Simple and effective modification of an existing model to make it suitable for the task. The model is mostly explained clearly. - Strong results as compared to prior art in this task.  Weaknesses:  - 071: This formulation of argumentation mining is just one of several proposed subtask divisions, and this should be mentioned. For example, in [1], claims are detected and classified before any supporting evidence is detected. Furthermore, [2] applied neural networks to this task, so it is inaccurate to say (as is claimed in the abstract of this paper) that this work is the first NN-based approach to argumentation mining. - Two things must be improved in the presentation of the model: (1) What is the pooling method used for embedding features (line 397)? and (2) Equation (7) in line 472 is not clear enough: is E_i the random variable representing the *type* of AC i, or its *identity*? Both are supposedly modeled (the latter by feature representation), and need to be defined. Furthermore, it seems like the LHS of equation (7) should be a conditional probability. - There are several unclear things about Table 2: first, why are the three first baselines evaluated only by macro f1 and the individual f1 scores are missing? This is not explained in the text. Second, why is only the  PN  model presented? Is this the same PN as in Table 1, or actually the Joint Model? What about the other three? - It is not mentioned which dataset the experiment described in Table 4 was performed on.  General Discussion:  - 132: There has to be a lengthier introduction to pointer networks, mentioning recurrent neural networks in general, for the benefit of readers unfamiliar with  sequence-to-sequence models . Also, the citation of Sutskever et al. (2014) in line 145 should be at the first mention of the term, and the difference with respect to recursive neural networks should be explained before the paragraph starting in line 233 (tree structure etc.). - 348: The elu activation requires an explanation and citation (still not enough well-known). - 501:  MC ,  Cl  and  Pr  should be explained in the label. - 577: A sentence about how these hyperparameters were obtained would be appropriate. - 590: The decision to do early stopping only by link prediction accuracy should be explained (i.e. why not average with type accuracy, for example?). - 594: Inference at test time is briefly explained, but would benefit from more details. - 617: Specify what the length of an AC is measured in (words?). - 644: The referent of  these  in  Neither of these  is unclear. - 684:  Minimum  should be  Maximum . - 694: The performance w.r.t. the amount of training data is indeed surprising, but other models have also achieved almost the same results - this is especially surprising because NNs usually need more data. It would be good to say this. - 745: This could alternatively show that structural cues are less important for this task. - Some minor typos should be corrected (e.g.  which is show , line 161).  [1] Rinott, Ruty, et al.  Show Me Your Evidence-an Automatic Method for Context Dependent Evidence Detection.  EMNLP. 2015.  [2] Laha, Anirban, and Vikas Raykar.  An Empirical Evaluation of various Deep Learning Architectures for Bi-Sequence Classification Tasks.  COLING. 2016.


Automatic Annotation and Evaluation of Error Types for Grammatical Error Correction
Until now, error type performance for Grammatical Error Correction (GEC) systems could only be measured in terms of recall because system output is not annotated. In this paper, we overcome this problem by using a linguisticallyenhanced alignment to automatically extract the edits between parallel original and corrected sentences and then classify them using a new dataset-independent rule-based classifier. As human experts rated the predicted error types as “Good” or “Acceptable” in at least 95% of cases, we applied our approach to the system output produced in the CoNLL-2014 shared task to carry out a detailed analysis of system error type performance for the first time.
1 Introduction
The Conference on Natural Language Learning (CoNLL) shared task of 2014 (Ng et al., 2014) required teams to build systems that were capable of correcting all types of grammatical errors in learner text. While the submitted systems were evaluated against text that had been explicitly annotated with error type information, the teams themselves were not required to annotate their output in a similar way. This mismatch ultimately meant that a detailed error type analysis of each system was impossible and that error type performance could only be measured in terms of recall.
The main aim of this paper is to rectify this situation and provide a method by which unannotated error correction data can be automatically annotated with error type information. This is important because some systems may be more effective at correcting certain error types than oth-
ers, yet this information is otherwise concealed in an overall score. Although several new metrics and methodologies for Grammatical Error Correction (GEC) have been proposed since the end of the CoNLL-2014 shared task (Felice and Briscoe, 2015; Bryant and Ng, 2015; Napoles et al., 2015; Grundkiewicz et al., 2015), none of these are currently capable of producing individual error type scores.
Our approach consists of two main steps. First, we automatically extract the edits between parallel original and corrected sentences by means of a linguistically-enhanced alignment algorithm (Felice et al., 2016), and second, we classify them according to a new rule-based framework specifically designed with error type evaluation in mind. This enables us to automatically annotate system hypothesis corrections with the same alignment and error type information as the reference and hence carry out a more detailed evaluation. The tool we use to do this will be released with this paper.
2 Edit Extraction
The first stage of automatic annotation is edit extraction. Specifically, given an original and corrected sentence pair, we need to determine the start and end boundaries of any edits. This is fundamentally an alignment problem:
The first attempt at automatic edit extraction was made by Swanson and Yamangil (2012), who simply used the Levenshtein distance to align original and corrected sentence pairs. As the Levenshtein distance only aligns individual tokens how-
2
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
ever, they also merged all adjacent non-matches in an effort to capture multi-token edits. Xue and Hwa (2014) subsequently improved on Swanson and Yamangil’s work by training a maximum entropy classifier to predict whether edits should be merged or not.
Most recently, Felice et al. (2016) proposed a new method of edit extraction using a linguistically-enhanced alignment supported by a set of merging rules. In particular, they incorporated various linguistic information, such as partof-speech and lemma, into the cost function of the Damerau-Levenshtein1 algorithm to make it more likely that tokens with similar linguistic properties align. This approach ultimately proved most effective at approximating human edits in several datasets (80-85 F1), and so we use it in the present study.
3 Automatic Error Typing
Having extracted the edits, the next step is to assign them error types. While Swanson and Yamangil (2012) did this by means of maximum entropy classifiers, one disadvantage of this approach is that such classifiers are biased towards their particular training corpora. In particular, the fact that different datasets are annotated according to different standards means that it is inappropriate to evaluate the predicted error types of an in-domain corpus against the predicted error types of an out-of-domain corpus (c.f. Xue and Hwa (2014)). Instead, a dataset-agnostic error type evaluation is much more desirable.
To solve this problem, we took inspiration from Swanson and Yamangil’s (2012) observation that most error types are based on part-of-speech (POS) categories, and wrote a rule to classify an edit based only on its automatic POS tags. We then added another rule to differentiate Missing, Unnecessary and Replacement errors depending on whether tokens were inserted, deleted or substituted. From there, we extended our approach to classify errors that are not well-characterised by POS alone (such as Spelling or Word Order) and ensured that all types are assigned based only on automatically obtained properties of the data.
One of the key strengths of this approach is that by being dependent only on automatic mark-up information, our classifier is entirely dataset in-
1Damerau-Levenshtein is an extension of Levenshtein that also handles transpositions; e.g. AB→BA
dependent and does not require labelled training data. This is in contrast with machine learning approaches which require different classifiers for different datasets and which ultimately may not be entirely compatible with each other. Instead, our approach is analogous to automating the annotation guidelines given to human annotators.
A second significant advantage of our approach is that it is also always possible to determine precisely why an edit was assigned to a particular error category. In contrast, human and machine learning classification decisions are often less transparent and may furthermore be subject to annotator bias. Moreover, by being fully deterministic, our approach bypasses bias effects altogether and should hence be more consistent.
3.1 Automatic Markup
The prerequisites for our rule-based classifier are that each token in both the original and corrected sentence is POS tagged, lemmatized, stemmed and dependency parsed. We use spaCy2 v1.6 for all but the stemming, which is performed by the Lancaster Stemmer in NLTK.3 Since fine-grained POS tags are often too detailed for the purposes of error evaluation, we also map spaCy’s Penn Treebank style tags to the coarser set of Universal Dependency tags.4 We use the latest Hunspell GB-large dictionary5 to help classify non-word errors. The marked-up tokens in an edit span are then input to our classifier and an error type is returned.
3.2 Error Categories
The complete list of 25 error types in our new framework is shown in Table 2. Note that most of them can be prefixed with ‘M:’, ‘R:’ or ‘U:’, depending on whether they describe a Missing, Replacement, or Unnecessary edit, to enable evaluation at different levels of granularity (See Appendix A for all valid combinations). This means we can choose to evaluate, for example, only replacement errors (anything prefixed by ‘R:’), only noun errors (anything suffixed with ‘NOUN’) or only replacement noun errors (‘R:NOUN’). This flexibility allows us to make more detailed observations about different aspects of system perfor-
2https://spacy.io/ 3http://www.nltk.org/ 4http://universaldependencies.org/tagset-conversion/
en-penn-uposf.html 5https://sourceforge.net/projects/wordlist/files/speller/ 2016.11.20/
3
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
Code Meaning Description / Example ADJ Adjective big → wide ADJ:FORM Adjective Form Comparative or superlative adjective errors.goodest → best, bigger → biggest, more easy → easier ADV Adverb speedily → quickly CONJ Conjunction and → but CONTR Contraction n’t → not DET Determiner the → a MORPH Morphology Tokens have the same lemma, but nothing else in common.quick (adj) → quickly (adv) NOUN Noun person → people NOUN:INFL Noun Inflection Count-mass noun errors.informations → information NOUN:NUM Noun Number cat → cats NOUN:POSS Noun Possessive friends → friend’s ORTH Orthography Case and/or whitespace errors.Bestfriend → best friend OTHER Other Errors that do not fall into any other category (e.g. paraphrasing).at his best → well, job → professional PART Particle (look) in → (look) at PREP Preposition of → at PRON Pronoun ours → ourselves PUNCT Punctuation ! → . SPELL Spelling genectic → genetic, color → colour UNK Unknown The annotator detected an error but was unable to correct it. VERB Verb ambulate → walk VERB:FORM Verb Form Infinitives (with or without “to”), gerunds (-ing) and participles.to eat → eating, dancing → danced VERB:INFL Verb Inflection Misapplication of tense morphology.getted → got, fliped → flipped VERB:SVA Subject-Verb Agreement (He) have → (He) has VERB:TENSE Verb Tense Includes inflectional and periphrastic tense, modal verbs and passivization.eats → ate, eats → has eaten, eats → can eat, eats → was eaten WO Word Order only can → can only
mance. One caveat concerning error scheme design is that it is always possible to add new categories for increasingly detailed error types; for instance, we currently label [could → should] a tense error, when it might otherwise be considered a modal error. The reason we do not call it a modal error, however, is because it would then become less clear how to handle other cases such as [can → should] and [has eaten → should eat], which might be considered a more complex combination of a modal and tense error. As it is impractical to create new categories and rules to differentiate between such narrow distinctions however, our final framework aims to be a compromise between informativeness and practicality.
3.3 Classifier Evaluation
As our new error scheme is based only on automatically obtained properties of the data, there are no gold standard labels against which to evaluate classifier performance. For this reason, we instead carried out a small-scale manual evaluation, where
we simply asked 5 GEC researchers to rate the appropriateness of the predicted error categories for 200 randomly chosen edits in context (100 from FCE-test (Yannakoudakis et al., 2011) and 100 from CoNLL-2014) as “Good”, “Acceptable” or “Bad”. “Good’ meant the chosen category was the most appropriate for the given edit, “Acceptable” meant the chosen category was appropriate, but probably not optimum, while “Bad” meant the chosen category was not appropriate for the edit. Raters were warned that edit boundaries had been determined automatically, and hence might be unusual, but that they should focus on the appropriateness of the error category regardless of whether they agreed with the boundary or not.
The result of this evaluation is shown in Table 3. Significantly, all 5 raters individually considered at least 95% of our rule-based error types to be either “Good” or “Acceptable”, despite the degree of noise introduced by automatic edit extraction. Furthermore, whenever raters judged an edit as “Bad”, this could usually be traced back to a mistake made by the POS tagger; e.g. [ring
4
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
Rater Good Acceptable Bad 1 92.0% 4.0% 4.0% 2 89.5% 6.5% 4.0% 3 83.0% 13.0% 4.0% 4 84.5% 11.0% 4.5% 5 82.5% 15.5% 2.0% OVERALL 86.3% 10.0% 3.7%
Table 3: The percent distribution for how each expert rated the appropriateness of the predicted error types. E.g. Rater 3 considered 83% of all predicted types to be “Good”.
→ rings] might be considered a NOUN:NUM or VERB:SVA error depending on whether the tagger considered both sides of the edit nouns or verbs. Inter-annotator agreement was good at 0.724 κfree (Randolph, 2005).
In contrast, the best results using a classifier were between 50-70 F1 (Felice et al., 2016). Although these results are incomparable with previous approaches which were evaluated using a different metric and error scheme, we nevertheless believe that the high scores awarded by the raters validate the efficacy of our rule-based approach.
4 CoNLL-2014 Shared Task Analysis
To demonstrate the value of our approach, we applied our automatic annotation tool to the data produced in the CoNLL-2014 shared task (Ng et al., 2014). In particular, we used our tool to generate annotated versions of the system output files produced by each participating team.6 Although our approach can be applied to any dataset, we chose CoNLL-2014 because it constitutes the largest collection of publicly available GEC system output.
One benefit of explicitly annotating the hypothesis files is that it makes evaluation much more straightforward. Specifically, if both the hypothesis and reference files are annotated in the same format, we need only compare the edits in each file to produce an F-score. In particular, for a given sentence, any edit with the same span and correction in both files is a true positive (TP), while the remaining edits in the hypothesis are false positives (FP) and the remaining edits in the reference are false negatives (FN). This is in contrast with all other metrics in GEC, which typically incorporate some sort of edit extraction or alignment component directly into their evalua-
6http://www.comp.nus.edu.sg/∼nlp/conll14st.html
M2 Scorer Our Approach Team Gold Auto Gold Auto AMU 35.01 35.06 32.67 32.22 CAMB 37.33 37.32 34.92 33.99 CUUI 36.79 37.64 34.15 34.68 IITB 5.90 5.97 5.77 5.74 IPN 7.09 7.69 6.12 6.15 NTHU 29.92 29.85 26.74 25.74 PKU 25.32 25.40 23.95 23.62 POST 30.88 31.02 28.43 28.00 RAC 26.68 26.89 23.39 23.21 SJTU 15.19 15.24 15.15 14.90 UFC 7.84 7.90 7.97 7.90 UMC 25.37 25.46 23.77 23.53
Table 4: Overall scores for each team in CoNLL2014 using gold and auto references with both the M2 scorer and our simpler edit comparison approach. All scores are in terms of F0.5.
tion algorithms (Dahlmeier and Ng, 2012; Felice and Briscoe, 2015; Napoles et al., 2015). Our approach, on the other hand, treats edit extraction and evaluation as separate tasks.
4.1 Gold Reference vs. Auto Reference
Before evaluating the newly annotated hypothesis files against the reference, we must also address another mismatch: namely that the hypothesis edits were aligned and classified automatically, while the reference edits were aligned and classified manually using a different framework. Since evaluation is now a straightforward comparison between two files however, it is especially important that both files are processed in the same way. For instance, a hypothesis edit [have eating → has eaten] will not match the reference edits [have → has] and [eating → eaten] because the former is one edit while the latter is two edits, even though they equate to the same thing.
We can solve this problem by reprocessing the reference file in the same way as the hypothesis file. This means all the reference edits are subject to the same alignment and classification criteria as the hypothesis edits. While it may seem unorthodox to discard gold reference information in favour of automatic reference information, Table 4 shows that this has no significant impact on the results when using either the M2 scorer, the de facto standard of GEC evaluation (Dahlmeier and Ng, 2012), or our own approach.
5
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
458
459
460
461
462
463
464
465
466
467
468
469
470
471
472
473
474
475
476
477
478
479
480
481
482
483
484
485
486
487
488
489
490
491
492
493
494
495
496
497
498
499
AMU CAMB CUUI IITB Type P R F0.5 P R F0.5 P R F0.5 P R F0.5
Missing 43.61 14.36 30.98 46.32 30.00 41.77 26.71 18.62 24.57 15.38 0.59 2.56 Replacement 37.19 26.90 34.54 37.37 28.07 35.05 45.78 22.89 38.15 29.85 1.49 6.22 Unnecessary - - - 25.51 27.59 25.90 34.20 33.91 34.14 46.15 1.55 6.83
IPN NTHU PKU POST Type P R F0.5 P R F0.5 P R F0.5 P R F0.5
Missing 2.86 0.29 1.04 35.34 11.60 25.08 33.33 4.37 14.34 31.14 13.13 24.44 Replacement 9.87 3.86 7.53 27.61 19.15 25.37 29.62 18.32 26.37 33.16 19.32 29.00 Unnecessary 0.00 0.00 0.00 34.57 16.17 28.16 0.00 0.00 0.00 26.32 33.12 27.44
RAC SJTU UFC UMC Type P R F0.5 P R F0.5 P R F0.5 P R F0.5
Missing 1.49 0.27 0.78 62.50 4.42 17.24 - - - 40.08 23.57 35.16 Replacement 29.50 20.87 27.25 50.54 3.43 13.47 72.00 2.64 11.52 34.62 9.69 22.87 Unnecessary 0.00 0.00 0.00 17.65 11.51 15.95 - - - 16.89 17.33 16.98
Table 5: Precision, recall and F0.5 for Missing, Unnecessary, and Replacement errors for each team. A dash indicates the team’s system did not attempt to correct the given error type (TP+FP = 0).
We validated this hypothesis for each team by means of bootstrap significance testing (Efron and Tibshirani, 1993) and found no statistically significant difference between auto and gold references (1,000 iterations, p > .05). This leads us to conclude that our auto annotations are qualitatively as good as human annotations.
Table 4 also shows that M2 scores tend to be higher than our own, which initially led us to believe that our approach was underestimating performance. We subsequently found, however, that the M2 scorer in fact tends to overestimate performance (c.f. Felice and Briscoe (2015) and Napoles et al. (2015)).
In particular, given a choice between matching [have eating → has eaten] from Annotator 1 or [have → has] and [eating → eaten] from Annotator 2, the M2 scorer will always choose Annotator 2 because two true positives (TP) are worth more than one. Similarly, whenever the scorer encounters two false positives (FP) within a certain distance of each other,7 it merges them and treats them as one false positive; e.g. [is a cat → are a cats] is selected over [is → are] and [cat → cats] even though these edits are best handled separately. Ultimately, it can be said that the M2 scorer exploits its dynamic edit boundary prediction in order to maximise true positives and minimise false positives and hence produces slightly inflated scores.
7The distance is controlled by the max unchanged words parameter which is set to 2 by default.
4.2 Operation Tier
In our first category experiment, we simply investigated the performance of each system in terms of Unnecessary, Missing or Replacement edits. The results are shown in Table 5.
The most surprising result is that 5 teams failed to correctly resolve any unnecessary token errors at all (AMU, IPN, PKU, RAC, UFC). This is especially surprising given that we would expect unnecessary token errors to be easier to correct than others; a system need only detect and delete without having to propose any alternative. There is also no obvious explanation as to why these teams had difficulty with this error type because each of them employed different combinations of correction strategies including machine translation (MT), language modelling, classifiers and rules.
In contrast, CUUI’s classifier approach (Rozovskaya et al., 2014) was the most successful at correcting not only unnecessary token errors, but also replacement token errors, while CAMB’s hybrid MT approach (Felice et al., 2014) significantly outperformed all others in terms of missing token errors. It would hence make sense to combine these two approaches, and indeed recent research has shown this improves overall performance (Rozovskaya and Roth, 2016).
4.3 General Error Types
Table 6 shows precision, recall and F0.5 for each of the error types in our proposed framework for each team in CoNLL-2014. We refer the reader to the shared task paper for more information about
6
501
502
503
504
505
506
507
508
509
510
511
512
513
514
515
516
517
518
519
520
521
522
523
524
525
526
527
528
529
530
531
532
533
534
535
536
537
538
539
540
541
542
543
544
545
546
547
548
549
550
551
552
553
554
555
556
557
558
559
560
561
562
563
564
565
566
567
568
569
570
571
572
573
574
575
576
577
578
579
580
581
582
583
584
585
586
587
588
589
590
591
592
593
594
595
596
597
598
599
AMU CAMB CUUI IITB IPN NTHU PKU POST RAC SJTU UFC UMC
ADJ P 7.14 9.09 - 0.00 0.00 0.00 50.00 0.00 11.11 0.00 - 4.35 R 9.38 12.82 - 0.00 0.00 0.00 6.67 0.00 3.33 0.00 - 3.57
F0.5 7.50 9.65 - 0.00 0.00 0.00 21.74 0.00 7.58 0.00 - 4.17
ADJ:FORM P 60.00 75.00 100.00 100.00 0.00 33.33 100.00 50.00 11.54 - - 80.00 R 60.00 50.00 27.27 33.33 0.00 33.33 33.33 11.11 42.86 - - 57.14
F0.5 60.00 68.18 65.22 71.43 0.00 33.33 71.43 29.41 13.51 - - 74.07
ADV P 11.76 12.66 0.00 0.00 0.00 0.00 0.00 - 0.00 4.76 - 7.27 R 5.88 23.26 0.00 0.00 0.00 0.00 0.00 - 0.00 3.03 - 10.00
F0.5 9.80 13.93 0.00 0.00 0.00 0.00 0.00 - 0.00 4.27 - 7.69
CONJ P 6.25 0.00 - - 0.00 0.00 - 0.00 - 0.00 - 0.00 R 7.69 0.00 - - 0.00 0.00 - 0.00 - 0.00 - 0.00
F0.5 6.49 0.00 - - 0.00 0.00 - 0.00 - 0.00 - 0.00
CONTR P 29.17 40.00 46.15 - 0.00 - - 33.33 0.00 66.67 - 28.57 R 87.50 28.57 75.00 - 0.00 - - 50.00 0.00 33.33 - 28.57
F0.5 33.65 37.04 50.00 - 0.00 - - 35.71 0.00 55.56 - 28.57
DET P 33.55 36.44 30.92 21.43 0.00 35.91 29.35 26.12 0.00 43.88 - 36.36 R 14.13 43.17 52.03 0.92 0.00 28.53 7.87 49.41 0.00 12.57 - 23.72
F0.5 26.32 37.61 33.65 3.93 0.00 34.14 18.99 28.84 0.00 29.29 - 32.86
MORPH P 54.67 57.35 52.94 15.38 2.25 25.00 21.43 30.30 27.27 50.00 36.36 34.48 R 45.56 45.35 20.00 2.70 2.78 20.25 32.14 12.82 15.19 2.74 5.06 11.63
F0.5 52.56 54.47 39.82 7.94 2.34 23.88 22.96 23.81 23.53 11.24 16.26 24.75
NOUN P 25.35 28.42 0.00 25.00 8.33 0.00 3.45 10.00 30.43 0.00 - 28.57 R 15.52 22.13 0.00 2.22 4.30 0.00 0.96 1.90 6.54 0.00 - 10.00
F0.5 22.50 26.89 0.00 8.20 7.02 0.00 2.27 5.41 17.59 0.00 - 20.83
NOUN:INFL P 55.56 60.00 50.00 - 0.00 100.00 57.14 80.00 60.00 0.00 - - R 83.33 75.00 66.67 - 0.00 40.00 57.14 66.67 60.00 0.00 - -
F0.5 59.52 62.50 52.63 - 0.00 76.92 57.14 76.92 60.00 0.00 - -
NOUN:NUM P 48.24 42.59 43.57 43.75 12.84 44.05 28.92 30.52 27.72 54.29 - 44.93 R 55.66 53.00 59.91 3.95 10.05 48.54 42.34 56.54 35.92 10.50 - 17.32
F0.5 49.56 44.33 46.09 14.52 12.16 44.88 30.88 33.62 29.04 29.60 - 34.07
NOUN:POSS P 20.00 66.67 - - - - 20.00 0.00 0.00 25.00 - 50.00 R 14.29 10.53 - - - - 5.26 0.00 0.00 4.55 - 5.00
F0.5 18.52 32.26 - - - - 12.82 0.00 0.00 13.16 - 17.86
ORTH P 60.00 66.67 73.81 - 3.45 0.00 28.57 49.32 16.67 - - 50.00 R 11.11 40.00 59.62 - 4.55 0.00 6.90 64.29 49.12 - - 17.24
F0.5 31.91 58.82 70.45 - 3.62 0.00 17.54 51.72 19.20 - - 36.23
OTHER P 19.33 23.57 16.13 12.50 2.38 1.40 11.11 13.95 0.00 0.00 - 13.54 R 6.65 9.87 1.37 0.30 0.31 0.58 0.58 1.69 0.00 0.00 - 3.74
F0.5 13.99 18.44 5.12 1.39 1.02 1.09 2.40 5.70 0.00 0.00 - 8.88
PART P 71.43 26.67 0.00 - - 12.90 - - - 40.00 - 18.18 R 20.00 16.00 0.00 - - 16.00 - - - 9.09 - 10.00
F0.5 47.17 23.53 0.00 - - 13.42 - - - 23.81 - 15.63
PREP P 47.62 41.70 32.69 75.00 0.00 10.95 - 21.74 0.00 37.50 - 20.53 R 16.53 35.91 13.65 1.44 0.00 12.81 - 2.18 0.00 7.18 - 13.42
F0.5 34.60 40.40 25.56 6.67 0.00 11.28 - 7.79 0.00 20.33 - 18.56
PRON P 43.75 20.00 0.00 0.00 11.11 50.00 100.00 27.27 4.76 0.00 - 22.45 R 9.72 13.25 0.00 0.00 1.72 2.86 1.56 4.76 1.54 0.00 - 14.10
F0.5 25.74 18.15 0.00 0.00 5.32 11.63 7.35 14.02 3.36 0.00 - 20.07
PUNCT P 25.00 60.47 39.53 100.00 0.00 48.28 - 27.27 0.00 5.00 - 43.02 R 3.57 15.66 11.33 1.85 0.00 9.72 - 6.34 0.00 0.97 - 23.13
F0.5 11.36 38.46 26.40 8.62 0.00 26.92 - 16.42 0.00 2.73 - 36.71
SPELL P 77.78 78.43 50.00 0.00 30.77 0.00 44.58 68.27 74.60 - - 100.00 R 64.95 42.55 2.60 0.00 5.41 0.00 71.15 70.30 86.24 - - 1.32
F0.5 74.82 67.11 10.75 0.00 15.87 0.00 48.18 68.67 76.67 - - 6.25
VERB P 18.84 16.09 - 0.00 7.69 0.00 14.29 0.00 0.00 0.00 - 18.87 R 8.07 8.86 - 0.00 0.71 0.00 0.68 0.00 0.00 0.00 - 6.58
F0.5 14.87 13.83 - 0.00 2.60 0.00 2.87 0.00 0.00 0.00 - 13.74
VERB:FORM P 34.85 38.24 70.59 50.00 8.77 36.84 30.77 20.00 35.42 30.77 100.00 34.04 R 23.71 26.26 26.37 1.15 5.75 36.84 35.16 3.45 34.69 4.65 1.22 18.18
F0.5 31.86 35.04 52.86 5.26 7.94 36.84 31.56 10.20 35.27 14.49 5.81 28.99
VERB:INFL P 100.00 100.00 - - 100.00 100.00 50.00 100.00 100.00 - 0.00 - R 100.00 100.00 - - 50.00 50.00 50.00 50.00 100.00 - 0.00 -
F0.5 100.00 100.00 - - 83.33 83.33 50.00 83.33 100.00 - 0.00 -
VERB:SVA P 49.06 42.68 54.71 50.00 24.53 50.58 57.14 33.33 34.83 59.09 82.86 58.33 R 27.37 31.82 70.45 1.15 13.98 66.92 17.20 16.67 31.00 14.13 28.16 14.29
F0.5 42.35 39.95 57.27 5.26 21.31 53.18 39.02 27.78 33.99 36.11 59.67 36.08
VERB:TENSE P 20.55 26.05 75.00 66.67 7.14 38.89 10.61 20.00 23.27 15.38 100.00 30.51 R 8.82 17.92 5.33 1.27 1.24 4.22 4.35 2.34 21.26 2.52 1.26 10.98
F0.5 16.23 23.88 20.74 5.92 3.66 14.71 8.24 7.97 22.84 7.60 5.99 22.50
WO P - 38.89 0.00 66.67 - - - 0.00 0.00 - - 41.18 R - 33.33 0.00 14.29 - - - 0.00 0.00 - - 35.00
F0.5 - 37.63 0.00 38.46 - - - 0.00 0.00 - - 39.77
Table 6: Precision, recall and F0.5 for each team and error type. A dash indicates the team’s system did not attempt to correct the given error type (TP+FP = 0). The highest F-score for each type is highlighted.
7
601
602
603
604
605
606
607
608
609
610
611
612
613
614
615
616
617
618
619
620
621
622
623
624
625
626
627
628
629
630
631
632
633
634
635
636
637
638
639
640
641
642
643
644
645
646
647
648
649
663
664
665
666
667
668
669
670
671
672
673
674
675
676
677
678
679
680
681
682
683
684
685
686
687
688
689
690
691
692
693
694
695
696
697
698
699
each team’s system (Ng et al., 2014).
Overall, CAMB was the most successful team in terms of error types, achieving the highest Fscore in 10 (out of 24) error categories, followed by AMU (Junczys-Dowmunt and Grundkiewicz, 2014), who scored highest in 6 categories. All but 2 teams (IITB and IPN) achieved the best score in at least 1 category, which suggests that different approaches to GEC complement different error types. Only CAMB attempted to correct at least 1 error from every category.
Regarding individual error categories: PKU’s language model approach significantly outperformed all others in handling ADJ errors (21.74 F0.5). ADV and CONJ errors proved extremely difficult for all teams, with the best results at 13.93 F0.5 (CAMB) and 6.49 F0.5 (AMU) respectively. Although several teams built specialist classifiers for DET errors, CAMB’s hybrid MT system still slightly outperformed them (37.61 F0.5). MT approaches were most effective at correcting NOUN errors (AMU, CAMB, UMC), while fairly high scores for NOUN:NUM errors showed that this category could be successfully handled by MT (AMU, CAMB), classifiers (CUUI) or language model approaches (NTHU). Few teams attempted to correct NOUN:POSS errors, but CAMB’s system handled them the best (32.26 F0.5). CUUI’s classifier for ORTH errors significantly outperformed all other teams at 70.45 F0.5. As with DET errors, several teams employed specialist classifiers to tackle PREP errors, but CAMB’s hybrid system still worked best overall (40.40 F0.5). AMU’s MT system was most successful at correcting PRON errors (25.74 F0.5), while CAMB was most successful at correcting PUNCT errors (38.46 F0.5). Although spell checkers are widespread nowadays, many teams did not seem to employ them; this would have been an easy way to boost overall performance. CUUI’s classifier approach to VERB:FORM errors significantly outperformed other approaches (52.86 F0.5), suggesting a classifier is well-suited to this category. While UFC’s rule-based approach achieved the highest score for VERB:SVA errors (59.67 F0.5), it is worth noting that CUUI’s classifier approach was not far behind (57.27 F0.5). Finally, only 3 teams were successful at handling WO errors (CAMB, IITB and UMC), all of whom achieved similar scores of just under 40 F0.5 using MT.
4.4 Detailed Error Types
In addition to analysing general error types, the modular design of our framework also allows us to evaluate error type performance at an even greater level of detail. For example, Table 7 shows the breakdown of Determiner errors for two teams using different approaches in terms of edit operation. Note that this is a representative example of detailed error type performance as an analysis of all error type combinations for all teams would take up too much space.
While CAMB’s hybrid MT approach achieved a higher score than CUUI’s classifier approach overall (37.61 F0.5 vs. 33.65 F0.5), our more detailed evaluation reveals that actually CUUI’s approach performed better at Replacement Determiner errors than CAMB (26.53 F0.5 vs. 21.39 F0.5). This shows that even though one approach might be better than another overall, other approaches may still have complementary strengths. In fact the main weakness of CUUI’s classifier seems to be that a high recall for missing and unnecessary determiners is counterbalanced by a low precision, which suggests that minimising false positives in these categories is the most obvious avenue for improvement.
4.5 Multi Token Errors
Another benefit of explicitly annotating all hypothesis edits is that edit spans become fixed; this means we can evaluate system performance in terms of edit size. Table 8 hence shows the overall performance for each team at correcting multitoken edits, where a multi-token edit is an edit that affects at least two tokens on either the source or
8
701
702
703
704
705
706
707
708
709
710
711
712
713
714
715
716
717
718
719
720
721
722
723
724
725
726
727
728
729
730
731
732
733
734
735
736
737
738
739
740
741
742
743
744
745
746
747
748
749
767
768
769
770
771
772
773
774
775
776
777
778
779
780
781
782
783
784
785
786
787
788
789
790
791
792
793
794
795
796
797
798
799
Team P R F0.5 AMU 17.14 5.33 11.88 CAMB 27.22 17.06 24.32 CUUI 15.69 3.67 9.48 IITB 28.57 0.94 4.15 IPN 3.33 0.47 1.51 NTHU 0.00 0.00 0.00 PKU 25.00 1.40 5.73 POST 12.77 2.82 7.48 RAC 2.96 2.82 2.93 SJTU 10.00 0.47 1.99 UFC - - - UMC 19.82 9.82 16.47
Table 8: Each team’s performance at correcting multi-token edits; i.e. there are at least 2 tokens on one side of the edit.
target side. In the CoNLL-2014 test set, there are roughly 220 such edits (about 10% of all edits).
In general, teams did not do well at multi-token edits. In fact only three teams achieved scores greater than 10 F0.5 and all of them used MT (AMU, CAMB, UMC). This is significant because recent work has suggested that the main goal of GEC should be to produce fluent-sounding, rather than just grammatical, sentences, even though this often requires complex multi-token edits (Sakaguchi et al., 2016). If no system is particularly adept at correcting multi-token errors however, robust fluency correction will likely require more sophisticated methods than are currently available.
4.6 Detection vs. Correction
Another important aspect of GEC that is less frequently reported in the literature is that of error detection; i.e. the extent to which a system can identify erroneous tokens in text. This can be calculated by comparing the edit overlap between the hypothesis and reference files regardless of the proposed correction in a manner similar to Recognition evaluation in the HOO shared tasks for GEC (Dale and Kilgarriff, 2011).
Figure 1 hence shows how each team’s score for detection differed in relation to their score for correction. While CAMB still scored highest for detection overall, it is interesting to note that the difference between the 2nd and 3rd place contenders (CUUI and AMU) is a lot narrower. This suggests that even though AMU detected roughly as many errors as CUUI, it was less successful at correcting them. IPN and PKU are also notable for detecting
many more errors than they were able to correct. Although we do not do so here, our scorer is also capable of providing a detailed error type breakdown for detection.
5 Conclusion
In this paper, we have described a method to automatically annotate parallel error correction data with explicit edit spans and error type information. This can be used to standardise existing error correction corpora or facilitate a detailed error type evaluation. The tool we use to do this will be released with this paper.
Our approach makes use of previous work to align sentences based on linguistic intuition and then introduces a new rule-based framework to classify edits. This framework is entirely dataset independent, and relies only on automatically obtained information such as POS tags and lemmas. A small-scale evaluation of our classifier found that each rater considered >95% of the predicted error types as either “Good” (85%) or “Acceptable” (10%).
We demonstrated the value of our approach by carrying out a detailed evaluation of system error type performance for the first time for all teams in the CoNLL-2014 shared task on Grammatical Error Correction. We found that different systems had different strengths and weaknesses which we hope researchers can exploit to further improve general performance.
9
801
802
803
804
805
806
807
808
809
810
811
812
813
814
815
816
817
818
819
820
821
822
823
824
825
826
827
828
829
830
831
832
833
834
835
836
837
838
839
840
841
842
843
844
845
846
847
848
849
850
851
852
853
854
855
856
857
858
859
860
861
862
863
864
865
866
867
868
869
870
871
872
873
874
875
876
877
878
879
880
881
882
883
884
885
886
887
888
889
890
891
892
893
894
895
896
897
898
899
A Complete list of valid error code combinations
Operation Tier Type Missing Unnecessary Replacement
To ke
n Ti
er
Pa rt
O fS
pe ec h Adjective M:ADJ U:ADJ R:ADJ Adverb M:ADV U:ADV R:ADV Conjunction M:CONJ U:CONJ R:CONJ Determiner M:DET U:DET R:DET Noun M:NOUN U:NOUN R:NOUN Particle M:PART U:PART R:PART Preposition M:PREP U:PREP R:PREP Pronoun M:PRON U:PRON R:PRON Punctuation M:PUNCT U:PUNCT R:PUNCT Verb M:VERB U:VERB R:VERB
O th
er Contraction M:CONTR U:CONTR R:CONTR Morphology - - R:MORPH Orthography - - R:ORTH Other M:OTHER U:OTHER R:OTHER Spelling - - R:SPELL Word Order - - R:WO
M or
ph ol
og y
Ti er
Adjective Form - - R:ADJ:FORM Noun Inflection - - R:NOUN:INFL Noun Number - - R:NOUN:NUM Noun Possessive M:NOUN:POSS U:NOUN:POSS R:NOUN:POSS Verb Form M:VERB:FORM U:VERB:FORM R:VERB:FORM Verb Inflection - - R:VERB:INFL Verb Agreement - - R:VERB:SVA Verb Tense M:VERB:TENSE U:VERB:TENSE R:VERB:TENSE
Table 9: There are 55 total possible error types. This table shows all of them except UNK, which indicates an uncorrected error. A dash indicates an impossible combination.

The paper presents a novel approach for evaluating grammatical error correction (GEC) systems. This approach makes it possible to assess the performance of GEC systems by error type not only in terms of recall but also in terms of precision, which was previously not possible in general since system output is usually not annotated with error categories.  Strengths:   - The proposed evaluation is an important stepping stone for    analyzing GEC system behavior.  - The paper includes evaluation for a variety of systems.  - The approach has several advantages over previous work:    - it computes precision by error type    - it is independent of manual error annotation    - it can assess the performance on multi token errors  - The automatically selected error tags for pre-computed error spans    are mostly approved of by human experts  Weaknesses:   - A key part   the rules to derive error types   are not described.  - The classifier evaluation lacks a thorough error analysis and based    upon that it lacks directions of future work on how to improve the    classifier.  - The evaluation was only performed for English and it is unclear how    difficult it would be to use the approach on another language.  Classifier and Classifier Evaluation ====================================  It is unclear on what basis the error categories were devised. Are they based on previous work?  Although the approach in general is independent of the alignment algorithm, the rules are probably not, but the authors don't provide details on that.  The error categories are a major part of the paper and the reader should at least get a glimpse of how a rule to assign an error type looks like.  Unfortunately, the paper does not apply the proposed evaluation on languages other than English.  It also does not elaborate on what changes would be necessary to run the classifier on other languages. I assume that the rules used for determining edit boundaries as well as for determining the error tags depend on the language the pre-processing pipeline to a certain extent and therefore need to be adapted. Also, the error categories might need to be changed.  The authors do not provide any detail on the rules for assigning error categories (how many are there overall per error type? how complex are they?) to estimate the effort necessary to use the approach on another language.  The error spans computed in the pre-processing step seem to be inherently continuous (which is also the case with the M2 scorer), which is problematic since there are errors which can only be tagged accurately when the error span is discontinuous. In German, for example, verbs with separable prefixes are separated from each other in the main clause: [1st constituent] [verb] [other constituents] [verb prefix]. Would the classifier be able to tag discontinuous edit spans?  The authors write that all human judges rated at least 95  % of the automatically assigned error tags as appropriate  despite the degree of noise introduced by automatic edit extraction  (295). I would be more cautious with this judgment since the raters might also have been more forgiving when the boundaries were noisy. In addition, they were not asked to select a tag without knowing the system output but could in case of noisy boundaries be more biased towards the system output. Additionally, there was no rating option between  Bad (Not Appropriate)  and  Appropriate , which might also have led raters to select  Appropriate  over  Bad . To make the evaluation more sound, the authors should also evaluate how the human judges rate the classifier output if the boundaries were manually created, i.e. without the noise introduced by faulty boundaries.  The classifier evaluation lacks a thorough error analysis. It is only mentioned that  Bad  is usually traced back to a wrong POS tag. Questions I'd like to see addressed: When did raters select  Bad , when  Appropriate ? Does the rating by experts point at possibilities to improve the classifier?  Gold Reference vs. Auto Reference =================================  It is unclear on what data the significance test was performed exactly. Did you test on the F0.5 scores? If so, I don't think this is a good idea since it is a derived measure with weak discriminative power (the performance in terms of recall an precision can be totally different but have the same F0.5 score). Also, at the beginning of Section 4.1 the authors refer to the mismatch between automatic and reference in terms of alignment and classification but as far as I can tell, the comparison between gold and reference is only in terms of boundaries and not in terms of classification.  Error Type Evaluation =====================  I do not think it is surprising that 5 teams (~line 473) failed to correct any unnecessary token error. For at least two of the systems there is a straightforward explanation why they cannot handle superfluous words. The most obvious is UFC: Their rule-base approach works on POS tags (Ng et al., 2014) and it is just not possible to determine superfluous words based on POS alone. Rozovskaya & Roth (2016) provide an explanation why AMU performs poorly on superfluous words.  The authors do not analyze or comment the results in Table 6 with respect to whether the systems were designed to handle the error type. For some error types, there is a straight-forward mapping between error type in the gold standard and in the auto reference, for example for word order error. It remains unclear whether the systems failed completely on specific error types or were just not designed to correct them (CUUI for example is reported with precision+recall=0.0, although it does not target word order errors). In the CUUI case (and there are probably similar cases), this also points at an error in the classification which is neither analyzed nor discussed.  Please report also raw values for TP, FP, TN, FN in the appendix for Table 6. This makes it easier to compare the systems using other measures. Also, it seems that for some error types and systems the results in Table 6 are based only on a few instances. This would also be made clear when reporting the raw values.  Your write  All but 2 teams (IITB and IPN) achieved the best score in at least 1 category, which suggests that different approaches to GEC complement different error types.  (606) It would be nice to mention here that this is in line with previous research.  Multi-token error analysis is helpful for future work but the result needs more interpretation: Some systems are probably inherently unable to correct such errors but none of the systems were trained on a parallel corpus of learner data and fluent (in the sense of Sakaguchi et al, 2016) corrections.  Other =====  - The authors should have mentioned that for some of the GEC   approaches, it was not impossible before to provide error   annotations, e.g. systems with submodules for one error type each.   Admittedly, the system would need to be adapted to include the   submodule responsible for a change in the system output. Still, the   proposed approach enables to compare GEC systems for which producing   an error tagged output is not straightforward to other systems in a   unified way. - References: Some titles lack capitalizations. URL for Sakaguchi et   al. (2016) needs to be wrapped. Page information is missing for   Efron and Tibshirani (1993).  Author response ===============  I agree that your approach is not  fatally flawed  and I think this review actually points out quite some positive aspects. The approach is good, but the paper is not ready.  The basis for the paper are the rules for classifying errors and the lack of description is a major factor.        This is not just a matter about additional examples. If the rules are not seen as a one-off implementation, they need to be described to be replicable or to adapt them.  Generalization to other languages should not be an afterthought.  It would be serious limitation if the approach only worked on one language by design.  Even if you don't perform an adaption for other languages, your approach should be transparent enough for others to estimate how much work such an adaptation would be and how well it could reasonably work.  Just stating that most research is targeted at ESL only reinforces the problem.  You write that the error types certain systems tackle would be  usually obvious from the tables .  I don't think it is as simple as that -- see the CUUI example mentioned above as well as the unnecessary token errors.  There are five systems that don't correct them (Table 5) and it should therefore be obvious that they did not try to tackle them. However, in the paper you write that  There is also no obvious explanation as to why these teams had difficulty with this error type .