A Full Non-Monotonic Transition System for Unrestricted Non-Projective Parsing
Restricted non-monotonicity has been shown beneficial for the projective arceager dependency parser in previous research, as posterior decisions can repair mistakes made in previous states due to the lack of information. In this paper, we propose a novel, fully non-monotonic transition system based on the non-projective Covington algorithm. As a non-monotonic system requires exploration of erroneous actions during the training process, we develop several non-monotonic variants of the recently defined dynamic oracle for the Covington parser, based on tight approximations of the loss. Experiments on datasets from the CoNLL-X and CoNLL-XI shared tasks show that a non-monotonic dynamic oracle outperforms the monotonic version in the majority of languages.
1 Introduction
Greedy transition-based dependency parsers are widely used in different NLP tasks due to their speed and efficiency. They parse a sentence from left to right by greedily choosing the highestscoring transition to go from the current parser configuration or state to the next. The resulting sequence of transitions incrementally builds a parse for the input sentence. The scoring of the transitions is provided by a statistical model, previously trained to approximate an oracle, a function that selects the needed transitions to parse a gold tree.
Unfortunately, the greedy nature that grants these parsers their efficiency also represents their main limitation. McDonald and Nivre (2007) show that greedy transition-based parsers lose accuracy to error propagation: a transition erroneously chosen by the greedy parser can place it
in an incorrect and unknown configuration, causing more mistakes in the rest of the transition sequence. Training with a dynamic oracle (Goldberg and Nivre, 2012) improves robustness in these situations, but in a monotonic transition system, erroneous decisions made in the past are permanent, even when the availability of further information in later states might be useful to correct them.
Honnibal et al. (2013) show that allowing some degree of non-monotonicity, by using a limited set of non-monotonic actions that can repair past mistakes and replace previously-built arcs, can increase the accuracy of a transition-based parser. In particular, they present a modified arc-eager transition system where the Left-Arc and Reduce transitions are non-monotonic: the former is used to repair invalid attachments made in previous states by replacing them with a leftward arc, and the latter allows the parser to link two words with a rightward arc that were previously left unattached due to an erroneous decision. Since the Right-Arc transition is still monotonic and leftward arcs can never be repaired because their dependent is removed from the stack by the arc-eager parser and rendered inaccessible, this approach can only repair certain kinds of mistakes: namely, it can fix erroneous rightward arcs by replacing them with a leftward arc, and connect a limited set of unattached words with rightward arcs. In addition, they argue that non-monotonicity in the training oracle can be harmful for the final accuracy and, therefore, they suggest to apply it only as a fallback component for a monotonic oracle, which is given priority over the non-monotonic one. Thus, this strategy will follow the path dictated by the monotonic oracle the majority of the time. Honnibal and Johnson (2015) present an extension of this transition system with an Unshift transition allowing it some extra flexibility to correct past errors. However, the restriction that only rightward
2
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
arcs can be deleted, and only by replacing them with leftward arcs, is still in place. Furthermore, both versions of the algorithm are limited to projective trees.
In this paper, we propose a non-monotonic transition system based on the non-projective Covington parser, together with a dynamic oracle to train it with erroneous examples that will need to be repaired. Unlike the system developed in (Honnibal et al., 2013; Honnibal and Johnson, 2015), we work with full non-monotonicity. This has a twofold meaning: (1) our approach can repair previous erroneous attachments regardless of their original direction, and it can replace them either with a rightward or leftward arc as both arc transitions are non-monotonic;1 and (2) we use exclusively a non-monotonic oracle, without the interferences of monotonic decisions. These modifications are feasible because the non-projective Covington transition system is less rigid than the arc-eager algorithm, as words are never deleted from the parser’s data structures and can always be revisited, making it a better option to exploit the full potencial of non-monotonicity. To our knowledge, the presented system is the first nonmonotonic parser that can produce non-projective dependency analyses. Another novel aspect is that our dynamic oracle is approximate, i.e., based on efficiently-computable approximations of the loss due to the complexity of calculating its actual value in a non-monotonic and non-projective scenario. However, this is not a problem in practice: experimental results show how our parser and oracle can use non-monotonic actions to repair erroneous attachments, outperforming the monotonic version developed by Gómez-Rodrı́guez and Fernández-González (2015) in a large majority of the datasets tested.
2 Preliminaries
2.1 Non-Projective Covington Transition System
The non-projective Covington parser was originally defined by Covington (2001), and then recast by Nivre (2008) under the transition-based parsing framework.
1The only restriction is that parsing must still proceed in left-to-right order. For this reason, a leftward arc cannot be repaired with a rightward arc, because this would imply going back in the sentence. The other three combinations (replacing leftward with leftward, rightward with leftward or rightward with rightward arcs) are possible.
The transition system that defines this parser is as follows: each parser configuration is of the form c = 〈λ1, λ2, B,A〉, such that λ1 and λ2 are lists of partially processed words, B is another list (called the buffer) containing currently unprocessed words, and A is the set of dependencies that have been built so far. Suppose that our input is a string w1 · · ·wn, whose word occurrences will be identified with their indices 1 · · ·n for simplicity. Then, the parser will start at an initial configuration cs(w1 . . . wn) = 〈[], [], [1 . . . n], ∅〉, and execute transitions chosen from those in Figure 1 until a terminal configuration of the form {〈λ1, λ2, [], A〉 ∈ C} is reached. At that point, the sentence’s parse tree is obtained from A.2
These transitions implement the same logic as the double nested loop traversing word pairs in the original formulation by Covington (2001). When the parser’s configuration is 〈λ1|i, λ2, j|B,A〉, we say that it is considering the focus words i and j, located at the end of the first list and at the beginning of the buffer. At that point, the parser must decide whether these two words should be linked with a leftward arc i ← j (Left-Arc transition), a rightward arc i → j (Right-Arc transition), or not linked at all (No-Arc transition). However, the two transitions that create arcs will be disallowed in configurations where this would cause a violation of the single-head constraint (a node can have at most one incoming arc) or the acyclicity constraint (the dependency graph cannot have cycles). After applying any of these three transitions, i is moved to the second list to make i − 1 and j the focus words for the next step. As an alternative, we can instead choose to execute a Shift transition which lets the parser read a new input word, placing the focus on j and j + 1.
The resulting parser can generate any possible dependency tree for the input, including arbitrary non-projective trees. While it runs in quadratic worst-case time, in theory worse than lineartime transition-based parsers (e.g. (Nivre, 2003; Gómez-Rodrı́guez and Nivre, 2013)), it has been shown to outspeed linear algorithms in practice, thanks to feature extraction optimizations that cannot be implemented in other parsers (Volokh and Neumann, 2012). In fact, one of the fastest dependency parsers ever reported uses this algorithm
2In general A is a forest, but it can be converted to a tree by linking headless nodes as dependents of an artificial root node at position 0. When we refer to parser outputs as trees, we assume that this transformation is being implicitly made.
3
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
Shift: 〈λ1, λ2, j|B,A〉 ⇒ 〈λ1 · λ2|j, [], B,A〉 No-Arc: 〈λ1|i, λ2, B,A〉 ⇒ 〈λ1, i|λ2, B,A〉 Left-Arc: 〈λ1|i, λ2, j|B,A〉 ⇒ 〈λ1, i|λ2, j|B,A ∪ {j → i}〉 only if @k | k → i ∈ A (single-head) and i→∗ j 6∈ A (acyclicity). Right-Arc: 〈λ1|i, λ2, j|B,A〉 ⇒ 〈λ1, i|λ2, j|B,A ∪ {i→ j}〉
only if @k | k → j ∈ A (single-head) and j →∗ i 6∈ A (acyclicity).
Figure 1: Transitions of the monotonic Covington non-projective dependency parser. The notation i→∗ j ∈ A means that there is a (possibly empty) directed path from i to j in A.
(Volokh, 2013).
2.2 Monotonic Dynamic Oracle
A dynamic oracle is a function that maps a configuration c and a gold tree tG to the set of transitions that can be applied in c and lead to some parse tree t minimizing the Hamming loss with respect to tG (the amount of nodes whose head is different in t and tG). Following Goldberg and Nivre (2013), we say that an arc set A is reachable from configuration c, and we write c A, if there is some (possibly empty) path of transitions from c to some configuration c′ = 〈λ1, λ2, B,A′〉, with A ⊆ A′. Then, we can define the loss of configuration c as
`(c) = min t|c t
L(t, tG),
and therefore, a correct dynamic oracle will return the set of transitions
od(c, tG) = {τ | `(c)− `(τ(c)) = 0},
i.e., the set of transitions that do not increase configuration loss, and thus lead to the best parse (in terms of loss) reachable from c. Hence, implementing a dynamic oracle reduces to computing the loss `(c) for each configuration c.
Goldberg and Nivre (2013) show a straightforward method to calculate loss for parsers that are arc-decomposable, i.e., those where every arc set A that can be part of a well-formed parse verifies that if c (i → j) for every i → j ∈ A (i.e., each of the individual arcs of A is reachable from a given configuration c), then c A (i.e., the set A as a whole is reachable from c). If this holds, then the loss of a configuration c equals the number of gold arcs that are not individually reachable from c, which is easy to compute in most parsers.
Gómez-Rodrı́guez and Fernández-González (2015) show that the non-projective Covington parser is not arc-decomposable because sets of individually reachable arcs may form cycles together with already-built arcs, preventing them
from being jointly reachable due to the acyclicity constraint. In spite of this, they prove that a dynamic oracle for the Covington parser can be efficiently built by counting individually unreachable arcs, and correcting for the presence of such cycles. Concretely, the loss is computed as:
`(c) = |U(c, tG)|+ nc(A ∪ I(c, tG))
where I(c, tG) = {x → y ∈ tG | c (x → y)} is the set of individually reachable arcs of tG from configuration c; U(c, tG) is the set of individually unreachable arcs of tG from c, computed as tG\I(c, tG); and nc(G) denotes the number of cycles in a graph G.
Therefore, to calculate the loss of a configuration c, we only need to compute the two terms |U(c, tG)| and nc(A ∪ I(c, tG)). To calculate the first term, given a configuration cwith focus words i and j (i.e., c = 〈λ1|i, λ2, j|B,A〉), an arc x→ y will be in U(c, tG) if it is not in A, and at least one of the following holds:
• j > max(x, y), (i.e., we have read too far in the string and can no longer get max(x, y) as right focus word), • j = max(x, y) ∧ i < min(x, y), (i.e., we
have max(x, y) as the right focus word but the left focus word has already moved left past min(x, y), and we cannot go back), • there is some z 6= 0, z 6= x such that z → y ∈ A, (i.e., we cannot create x→ y because it would violate the single-head constraint), • x and y are on the same weakly connected
component of A (i.e., we cannot create x → y due to the acyclicity constraint).
The second term of the loss, nc(A ∪ I(c, tG)), can be computed by first obtaining I(c, tG) as tG \ U(c, tG). Since the graph I(c, tG) has indegree 1, the algorithm by Tarjan (1972) can then be used to find and count the cycles in O(n) time.
4
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
Algorithm 1 Computation of the loss of a configuration in the monotonic oracle. 1: function LOSS(c = 〈λ1|i, λ2, j|B,A〉, tG) 2: U ← ∅ . Variable U is for U(c, tG) 3: for each x→ y ∈ (tG \A) do 4: left ← min(x, y) 5: right ← max(x, y) 6: if j > right ∨ 7: (j = right ∧ i < left)∨ 8: (∃z > 0, z 6= x : z → y ∈ A)∨ 9: WEAKLYCONNECTED(A, x, y) then 10: U ← u ∪ {x→ y} 11: I ← tG \U . Variable I is for I(c, tG) 12: return |U |+ COUNTCYCLES(A ∪ I )
Algorithm 1 shows the resulting loss calculation algorithm, where COUNTCYCLES is a function that counts the number of cycles in the given graph and WEAKLYCONNECTED returns whether two given nodes are weakly connected in A.
3 Non-Monotonic Transition System for the Covington Non-Projective Parser
We now define a non-monotonic variant of the Covington non-projective parser. To do so, we allow the Right-Arc and Left-Arc transitions to create arcs between any pair of nodes without restriction. If the node attached as dependent already had a previous head, the existing attachment is discarded in favor of the new one. This allows the parser to correct erroneous attachments made in the past by assigning new heads, while still enforcing the single-head constraint, as only the most recent head assigned to each node is kept.
To enforce acyclicity, one possibility would be to keep the logic of the monotonic algorithm, forbidding the creation of arcs that would create cycles. However, this greatly complicates the definition of the set of individually unreachable arcs, which is needed to compute the loss bounds that will be used by the dynamic oracle. This is because a gold arc x → y may superficially seem unreachable due to forming a cycle together with arcs in A, but it might in fact be reachable if there is some transition sequence that first breaks the cycle using non-monotonic transitions to remove arcs from A, to then create x → y. We do not know of a way to characterize the conditions under which such a transition sequence exists, and thus cannot estimate the loss efficiently.
Instead, we enforce the acyclicity constraint in a similar way to the single-head constraint: Right-Arc and Left-Arc transitions are always allowed, even if the prospective arc would create a
cycle in A. However, if the creation of a new arc x→ y generates a cycle in A, we immediately remove the arc of the form z → x from A (which trivially exists, and is unique due to the singlehead constraint). This not only enforces the acyclicity constraint while keeping the computation of U(c, tG) simple and efficient, but also produces a straightforward, coherent algorithm (arc transitions are always allowed, and both constraints are enforced by deleting a previous arc) and allows us to exploit non-monotonicity to the maximum (we can not only recover from assigning a node the wrong head, but also from situations where previous errors together with the acyclicity constraint prevent us from building a gold arc, keeping with the principle that later decisions override earlier ones).
In Figure 2, we can see the resulting nonmonotonic transition system for the non-projective Covington algorithm, where, unlike the monotonic version, all transitions are allowed at each configuration, and the single-head and acyclicity constraints are kept in A by removing offending arcs.
4 Non-Monotonic Approximate Dynamic Oracle
To successfully train a non-monotonic system, we need a dynamic oracle with error exploration, so that the parser will be put in erroneous states and need to apply non-monotonic transitions in order to repair them. To achieve that, we modify the dynamic oracle defined by Gómez-Rodrı́guez and Fernández-González (2015) so that it can deal with non-monotonicity. Our modification is an approximate dynamic oracle: due to the extra flexibility added to the algorithm by non-monotonicity, we do not know of an efficient way of obtaining an exact calculation of the loss of a given configuration. Instead, we use upper or lower bounds on the loss, which we empirically show to be very tight (less that 1% relative error with respect to the real loss) and are sufficient for the algorithm to provide better accuracy than the exact monotonic oracle.
First of all, we adapt the computation of the set of individually unreachable arcs U(c, tG) to the new algorithm. In particular, if c has focus words i and j (i.e., c = 〈λ1|i, λ2, j|B,A〉), then an arc x → y is in U(c, tG) if it is not in A, and at least one of the following holds: • j > max(x, y), (i.e., we have read too far in
the string and can no longer get max(x, y) as
5
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
458
459
460
461
462
463
464
465
466
467
468
469
470
471
472
473
474
475
476
477
478
479
480
481
482
483
484
485
486
487
488
489
490
491
492
493
494
495
496
497
498
499
Shift: 〈λ1, λ2, j|B,A〉 ⇒ 〈λ1 · λ2|j, [], B,A〉 No-Arc: 〈λ1|i, λ2, B,A〉 ⇒ 〈λ1, i|λ2, B,A〉 Left-Arc: 〈λ1|i, λ2, j|B,A〉 ⇒ 〈λ1, i|λ2, j|B, (A ∪ {j → i}) \{x→ i ∈ A} \ {k → j ∈ A | i→∗ k ∈ A}〉 Right-Arc: 〈λ1|i, λ2, j|B,A〉 ⇒ 〈λ1, i|λ2, j|B,A ∪ {i→ j}
\{x→ j ∈ A} \ {k → i ∈ A | j →∗ k ∈ A}〉
Figure 2: Transitions of the non-monotonic Covington non-projective dependency parser. The notation i→∗ j ∈ A means that there is a (possibly empty) directed path from i to j in A.
right focus word), • j = max(x, y) ∧ i < min(x, y) (i.e., we
have max(x, y) as the right focus word but the left focus word has already moved left past min(x, y), and we cannot move it back).
Note that, since the head of a node can change during the parsing process and arcs that produce cycles in A can be built, the two last conditions present in the monotonic scenario for computing U(c, tG) are not needed when we use nonmonotonicity and, as a consequence, the set of individually reachable arcs I(c, tG) is larger: due to the greater flexibility provided by nonmonotonicity, we can reach arcs that would be unreachable for the monotonic version.
Since arcs that are in this new U(c, tG) are unreachable even by the non-monotonic parser, |U(c, tG)| is trivially a lower bound of the loss `(c). It is worth noting that there always exists at least one transition sequence that builds every arc in I(c, tG) at some point (although not all of them necessarily appear in the final tree, due to non-monotonicity). This can be easily shown based on the fact that the non-monotonic parser does not forbid transitions at any configuration. Thanks to this, we can can generate one such sequence by just applying the original Covington (2001) criteria (choose an arc transition whenever the focus words are linked in I(c, tG), and otherwise Shift or No-Arc depending on whether the left focus word is the first word in the sentence or not), although this sequence is not necessarily optimal in terms of loss. In such a transition sequence, the gold arcs that are missed are (1) those in U(c, tG), and (2) those that are removed by the cycle-breaking in Left-Arc and Right-Arc transitions. In practice configurations where (2) is needed are uncommon, so this lower bound is a very close approximation of the real loss, as will be seen empirically below.
This reasoning also helps us calculate an up-
per bound of the loss: in a transition sequence as described, if we only build the arcs in I(c, tG) and none else, the amount of arcs removed by breaking cycles (2) cannot be larger than the number of cycles in A ∪ I(c, tG). This means that |U(c, tG)|+nc(A∪I(c, tG)) is an upper bound of the loss `(c). Note that, contrary to the monotonic case, this expression does not always give us the exact loss, for several reasons: firstly, A∪I(c, tG) can have non-disjoint cycles (a node may have different heads in A and I since attachments are not permanent, contrary to the monotonic version) and thus removing a single arc may break more than one cycle; secondly, the removed arc can be a non-gold arc of A and therefore not incur loss; and thirdly, there may exist alternative transition sequences where a cycle in A∪I(c, tG) is broken early by non-monotonic configurations that change the head of a wrongly-attached node in A to a different (and also wrong) head,3 removing the cycle before the cycle-breaking mechanism needs to come into play without incurring in extra errors. Characterizing the situations where such an alternative exists is the main difficulty for an exact calculation of the loss.
However, it is possible to obtain a closer upper bound to the real loss if we consider the following: for each cycle in A ∪ I(c, tG) that will be broken by the transition sequence described above, we can determine exactly which is the arc removed by cycle-breaking (if x → y is the arc that will close the cycle according to the Covington arc-building order, then the affected arc is the one of the form z → x). The cycle can only cause the loss of a gold arc if that arc z → x is gold, which can be trivially checked. Hence, if we call cycles where that holds problematic cycles, then the expression
3Note that, in this scenario, the new head must also be wrong because otherwise the newly created arc would be an arc of I(c, tG) (and therefore, would not be breaking a cycle in A ∪ I(c, tG)). However, replacing a wrong attachment with another wrong attachment need not increase loss.
6
501
502
503
504
505
506
507
508
509
510
511
512
513
514
515
516
517
518
519
520
521
522
523
524
525
526
527
528
529
530
531
532
533
534
535
536
537
538
539
540
541
542
543
544
545
546
547
548
549
550
551
552
553
554
555
556
557
558
559
560
561
562
563
564
565
566
567
568
569
570
571
572
573
574
575
576
577
578
579
580
581
582
583
584
585
586
587
588
589
590
591
592
593
594
595
596
597
598
599
average value relative difference to loss Language lower loss pc upper upper lower pc upper upper Arabic 0.66925 0.67257 0.67312 0.68143 0.00182 0.00029 0.00587 Basque 0.58260 0.58318 0.58389 0.62543 0.00035 0.00038 0.02732 Catalan 0.58009 0.58793 0.58931 0.60644 0.00424 0.00069 0.00961 Chinese 0.56515 0.56711 0.57156 0.62921 0.00121 0.00302 0.03984 Czech 0.57521 0.58357 0.59401 0.62883 0.00476 0.00685 0.02662 English 0.55267 0.56383 0.56884 0.59494 0.00633 0.00294 0.01767 Greek 0.56123 0.57443 0.57983 0.61256 0.00731 0.00296 0.02256 Hungarian 0.46495 0.46672 0.46873 0.48797 0.00097 0.00114 0.01165 Italian 0.62033 0.62612 0.62767 0.64356 0.00307 0.00082 0.00883 Turkish 0.60143 0.60215 0.60660 0.63560 0.00060 0.00329 0.02139 Bulgarian 0.61415 0.62257 0.62433 0.64497 0.00456 0.00086 0.01233 Danish 0.67350 0.67904 0.68119 0.69436 0.00291 0.00108 0.00916 Dutch 0.69201 0.70600 0.71105 0.74008 0.00709 0.00251 0.01862 German 0.54581 0.54755 0.55080 0.58182 0.00104 0.00208 0.02033 Japanese 0.60515 0.60515 0.60515 0.60654 0.00000 0.00000 0.00115 Portuguese 0.58880 0.60063 0.60185 0.61780 0.00651 0.00067 0.00867 Slovene 0.56155 0.56860 0.57135 0.60373 0.00396 0.00153 0.01979 Spanish 0.58247 0.59119 0.59277 0.61273 0.00487 0.00089 0.01197 Swedish 0.57543 0.58636 0.58933 0.61104 0.00585 0.00153 0.01383 Average 0.59009 0.59656 0.59954 0.62416 0.00355 0.00176 0.01513
Table 1: Average value of the different bounds and the loss, and of the relative differences from each bound to the loss, on CoNLL-XI (first block) and CoNLL-X (second block) datasets during 100,000 transitions. For each language, we show in boldface the average value and relative difference of the bound that is closer to the loss.
|U(c, tG)|+npc(A∪I(c, tG)), where “pc” stands for problematic cycles, is a closer upper bound to the loss `(c) and the following holds:
|U(c, tG)| ≤ `(c) ≤ |U(c, tG)|+npc(A∪I(c, tG))
≤ |U(c, tG)|+ nc(A ∪ I(c, tG))
As mentioned before, unlike the monotonic approach, a node can have a different head in A than in I(c, tG) and, as a consequence, the resulting graph A ∪ I(c, tG) has maximum in-degree 2 rather than 1, and there can be overlapping cycles. Therefore, the computation of the non-monotonic terms nc(A ∪ I(c, tG)) and npc(A ∪ I(c, tG)) requires an algorithm such as the one by Johnson (1975) to find all elementary cycles in a directed graph. This runs in O((n + e)(c + 1)), where n is the number of vertices, e is the number of edges and c is the number of elementary cycles in the graph. This implies that the calculation of the two non-monotonic upper bounds is less efficient than the linear loss computation in the monotonic scenario. However, a non-monotonic algorithm that uses the lower bound as loss expression is the fastest option (even faster than the monotonic approach) as the oracle does not need to compute cycles at all, speeding up the training process.
Algorithm 2 shows the non-monotonic variant of Algorithm 1, where COUNTRELEVANTCYCLES is a function that counts the number of cycles or problematic cycles in the given graph,
Algorithm 2 Computation of the approximate loss of a non-monotonic configuration. 1: function LOSS(c = 〈λ1|i, λ2, j|B,A〉, tG) 2: U ← ∅ . Variable U is for U(c, tG) 3: for each x→ y ∈ (tG \A) do 4: left ← min(x, y) 5: right ← max(x, y) 6: if j > right ∨ 7: (j = right ∧ i < left) then 8: U ← u ∪ {x→ y} 9: I ← tG \U . Variable I is for I(c, tG) 10: return |U |+ COUNTRELEVANTCYCLES(A ∪ I )
depending on the upper bound implemented, and will return 0 in case we use the lower bound.
5 Evaluation of the loss bounds
To determine how close the lower bound |U(c, tG)| and the upper bounds |U(c, tG)| + npc(A∪I(c, tG)) and |U(c, tG)|+nc(A∪I(c, tG)) are to the actual loss in practical scenarios, we use exhaustive search to calculate the real loss of a given configuration, to then compare it with the bounds. This is feasible because the lower and upper bounds allow us to prune the search space: if an upper and a lower bound coincide for a configuration we already know the loss and need not keep searching, and if we can branch to two configurations such that the lower bound of one is greater or equal than an upper bound of the other, we can discard the former as it will never lead to smaller loss than the latter. Therefore, this ex-
7
601
602
603
604
605
606
607
608
609
610
611
612
613
614
615
616
617
618
619
620
621
622
623
624
625
626
627
628
629
630
631
632
633
634
635
636
637
638
639
640
641
642
643
644
645
646
647
648
649
650
651
652
653
654
655
656
657
658
659
660
661
662
663
664
665
666
667
668
669
670
671
672
673
674
675
676
677
678
679
680
681
682
683
684
685
686
687
688
689
690
691
692
693
694
695
696
697
698
699
Unigrams L0w; L0p; L0wp; L0l; L0hw; L0hp; L0hl; L0l′w; L0l′p; L0l′ l; L0r′w; L0r′p; L0r′ l; L0h2w; L0h2p; L0h2l; L0lw; L0lp; L0ll; L0rw; L0rp; L0rl; L0wd; L0pd; L0wvr ; L0pvr ; L0wvl; L0pvl; L0wsl; L0psl; L0wsr ; L0psr ; L1w; L1p; L1wp; R0w; R0p; R0wp; R0hw; R0hp;R0hl; R0h2w; R0h2p; R0l′w; R0l′p; R0l′ l; R0lw; R0lp; R0ll; R0wd; R0pd; R0wvl; R0pvl; R0wsl; R0psl; R1w; R1p; R1wp; R2w; R2p; R2wp; CLw; CLp; CLwp; CRw; CRp; CRwp; Pairs L0wp+R0wp; L0wp+R0w; L0w+R0wp; L0wp+R0p; L0p+R0wp; L0w+R0w; L0p+R0p; R0p+R1p; L0w+R0wd; L0p+R0pd; Triples R0p+R1p+R2p; L0p+R0p+R1p; L0hp+L0p+R0p; L0p+L0l′p+R0p; L0p+L0r′p+R0p; L0p+R0p+R0l′p; L0p+L0l′p+L0lp; L0p+L0r′p+L0rp; L0p+L0hp+L0h2p; R0p+R0l′p+R0lp;
Table 2: Feature templates. L0 and R0 denote the left and right focus words; L1, L2, . . . are the words to the left of L0 and R1, R2, . . . those to the right of R0. Xih means the head of Xi, Xih2 the grandparent, Xil and Xil′ the farthest and closest left dependents, and Xir and Xir′ the farthest and closest right dependents, respectively. CL and CR are the first and last words between L0 andR0 whose head is not in the interval [L0, R0]. Finally, w stands for word form; p for PoS tag; l for dependency label; d is the distance between L0 and R0; vl, vr are the left/right valencies (number of left/right dependents); and sl, sr the left/right label sets (dependency labels of left/right dependents).
haustive search with pruning guarantees to find the exact loss.
Due to the time complexity of this process, we undertake the analysis of only the first 100,000 transitions on each dataset of the nineteen languages available from CoNLL-X and CoNLL-XI shared tasks (Buchholz and Marsi, 2006; Nivre et al., 2007). In Table 1, we present the average values for the lower bound, both upper bounds and the loss, as well as the relative differences from each bound to the real loss. After those experiments, we conclude that the lower and the closer upper bounds are a tight approximation of the loss, with both bounds incurring relative errors below 0.8% in all datasets. If we compare them, the real loss is closer to the upper bound |U(c, tG)| + npc(A ∪ I(c, tG)) in the majority of datasets (12 out of 18 languages, excluding Japanese where both bounds were exactly equal to the real loss in the whole sample of configurations). This means that the term npc(A∪I(c, tG)) provides a close approximation of the gold arcs missed by the presence of cycles in A. Regarding the upper bound |U(c, tG)|+nc(A∪I(c, tG)),
it presents a more variable relative error, ranging from 0.1% to 4.0%.
Thus, although we do not know an algorithm to obtain the exact loss which is fast enough to be practical, any of the three studied loss bounds can be used to obtain a feasible approximate dynamic oracle with full non-monotonicity.
6 Experiments
To prove the usefulness of our approach, we implement the static, dynamic monotonic and nonmonotonic oracles for the non-projective Covington algorithm and compare their accuracies on nine datasets from the CoNLL-X shared task (Buchholz and Marsi, 2006) and all datasets from the CoNLL-XI shared task (Nivre et al., 2007). For the non-monotonic algorithm, we test the three different loss expressions defined in the previous section. We train an averaged perceptron model for 15 iterations and use the same feature templates for all languages, which are listed in detail in Table 2.
The accuracies obtained by the non-projective Covington parser with the three available oracles are presented in Table 3. For the non-monotonic dynamic oracle, three variants are shown, one for each loss expression implemented. As we can see, the novel non-monotonic oracle improves over the accuracy of the monotonic version on 14 out of 19 languages (0.32 in UAS on average) with the best loss calculation |U(c, tG)|+nc(A∪I(c, tG)), where 6 of these improvements are statistically significant at the .05 level (Yeh, 2000). The other two loss calculation methods also achieve good results, outperforming the monotonic algorithm on 12 out of 19 datasets tested.
The loss expression |U(c, tG)| + nc(A ∪ I(c, tG)) obtains greater accuracy on average than the other two loss expressions, including the more adjusted upper bound that is provably closer to the real loss. This could be explained by the fact that identifying problematic cycles is a difficult task to learn for the parser, and for this reason a more straightforward approach, which tries to avoid all kinds of cycles (regardless of whether they will cost gold arcs or not), can perform better. This also leads us to hypothesize that, even if it were feasible to build an oracle with the exact loss, it would not provide practical improvements over these approximate oracles; as it appears difficult for a statistical model to learn the situations where repla-
8
701
702
703
704
705
706
707
708
709
710
711
712
713
714
715
716
717
718
719
720
721
722
723
724
725
726
727
728
729
730
731
732
733
734
735
736
737
738
739
740
741
742
743
744
745
746
747
748
749
750
751
752
753
754
755
756
757
758
759
760
761
762
763
764
765
766
767
768
769
770
771
772
773
774
775
776
777
778
779
780
781
782
783
784
785
786
787
788
789
790
791
792
793
794
795
796
797
798
799
dynamic dynamic non-monotonic static monotonic lower pc upper upper
Language UAS LAS UAS LAS UAS LAS UAS LAS UAS LAS Arabic 80.67 66.51 82.76∗ 68.48∗ 83.29∗ 69.14∗ 83.18∗ 69.05∗ 83.40† 69.29† Basque 76.55 66.05 77.49† 67.31† 74.61 65.31 74.69 65.18 74.27 64.78 Catalan 90.52 85.09 91.37∗ 85.98∗ 90.51 85.35 90.40 85.30 90.44 85.35 Chinese 84.93 80.80 85.82 82.15 86.55∗ 82.53∗ 86.29∗ 82.27∗ 86.60∗ 82.51∗ Czech 78.49 61.77 80.21∗ 63.52∗ 81.32† 64.89† 81.33† 64.81† 81.49† 65.18† English 85.35 84.29 87.47∗ 86.55∗ 88.44† 87.37† 88.23† 87.22† 88.50† 87.55† Greek 79.47 69.35 80.76 70.43 80.90 70.46 80.84 70.34 81.02∗ 70.49∗ Hungarian 77.65 68.32 78.84∗ 70.16∗ 78.67∗ 69.83∗ 78.47∗ 69.66∗ 78.65∗ 69.74∗ Italian 84.06 79.79 84.30 80.17 84.38 80.30 84.64 80.52 84.47 80.32 Turkish 81.28 70.97 81.14 71.38 80.65 71.15 80.80 71.29 80.60 71.07 Bulgarian 89.13 85.30 90.45∗ 86.86∗ 91.36† 87.88† 91.33† 87.89† 91.73† 88.26† Danish 86.00 81.49 86.91∗ 82.75∗ 86.83∗ 82.63∗ 86.89∗ 82.74∗ 86.94∗ 82.68∗ Dutch 81.54 78.46 82.07 79.26 82.78∗ 79.64∗ 82.80∗ 79.68∗ 83.02† 79.92† German 86.97 83.91 87.95∗ 85.17∗ 87.31 84.37 87.18 84.22 87.48 84.54 Japanese 93.63 92.20 93.67 92.33 94.02 92.68 94.02 92.68 93.97 92.66 Portuguese 86.55 82.61 87.45∗ 83.62∗ 87.17∗ 83.47∗ 87.12∗ 83.45∗ 87.40∗ 83.71∗ Slovene 76.76 63.53 77.86 64.43 80.39† 67.04† 80.56† 67.10† 80.47† 67.10† Spanish 79.20 76.00 80.12∗ 77.24∗ 81.36∗ 78.30∗ 81.12∗ 77.99∗ 81.33∗ 78.16∗ Swedish 87.43 81.77 88.05∗ 82.77∗ 88.20∗ 83.02∗ 88.09∗ 82.87∗ 88.36∗ 83.16∗ Average 83.48 76.75 84.46 77.92 84.67 78.18 84.63 78.12 84.74 78.24
Table 3: Parsing accuracy (UAS and LAS, including punctuation) of the Covington non-projective parser with static, and dynamic monotonic and non-monotonic oracles on CoNLL-XI (first block) and CoNLLX (second block) datasets. For the dynamic non-monotonic oracle, we show the performance with the three loss expressions, where lower stands for the lower bound |U(c, tG)|, pc upper for the upper bound |U(c, tG)| + npc(A ∪ I(c, tG)), and upper for the upper bound |U(c, tG)| + nc(A ∪ I(c, tG)). For each language, we run five experiments with the same setup but different seeds and report the averaged accuracy. Best results for each language are shown in boldface. Statistically significant improvements (α = .05) of both dynamic oracles are marked with ∗ if they are only over the static oracle, and with † if they are over the opposite dynamic oracle too.
cing a wrong arc with another indirectly helps due to breaking prospective cycles.
It is also worth mentioning that the nonmonotonic dynamic oracle with the best loss expression accomplishes an average improvement over the static version (1.26 UAS) greater than that obtained by the monotonic oracle (0.98 UAS), resulting in 13 statistically significant improvements achieved by the non-monotonic variant over the static oracle in comparison to the 12 obtained by the monotonic system. Finally, note that, despite this remarkable performance, the non-monotonic version (regardless of the loss expression implemented) has an inexplicable drop in accuracy in Basque in comparison to the other two oracles.
7 Conclusion
We presented a novel, fully non-monotonic variant of the well-known non-projective Covington parser, trained with a dynamic oracle. Due to the unpredictability of a non-monotonic scenario, the real loss of each configuration cannot be computed. To overcome this, we proposed three different loss expressions that closely bound the loss and enable us to implement a practical non-monotonic
dynamic oracle.
On average, our non-monotonic algorithm obtains better perfomance than the monotonic version, regardless of the loss calculation used. In particular, one of the loss expressions developed proved very promising by providing the best average accuracy, in spite of being the farthest approximation from the actual loss. On the other hand, the proposed lower bound makes the nonmonotonic system the fastest one among all dynamic oracles developed for the non-projective Covington algorithm.
To our knowledge, this is the first implementation of non-monotonicity for a nonprojective parsing algorithm, and the first approximate dynamic oracle that uses close, efficientlycomputable approximations of the loss, showing this to be a feasible alternative when it is not practical to compute the actual loss.
While we used a perceptron classifier for our experiments, our oracle could also be used in neuralnetwork implementations of greedy transitionbased parsing (Chen and Manning, 2014; Dyer et al., 2015), providing an interesting avenue for future work.
9
801
802
803
804
805
806
807
808
809
810
811
812
813
814
815
816
817
818
819
820
821
822
823
824
825
826
827
828
829
830
831
832
833
834
835
836
837
838
839
840
841
842
843
844
845
846
847
848
849
850
851
852
853
854
855
856
857
858
859
860
861
862
863
864
865
866
867
868
869
870
871
872
873
874
875
876
877
878
879
880
881
882
883
884
885
886
887
888
889
890
891
892
893
894
895
896
897
898
899

- Strengths:  The paper makes several novel contributions to (transition-based) dependency parsing by extending the notion of non-monotonic transition systems and dynamic oracles to unrestricted non-projective dependency parsing. The theoretical and algorithmic analysis is clear and insightful, and the paper is admirably clear.  - Weaknesses:  Given that the main motivation for using Covington's algorithm is to be able to recover non-projective arcs, an empirical error analysis focusing on non-projective structures would have further strengthened the paper. And even though the main contributions of the paper are on the theoretical side, it would have been relevant to include a comparison to the state of the art on the CoNLL data sets and not only to the monotonic baseline version of the same parser.  - General Discussion:  The paper extends the transition-based formulation of Covington's dependency parsing algorithm (for unrestricted non-projective structures) by allowing non-monotonicity in the sense that later transitions can change structure built by earlier transitions. In addition, it shows how approximate dynamic oracles can be formulated for the new system. Finally, it shows experimentally that the oracles provide a tight approximation and that the non-monotonic system leads to improved parsing accuracy over its monotonic counterpart for the majority of the languages included in the study.  The theoretical contributions are in my view significant enough to merit publication, but I also think the paper could be strengthened on the empirical side. In particular, it would be relevant to investigate, in an error analysis, whether the non-monotonic system improves accuracy specifically on non-projective structures. Such an analysis can be motivated on two grounds: (i) the ability to recover non-projective structures is the main motivation for using Covington's algorithm in the first place; (ii) non-projective structures often involved long-distance dependencies that are hard to predict for a greedy transition-based parser, so it is plausible that the new system would improve the situation.   Another point worth discussion is how the empirical results relate to the state of the art in light of recent improvements thanks to word embeddings and neural network techniques. For example, the non-monotonicity is claimed to mitigate the error propagation typical of classical greedy transition-based parsers. But another way of mitigating this problem is to use recurrent neural networks as preprocessors to the parser in order to capture more of the global sentence context in word representations. Are these two techniques competing or complementary? A full investigation of these issues is clearly outside the scope of the paper, but some discussion would be highly relevant.  Specific questions:  Why were only 9 out of the 13 data sets from the CoNLL-X shared task used? I am sure there is a legitimate reason and stating it explicitly may prevent readers from becoming suspicious.   Do you have any hypothesis about why accuracy decreases for Basque with the non-monotonic system? Similar (but weaker) trends can be seen also for Turkish, Catalan, Hungarian and (perhaps) German.  How do your results compare to the state of the art on these data sets? This is relevant for contextualising your results and allowing readers to estimate the significance of your improvements.  Author response:  I am satisfied with the author's response and see no reason to change my previous review.


Recurrent Neural Networks are showing much promise in many sub-areas of natural language processing, ranging from document classification to machine translation to automatic question answering. Despite their early promise, many recurrent models have to read the whole text sequentially, making it difficult to apply them to long documents. For example, it is difficult to use a recurrent network to read a book and answer questions about it. In this paper, we present an approach of reading text non-sequentially, thereby skipping irrelevant information if needed. The underlying model is a recurrent network that learns how far to jump after reading a few words of the input text. We employ a standard policy gradient method to train the model to make discrete jumping decisions. In our benchmarks on four different tasks, including number prediction, sentiment analysis, news article classification and automatic Q&A, our proposed model, a modified non-sequential LSTM, is up to 6 times faster than the standard sequential LSTM, while maintaining the same or even better accuracy.
1 Introduction
The last few years have seen much success of applying neural networks to many important applications in natural language processing, e.g., partof-speech tagging, chunking, named entity recognition (Collobert et al., 2011), sentiment analysis (Socher et al., 2011, 2013), document classification (Kim, 2014; Le and Mikolov, 2014; Zhang et al., 2015; Dai and Le, 2015), machine translation (Kalchbrenner and Blunsom, 2013; Sutskever
et al., 2014; Bahdanau et al., 2014; Sennrich et al., 2015; Wu et al., 2016), conversational/dialogue modeling (Sordoni et al., 2015; Vinyals and Le, 2015; Shang et al., 2015), document summarization (Rush et al., 2015; Nallapati et al., 2016), parsing (Andor et al., 2016) and automatic question answering (Q&A) (Weston et al., 2015; Hermann et al., 2015; Wang and Jiang, 2016; Wang et al., 2016; Trischler et al., 2016; Lee et al., 2016; Seo et al., 2016; Xiong et al., 2016). An important trait of all these models is that they read all the text available to them. While it is essential for certain applications, such as machine translation, this trait also makes it difficult to apply these models to scenarios that have long input text, such as document classification or automatic Q&A.
In this paper, we consider the problem of understanding long documents with partial reading, and propose a modification to the basic neural architectures that allows them to read input text nonsequentially. The main benefit of this approach is faster inference because it skips irrelevant information. An unexpected benefit of this approach is that it also helps the models generalize better.
In our approach, the model is a recurrent network, which learns to predict the number of jumping steps after it reads one or several input tokens. Such a discrete model is therefore not fully differentiable, but it can be trained by a standard policy gradient algorithm, where the reward can be the accuracy or its proxy during training.
In our experiments, we use the basic LSTM recurrent networks (Hochreiter and Schmidhuber, 1997) as the base model and benchmark the proposed algorithm on a range of document classification or reading comprehension tasks, such as Rotten Tomatoes (Pang and Lee, 2005), IMDB (Maas et al., 2011), AG News (Zhang et al., 2015) and Children’s Book Test (Hill et al., 2015). We find that the proposed approach of non-sequential read-
2
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
Figure 1: An example of the proposed model processing a text document. Here, the maximum size of jump is K = 5, the number of tokens read before a jump is R = 2 and the number of jumps allowed is N = 10. The green softmax are for jumping predictions. The processing stops if a) the jumping predicts a 0 or b) the jump times exceed N or c) the network processed the last token. We only show case a) in this figure.
ing speeds up its sequential counterpart by two to six times. Surprisingly, we also observe our model beats the standard LSTM in terms of accuracy.
In summary, the main contribution of our work is to design an architecture that learns to read text non-sequentially and show that it is both faster and more accurate in practical applications of text processing. Our model is simple and flexible enough that we anticipate it be able to incorporate to recurrent nets with more sophisticated structures to achieve even better performance in the future.
2 Methodology
In this section, we introduce the proposed model named LSTM-Jump. We will first describe its main structure followed by the difficulty of estimating part of the model parameters because of non-differentiability. To address this issue, we appeal to a reinforcement learning formulation and adopt a policy gradient method.
2.1 Model Overview
The main architecture of the proposed model is shown in Figure 1, which is based on an LSTM recurrent neural network. Before training, the number of jumps N allowed, the number of tokens read between every two jumps R and the maximum size of jumping K are chosen ahead of time. While K is a fixed parameter of the model, N and R are hyperparameters that can vary between training and testing. We summarize those notations in Table 2, such that one can refer to when reading the experiment results in Section 3. Also, throughout the paper, we would use d1:p to denote a sequence d1, d2, ..., dp.
In the following, we describe in detail how the model operates when processing text. Given a
training example x1:T , the recurrent network will read the embedding of the first R tokens x1:R and output the hidden state. Then this state is used to compute the jumping softmax that determines a distribution over the jumping steps between 1 and K. The model then samples from this distribution a jumping step, which is used to decide the next token to be read into the model. Let κ be the sampled value, then the next starting token is xR+κ. Such process continues until either
a) the jump softmax samples a 0; or b) the number of jumps exceeds N ; or c) the model reaches the last token xT .
After stopping, the latest hidden state is further used for predicting desired targets. How to leverage the hidden state depends on the specifics of the task at hand. For example, in classification problems of Section 3.1, 3.2 and 3.3, it is directly applied to produce a softmax for classification, while in automatic Q&A problem of Section 3.4, it is used to compute the correlation with the candidate answers in order to select the best one. Figure 1 gives an example withK = 5, R = 2 andN = 10 terminating on condition a).
2.2 Training with REINFORCE
Our goal for training is to estimate the LSTM and possibly word embedding parameters θm, together with the jumping action parameters θa. Once obtained, they can be used for inference.
The estimation of θm is straightforward in the tasks that can be reduced as classification problems (which is exactly what our experiments cover), as the cross entropy objective J1(θm) is differentiable over θm that we can directly apply backpropagation to minimize.
However, the nature of discrete jumping deci-
3
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
sions made at every step makes it difficult to estimate θa, as cross entropy is no longer differentiable over θa. Therefore, we are tempted to formulate it as a reinforcement learning problem and apply policy gradient method to train the model. Specifically, we need to maximize a reward function over θa which can be constructed as follows.
Let j1:N be the jumping actions sequence during the training with an example x1:T . Suppose hi is a hidden state of the LSTM right before the i-th jump ji,1 then it is a function of j1:i−1 and thus can be denoted as hi(j1:i−1). Now the jump is attained by sampling from the multinomial distribution p(ji|hi(j1:i−1); θa), which is determined by the jump softmax. We can receive a reward R after processing x1:T under the current jumping strategy.2 The reward should be positive if the output is favorable or non-positive otherwise. In our experiments, we choose
R = { 1 if prediction correct; −1 otherwise.
Then the objective function of θa we want to maximize is the expected reward under the distribution defined by the current jumping policy, i.e.,
J2(θa) = Ep(j1:N ;θa)[R]. (1)
where p(j1:N ; θa) = ∏ i p(j1:i|hi(j1:i−1); θa).
Optimizing this objective numerically requires computing its gradient, whose exact value is intractable to obtain as the expectation is over high dimensional interaction sequences. By running S examples, an approximated gradient can be computed by the following REINFORCE algorithm (Williams, 1992): ∇θaJ2(θa) = N∑ i=1 Ep(j1:N ;θa)[∇θa log p(j1:i|hi; θa)R]
≈ 1 S S∑ s=1 N∑ i=1 [∇θa log p(js1:i|hsi ; θa)Rs]
where the superscript s denotes a quantity belonging to the s-th example. Now the term ∇θa log p(j1:i|hi; θa) can be computed by standard backpropagation.
1The i-th jumping step is usually not xi. 2In the general case, one may receive (discounted) intermediate rewards after each jump. But in our case, we only consider final reward. It is equivalent to a special case that all intermediate rewards are identical and without discount.
Although the above estimation of ∇θaJ2(θa) is unbiased, it may have very high variance. One widely used remedy to reduce the variance is to subtract a baseline value bsi from the reward R
s, such that the approximated gradient becomes
∇θaJ2(θa) ≈ 1
S S∑ s=1 N∑ i=1 [∇θa log p(js1:i|hsi ; θ)(Rs−bsi )]
It is shown (Williams, 1992; Zaremba and Sutskever, 2015) that any number bsi will yield an unbiased estimation. Here, we adopt the strategy of Mnih et al. (2014) that bsi = wbh s i + cb and the parameter θb = {wb, cb} is learned by minimizing (Rs− bsi )2. Now the final objective to minimize is J(θm, θa, θb) = J1(θm)−J2(θa)+ S∑ s=1 N∑ i=1 (Rs−bsi )2,
which is fully differentiable and can be solved by standard backpropagation.
2.3 Inference
During inference, we can either use sampling or greedy evaluation by selecting the most probable jumping step suggested by the jump softmax and follows that path. In the our experiments, we will adopt the sampling scheme.
3 Experimental Results
In this section, we present our empirical studies to understand the efficiency of the proposed model in reading text. The tasks under experimentation are: synthetic number prediction, sentiment analysis, news topic classification and automatic question answering. Those are representative tasks in text reading involving different sizes of datasets and various levels of text processing, from character to word and to sentence. Table 1 summarizes the statistics of the dataset in our experiments.
To exclude the potential impact of advanced models, we restrict our comparison between the vanilla LSTM (Hochreiter and Schmidhuber, 1997) and our model, which is referred to as LSTM-Jump. In a nutshell, we show that, while achieving the same or even better testing accuracy, our model is up to 6 times and 66 times faster than the baseline LSTM model in real and synthetic datasets, respectively, as we are able to selectively skip a large fraction of text.
In fact, the proposed model can be readily extended to other recurrent neural networks with sophisticated mechanisms such as attention and/or
4
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
Task Dataset Level Vocab AvgLen #train #valid #test #class Number Prediction synthetic word 100 100 words 1M 10K 10K 100 Sentiment Analysis Rotten Tomatoes word 18,764 22 words 8,835 1,079 1,030 2 Sentiment Analysis IMDB word 112,540 241 words 21,143 3,857 25,000 2 News Classification AG character 70 200 characters 101,851 18,149 7,600 4
Q/A Children Book Test-NE sentence 53,063 20 sentences 108,719 2,000 2,500 10 Q/A Children Book Test-CN sentence 53,185 20 sentences 120,769 2,000 2,500 10
Table 1: Tasks and datasets statistics.
hierarchical structure to achieve higher accuracy than those presented below. However, this is orthogonal to the main focus of this work and would be left as an interesting future work.
General Experiment Settings We use the Adam optimizer (Kingma and Ba, 2014) with a learning rate of 0.001 in all experiments. We also apply gradient clipping to all the trainable variables with the threshold of 1.0. The dropout rate between the LSTM layers is 0.2 and the embedding dropout rate is 0.1. We repeat the notations N,K,R defined previously in Table 2, such that reader can easily refer to when looking at in Tables 4,5,6,7. While K is fixed during both training and testing, we would fix R and N at training but vary their values during test to see how the change of parameters affects the result. Besides, the reported test time is measured by running one pass of the whole test set and the speedup is over the base LSTM model. The code is written with TensorFlow.3
Notation Meaning N number of jumps allowed K maximum size of jumping R number of tokens read before a jump
Table 2: Notations referred to in experiments.
3.1 Number Prediction with a Synthetic Dataset
First of all, we carry out the sanity check of whether LSTM-Jump is indeed able to learn how to jump if a very clear jumping signal is given in the text. The input of the task is a sequence of L positive integers x0:T−1 and the output is simply xx0 . That is, the output is chosen from the input sequence, with index determined by x0 . Here are
3https://www.tensorflow.org/
two examples to illustrate this idea:
input1 : 4, 5, 1, 7, 6, 2. output1 : 6
input2 : 2, 4, 9, 4, 5, 6. output2 : 9
One can see that a0 is essentially the indicator of how many steps the reading should jump to get the exact output and obviously, the remaining number of the sequence are useless. After reading the first token, a “smart” network should be able to learn from the training examples to jump to the output position, skipping the rest.
We generate 1 million training and 10,000 validation examples with the rule above, each with sequence length T = 100. We also impose 1 ≤ x0 < T to ensure the index is valid. We find that directly training the LSTM-Jump with full sequence is unlikely to converge, therefore, we adopt a curriculum training scheme. More specifically, we generate sequences with lengths {10, 20, 30, 40, 50, 60, 70, 80, 90, 100} and train the model starting from the shortest. Whenever the training accuracy reaches a threshold, we shift to longer sequences. The training stops when the validation accuracy is larger than 98%. We also train an LSTM with the same curriculum training scheme to conduct the prediction. All the networks are single layered, with hidden size 512, embedding size 32 and batch size 100. During testing, we generate sequences of lengths 10, 100 and 1000 with the same rule, each having 10,000 examples. As the training size is huge, we do not need to worry about overfitting so dropout is not applied. In fact, we find that the training, validation and testing accuracies are almost the same.
The results of LSTM and our method, LSTMJump, are shown in Table 3. The first observation is that LSTM-Jump is faster than LSTM; the longer the sequence is, the more significant speedup LSTM-Jump can gain. This is because LSTMJump is aware of the jumping signal at the first token and hence can directly jump to the output position to make prediction, while LSTM is agnostic to the signal and has to read the whole se-
5
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
458
459
460
461
462
463
464
465
466
467
468
469
470
471
472
473
474
475
476
477
478
479
480
481
482
483
491
492
493
494
495
496
497
498
499
Seq length LSTM-Jump LSTM Speedup Test accuracy
10 98% 96% n/a 100 98% 96% n/a
1000 90% 80% n/a Test time (Avg tokens read)
10 13.5s (2.1) 18.9s (10) 1.40x 100 13.9s (2.2) 120.4s (100) 8.66x
1000 18.9s (3.0) 1250s (1000) 66.14x
Table 3: Testing accuracy and time of synthetic number prediction problem. The jumping level is word.
quence. Thanks to this fact, the reading speed of LSTM-Jump is hardly affected by the length of sequence, but that of LSTM is linear with respect to length. Besides, LSTM-Jump also outperforms LSTM in terms of test accuracy under all cases. This is not surprising either, as LSTM has to read a large amount of tokens that are potentially not helpful and could interfere with the prediction. In summary, the results indicate LSTM-Jump is able to learn to “jump” when the signal is very clear.
3.2 Word Level Sentiment Analysis with Rotten Tomatoes and IMDB datasets
As LSTM-Jump has shown great speedups in the synthetic dataset, we would like to understand whether it could carry this benefit to real-world data, where “jumping” signal is not explicit. So in this section, we conduct sentiment analysis on two movie review datasets, both containing equal numbers of positive and negative reviews.
The first dataset is Rotten Tomatoes, which contains 10,662 documents. Since there is not a standard split, we randomly select around 80% for training, 10% for validation, and 10% for testing. The average and maximum lengths of the reviews are 22 and 56 words respectively, and we pad each of them to 60. We choose the pre-trained word2vec embeddings4 (Mikolov et al., 2013) as our fixed word embedding that we do not update this matrix during training. Both LSTM-Jump and LSTM contain 2 layers, 256 hidden units and the batch size is 100. As the amount of training data is small, we slightly augment the data by sampling a continuous 50-word sequence in each padded reviews as one training sample. During training, we enforce LSTM-Jump to read 8 tokens before a jump (R = 8), and the maximum skipping to-
4https://code.google.com/archive/p/ word2vec/
kens per jump is 10 (K = 10), while the number of jumps allowed is 3 (N = 3).
The testing result is reported in Table 4. In a nutshell, LSTM-Jump is always faster than LSTM under different combinations of R and N . At the same time, the accuracy is on par with that of LSTM. In particular, the combination of (R,N) = (7, 4) even achieves slightly better accuracy than LSTM while having a 1.5x speedup.
Model (R,N) Accuracy Time Speedup
LSTM-Jump (9, 2) 0.783 6.3s 1.98x (8, 3) 0.789 7.3s 1.71x (7, 4) 0.793 8.1s 1.54x
LSTM n/a 0.791 12.5s 1x
Table 4: Testing time and accuracy on the Rotten Tomatoes review classification dataset. The maximum size of jumping K is set to 10 for all the settings. The jumping level is word.
The second dataset of interest is IMDB (Maas et al., 2011),5 which contains 25,000 training and 25,000 testing movie reviews, where the average length of text is 240 words, much longer than that of Rotten Tomatoes. We randomly set aside about 15% of training data as validation set. Both LSTM-Jump and LSTM has one layer and 128 hidden units, and the batch size is 50. Again, we use pretrained word2vec embeddings as initialization but they are updated during training. We either pad a short sequence to 400 words or randomly select a 400-word segment from a long sequence as a training example. The number of tokens read before a jump is set to R = 20, maximum skipping tokens per jump is K = 40 and the maximum number of jumps is N = 5.
As Table 5 shows, the result exhibits a similar trend as found in Rotten Tomatoes that LSTM-
5http://ai.Stanford.edu/amaas/data/ sentiment/index.html
6
501
502
503
504
505
506
507
508
509
510
511
512
513
514
515
516
517
518
519
520
521
522
523
524
525
526
527
528
529
530
531
532
533
534
535
536
537
538
539
540
541
542
543
544
545
546
547
548
549
550
551
552
553
554
555
562
563
564
565
566
567
568
569
570
571
572
573
574
575
576
577
578
579
580
581
582
583
584
585
586
587
588
589
590
591
592
593
594
595
596
597
598
599
Jump is uniformly faster than LSTM under many settings. The various (R,N) combinations again display the trade-off between efficiency and accuracy. If one cares more about accuracy, then allowing LSTM-Jump to read and jump more times is a good choice. Otherwise, shrinking either one would bring a significant speedup though at the price of losing some accuracy. Nevertheless, the configuration with the highest accuracy still enjoys a 1.6x speedup compared to LSTM. With a slight loss of accuracy, LSTM-Jump can be 2.5x faster .
3.3 Character Level News Article Classification with AG dataset
We now present results on testing the character level jumping with a news article classification problem. The data contains four classes of topics (World, Sports, Business, Sci/Tech) from the AG’s news corpus,6 a collection of more than 1 million news articles. The data we use is the subset constructed by Zhang et al. (2015) for classification with character-level convolutional networks. There are 30,000 training and 1900 testing examples for each class respectively, where 15% of training are set aside as validation. The nonspace alphabet under use are:
abcdefghijklmnopqrstuvwxyz0123456 789-,;.!?:/\|_@#$%&* +-=<>()[]{}
Since the vocabulary size is small, we choose 16 as the embedding size. The initialized entries of the embedding matrix are drawn from a uniform distribution in [−0.25, 0.25], which are progressively updated during training. Both LSTM-Jump and LSTM have 1 layer and 64 hidden units and the batch sizes are 20 and 100 respectively. The training sequence is again of length 400 that it is either padded from a short sequence or sampled from a long one. The number of characters read before a jump is R = 30, the maximum span per jump is K = 40 and N = 5 jumps are allowed at training.
The result is summarized in Table 6. Not surprisingly, LSTM-Jump outperforms LSTM in terms of both efficiency and accuracy, although the advantage in speedup is not as significant as that in the previous tasks. This is mainly due to the fact that the embedding size and hidden are both much smaller than those used previously, and accordingly the processing of a token is much faster. In that case, other computation overhead such as
6http://www.di.unipi.it/˜gulli/AG_ corpus_of_news_articles.html
calculating and sampling from the jump softmax might become a dominating factor of efficiency. By this cross-tasks comparison, we can see that the larger the recurrent neural network and the embedding are, the more speedup LSTM-Jump can gain, which is also confirmed by the task below.
3.4 Sentence Level Automatic Question Answering with Children’s Book Test dataset
The last task is automatic question answering, in which we aim to test the sentence level skimming of LSTM-Jump. We benchmark on the data set Children’s Book Test (CBT) (Hill et al., 2015).7 In each document, there are 20 contiguous sentences (context) extracted from a children’s book followed by a query sentence. A word of the query is deleted and the task is to select the best fit for this position from 10 candidates. Originally, there are 4 types of tasks according to the part of speech of the missing word, from which, we choose the most difficult two, i.e., the name entity (NE) and common noun (CN) as our focus, since simple language models can already achieve human-level performance for the rest two types .
The models, LSTM or LSTM-Jump, firstly read the whole query, then the context sentences and finally output the predicted word. While LSTM reads everything, our jumping model would decide how many context sentences should skip after reading one sentence. Whenever a model finishes reading, the context and query are encoded in its hidden state ho, and the best answer from the candidate words has the same index that maximizes the following:
softmax(CWho) ∈ R10,
7http://www.thespermwhale.com/ jaseweston/babi/CBTest.tgz
7
601
602
603
604
605
606
607
608
609
610
611
612
613
614
615
616
617
618
619
620
621
622
623
624
625
626
627
628
629
630
631
632
633
634
635
636
637
638
639
640
641
642
643
644
645
646
647
648
649
650
651
652
653
654
655
656
657
658
659
660
661
662
663
664
665
666
667
668
669
670
671
672
673
674
675
676
677
678
679
680
681
682
683
684
685
686
687
688
689
690
691
692
693
694
695
696
697
698
699
where C ∈ R10×d is the word embedding matrix of the 10 candidates and W ∈ Rd×hidden size is a trainable weight variable. Using such bilinear form to select answer basically follows the idea of Chen et al. (2016), as it is shown to have good performance. The task is now distilled to a classification problem of 10 classes.
We either truncate or pad each context sentence, such that they all have length 20. The same preprocessing is applied to the query sentences except that the length is set 30. For both models, the number of layers is 2, the hidden units are 256 and the batch size is 32. Pretrained word2vec embeddings are again used and they are not adjusted during training. The maximum number of context sentences LSTM-Jump can skip per time is K = 5 while the number of total jumping is limited to N = 5. We let the model jump after reading every sentence, so R = 1 (20 words).
The result is reported in Table 7. The performance of LSTM-Jump is superior to LSTM in terms of both accuracy and efficiency under all settings in our experiments. In particular, the fastest LSTM-Jump configuration achieves a remarkable 6x speedup over LSTM, while also having respectively 1.4% and 4.4% higher accuracy in Children’s Book Test - Named Entity and Children’s Book Test - Common Noun.
Model (R,N) Accuracy Time Speedup Children’s Book Test - Named Entity
LSTM-Jump (1, 5) 0.468 40.9s 3.04x (1, 3) 0.464 30.3s 4.11x (1, 1) 0.452 19.9s 6.26x
LSTM n/a 0.438 124.5s n/a Children’s Book Test - Common Noun
LSTM-Jump (1, 5) 0.493 39.3s 3.09x (1, 3) 0.487 29.7s 4.09 (1, 1) 0.497 19.8s 6.14x
LSTM n/a 0.453 121.5s 1x
Table 7: Testing time and accuracy on the Children’s Book Test dataset. The maximum size of jumping K is set to 5 for all the settings. The jumping level is sentence.
The dominant performance of LSTM-Jump over LSTM might be interpreted as follows. After reading the query, both LSTM and LSTM-Jump know what the question is. However, LSTM still has to process the remaining 20 sentences and thus at the very end of the last sentence, the long dependency between the question and output might become weak that the prediction is hampered. On the contrary, the question can guide LSTM-Jump
on how to read selectively and stop early when the answer is clear. Therefore, when it comes to the output stage, the “memory” is both fresh and uncluttered that a more accurate answer is likely to be picked.
Below is an example of how the model reads a test context given a query (bold face sentences are those read by our model in the increasing order). XXXXX is the missing word we want to fill.
(a) Query: ‘XXXXX! (b) Context: 1. said Big Klaus, and he ran off at once to
Little Klaus. 2. ‘Where did you get so much money from?’ 3. ‘Oh, that was from my horse-skin. 4. I sold it yesterday evening.’ 5. ‘That ’s certainly a good price!’ 6. said Big Klaus; and running home in great
haste, he took an axe, knocked all his four 7. ‘Skins! 8. skins! 9. Who will buy skins?’
10. he cried through the streets. 11. All the shoemakers and tanners came running
to ask him what he wanted for them.’ 12. A bushel of money for each,’ said Big
Klaus. 13. ‘Are you mad?’ 14. they all exclaimed. 15. ‘Do you think we have money by the bushel?’ 16. ‘Skins! 17. skins! 18. Who will buy skins?’ 19. he cried again, and to all who asked him what
they cost, he answered,’ A bushel 20. ‘He is making game of us,’ they said; and the
shoemakers seized their yard measures and (c) Candidates: Klaus | Skins | game | haste |
head | home | horses | money | price| streets (d) Answer: Skins
The reading behavior might be interpreted as follows. The model tries to search for clues, and after reading sentence 8, it realizes that the most plausible answer is “Klaus” or “Skins”, as they both appear twice. “Skins” is more likely to be the answer as it is followed by a “!”. The model searches further to see if ”Klaus!” is mentioned somewhere, but it only finds “Klaus” without “!” for the third time. After the last attempt at sentence 14, it is confident about the answer and stops to output with “Skins”.
8
701
702
703
704
705
706
707
708
709
710
711
712
713
714
715
716
717
718
719
720
721
722
723
724
725
726
727
728
729
730
731
732
733
734
735
736
737
738
739
740
741
742
743
744
745
746
747
748
749
750
751
752
753
754
755
756
757
758
759
760
761
762
763
764
765
766
767
768
769
770
771
772
773
774
775
776
777
778
779
780
781
782
783
784
785
786
787
788
789
790
791
792
793
794
795
796
797
798
799
4 Related Work
Closely related to our work is the idea of learning visual attention with neural networks (Mnih et al., 2014; Ba et al., 2014; Sermanet et al., 2014), where a recurrent model is used to combine visual evidence at multiple fixations processed by a convolutional neural network. Similar to our approach, the model is trained end-to-end using the REINFORCE algorithm (Williams, 1992). However, a major difference between those work and ours is that we have to sample from discrete jumping distribution, while they can sample from continuous distribution such as Gaussian. The difference is mainly due to the inborn characteristics of text and image. In fact, as pointed out by Mnih et al. (2014), it was difficult to learn policies over more than 25 possible discrete locations.
This idea has recently been explored in the context of natural language processing applications, where the main goal is to filter irrelevant content using a small network (Choi et al., 2016). Perhaps most closely related to our work is the concurrent work on learning to reason with reinforcement learning (Shen et al., 2016). The key difference between our work and Shen et al. (2016) is that they focus on early stopping after multiple pass of data to ensure accuracy whereas our method focuses on selective reading with single pass to enable fast processing.
The concept of “hard” attention has also used successfully in the context of making neural network predictions more interpretable (Lei et al., 2016). The key difference between our work and Lei et al. (2016)’s method is that our method optimizes for faster inference, and is more dynamic in its jumping. Likewise is the difference between our approach and the “soft” attention approach by (Bahdanau et al., 2014).
Our method belongs to adaptive computation of neural networks, whose idea is recently explored by (Graves, 2016; Jernite et al., 2016), where different amount of computations are allocated dynamically per time step. The main difference between our method and Graves; Jernite et al.’s methods is that our method can set the amount of computation to be exactly zero for many steps, thereby achieving faster scanning over texts. Even though our method requires policy gradient methods to train, which is a disadvantage compared to (Graves, 2016; Jernite et al., 2016), we do not find training with policy gradient methods prob-
lematic in our experiments. At the high-level, our model can be viewed as a simplified trainable Turing machine, where the controller can move on the input tape. It is therefore related to the prior work on Neural Turing Machines (Graves et al., 2014) and especially its RL version (Zaremba and Sutskever, 2015). Compared to (Zaremba and Sutskever, 2015), the output tape in our method is more simple and reward signals in our problems are less sparse, which explains why our model is easy to train. It is worth noting that Zaremba and Sutskever report difficulty in using policy gradients to train their model.
Our method, by skipping irrelevant content, shortens the length of recurrent networks, thereby addressing the vanishing or exploding gradients in them (Hochreiter et al., 2001). The baseline method itself, Long Short Term Memory (Hochreiter and Schmidhuber, 1997), belongs to the same category of methods. In this category, there are several recent methods that try to achieve the same goal, such as having recurrent networks that operate in different frequency (Koutnik et al., 2014) or is organized in a hierarchical fashion (Chan et al., 2015; Chung et al., 2016).
5 Conclusion and Future Work
In this paper, we focus on learning how to skim text for fast reading. In particular, we propose a “jumping” model that after reading every few tokens, it decides how many tokens should be skipped by sampling from a softmax. Such jumping behavior is modeled as a discrete decision making process, which can be trained by reinforcement learning algorithm such as REINFORCE. In four different tasks with six datasets, we test the efficiency of the proposed method on various levels of text jumping, from character to to word and to sentence. The results indicate our model is several times faster than, while the accuracy is on par with the baseline LSTM model.
As an important future work, we hope to extend our model to a bidirectional jumping network, such that it can jump back and forth and possibly pay more attention to the important part of text. We would also like to incorporate our model with advanced neural networks such as memory network (Weston et al., 2014), and/or with sophisticated mechanisms like attention (Bahdanau et al., 2014), and/or with hierarchical structure (Choi et al., 2016) to build a more accurate model.
9
801
802
803
804
805
806
807
808
809
810
811
812
813
814
815
816
817
818
819
820
821
822
823
824
825
826
827
828
829
830
831
832
833
834
835
836
837
838
839
840
841
842
843
844
845
846
847
848
849
850
851
852
853
854
855
856
857
858
859
860
861
862
863
864
865
866
867
868
869
870
871
872
873
874
875
876
877
878
879
880
881
882
883
884
885
886
887
888
889
890
891
892
893
894
895
896
897
898
899

The paper proposes a recurrent neural architecture that can skip irrelevant input units. This is achieved by specifying R (# of words to read at each  skim ), K (max jump size), and N (max # of jumps allowed). An LSTM processes R words, predicts the jump size k in {0, 1...K} (0 signals stop), skips the next k-1 words and continues until either the number of jumps reaches N or the model reaches the last word. While the model is not differentiable, it can be trained by standard policy gradient. The work seems to have been heavily influenced by Shen et al. (2016) who apply a similar reinforcement learning approach (including the same variance stabilization) to multi-pass machine reading.   - Strengths:  The work simulates an intuitive  skimming  behavior of a reader, mirroring Shen et al. who simulate (self-terminated) repeated reading. A major attribute of this work is its simplicity. Despite the simplicity, the approach yields favorable results. In particular, the authors show through a well-designed synthetic experiment that the model is indeed able to learn to skip when given oracle jump signals. In text classification using real-world datasets, the model is able to perform competitively with the non-skimming model while being clearly faster.   The proposed model can potentially have meaningful practical implications: for tasks in which skimming suffices (e.g., sentiment classification), it suggests that we can obtain equivalent results without consuming all data in a completely automated fashion. To my knowledge this is a novel finding.   - Weaknesses:  It's a bit mysterious on what basis the model determines its jumping behavior so effectively (other than the synthetic dataset). I'm thinking of a case where the last part of the given sentence is a crucial evidence, for instance:    The movie was so so and boring to the last minute but then its ending blew me away.    In this example, the model may decide to skip the rest of the sentence after reading  so so and boring . But by doing so it'll miss the turning point  ending blew me away  and mislabel the instance as negative. For such cases a solution can be running the skimming model in both directions as the authors suggest as future work. But in general the model may require more sophisticated architecture for controlling skimming.  It seems one can achieve improved skimming by combining it with multi-pass reading (presumably in reverse directions). That's how humans read to understand text that can't be digested in one skim; indeed, that's how I read this draft.   Overall, the work raises an interesting problem and provides an effective but intuitive solution.