While sequence-to-sequence (seq2seq) models have been broadly used, their application to Abstract Meaning Representation (AMR) parsing and AMR realization has been limited, at least in part because data sparsity was thought to pose a significant challenge. In contrast, we show that with careful preprocessing and a novel training procedure that allows us to incorporate millions of unlabeled sentences, we can significantly reduce the impact of sparsity. For parsing, we obtain competitive results of 61.9 SMATCH, the current best reported without significant use of external annotated semantic resources. For realization, we outperform state of the art by over 5 points, achieving 32.3 BLEU. We also present extensive ablative and qualitative analysis in addition to showing strong evidence that seq2seq models are robust to artifacts introduced by converting AMR graphs to sequences.
1 000 001 002 003 004 005 006 007 008 009 010 011 012 013 014 015 016 017 018 019 020 021 022 023 024 025 026 027 028 029 030 031 032 033 034 035 036 037 038 039 040 041 042 043 044 045 046 047 048 049
050
051
052
053
054
055
056
057
058
059
060
061
062
063
064
065
066
067
068
069
070
071
072
073
074
075
078
079
080
081
082
083
084
085
086
087
088
089
090
091
092
093
094
095
096
097
098
099
Neural AMR: Sequence-to-Sequence Models for Parsing and Generation
Anonymous ACL submission
Abstract
While sequence-to-sequence (seq2seq) models have been broadly used, their application to Abstract Meaning Representation (AMR) parsing and AMR realization has been limited, at least in part because data sparsity was thought to pose a significant challenge. In contrast, we show that with careful preprocessing and a novel training procedure that allows us to incorporate millions of unlabeled sentences, we can significantly reduce the impact of sparsity. For parsing, we obtain competitive results of 61.9 SMATCH, the current best reported without significant use of external annotated semantic resources. For realization, we outperform state of the art by over 5 points, achieving 32.3 BLEU. We also present extensive ablative and qualitative analysis in addition to showing strong evidence that seq2seq models are robust to artifacts introduced by converting AMR graphs to sequences.
1 Introduction
Abstract Meaning Representation (AMR) is a graph based formalism that encodes many aspects of the meaning of a natural language sentence (for example, see Figure 1). AMR has been used as an intermediate representation for machine translation (Jones et al., 2012), summarization (Liu et al., 2015), sentence compression (Takase et al., 2016), event extraction (Huang et al., 2016), and has potential applications in dialogue, or human-robotic interaction. While AMR is extremely expressive, annotation is expensive and training data is limited, making application of neural methods challenging (Misra and Artzi, 2016; Peng and Xue, 2017; Barzdins and Gosko, 2016).
Obama was elected and his voters celebrated
Obama
elect.01 celebrate.01
voters
and * op1 op2
arg0
poss
arg0
person name
name op1
In this work, we tackle both AMR parsing and AMR realization together, showing the first successful sequence-to-sequence (seq2seq) models for both problems. While seq2seq models have been broadly used (Wu et al., 2016; Bahdanau et al., 2014; Luong et al., 2015; Vinyals et al., 2015), application to AMR has been limited, at least in part because effective linearization (encoding graphs as linear sequences) and data sparsity were thought to pose significant challenges. We show these challenges can be easily overcome, by demonstrating that seq2seq models can be trained using any graph-isomorphic linearization and that unlabeled text can be used to significantly reduce sparsity.
Our approach is two-fold. First, we carefully preprocess the AMR, anonymizing entities and dates, grouping entity categories, and encoding nesting information in concise ways, for example see Figure 2(d). Under such a representation, we show that any depth first traversal of the AMR is effective for constructing a linearization, and it is even possible to use a different random or-
2
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
der for each example. Second, we introduce a novel paired training procedure (Algorithm 1) for constructing an AMR parser and realizer. First we use self-training to bootstrap a high quality AMR parser from millions of unlabeled Gigaword (Napoles et al., 2012) sentences and then use it to pre-train an AMR realizer. The paired training allows both the parser and realizer to learn high quality representations of input and output language, respectively, from millions of weakly labeled examples, that are then further improved by training on human annotated AMR data.
Experiments on the LDC2015E86 AMR corpus (SemEval-2016 Task 8) demonstrate the effectiveness of the overall approach. For parsing, we are able to obtain competitive performance of 61.9 SMATCH without using any external annotated examples other than the output of a NER system, an improvement of over 9 points relative to neural models with a comparable setup. For realization, we are able to substantially outperform state of the art, providing gains of over 5 points, for a final performance of 32.30 BLEU. We also provide extensive ablative and qualitative analysis, quantifying the contributions that come from preprocessing and the paired training procedure.
2 Related Work
Alignment-based Parsing Flanigan et al. (2014) (JAMR) pipeline concept and relation identification with a graph-based algorithm that attempts to find the maximum spanning sub-graph. Zhou et al. (2016), extend JAMR by performing the concept and relation identification tasks jointly with an incremental model. Both systems rely on features based on a set of alignments produced using bi-lexical cues and hand-written rules. In contrast, our models train directly on parallel corpora, and make only minimal use of alignments to anonymize named entities.
Grammar-based Parsing Wang et al. (2016) (CAMR) perform a series of shift-reduce transformations on the output of an externally-trained dependency parser, similar to Brandt et al. (2016),Puzikov et al. (2016), and Goodman et al. (2016). Artzi et al. (2015) use a grammar induction approach with Combinatory Categorical Grammar (CCG), which relies on pre-trained CCGBank categories, like Bjerva et al. (2016). Pust et al. (2015) recast parsing as a string-to-tree Machine Translation problem, using unsupervised
alignments (Pourdamghani et al., 2014), and employing several external semantic resources. Our neural approach is engineering lean, relying only on a large unannotated corpus of English and algorithms to find and canonicalize named entities.
Neural Parsing Recently there have been a few seq2seq systems for AMR parsing (Barzdins and Gosko, 2016; Peng and Xue, 2017). Similar to our approach, Peng and Xue (2017) deal with sparsity by anonymizing named entities and typing low frequency words, resulting in a very compact vocabulary (2k tokens). We avoid reducing our vocabulary by introducing a large set of unlabeled sentences from an external corpus. AMR Realization Flanigan et al. (2016) specify a number of tree-to-string transduction rules based on alignments and POS-based features, that are used to drive a tree-based SMT system. Pourdamghani et al. (2016) also use an MT decoder; they learn a classifier that linearizes the input AMR graph in an order that follows the output sentence, effectively reducing the number of alignment crossings of the phrase-based decoder. Song et al. (2016) recast generation as a traveling salesman problem, after partitioning the graph into fragments, and finding the best linearization order. Our models do not need to rely on a particular linearization of the input, attaining comparable performance even with a per example random traversal of the graph. Finally, all three systems intersect with a large LM trained on Gigaword. We show that our seq2seq model has the capacity to learn the same information as a language model, especially after pretraining on the external corpus.
Data Augmentation Our paired training procedure is largely inspired by Sennrich et al. (2016). They improve neural MT performance for low resource language pairs, by creating synthetic output for a large monolingual corpus of the target language, using a back-translation MT system and mix it with the human translations. We instead pre-train on the external corpus first, and then finetune on the original dataset.
3 Methods
In this section we describe: (1) the tasks of AMR parsing and realization formally, (2) the form of sequence-to-sequence models we use, and (3) our paired training procedure, Algorithm 1.
3
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
3.1 Tasks
We assume access to training dataset D where each example pairs a natural language sentence s with an AMR a. The AMR is a rooted directed acylical graph. It contains nodes whose names correspond to sense-identified verbs, nouns, or AMR specific concepts, for example elect.01, Obama, and person in Figure 1. One of these nodes is a distinguished root, for example, the node and in Figure 1. Furthermore, the graph contains labeled edges, which correspond to ProbBank-style (Palmer et al., 2005) semantic roles for verbs or other relations introduced for AMR, for example, arg0 or op1 in Figure 1. The set of node and edge names in an AMR graph is drawn from a set of concepts C, and every token in a sentence is drawn from a vocabulary W .
We study the task of training an AMR parser, i.e., finding a set of parameters θP for model f , that predicts an AMR graph â, given a sentence s:
â = argmax a
f ( a|s; θP ) (1)
We also consider the reverse task, training an AMR realizer by finding a set of parameters θG, for a model f that predicts a sentence ŝ, given an AMR graph a:
ŝ = argmax s
f ( s|a; θG ) (2)
In both cases, we use the same family of predictors f , sequence-to-sequence models that use global attention, but the models have independent parameters , θP and θG.
3.2 Sequence-to-sequence Model
For both tasks, we use a stacked-LSTM sequenceto-sequence neural architecture employed in neural machine translation (Bahdanau et al., 2014; Wu et al., 2016). Our model uses a global attention decoder and unknown word replacement with small modifications (Luong et al., 2015).
The model uses a stacked bidirectional-LSTM encoder to encode an input sequence and a stacked LSTM to decode from the hidden states produced by the encoder. We make two modifications to the encoder: (1) we concatenate the forward and backward hidden states at every level of the stack instead of at the top of the stack, and (2) introduce dropout in the first layer of the encoder. The decoder predicts an attention vector over the encoder hidden states using previous decoder states.
Algorithm 1 Paired Training Procedure Input: Training set of sentences and AMR graphs (s, a) ∈
D, an unannotated external corpus of sentences Se, a number of self training iterations, N , and a initial sample size k. Output: Model parameters for AMR parser θP and AMR realizer θG.
1: θP ← Train parser on D . Self-train AMR parser. 2: S1e ← sample k sentences from Se 3: for i = 1 to N do 4: Aie ← Parse Sie using parameters θP
. Pre-train AMR parser. 5: θP ← Train parser on (Aie, Sie)
. Fine tune AMR parser. 6: θP ← Train parser on D with initial parameters θP 7: Si+1e ← sample k · 10i new sentences from Se 8: end for 9: SNe ← sample k · 10N new sentences from Se
. Pre-train AMR realizer. 10: Ae ← Parse SNe using parameters θP 11: θG ← Train realizer on (ANe , SNe )
. Fine tune AMR realizer. 12: θG ← Train realizer on D using initial parameters θG 13: return θP , θG
The attention is then used to weight the hidden states of the encoder and then predict a token in the output sequence. The weighted hidden states, the decoded token, and an attention signal from the previous time step (input feeding) are then fed together as input to the next decoder state. The decoder can optionally choose to output an unknown word symbol, in which case the predicted attention is used to copy a token directly from the input sequence into the output sequence.
3.3 Linearization
Our seq2seq models require that both the input and target be presented as a linear sequence of tokens. We define a linearization order for an AMR graph as any sequence of its nodes and edges (potentially with repeats). A linearization is defined as (1) a linearization order and (2) a rendering function that generates any number of tokens when applied to an element in the linearization order. Furthermore, for parsing, a valid AMR graph must be recoverable from the linearization.
3.4 Paired Training
Obtaining a corpus of jointly annotated pairs of sentences and AMR graphs is expensive and current datasets only extend to thousands of examples. Neural sequence-to-sequence models suffer from sparsity with so few training pairs. To reduce the effect of sparsity, we use an external unannotated corpus of sentences Se, and a procedure which pairs the training of the parser and realizer.
4
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
Our procedure is described in Algorithm 1, and first trains a parser on the datasetD of pairs of sentences and AMR graphs. Then it uses self-training to improve the initial parser. Every iteration of self-training has three phases: (1) parsing samples from a large, unlabeled corpus Se, (2) creating a new set of parameters by training on Se, and (3) fine-tuning those parameters on the original paired data. After each iteration, we increase the size of the sample from Se by an order of magnitude. After we have the best parser from self-training, we use it to label AMRs for Se and pre-train the realizer. The final step of the procedure fine-tunes the realizer on the original dataset D.
4 Data Handling
In the following sections, we explain a series of data preparation steps, including AMR linerization, anonymization, and other modifications we make to sentence-graph pairs to produce input and output for our seq2seq model. Our methods have two goals: (1) reduce the length of sentences and linearizations to make learning easier while maintaining enough original information, and (2) address sparsity from certain open class vocabulary entries, such as named entities (NEs) and quantities. Figure 2(d) contains example inputs and outputs with all of our preprocessing techniques.
Basic Preprocessing In order to reduce the overall length of the linearized graph, we first remove variable names and the instance-of relation ( / ) before every concept. In case of re-entrant nodes we replace the variable mention with its co-referring concept. Even though this replacement incurs loss of information, often the surrounding context helps recover the correct realization, e.g., the possessive role :poss in the example of Figure 1 is strongly correlated with the surface form his. Following Pourdamghani et al. (2016) we also remove senses from all concepts for AMR realization only. Figure 2(a) contains an example output after this stage.
4.1 Anonymization of Named Entities
Open-class types including NEs, dates, and numbers account for the 9.6% of tokens in the sentences of the training corpus, and 31.22% of the NL vocabulary. 83.36% of them occur fewer than 5 times in the dataset. In order to reduce sparsity and be able to account for new unseen entities, we perform extensive anonymization.
First, we anonymize sub-graphs headed by one of AMR’s over 140 fine-grained entity types that contain a :name role. This captures structures referring to entities such as person, country, miscellaneous entities marked with *-enitity, and typed numerical values, *-quantity. We exclude date entities (see the next section). We then replace these sub-graphs with a token indicating fine-grained type and an index, i, indicating it is the ith occurrence of that type 1. For example, in Figure 2 the sub-graph headed by country gets replaced with country 0.
On the training set, we use JAMR and unsupervised alignments to find mappings of anonymized subgraphs to spans of text (Flanigan et al., 2014; Pourdamghani et al., 2014) and replace mapped text with the anonymized token that we inserted into the AMR graph. We record this mapping for use during testing of realization models. If a realization model predicts an anonymization token, we find the corresponding token in the AMR graph and replace the model’s output with the most frequent mapping observed during training for the entity name. If the entity was never observed, we copy its name directly from the AMR graph.
Anonymizing Dates For dates in AMR graphs, we use separate anoymization tokens for year, month-number, month-name, day-number and day-name, indicating whether the date is mentioned by word or by number.2 In AMR realization, we render the corresponding format when predicted. Figure 2(b) contains an example of all preprocessing up to this stage.
Named Entity Clusters When performing AMR realization, each of the AMR fine-grained entity types is manually mapped to one of the four coarse entity types. We adopt the same types as used in the Stanford NER system (Finkel et al., 2005): person, location, organization and misc. This reduces the sparsity associated with many rarely occurring entity types. Figure 2 (c) contains an example with named entity clusters.
NER for Parsing When parsing, we must normalize test sentences to match our anonymized training data. To produce fine-grained named entities, we run the Stanford NER system (Finkel
1In practice we only used three groups of ids: a different one for NEs, dates and constants/numbers.
2We additionally use three date format markers that appear in the text as: YYYYMMDD, YYMMDD, and YYYY-MMDD.
5
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
458
459
460
461
462
463
464
465
466
467
468
469
470
471
472
473
474
475
476
477
478
479
480
481
482
483
484
485
486
487
488
489
490
491
492
493
494
495
496
497
498
499
US officials held an expert group meeting in January 2002 in New York. (h / hold-04
:ARG0 (p2 / person
:ARG0-of (h2 / have-org-role-91
:ARG1 (c2 / country
:name (n3 / name
:op1 “United" op2: “States”))
:ARG2 (o / official)))
:ARG1 (m / meet-03
:ARG0 (p / person
:ARG1-of (e / expert-01)
:ARG2-of (g / group-01)))
:time (d2 / date-entity :year 2002 :month 1)
:location (c / city
:name (n / name :op1 “New" :op2 “York”)))
hold :ARG0 person :ARG0-of have-org-role :ARG1 loc_0 :ARG2 official :ARG1 meet :ARG0 person :ARG1-of expert :ARG2-of group :time date-entity year_0 month_0 :location loc_1
hold :ARG0 person :ARG0-of have-org-role :ARG1 country_0 :ARG2 official :ARG1 meet :ARG0 person :ARG1-of expert :ARG2-of group :time date-entity year_0 month_0 :location city_1
hold :ARG0 person :ARG0-of have-org-role :ARG1 country :name name :op1 United :op2 States :ARG2 official :ARG1 meet :ARG0 person :ARG1-of expert :ARG2-of group :time date-entity :year 2002 :month 1 :location city :name name :op1 New :op2 York
hold :ARG0 ( person :ARG0-of ( have-org-role :ARG1 loc_0 :ARG2 official ) ) :ARG1 ( meet :ARG0 ( person :ARG1-of expert :ARG2-of group ) ) :time ( date-entity year_0 month_0 ) :location loc_1
US officials held an expert group meeting in January 2002 in New York.
country_0 officials held an expert group meeting in month_0 year_0 in city_1.
loc_0 officials held an expert group meeting in month_0 year_0 in loc_1.
loc_0 officials held an expert group meeting in month_0 year_0 in loc_1.
(a)
(b)
(c)
(d)
Figure 2: Preprocessing methods applied to sentence (top row) - AMR graph (left column) pairs. Sentence-graph pairs after (a) basic preprocessing, (b) named entity anonymization, (c) named entity clustering, and (d) insertion of scope markers.
et al., 2005) and first try to replace any identified span with a fine-grained category based on alignments observed during training. If this fails, we anonymize the sentence using the coarse categories predicted by the NER system, which are also categories in AMR. After parsing, we deterministically generate AMR for anonymizations using the corresponding text span.
4.2 Linearization
A linearization is specified by (1) a linearization order, a list of nodes and edges to visit, possibly with repeats and (2) a rendering function that emits any number of tokens given an element in the linearization order.
Linearization Order Our linearization order is analogous to the order of nodes visited by depth first search, including backward traversing steps. For example, in Figure 2, starting at meet the order contains meet, :ARG0, person, :ARG1-of, expert, :ARG2-of, group, :ARG2-of, :ARG1-of, :ARG0.3 The order traverses children in the sequence they are presented in the AMR. We consider alternative orderings of children in Section 7 but always follow the pattern demonstrated above.
Rendering Function Our rendering function marks scope, and generates tokens based on two cases. (1) If the element is a node, it emits the
3Sense and instance-of information has been removed at the point of linearization
type of the node. (2) If the element is an edge, the first time an edge is encountered, if the edge leads to a node with children it emits a left parenthesis “(” and then emits the type of the edge. The second time an edge is encountered, if a “(” was emitted previously, a right parenthesis “)” is emitted. This rendering function omits our scope markers “(” and “)” in cases when a node only has one child, significantly reducing the number of tokens it generates. Figure 2(d) contains an example showing all of the preprocessing techniques and scope markers that we use in our full model.
5 Experimental Setup
We conduct all experiments on the AMR corpus used in SemEval-2016 Task 8 (LDC2015E86), which contains 16,833/1,368/1,371 train/dev/test examples. For the paired training procedure of Algorithm 1 we use Gigaword as our external corpus and sample sentences that only contain tokens from the AMR corpus. We evaluate AMR parsing with SMATCH (Cai and Knight, 2013), and AMR realization using BLEU (Papineni et al., 2002) 4.
We validated word embedding sizes and RNN hidden representation sizes by maximizing development performance just using data in the AMR corpus, Algorithm 1, line 1. We searched over the set {128, 256, 500, 1024} for the best combinations of sizes and set both to 500. Models were trained by optimizing cross-entropy loss with
4We use the multi BLEU script.
6
501
502
503
504
505
506
507
508
509
510
511
512
513
514
515
516
517
518
519
520
521
522
523
524
525
526
527
528
529
530
531
532
533
534
535
536
537
538
539
540
541
542
543
544
545
546
547
548
549
550
551
552
553
554
555
556
557
558
559
560
561
562
563
564
565
566
567
568
569
570
571
572
573
574
575
576
577
578
579
580
581
582
583
584
585
586
587
588
589
590
591
592
593
594
595
596
597
598
599
Model Prec Rec F1 Prec Rec F1 SBMT (Pust et al., 2015) - - 69.0 - - 67.1 CAMR (Wang et al., 2016) 72.3 61.4 66.6 70.4 63.1 66.5 CCG* (Artzi et al., 2015) 67.2 65.1 66.1 66.8 65.7 66.3 JAMR (Flanigan et al., 2014) - - - 64.0 53.0 58.0 GIGA-2M 61.9 64.8 63.3 60.2 63.6 61.9 GIGA-200k 59.7 62.9 61.3 57.8 60.9 59.3 AMR-ONLY 54.9 60.0 57.4 53.1 58.1 55.5 SEQ2SEQ (Peng and Xue, 2017) - - - 55.0 50.0 52.0 CHAR-LSTM (Barzdins and Gosko, 2016) - - - - - 43.0
Table 1: SMATCH scores for AMR Parsing (TEST set). *Reported numbers are on the newswire portion of a previous release of the corpus (LDC2014T12).
Model Dev Test GIGA-2M 31.8 32.3 GIGA-200k 27.2 27.4 AMR-ONLY 21.7 22.0 PBMT* (Pourdamghani et al., 2016) 27.2 26.9 TSP (Song et al., 2016) 21.1 22.4 TREETOSTR (Flanigan et al., 2016) 23.0 23.0
Table 2: BLEU results for AMR Realization. *Model has been trained on a previous release of the corpus (LDC2014T12).
stochastic gradient descent, using a batch size of 100, and dropout rate of 0.5. Across all models when performance does not improve on the AMR dev set, we decay the learning rate by 0.8.
For the initial parser trained on the AMR corpus, Algorithm 1, line 1, we use a single stack version of our model, set initial learning rate to 0.5 and train for 60 epochs, taking the best performing model on the development set. All subsequent models benefited from increased depth and we used 2-layer stacked versions, maintaining same embedding sizes. We set the initial Gigaword sample size to k = 200, 000 and executed a maximum of 2 iterations of self-training. For pre-training the parser and realizer, Algorithm 1, lines 4 and 9, we used an initial learning rate of 1.0, and ran for 20 epochs. We attempt to fine-tune the parser and realizer, respectively, after every epoch of pretraining, setting the initial learning rate to 0.1. We select the best performing model on the development set among all of these fine-tuning attempts.
6 Results
Parsing Results Table 1 summarizes our development results for different rounds of self-training and test results for our final system, self-trained
Model BLEU Full 21.76 Full - scope 19.68 Full - scope - ne 19.53 Full - scope - ne - anon 18.71
Table 3: BLEU scores for AMR realization ablation results on preprocessing (DEV set).
on 2M unlabeled Gigaword sentences. Through every round of self-training, our parser improves. Our final parser outperforms comparable seq2seq and character LSTM models by over 9.8 points. While much of this improvement comes from selftraining, our model without Gigaword data outperforms these approaches by 5.4 points on F1. All other models that we compare against use semantic resources, such as WordNet, dependency parsers or CCG parsers (models marked with * were trained with less data, but only evaluate on newswire text; the rest evaluate on the full test set, containing text from blogs). Our full models outperform JAMR, a graph-based model but still lags behind other parser-dependent systems (CAMR), and resource heavy approaches (SBMT).
Realization Results Table 2 summarizes our AMR realization results on development and test set. We outperform all previous state-of-the-art systems by the first round of self-training and further improve with the second round. Our final model trained on GIGA-2M outperforms previous models by 5.4 BLEU. Overall, our model incorporates less data than previous approaches as all reported methods train language models on the whole Gigaword corpus. We leave scaling our models to all of Gigaword for future work.
7
601
602
603
604
605
606
607
608
609
610
611
612
613
614
615
616
617
618
619
620
621
622
623
624
625
626
627
628
629
630
631
632
633
634
635
636
637
638
639
640
641
642
643
644
645
646
647
648
649
650
651
652
653
654
655
656
657
658
659
660
661
662
663
664
665
666
667
668
669
670
671
672
673
674
675
676
677
678
679
680
681
682
683
684
685
686
687
688
689
690
691
692
693
694
695
696
697
698
699
Model Prec Rec F1 Full 54.9 60.0 57.4 Full - anon 22.7 54.2 32.0
Table 4: SMATCH scores for AMR parsing ablation results on preprocessing (DEV set).
Linearization Order BLEU Human 21.67 Random 20.80 Stochastic 20.34
Table 5: BLEU on AMR realization for different linearization orders (DEV set).
Preprocessing Ablation Study We consider the contributation of each main component of our preprocessing stages while keeping our linearization order identical. Figure 2 contains examples of linearized AMR and sentence pairs we evaluate for each setting of our ablations. First, we evaluate with AMR linearized without parentheses for indicating scope, Figure 2(c), then additionally without named entity clusters, Figure 2(b), and additionally without any anonymization, Figure 2(a).
Tables 3 summarizes our evaluation on the AMR realization. Results indicate each of these components is required, and that scope markers and anonymization are the biggest contributors. We suspect without scope markers our seq2seq models are not as effective at capturing long range semantic relationships between elements of the AMR graph. We also evaluated the contribution of anonymization to AMR parsing, Table 4.Similar to previous work, we find seq2seq based AMR parsing is largely ineffective without anonymization (Peng and Xue, 2017).
7 Linearization
In this section we evaluate three strategies for converting AMR graphs into sequences in the context of AMR realization and show that our models are largely agnostic to linearization orders. Our results argue, unlike SMT-based AMR realization methods (Pourdamghani et al., 2016), that seq2seq models can learn to ignore artifacts of the conversion of graphs to linear sequences.
7.1 Linearization Orders
All linearizations we consider use the pattern described in Section 4.2, but differ on the order in which children are visited. Each linearization gen-
erates anonymized, scope marked output (see Section 4), of the form in Figure 2(d). Human The proposal traverses children in the order presented by human authored AMR annotations, exactly as shown in Figure 2(d). Random We construct a random global ordering of all edge types appearing in AMR graphs. We traverse children, based on the position in the global ordering of the edge leading to a child. Stochastic In this linearization we randomize our traversal of children, per example.
7.2 Results
We present AMR realization results for the three proposed realization orders in Table 5. Random linearization order performs only slightly worse than traversing the graph according to Human linearization order. Surprisingly, a per example stochastic linearization order performs nearly identically to a stable random order, arguing seq2seq models can learn to ignore artifacts of the conversion of graphs to linear sequences.
Human-authored AMR leaks information The small difference between stochastic and random linearizations argues that our models are largely agnostic to variation in linearization order. On the other hand, the model that follows the human order performs significantly better which leads us to suspect it carries extra information not apparent in the graphical structure of the AMR.
To further investigate, we compared the relative ordering of edge pairs under the same parent to relative position of children nodes derived from those edges, in a sentence, as reported by JAMR alignments. We found that the majority of pairs of AMR edges, 57.6%, always occurred in the same relative order, therefore revealing no extra realization order information.5 Of the examples corresponding to edge pairs that showed variation, 70.3% appeared in an order consistent with the order they were realized in the sentence. The relative ordering of some pairs of AMR edges was particularly indicative of realization order. For example, the relative ordering of edges with types location and time, was 17% more indicative of the realization order than the majority of realizing locations before time.6
5This is consistent with constraints encoded in the annotation tools used to collect AMR. For example, arg0 edges are always ordered before arg1 edges
6Consider the sentences “She went to school in New York
8
701
702
703
704
705
706
707
708
709
710
711
712
713
714
715
716
717
718
719
720
721
722
723
724
725
726
727
728
729
730
731
732
733
734
735
736
737
738
739
740
741
742
743
744
745
746
747
748
749
750
751
752
753
754
755
756
757
758
759
760
761
762
763
764
765
766
767
768
769
770
771
772
773
774
775
776
777
778
779
780
781
782
783
784
785
786
787
788
789
790
791
792
793
794
795
796
797
798
799
Error Type % Coverage 29 Disfluency 23 Anonymization 14 Sparsity 13 Attachment 12 Other 10
Table 6: Error analysis for AMR realization on a sample of 50 examples from the development set.
To compare to previous work we still report using human orderings. However, we note that any practical application requiring a system to generate an AMR representation with the intention to realize it later on, e.g., a dialogue agent, will need to be trained either using consistent, or stochasticderived linearization orders. Arguably, our models are agnostic to this choice.
8 Qualitative Results
Figure 3 shows example output of our realizer.The realization of the first graph is nearly perfect with only a small grammatical error due to anonymization. The second example is more challenging, with a deep right-branching structure, and a coordination of the verbs stabilize and push in the subordinate clause headed by state. The model omits some information from the graph, namely the concepts terrorist and virus. In the third example there are greater parts of the graph that are missing, such as the whole subgraph headed by expert. Also the model makes wrong attachment decisions in the last two subgraphs (it is the evidence that is unimpeachable and irrefutable, and not the equipment), mostly due to insufficient annotation (thing) thus making their realization harder.
Finally, Table 6 summarizes the proportions of error types we identified on 50 randomly selected examples from the development set. We found that the realizer mostly suffers from coverage issues, an inability to mention all tokens in the input, followed by fluency mistakes, as illustrated above. Attachment errors are less frequent, which supports our claim that the model is robust to graph linearization, and can successfully encode long range dependency information between concepts.
two years ago”, and “Two years ago, she went to school in New York”, where “two year ago” is the time modifying constituent for the verb went and “New York” is the location modifying constituent of went.
limit :arg0 ( treaty :arg0-of ( control :arg1 arms ) ) :arg1 ( number :arg1 ( weapon :mod conventional :arg1-of ( deploy :arg2 ( relative-pos :op1 loc_0 :dir west ) :arg1-of possible ) ) )
SYS: the arms control treaty limits the number of conventional weapons that can be deployed west of Ural Mountains .
REF: the arms control treaty limits the number of conventional weapons that can be deployed west of the Ural Mountains .
COMMENT: disfluency
state :arg0 ( person :arg0-of ( have-org-role :arg1 ( committee :mod technical ) :arg3 ( expert :arg1 person :arg2 missile :mod loc_0 ) ) ) :arg1 ( evidence :arg0 equipment :arg1 ( plan :arg1 ( transfer :arg1 ( contrast :arg1 ( missile :mod ( just :polarity - ) ) :arg2 ( capable :arg1 thing :arg2 ( make :arg1 missile ) ) ) ) ) :mod ( impeach :polarity - :arg1 thing ) :mod ( refute :polarity - :arg1 thing ) )
SYS: a technical committee expert on the technical committee stated that the equipment is not impeach , but it is not refutes .
REF: a technical committee of Indian missile experts stated that the equipment was unimpeachable and irrefutable evidence of a plan to transfer not just missiles but missile-making capability.
COMMENT: coverage , disfluency, attachment
state :arg0 report :arg1 ( obligate :arg1 ( government-organization :arg0-of ( govern :arg1 loc_0 ) ) :arg2 ( help :arg1 ( and :op1 ( stabilize :arg1 ( state :mod weak ) ) :op2 ( push :arg1 ( regulate :mod international :arg0-of ( stop :arg1 terrorist :arg2 ( use :arg1 ( information :arg2-of ( available :arg3-of free )) :arg2 ( and :op1 ( create :arg1 ( form :domain ( warfare :mod biology :example ( version :arg1-of modify :poss other_1 ) ) :mod new ) ) :op2 ( unleash :arg1 form ) ) ) ) ) ) ) ) )
REF: the report stated British government must help to stabilize weak states and push for international regulations that would stop terrorists using freely available information to create and unleash new forms of biological warfare such as a modified version of the influenza virus .
COMMENT: coverage , disfluency, attachment
SYS: the report stated that the Britain government must help stabilize the weak states and push international regulations to stop the use of freely available information to create a form of new biological warfare such as the modified version of the influenza .
Figure 3: Linearized AMR after preprocessing, reference sentence, and output of the realizer. We mark, with colors, common error types: disfluency, coverage (missing information from the input graph), and attachment (implying a semantic relation from the AMR between incorrect entities).
9
801
802
803
804
805
806
807
808
809
810
811
812
813
814
815
816
817
818
819
820
821
822
823
824
825
826
827
828
829
830
831
832
833
834
835
836
837
838
839
840
841
842
843
844
845
846
847
848
849
850
851
852
853
854
855
856
857
858
859
860
861
862
863
864
865
866
867
868
869
870
871
872
873
874
875
876
877
878
879
880
881
882
883
884
885
886
887
888
889
890
891
892
893
894
895
896
897
898
899

- Strengths:  The paper demonstrates that seq2seq models can be comparatively effectively applied to the tasks of AMR parsing and AMR realization by linearization of an engineered pre-processed version of the AMR graph and associated sentence, combined with 'Paired Training' (iterative back-translation of monolingual data combined with fine-tuning). While parsing performance is worse than other reported papers (e.g., Pust et al., 2015), those papers used additional semantic information.   On the task of AMR realization, the paper demonstrates that utilizing additional monolingual data (via back-translation) is effective relative to a seq2seq model that does not use such information. (See note below about comparing realization results to previous non-seq2seq work for the realization task.)  - Weaknesses:   At a high-level, the main weakness is that the paper aims for empirical comparisons, but in comparing to other work, multiple aspects dimensions are changing at the same time (in some cases, not comparable due to access to different information), complicating comparisons.   For example, with the realization results (Table 2), PBMT (Pourdamghani et al., 2016) is apparently trained on LDC2014T12, which consists of 13,051 sentences, compared to the model of the paper, which is trained on LDC2015E86, which consists of 19,572 sentences, according to http:  amr.isi.edu download.html. This is used in making the claim of over 5 points improvement over the state-of-the-art (PBMT) in line 28 29, 120 121, and line 595, and is only qualified in the caption of Table 2. To make a valid comparison, the approach of the paper or PBMT needs to be re-evaluated after using the same training data.  - General Discussion:  Is there any overlap between the sentences in your Gigaword sample and the test sentences of LDC2015E86? Apparently LDC2015E86 contains data from the ''proxy report data in LDC's DEFT Narrative Text Source Data R1 corpus (LDC2013E19)'' (Accessible with LDC account: https:  catalog.ldc.upenn.edu LDC2015E86). It seems LDC2013E19 contains data from Gigaword (https:  catalog.ldc.upenn.edu LDC2013E19). Apparently AMR corpus LDC2014T12 also contained ''data from newswire articles selected from the English Gigaword Corpus, Fifth Edition'' (publicly accessible link: https:  catalog.ldc.upenn.edu docs LDC2014T12 README.txt). Please check that there is no test set contamination.  Line 244-249: Did these two modifications to the encoder make a significant difference in effectiveness? What was the motivation behind these changes?  Please make it clear (in an appendix is fine) for replication purposes whether the implementation is based on an existing seq2seq framework.  Line 321: What was the final sequence length used? (Consider adding such details in an appendix.)  Please label the columns of Table 1 (presumably dev and test). Also, there is a mismatch between Table 1 and the text: ''Table 1 summarizes our development results for different rounds of self-training.'' It appears that only the results of the second round of self-training are shown.  Again, the columns for Table 1 are not labeled, but should the results for column 1 for CAMR instead be 71.2, 63.9, 67.3--the last line of Table 2 in http:  www.aclweb.org anthology S16-1181 which is the configuration for +VERB+RNE+SRL+WIKI? It looks like the second from last row of Table 2 in CAMR (Wang et al., 2016) is currently being used. On this note, how does your approach handle the wikification information introduced in LDC2015E86?   7.1.Stochastic is missing a reference to the example.  Line 713-715: This seems like a hypothesis to be tested empirically rather than a forgone conclusion, as implied here.  Given an extra page, please add a concluding section.  How are you performing decoding? Are you using beam search?  As a follow-up to line 161-163, it doesn't appear that the actual vocabulary size used in the experiments is mentioned. After preprocessing, are there any remaining unseen tokens in dev test? In other words, is the unknown word replacement mechanism (using the attention weights), as described in Section 3.2, ever used?   For the realization case study, it would be of interest to see performance on phenomena that are known limitations of AMR, such as quantification and tense (https:  github.com amrisi amr-guidelines blob master amr.md).  The paper would benefit from a brief discussion (perhaps a couple sentences) motivating the use of AMR as opposed to other semantic formalisms, as well as why the human-annotated AMR information signal might be useful as opposed to learning a model (e.g., seq2seq itself) directly for a task (e.g., machine translation).  For future work (not taken directly into account in the scores given here for the review, since the applicable paper is not yet formally published in the EACL proceedings): For parsing, what accounts for the difference from previous seq2seq approaches? Namely, between Peng and Xue, 2017 and AMR-only (as in Table 1) is the difference in effectiveness being driven by the architecture, the preprocessing, linearization, data, or some combination thereof? Consider isolating this difference. (Incidentally, the citation for Peng and Xue, 2017 [''Addressing the Data Sparsity Issue in Neural AMR Parsing''] should apparently be Peng et al. 2017 (http:  eacl2017.org index.php program accepted-papers; https:  arxiv.org pdf 1702.05053.pdf). The authors are flipped in the References section.  Proofreading (not necessarily in the order of occurrence; note that these are provided for reference and did not influence my scoring of the paper):  outperform state of the arttooutperform the state of the art  Zhou et al. (2016), extendtoZhou et al. (2016) extend  (2016),Puzikov et al.to(2016), Puzikov et al.  POS-based features, thattoPOS-based features that  language pairs, by creatingtolanguage pairs by creating  using a back-translation MT system and mix it with the human translations.tousing a back-translation MT system, and mix it with the human translations.  ProbBank-style (Palmer et al., 2005)toPropBank-style (Palmer et al., 2005)  independent parameters ,toindependent parameters,  for the 9.6% of tokenstofor 9.6% of tokens  maintaining same embedding sizestomaintaining the same embedding sizes  Table 4.SimilartoTable 4. Similar  realizer.Thetorealizer. The  Notation: Line 215, 216: The sets C and W are defined, but never subsequently referenced. (However, W could should be used in place of ''NL'' in line 346 if they are referring to the same vocabulary.)


Learning Character-level Compositionality with Visual Features
Previous work has modeled the compositionality of words by creating characterlevel models of meaning, reducing problems of sparsity for rare words. However, in many writing systems compositionality has an effect even on the character-level: the meaning of a character is derived by the sum of its parts. In this paper, we model this effect by creating embeddings for characters based on their visual characteristics, creating an image for the character and running it through a convolutional neural network to produce a visual character embedding. Experiments on a text classification task demonstrate that such model allows for better processing of instances with rare characters in languages such as Chinese, Japanese, and Korean. Additionally, qualitative analyses demonstrate that our proposed model learns to focus on the parts of characters that carry categorical content which resulting in embeddings that are coherent in visual space.
1 000
011
012
013
014
015
016
017
018
019
020
021
022
023
024
025
026
027
028
029
030
031
032
033
034
035
036
037
038
039
040
041
042
043
044
045
046
047
048
049
061
062
063
064
065
066
067
068
069
070
071
072
073
076
077
078
079
080
081
082
083
084
085
086
087
088
089
090
091
092
093
094
095
096
097
098
099
Previous work has modeled the compositionality of words by creating characterlevel models of meaning, reducing problems of sparsity for rare words. However, in many writing systems compositionality has an effect even on the character-level: the meaning of a character is derived by the sum of its parts. In this paper, we model this effect by creating embeddings for characters based on their visual characteristics, creating an image for the character and running it through a convolutional neural network to produce a visual character embedding. Experiments on a text classification task demonstrate that such model allows for better processing of instances with rare characters in languages such as Chinese, Japanese, and Korean. Additionally, qualitative analyses demonstrate that our proposed model learns to focus on the parts of characters that carry categorical content which resulting in embeddings that are coherent in visual space.
1 Introduction
Compositionality—the fact that the meaning of larger linguistic units is created by combining the meaning of smaller units—is a hallmark of every natural language (Szabó, 2010). Recently, neural models have provided a powerful tool for learning how to compose words together into a meaning representation of whole sentences for many downstream tasks. This is done using models of various levels of sophistication, from simpler bag-ofwords (Iyyer et al., 2015) and linear recurrent neural network (RNN) models (Sutskever et al., 2014; Kiros et al., 2015), to more sophisticated models
Kalb Kälbera
Do Do'(polite)
Calf CalvesLaurel
Whale
Salmon
Salmon
gui jing
gui gui
(a)
(b)
(c)
(d)
han'da ham''ni'''da
using tree-structured (Socher et al., 2013) or convolutional networks (Kalchbrenner et al., 2014). In fact, a growing body of evidence shows that it is essential to look below the word-level and consider compositionality within words themselves. For example, several works have proposed models that represent words by composing together the characters into a representation of the word itself (Ling et al., 2015; Zhang et al., 2015; Dhingra et al., 2016). Additionally, for languages with productive word formation (such as agglutination and compounding), models calculating morphologysensitive word representations have been found effective (Luong et al., 2013; Botha and Blunsom, 2014). Thesemodels help to learnmore robust representations for rare words by exploiting morphological patterns, as opposed to models that operate purely on the lexical level as the atomic units. For many languages, compositionality stops at the character-level: characters are atomic units of meaning or pronunciation in the language, and no further decomposition can be done.1 However, for other languages, character-level compositionality, where a character’s meaning or pronunciation
1In English, for example, this is largely the case.
2
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
Lang Geography Sports Arts Military Economics Transportation Chinese 32.4k 49.8k 50.4k 3.6k 82.5k 40.4k Japanese 18.6k 82.7k 84.1k 81.6k 80.9k 91.8k Korean 6k 580 5.74k 840 5.78k 1.68k Lang Medical Education Food Religion Agriculture Electronics Chinese 30.3k 66.2k 554 66.9k 89.5k 80.5k Japanese 66.5k 86.7k 20.2k 98.1k 97.4k 1.08k Korean 16.1k 4.71k 33 2.60k 1.51k 1.03k
Table 1: By-category statistics for the Wikipedia dataset. Note that Food is the abbreviation for “Food and Culture” and Religion is the abbreviation for “Religion and Belief”.
can be derived from the sum of its parts, is very much a reality. Perhaps the most compelling example of compositionality of sub-character units can be found in logographic writing systems such as the Han and Kanji characters used in Chinese and Japanese, respectively.2 As shown on the left side of Fig. 1, each part of a Chinese character (called a “radical”) potentially contributes to the meaning (i.e., Fig. 1(a)) or pronunciation (i.e., Fig. 1(b)) of the overall character. This is similar to how English characters combine into the meaning or pronunciation of an English word. Even in languages with phonemic orthographies, where each character corresponds to a pronunciation instead of a meaning, there are cases where composition occurs. Fig. 1(c) and (d) show the examples of Korean and German, respectively, where morphological inflection can cause single characters to make changes where some but not all of the component parts are shared. In this paper, we investigate the feasibility of modeling the compositionality of characters in a way similar to how humans do: by visually observing the character and using the features of its shape to learn a representation encoding its meaning. Our method is relatively simple, and generalizable to a wide variety of languages: we first transform each character from its Unicode representation to a rendering of its shape as an image, then calculate a representation of the image using Convolutional Neural Networks (CNNs) (Cun et al., 1990). These features then serve as inputs to a down-stream processing task and trained in an end-to-end manner, which first calculates a loss function, then back-propagates the loss back to the CNN.
2Other prominent examples are largely for extinct languages: Egyptian hieroglyphics, Mayan glyphs, and Sumerian cuneiform scripts (Daniels and Bright, 1996).
As demonstrated by our motivating examples in Fig. 1, in logographic languages, character-level semantic or phonetic similarity is often indicated by visual cues; we conjecture that CNNs can appropriately model these visual patterns. Consequently, characters with similar visual appearances will be biased to have similar embeddings, allowing our model to handle rare characters effectively, just as character-level models have been effective for rare words. To evaluate our model’s ability to learn representations, particularly for rare characters, we perform experiments on a downstream task of classifying Wikipedia titles for three Asian languages: Chinese, Japanese, and Korean. We show that our proposed framework outperforms baseline model that use standard character embeddings for instances containing rare characters. A qualitative analysis of the characteristics of the learned embeddings of our model demonstrates that visually similar characters share similar embeddings. We also show that the learned representations are particularly effective under low-resource scenarios and complementary with standard character embeddings; combining the two representations through three different fusion methods (Snoek et al., 2005; Karpathy et al., 2014) leads to consistent improvements over the strongest baseline without visual features.
2 Dataset
Before delving into the details of our model, we first describe a dataset we constructed to examine the ability of our model to capture the compositional characteristics of characters. Specifically, the dataset must satisfy two desiderata: (1) it must be necessary to fully utilize each character in the input in order to achieve high accuracy, and (2) there must be enough regularity and com-
3
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
100 101 102 103 104
106
100
101 102
103
104 105
Rank
Fr eq
ue nc
y ⎯⎯ Chinese ⎯⎯"Japanese ⎯⎯ Korean
Rank < 20% Freq. > 80%
Figure 2: The character rank-frequency distribution of the corpora we considered in this paper. All three languages have a long-tail distribution.
positionality in the characters of the language. To satisfy these desiderata, we create a text classification dataset where the input is a Wikipedia article title in Chinese, Japanese, or Korean, and the output is the category to which the article belongs. 3 This satisfies (1), because Wikipedia titles are short and thus each character in the title will be important to our decision about its category. It also satisfies (2), because Chinese, Japanese, and Korean have writing systems with large numbers of characters that decompose regularly as shown in Fig. 1. While this task in itself is novel, it is similar to previous work in named entity type inference using Wikipedia (Toral and Munoz, 2006; Kazama and Torisawa, 2007; Ratinov and Roth, 2009), which has proven useful for downstream named entity recognition systems.
2.1 Dataset Collection
As the labels we would like to predict, we use 12 different main categories from the Wikipedia web page: Geography, Sports, Arts, Military, Economics, Transportation, Health Science, Education, Food Culture, Religion and Belief, Agriculture and Electronics. Wikipedia has a hierarchical structure, where each of these main categories has a number of subcategories, and each subcategory has its own subcategories, etc. We traverse this hierarchical structure, adding each main category tag to all of its descendants in this subcategory tree structure. In the case that a particular article is the descendant of multiple main categories, we favor the main category that minimizes the depth of the
3This dataset and the crawling scripts are attached as supplementary material, and will be released publicly upon acceptance of the paper.
Geography Sports Arts Military Economics Transportation Health Science Education Food Culture Religion and Belief Agriculture Electronics Visual model (Image as input) Lookup model (Symbol as input) CNN CNN CNN Softmax GRU 36 36
Figure 3: An illustration of two models, our proposed Visual model at the top and the baseline Lookup model at the bottom using the same RNN architecture. A string of characters (e.g. “温病 学”), each converted into a 36x36 image, serves as input of our Visual model. dc is the dimension of the character embedding for the Lookup model.
article in the tree (e.g., if an article is two steps away from Sports and three steps away from Arts, it will receive the “Sports” label). We also perform some rudimentary filtering, removing pages that match the regular expression “.*:.*”, which catches special pages such as “title:agriculture”.
2.2 Statistics
For Chinese, Japanese, and Korean, respectively, the number of articles is 593k/810k/46.6k, and the average length and standard deviation of the title is 6.25±3.96/8.60±5.58/6.10±3.71. As shown in Fig. 2, the character rank-frequency distributions of all three languages follows the 80/20 rule (Newman, 2005) (i.e., top 20% ranked characters that appear more than 80% of total frequencies), demonstrating that the characters in these languages belong to a long tail distribution. We further split the dataset into training, validation, and testing sets with a 6:2:2 ratio. The category distribution for each language can be seen in Tab. 1. Chinese has two varieties of characters, traditional and simplified, and the dataset is a mix of the two. Hence, we transform this dataset into two separate sets, one completely simplified and the other completely traditional using the Chinese text converter provided with Mac OS.
3 Model
Our overall model for the classification task follows the encoder model by Sutskever et al. (2014). We calculate character representations, use a RNN
4
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
Layer# 3-layer CNN Configuration 1 Spatial Convolution (3, 3) → 32 2 ReLu 3 MaxPool (2, 2) 4 Spatial Convolution (3, 3) → 32 5 ReLu 6 MaxPool (2, 2) 7 Spatial Convolution (3, 3) → 32 8 ReLu 9 Linear (800, 128) 10 ReLu 11 Linear (128, 128) 12 ReLu
Table 2: Architecture of the CNN used in the experiments. All the convolutional layers have 32 3×3 filters.
to combine the character representations into a sentence representation, and then add a softmax layer after that to predict the probability for each class. As shown in Fig. 3, the baseline model, which we call it the Lookup model, calculates the representation for each character by looking it up in a character embedding matrix. Our proposed model, the Visual model instead learns the representation of each character from its visual appearance via CNN.
Lookupmodel Given a character vocabularyC, for the Lookup model as in the bottom part of Fig. 3, the input to the network is a stream of characters c1, c2, ...cN , where cn ∈ C. Each character is represented by a 1-of-|C| (one-hot) encoding. This one-hot vector is thenmultiplied by the lookupmatrix TC ∈ R|C|×dc , where dc is the dimension of the character embedding.
Visual model The proposed method aims to learn a representation that includes image information, allowing for better parameter sharing among characters, particularly characters that are less common. Different from the Lookup model, each character is first transformed into a 36-by-36 image based on its Unicode encoding as shown in the upper part of Fig 3. We then pass the image through a CNN to get the embedding for the image. The parameters for the CNN are learned through backpropagation from the classification loss. Because we are training embeddings based on this classification loss, we expect that the CNN will focus on parts of the image that contain semantic information useful for category classification, a hypothe-
sis that we examine in the experiments (see Section 5.5). In more detail, the specific structure of the CNN that we utilize consists of three convolution layers where each convolution layer is followed by the max pooling and ReLU nonlinear activation layers. The configurations of each layer are listed in Tab. 2. The output vector for the image embeddings also has size dc which is the same as the Lookupmodel.
Encoder and Classifier For both the Lookup and the Visual models, we adopt an RNN encoder usingGated Recurrent Units (GRUs) (Chung et al., 2014). Each of the GRU units processes the character embeddings sequentially. At the end of the sequence, the incremental GRU computation results in a hidden state e embedding the sentence. The encoded sentence embedding is passed through a linear layer whose output is the same size as the number of classes. We use a softmax layer to compute the posterior class probabilities:
P (y = j|e) = exp(wTj e+ bj)∑L i=1 exp(wTi e+ bj)
(1)
To train themodel, we use cross-entropy loss between predicted and true targets:
J = 1
B B∑ i=1 L∑ j=1 −ti,j log(pi,j) (2)
where ti,j ∈ {0, 1} represents the ground truth label of the j-th class in the i-thWikipedia page title. B is the batch size and L is the number of categories.
4 Fusion-based Models
One thing to note is that the Lookup and the Visual models have their own advantages. The Lookup model learns embedding that captures the semantics of each character symbol without sharing information with each other. On the contrary, the proposed Visual model directly learns embedding from visual information, which naturally shares information between visually similar characters. This characteristic gives the Visual model the ability to generalize better to rare characters, but also has the potential disadvantage of introducing noise for characters with similar appearances but different meanings. With the complementary nature of these two models in mind, we further combine the two embeddings to achieve better performances. We adopt
5
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
458
459
460
461
462
463
464
465
466
467
468
469
470
471
472
473
474
475
476
477
478
479
480
481
482
483
484
485
486
487
488
489
490
491
492
493
494
495
496
497
498
499
Lookup/Visual 100% 50% 12.5% zh_trad 0.55/0.54 0.53/0.50 0.48/0.47 zh_simp 0.55/0.54 0.53/0.52 0.48/0.46 ja 0.42/0.39 0.47/0.45 0.44/0.41 ko 0.47/0.42 0.44/0.39 0.37/0.36
Table 3: The classification results of the Lookup / Visual models for different percentages of full training size.
three fusion schemes, early fusion, late fusion (described by Snoek et al. (2005) and Karpathy et al. (2014)), and fallback fusion, a method specific to this paper.
Early Fusion Early fusion works by concatenating the two varieties of embeddings before feeding them into the RNN. In order to ensure that the dimensions of the RNN are the same after concatenation, the concatenated vector is fed through a hidden layer to reduce the size from 2× dc to dc. The whole model is then fine-tuned with training data.
Late Fusion Instead of learning a joint representation like early fusion, late fusion averages the model predictions. Specifically, it takes the output of the softmax layers from both models and averages the probabilities to create a final distribution used to make the prediction.
Fallback Fusion Our final fallback fusion method hypothesizes that our Visual model does better with instances which contain more rare characters. First, in order to quantify the overall rareness of an instance consisting of multiple characters, we calculate the average training set frequency of the characters therein. The fallback fusion method uses the Visual model to predict testing instances with average character frequency below or equal to a threshold (here we use 0.0 frequency as cutoff, which means all characters in the instance do not appear in the training set), and uses the Lookup model to predict the rest of the instances.
5 Experiments and Results
In this section, we compare our proposed Visual model with the baseline Lookup model through three different sets of experiments. First, we examine whether our model is capable of classifying text and achieving similar performance as the baseline model. Next, we examine the hypothesis that our model will outperform the baseline model
Lookup/Visual 100 1000 10000 zh_trad 0.22/0.49 0.35/0.35 0.40/0.39 zh_simp 0.25/0.53 0.39/0.37 0.41/0.40 ja 0.30/0.35 0.45/0.41 0.44/0.41 ko 0.44/0.33 0.44/0.33 0.48/0.42
Table 4: Classification results for the Lookup / Visual of the k lowest frequency instances across four datasets.
when dealing with low frequency characters. Finally, we examine the fusion methods described in Section 4.
5.1 Experimental Configurations
The dimension of the embeddings and batch size for both models are set to dc = 128 and B = 400, respectively. We build our proposed model using Torch (Collobert et al., 2002), and use Adam (Kingma and Ba, 2014) with a learning rate η = 0.001 for stochastic optimization. The length of each instance is cut off or padded to 10 characters for batch training.
5.2 Comparison with the baseline model
In this experiment, we examine whether our Visual model achieves similar performance with the baseline Lookup model in classification accuracy. The results in Tab. 3 show that the baseline model performs 1-2% better across four datasets; this is due to the fact that the Lookupmodel can directly learn character embeddings that capture the semantics of each character symbol for frequent characters. On the contrary, the Visual model learns embeddings from visual information, which constraints characters that has similar appearance to have similar embeddings. This is an advantage for rare characters, but a disadvantage for high frequency characters because being similar in appearance does not always lead to similar semantics. To demonstrate that this is in fact the case, besides looking at the overall classification accuracy, we also examine the performance on classifying low frequency instances which are sorted according to the average training set frequency of the characters therein. Tab. 4 and Fig. 4 (blue lines) both show that our model performs better in the 100 lowest frequency instances (the intersection point of the two models); this confirms that the Visual model can share visual information among characters and help to classify low frequency instances.
6
501
502
503
504
505
506
507
508
509
510
511
512
513
514
515
516
517
518
519
520
521
522
523
524
525
526
527
528
529
530
531
532
533
534
535
536
537
538
539
540
541
542
543
544
545
546
547
548
549
550
551
552
553
554
555
556
557
558
559
560
561
562
563
564
565
566
567
568
569
570
571
572
573
574
575
576
577
578
579
580
581
582
583
584
585
586
587
588
589
590
591
592
593
594
595
596
597
598
599
101 102 103 100
101
102
103
102 103 100
101
102
103 102 103
100
101
102
103
102 103 100
101
102
103
A cc
um ul
at ed
N um
be r o f C or re ct ly P re di ct ed In st an
ce s
Rank
(a) (b)
(c) (d)
⎯⎯!Visual,(TP(=(100% ⎯⎯!Visual,(TP(=(50% ⎯⎯!Visual,(TP(=(12.5% ⎯!⎯!Lookup,(TP(=(100% ⎯!⎯!Lookup,(TP(=(50% ⎯!⎯!Lookup,(TP(=(12.5%
⎯⎯!Visual,(TP(=(100% ⎯⎯!Visual,(TP(=(50% ⎯⎯!Visual,(TP(=(12.5% ⎯!⎯!Lookup,(TP(=(100% ⎯!⎯!Lookup,(TP(=(50% ⎯!⎯!Lookup,(TP(=(12.5%
⎯⎯!Visual,(TP(=(100% ⎯⎯!Visual,(TP(=(50% ⎯⎯!Visual,(TP(=(12.5% ⎯!⎯!Lookup,(TP(=(100% ⎯!⎯!Lookup,(TP(=(50% ⎯!⎯!Lookup,(TP(=(12.5% ⎯⎯!Visual,(TP(=(100% ⎯⎯!Visual,(TP(=(50% ⎯⎯!Visual,(TP(=(12.5% ⎯!⎯!Lookup,(TP(=(100% ⎯!⎯!Lookup,(TP(=(50% ⎯!⎯!Lookup,(TP(=(12.5%
Figure 4: Experiments on different training sizes for four different datasets. More specifically, we consider three different training data size percentages (TPs) (100%, 50%, and 12.5%) and four datasets: (a) traditional Chinese, (b) simplified Chinese, (c) Japanese, and (d) Korean. We calculate the accumulated number of correctly predicted instances for the Visualmodel (solid lines) and the Lookup model (dashed lines).
5.3 Experiments on different training data sizes
In our second experiment, we consider two smaller training sizes (i.e., 50% and 12.5% of the full training size) indicated by green and red lines in Fig. 4. We performed this experiment under the hypothesis that because the proposed method was more robust to infrequent characters, the proposed model may perform better in low-resourced scenarios. If this is the case, the intersection point of the two models will shift right because of the increase of the number of instances with low average character frequency. As we can see in Fig. 4, the intersection point for 100% training data lies between the intersection point for 50% training data and 12.5%. This disagrees with our hypothesis; this is likely because while the number of low-frequency characters increases, smaller amounts of data also adversely impact the ability of CNN to learn useful visual features, and thus there is not a clear gain nor loss when using the proposed method. As a more extreme test of the ability of our proposed framework to deal with the unseen characters in the test set, we use traditional Chinese as our training data and simplified Chinese as our testing
zh_trad zh_simp ja ko Lookup 0.5503 0.5543 0.4914 0.4765 Visual 0.5434 0.5403 0.4775 0.4207 early 0.5520 0.5546 0.4896 0.4796 late 0.5658 0.5685 0.5029 0.4869 fall 0.5507 0.5547 0.4914 0.4766
Table 5: Experiment results for three different fusion methods across 4 datasets.
data. The model was able to achieve around 40% classification accuracy when we use the full training set, compared to 55%, which is achieved by the model trained on simplified Chinese. This result demonstrates that the model is able to transfer between similar scripts, similarly to howmost Chinese speakers can guess the meaning of the text, even if it is written in the other script.
5.4 Experiment on Different Fusion Methods
Results of different fusion methods can be found in Tab. 5. The results show that late fusion gives the best performance among all the fusion schemes combining the Lookup model and the proposed Visual model. Early fusion achieves small improvements for all languages except Japanese, where it displays a slight drop. Unsurprisingly,
7
601
602
603
604
605
606
607
608
609
610
611
612
613
614
615
616
617
618
619
620
621
622
623
624
625
626
627
628
629
630
631
632
633
634
635
636
637
638
639
640
641
642
643
644
645
646
647
648
649
650
651
652
653
654
655
656
657
658
659
660
661
662
663
664
665
666
667
668
669
670
671
672
673
674
675
676
677
678
679
680
681
682
683
684
685
686
687
688
689
690
691
692
693
694
695
696
697
698
699
Iron Bronze Salmon Serranidae
Silk Coil Rhyme Pleased
Wave Put on Cypress Pillar
Cuckoo Eagle Mosquito Ant
Figure 5: Examples of how much each part of the character contributes to its embedding (the darker the more). Two characters are shown per radical to emphasize that characters with same radical have similar patterns.
fallback fusion performs better than the Lookup model and the Visual model alone, since it directly targets the weakness of the Lookup model (e.g., rare characters) and replaces the results with the Visual model. These results show that simple integration, no matter which schemes we use, is beneficial, demonstrating that both methods are capturing complementary information.
5.5 Visualization of Character Embeddings
Finally, we qualitatively examine what is learned by our proposed model in two ways. First, we visualize which parts of the image are most important to the Visual model’s embedding calculation. Second, we show the 6-nearest neighbor results for characters using both the Lookup and the Visual embeddings.
Emphasis of the Visual Model In order to delve deeper into what the Visual model has learned, we measure a modified version of the occlusion sensitivity proposed by Zeiler and Fergus (2014) by masking the original character image in four ways, and examine the importance of each part of the character to the model’s calculated representations. Specifically, we leave only the upper half, bottom half, left half, or right half of the image, and mask the remainder with white pixels since Chinese characters are usually formed by combining two radicals vertically or horizontally. We run these four images forward through the CNN part of the model and calculate the L2 distance between
! !
! ! !
!
Visual'model Lookup'model Visual'model Lookup'model
Figure 6: Visualization of the Chinese traditional characters by finding the 6-nearest neighbors of the query (i.e., center) characters. The highlighted red indicates the radical along with the meaning of the characters.
the masked image embeddings with the full image embedding. The larger the distance, the more the masked part of the character contributes to the original embedding. The contribution of each part (e.g. the L2 distance) is represented as a heat map, and then it is normalized to adjust the opacity of the character strokes for better visualization. The value of each corner of the heatmap is calculated by adding the two L2 distances that contribute to this corner. The visualization is shown in Fig. 5. The meaning of each Chinese character in English is shown below the Chinese character. The opacity of the character strokes represent how much the corresponding parts contribute to the original embedding (the darker the more). In general, the darker part of the character is related to its semantics. For example, “金” means gold in Chinese, which is highlighted in both “鐵” (Iron) and “銅” (Bronze). We can also find similar results for other examples shown in Fig. 5.
K-nearest neighbors Finally, to illustrate the difference of the learned embeddings between the two models, we display 6-nearest neighbors (L2 distance) for selected characters in Fig. 6. As can be seen, the Visual embedding for characters with similar appearances are close to each other. In addition, similarity in the radical part indicates semantic similarity between the characters. For example, the characters with radical “鳥” all refer to different type of birds. The Lookup embedding do not show such feature, as it learns the embedding individually for each symbol and relies heavily on the training set and the task. In fact, the characters shown in Fig. 6
8
701
702
703
704
705
706
707
708
709
710
711
712
713
714
715
716
717
718
719
720
721
722
723
724
725
726
727
728
729
730
731
732
733
734
735
736
737
738
739
740
741
742
743
744
745
746
747
748
749
750
751
752
753
754
755
756
757
758
759
760
761
762
763
764
765
766
767
768
769
770
771
772
773
774
775
776
777
778
779
780
781
782
783
784
785
786
787
788
789
790
791
792
793
794
795
796
797
798
799
for the Lookupmodel do not exhibit semantic similarity either. There are two potential explanations for this: First, the category classification task that we utilized do not rely heavily on the finegrained semantics of each character, and thus the Lookup model was able to perform well without exactly capturing the semantics of each character precisely. Second, the Wikipedia dataset contains a large number of names and location and the characters therein might not have the same semantic meaning used in daily vocabulary.
6 Related Work
Methods that utilize neural networks to learn distributed representations of words or characters have been widely developed. However, word2vec (Mikolov et al., 2013), for example, requires storing an extremely large table of vectors for all word types. For example, due to the size of word types in twitter tweets, work has been done to generate vector representations of tweets at character-level (Dhingra et al., 2016).
There is also work done in understanding mathematical expressions with a convolutional network for text and layout recognition by using an attention-based neural machine translation system (Deng et al., 2016). They tested on realworld rendered mathematical expressions paired with LaTeX markup and show the system is effective at generating accurate markup. Other than that, there are several works that combine visual information with text in improving machine translation (Sutskever et al., 2014), visual question answering, caption generation (Xu et al., 2015), etc. These works extract image representations from a pre-trained CNN (Zhu et al., 2016; Wang et al., 2016). Unrelated to images, CNNs have also been used for text classification (Kim, 2014; Zhang et al., 2015). These models look at the sequential dependencies at the word or character-level and achieve the state-of-the-art results. These works inspire us to use CNN to extract features from image and serve as the input to the RNN. Our model is able to directly back-propagate the gradient all the way through the CNN, which generates visual embeddings, in a way such that the embedding can contain both semantic and visual information. Several techniques for reducing the rare words effects have been introduced in the literature, including spelling expansion (Habash, 2008), dictio-
nary term expansion (Habash, 2008), proper name transliteration (Daumé and Jagarlamudi, 2011), treating words as a sequence of characters (Luong and Manning, 2016), subword units (Sennrich et al., 2015), and reading text as bytes (Gillick et al., 2015). However, most of these techniques still have no mechanism for handling low frequency characters, which are the target of this work. Finally, there is one work by Shi et al. (2015) on “radical embedding”, which explicitly splits Chinese characters into radicals based on a dictionary of what radicals are included in which characters. The motivation of this method is similar to ours, but is only applicable to Chinese, in contrast to the method in this paper, which works on any language for which we can render text.
7 Conclusion and Future Work
In this paper, we proposed a new framework that utilizes appearance of characters, convolutional neural networks, recurrent neural networks to learn embeddings that are compositional in the component parts of the characters. More specifically, we collected a Wikipedia dataset, which consists of short titles of three different languages and satisfies the compositionality in the characters of the language. Next, we proposed an end-to-end model that learns visual embeddings for characters using CNN and showed that the features extracted from the CNN include both visual and semantic information. Furthermore, we showed that our Visual model outperforms the Lookup baseline model in low frequency instances. Additionally, by examining the character embeddings visually, we found that our Visual model is able to learn visually related embeddings. In summary, we tackled the problem of rare characters by using embeddings learned from images. In the future, we hope to further generalize this method to other tasks such as pronunciation estimation, which can take advantage of the fact that pronunciation information is encoded in parts of the characters as demonstrated in Fig. 1, or machine translation, which could benefit from a wholistic view that considers both semantics and pronunciation. We also hope to apply the model to other languageswith complicated compositional writing systems, potentially including historical texts such as hieroglyphics or cuneiform.
9
801
802
803
804
805
806
807
808
809
810
811
812
813
814
815
816
817
818
819
820
821
822
823
824
825
826
827
828
829
830
831
832
833
834
835
836
837
838
839
840
841
842
843
844
845
846
847
848
849
850
851
852
853
854
855
856
857
858
859
860
861
862
863
864
865
866
867
868
869
870
871
872
873
874
875
876
877
878
879
880
881
882
883
884
885
886
887
888
889
890
891
892
893
894
895
896
897
898
899

- Update after rebuttal  I appreciate the authors taking the time to clarify their implementation of the baseline and to provide some evidence of the significance of the improvements they report. These clarifications should definitely be included in the camera-ready version. I very much like the idea of using visual features for these languages, and I am looking forward to seeing how they help more difficult tasks in future work.  - Strengths:  - Thinking about Chinese Japanese Korean characters visually is a great idea!  - Weaknesses:  - Experimental results show only incremental improvement over baseline, and the choice of evaluation makes it hard to verify one of the central arguments: that visual features improve performance when processing rare unseen words.  - Some details about the baseline are missing, which makes it difficult to interpret the results, and would make it hard to reproduce the work.  - General Discussion:  The paper proposes the use of computer vision techniques (CNNs applied to images of text) to improve language processing for Chinese, Japanese, and Korean, languages in which characters themselves might be compositional. The authors evaluate their model on a simple text-classification task (assigning Wikipedia page titles to categories). They show that a simple one-hot representation of the characters outperforms the CNN-based representations, but that the combination of the visual representations with standard one-hot encodings performs better than the visual or the one-hot alone. They also present some evidence that the visual features outperform the one-hot encoding on rare words, and present some intuitive qualitative results suggesting the CNN learns good semantic embeddings of the characters.  I think the idea of processing languages like Chinese and Japanese visually is a great one, and the motivation for this paper makes a lot of sense. However, I am not entirely convinced by the experimental results. The evaluations are quite weak, and it is hard to say whether these results are robust or simply coincidental. I would prefer to see some more rigorous evaluation to make the paper publication-ready. If the results are statistically significant (if the authors can indicate this in the author response), I would support accepting the paper, but ideally, I would prefer to see a different evaluation entirely.  More specific comments below:  - In Section 3, paragraph  lookup model , you never explicitly say which embeddings you use, or whether they are tuned via backprop the way the visual embeddings are. You should be more clear about how the baseline was implemented. If the baseline was not tuned in a task-specific way, but the visual embeddings were, this is even more concerning since it makes the performances substantially less comparable.  - I don't entirely understand why you chose to evaluate on classifying wikipedia page titles. It seems that the only real argument for using the visual model is its ability to generalize to rare unseen characters. Why not focus on this task directly? E.g. what about evaluating on machine translation of OOV words? I agree with you that some languages should be conceptualized visually, and sub-character composition is important, but the evaluation you use does not highlight weaknesses of the standard approach, and so it does not make a good case for why we need the visual features.   - In Table 5, are these improvements statistically significant?  - It might be my fault, but I found Figure 4 very difficult to understand. Since this is one of your main results, you probably want to present it more clearly, so that the contribution of your model is very obvious. As I understand it,  rank  on the x axis is a measure of how rare the word is (I think log frequency?), with the rarest word furthest to the left? And since the visual model intersects the x axis to the left of the lookup model, this means the visual model was  better  at ranking rare words? Why don't both models intersect at the same point on the x axis, aren't they being evaluated on the same set of titles and trained with the same data? In the author response, it would be helpful if you could summarize the information this figure is supposed to show, in a more concise way.   - On the fallback fusion, why not show performance for for different thresholds? 0 seems to be an edge-case threshold that might not be representative of the technique more generally.  - The simple traditional experiment for unseen characters is a nice idea, but is presented as an afterthought. I would have liked to see more eval in this direction, i.e. on classifying unseen words  - Maybe add translations to Figure 6, for people who do not speak Chinese?