Semi-supervised sequence tagging with bidirectional language models
Pre-trained word embeddings learned from unlabeled text have become a standard component of neural network architectures for NLP tasks. However, in most cases, the recurrent network that operates on word-level representations to produce context sensitive representations is trained on relatively little labeled data. In this paper, we demonstrate a general semi-supervised approach for adding pretrained context embeddings from bidirectional language models to NLP systems and apply it to sequence labeling tasks. We evaluate our model on two standard datasets for named entity recognition (NER) and chunking, and in both cases achieve state of the art results, surpassing previous systems that use other forms of transfer or joint learning with additional labeled data and task specific gazetteers.
1 Introduction
Due to their simplicity and efficacy, pre-trained word embedding have become ubiquitous in NLP systems. Many prior studies have shown that they capture useful semantic and syntactic information (Mikolov et al., 2013; Pennington et al., 2014) and including them in NLP systems has been shown to be enormously helpful for a variety of downstream tasks (Collobert et al., 2011).
However, in many NLP tasks it is essential to represent not just the meaning of a word, but also the word in context. For example, in the two phrases “A Central Bank spokesman” and “The Central African Republic”, the word ‘Central’ is used as part of both an Organization and Location. Accordingly, current state of the art sequence tagging models typically include a bidirectional re-
current neural network (RNN) that encodes token sequences into a context sensitive representation before making token specific predictions (Yang et al., 2017; Ma and Hovy, 2016; Lample et al., 2016; Hashimoto et al., 2016).
Although the token representation is initialized with pre-trained embeddings, the parameters of the bidirectional RNN are typically learned only on labeled data. Previous work has explored methods for jointly learning the bidirectional RNN with supplemental labeled data from other tasks (e.g., Søgaard and Goldberg, 2016; Yang et al., 2017).
In this paper, we explore an alternate semisupervised approach which does not require additional labeled data. We use a neural language model (LM) pre-trained on a large, unlabeled corpus to compute an encoding of the context at each position in the sequence (hereafter an LM embedding) and use it in the supervised sequence tagging model. Since the LM embeddings are used to compute the probability of future words in a neural LM, they are likely to encode both the semantic and syntactic roles of words in context.
Our main contribution is to show that the context sensitive representation captured in the LM embeddings is useful in the supervised sequence tagging setting. When we include the LM embeddings in our system overall performance increases from 90.87% to 91.93% F1 for the CoNLL 2003 NER task, a more then 1% absolute F1 increase, and a substantial improvement over the previous state of the art. We also establish a new state of the art result (96.37% F1) for the CoNLL 2000 Chunking task.
As a secondary contribution, we show that using both forward and backward LM embeddings boosts performance over a forward only LM. We also demonstrate that domain specific pre-training is not necessary by applying a LM trained in the news domain to scientific papers.
2
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
2 Language model augmented sequence taggers (TagLM)
2.1 Overview
The main components in our language-modelaugmented sequence tagger (TagLM) are illustrated in Fig. 1. After pre-training word embeddings and a neural LM on large, unlabeled corpora (Step 1), we extract the word and LM embeddings for every token in a given input sequence (Step 2) and use them in the supervised sequence tagging model (Step 3).
2.2 Baseline sequence tagging model
Our baseline sequence tagging model is a hierarchical neural tagging model, closely following a number of recent studies (Ma and Hovy, 2016; Lample et al., 2016; Yang et al., 2017; Chiu and Nichols, 2016) (left side of Figure 2).
Given a sentence of tokens (t1, t2, . . . , tN ) it first forms a representation, xk, for each token by concatenating a character based representation ck with a token embedding wk:
ck = C(tk; θc)
wk = E(tk; θw)
xk = [ck;wk] (1)
The character representation ck captures morphological information and is either a convolutional neural network (CNN) (Ma and Hovy, 2016; Chiu and Nichols, 2016) or RNN (Yang et al., 2017; Lample et al., 2016). It is parameterized by C(·, θc) with parameters θc. The token embeddings, wk, are obtained as a lookup E(·, θw), initialized using pre-trained word embeddings, and fine tuned during training (Collobert et al., 2011).
To learn a context sensitive representation, we employ multiple layers of bidirectional RNNs. For each token position, k, the hidden state hk,i of RNN layer i is formed by concatenating the hidden states from the forward ( −→ h k,i) and backward ( ←− h k,i) RNNs. As a result, the bidirectional RNN is able to use both past and future information to make a prediction at token k. More formally, for the first RNN layer that operates on xk to output hk,1:
−→ h k,1 = −→ R 1(xk, −→ h k−1,1; θ−→R1 ) ←− h k,1 = ←− R 1(xk, ←− h k+1,1; θ←−R1 )
hk,1 = [ −→ h k,1; ←− h k,1] (2)
The second RNN layer is similar and uses hk,1 to output hk,2. In this paper, we use L = 2 layers of RNNs in all experiments and parameterize Ri as either Gated Recurrent Units (GRU) (Cho et al., 2014) or Long Short-Term Memory units (LSTM) (Hochreiter and Schmidhuber, 1997) depending on the task.
Finally, the output of the final RNN layer hk,L is used to predict a score for each possible tag using a single dense layer. Due to the dependencies between successive tags in our sequence labeling tasks (e.g. using the BIOES labeling scheme, it is not possible for I-PER to follow B-LOC), it is beneficial to model and decode each sentence jointly instead of independently predicting the label for each token. Accordingly, we add another layer with parameters for each label bigram, computing the sentence conditional random field (CRF) loss (Lafferty et al., 2001) using the forward-backward algorithm at training time, and using the Viterbi algorithm to find the most likely tag sequence at test time, similar to Collobert et al. (2011).
3
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
New York is located ...
Neural net
Char CNN/ RNN
Embedding
Token embedding
RNNDense
E-LOCB-LOC CRF
bi-RNN (R2)
Token representation
New York is located ...
Forward LM
Backward LM
h1 LM
Concat LM embedding
Sequence tagging
Pre-trained bi-LM
bi-RNN (R1)
Sequence representation
Concatenation
Token representation
New York is located ...
Token representation
h1,1 h2 LMh2,1
h1,2 h2,2
Figure 2: Overview of TagLM, our language model augmented sequence tagging architecture. The top level embeddings from a pre-trained bidirectional LM are inserted in a stacked bidirectional RNN sequence tagging model. See text for details.
2.3 Bidirectional LM
A language model computes the probability of a token sequence (t1, t2, . . . , tN )
p(t1, t2, . . . , tN ) = N∏ k=1 p(tk | t1, t2, . . . , tk−1).
Recent state of the art neural language models (Józefowicz et al., 2016) use a similar architecture to our baseline sequence tagger where they pass a token representation (either from a CNN over characters or as token embeddings) through multiple layers of LSTMs to embed the history (t1, t2, . . . , tk) into a fixed dimensional vector−→ h LMk . This is the forward LM embedding of the token at position k and is the output of the top LSTM layer in the language model. Finally, the language model predicts the probability of token tk+1 using a softmax layer over words in the vocabulary.
The need to capture future context in the LM embeddings suggests it is beneficial to also consider a backward LM in additional to the traditional forward LM. A backward LM predicts the previous token given the future context. Given a sentence with N tokens, it computes p(t1, t2, . . . , tN ) = N∏ k=1 p(tk | tk+1, tk+2, . . . , tN ).
A backward LM can be implemented in an analogous way to a forward LM and produces the backward LM embedding ←− h LMk , for the sequence (tk, tk+1, . . . , tN ), the output embeddings of the top layer LSTM.
In our final system, after pre-training the forward and backward LMs separately, we remove the top layer softmax and concatenate the forward and backward LM embeddings to form bidirectional LM embeddings, i.e., hLMk = [ −→ h LMk ; ←− h LMk ]. Note that in our formulation, the forward and backward LMs are independent, without any shared parameters.
2.4 Combining LM with sequence model
Our combined system, TagLM, uses the LM embeddings as additional inputs to the sequence tagging model. In particular, we concatenate the LM embeddings hLM with the output from one of the bidirectional RNN layers in the sequence model. In our experiments, we found that introducing the LM embeddings at the output of the first layer performed the best. More formally, we simply replace (2) with
hk,1 = [ −→ h k,1; ←− h k,1;h LM k ]. (3)
4
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
3 Experiments
We evaluate our approach on two well benchmarked sequence tagging tasks, the CoNLL 2003 NER task (Sang and Meulder, 2003) and the CoNLL 2000 Chunking task (Sang and Buchholz, 2000). We report the official evaluation metric (micro-averaged F1). In both cases, we use the BIOES labeling scheme for the output tags, following previous work which showed it outperforms other options (e.g., Ratinov and Roth, 2009). Following Chiu and Nichols (2016), we use the Senna word embeddings (Collobert et al., 2011) and pre-processed the text by lowercasing all tokens and replacing all digits with 0.
3.0.1 CoNLL 2003 NER
The CoNLL 2003 NER task consists of newswire from the Reuters RCV1 corpus tagged with four different entity types (PER, LOC, ORG, MISC). It includes standard train, development and test sets. Following previous work (Yang et al., 2017; Chiu and Nichols, 2016) we trained on both the train and development sets after tuning hyperparameters on the development set.
The hyperparameters for our baseline model are similar to Yang et al. (2017). We use two bidirectional GRUs with 80 hidden units and 25 dimensional character embeddings for the token character encoder. The sequence layer uses two bidirectional GRUs with 300 hidden units each. For regularization, we add 25% dropout to the input of each GRU, but not to the recurrent connections.
3.0.2 CoNLL 2000 chunking
The CoNLL 2000 chunking task uses sections 15- 18 from the Wall Street Journal corpus for training and section 20 for testing. It defines 11 syntactic chunk types (e.g., NP, VP, ADJP) in addition to other. We randomly sampled 1000 sentences from the training set as a held-out development set.
The baseline sequence tagger uses 30 dimensional character embeddings and a CNN with 30 filters of width 3 characters followed by a tanh non-linearity for the token character encoder. The sequence layer uses two bidirectional LSTMs with 200 hidden units. Following Ma and Hovy (2016) we added 50% dropout to the character embeddings, the input to each LSTM layer (but not recurrent connections) and to the output of the final LSTM layer.
3.0.3 Pre-trained language models The primary bidirectional LMs we used in this study were trained on the 1B Word Benchmark (Chelba et al., 2014), a publicly available benchmark for large-scale language modeling. The training split has approximately 800 million tokens, about a 4000X increase over the number training tokens in the CoNLL datasets. Józefowicz et al. (2016) explored several model architectures and released their best single model and training recipes. Following Sak et al. (2014), they used linear projection layers at the output of each LSTM layer to reduce the computation time but still maintain a large LSTM state. Their single best model took three weeks to train on 32 GPUs and achieved 30.0 test perplexity. It uses a character CNN with 4096 filters for input, followed by two stacked LSTMs, each with 8192 hidden units and a 1024 dimensional projection layer. We use CNN-BIG-LSTM to refer to this language model in our results.
In addition to CNN-BIG-LSTM from Józefowicz et al. (2016),1 we used the same corpus to train two additional language models with fewer parameters: forward LSTM-2048-512 and backward LSTM-2048-512. Both language models use token embeddings as input to a single layer LSTM with 2048 units and a 512 dimension projection layer. We closely followed the procedure outlined in Józefowicz et al. (2016), except we used synchronous parameter updates across four GPUs instead of asynchronous updates across 32 GPUs and ended training after 10 epochs. The test set perplexities for our forward and backward LSTM-2048-512 language models are 47.7 and 47.3, respectively.2
3.0.4 Training All experiments use the Adam optimizer (Kingma and Ba, 2014) with gradient norms clipped at 5.0. In addition to explicit dropout regularization, we also use early stopping to prevent over-fitting and use the following process to determine when to stop training. We first train with a constant learning rate α = 0.001 on the training data and monitor the development set performance at each epoch. Then, at the epoch with the highest de-
1https://github.com/tensorflow/models/ tree/master/lm_1b
2Due to different implementations, the perplexity of the forward LM with similar configurations in Józefowicz et al. (2016) is different (45.0 vs. 47.7).
5
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
458
459
460
461
462
463
464
465
466
467
468
469
470
471
472
473
474
475
476
477
478
479
480
481
482
483
484
485
486
487
488
489
490
491
492
493
494
495
496
497
498
499
Model F1± std Chiu and Nichols (2016) 90.91± 0.20 Lample et al. (2016) 90.94 Ma and Hovy (2016) 91.37 Our baseline without LM 90.87± 0.13 TagLM 91.93± 0.19
Table 1: Test set F1 comparison on CoNLL 2003 NER task, using only CoNLL 2003 data and unlabeled text.
Model F1± std Yang et al. (2017) 94.66 Hashimoto et al. (2016) 95.02 Søgaard and Goldberg (2016) 95.28 Our baseline without LM 95.00± 0.08 TagLM 96.37± 0.05
Table 2: Test set F1 comparison on CoNLL 2000 Chunking task using only CoNLL 2000 data and unlabeled text.
velopment performance, we start a simple learning rate annealing schedule: decrease α an order of magnitude (i.e., divide by ten), train for five epochs, decrease α an order of magnitude again, train for five more epochs and stop.
Following Chiu and Nichols (2016), we train each final model configuration ten times with different random seeds and report the mean and standard deviation F1. It is important to estimate the variance of model performance since the test data sets are relatively small.
3.1 Overall system results
Tables 1 and 2 compare results from TagLM with previously published state of the art results without additional labeled data or task specific gazetteers. Tables 3 and 4 compare results of TagLM to other systems that include additional labeled data or gazetteers. In both tasks, TagLM establishes a new state of the art using bidirectional LMs (the forward CNN-BIG-LSTM and the backward LSTM-2048-512).
In the CoNLL 2003 NER task, our model scores 91.93 mean F1, which is a statistically significant increase over the previous best result of 91.62 ±0.33 from Chiu and Nichols (2016) that used gazetteers (at 95%, two-sided Welch t-test, p = 0.021).
In the CoNLL 2000 Chunking task, TagLM
achieves 96.37 mean F1, exceeding all previously published results without additional labeled data by more then 1% absolute F1. The improvement over the previous best result of 95.77 in Hashimoto et al. (2016) that jointly trains with Penn Treebank (PTB) POS tags is statistically significant at 95% (p < 0.001 assuming standard deviation of 0.1).
Importantly, the LM embeddings amounts to an average absolute improvement of 1.06 and 1.37 F1 in the NER and Chunking tasks, respectively.
Adding external resources. Although we do not use external labeled data or gazetteers, we found that TagLM outperforms previous state of the art results in both tasks when external resources (labeled data or task specific gazetteers) are available. Furthermore, Tables 3 and 4 show that, in most cases, the improvements we obtain by adding LM embeddings are larger then the improvements previously obtained by adding other forms of transfer or joint learning. For example, Yang et al. (2017) noted an improvement of only 0.06 F1 in the NER task when transfer learning from both CoNLL 2000 chunks and PTB POS tags and Chiu and Nichols (2016) reported an increase of 0.71 F1 when adding gazetteers to their baseline. In the Chunking task, previous work has reported from 0.28 to 0.75 improvement in F1 when including supervised labels from the PTB POS tags or CoNLL 2003 entities (Yang et al., 2017; Søgaard and Goldberg, 2016; Hashimoto et al., 2016).
3.2 Analysis
To elucidate the characteristics of our LM augmented sequence tagger, we ran a number of additional experiments on the CoNLL 2003 NER task.
How to use LM embeddings? In this experiment, we concatenate the LM embeddings at different locations in the baseline sequence tagger. In particular, we used the LM embeddings hLMk to:
• augment the input of the first RNN layer; i.e., xk = [ck;wk;h LM k ],
• augment the output of the first RNN layer; i.e., hk,1 = [ −→ h k,1; ←− h k,1;h LM k ], 3 and
• augment the output of the second RNN layer; i.e., hk,2 = [ −→ h k,2; ←− h k,2;h LM k ].
3This configuration the same as Eq. 3 in §2.4. It was reproduced here for convenience.
6
501
502
503
504
505
506
507
508
509
510
511
512
513
514
515
516
517
518
519
520
521
522
523
524
525
526
527
528
529
530
531
532
533
534
535
536
537
538
539
540
541
542
543
544
545
546
547
548
549
568
569
570
571
572
573
574
575
576
577
578
579
580
581
582
583
584
585
586
587
588
589
590
591
592
593
594
595
596
597
598
599
F1 F1 Model External resources Without With ∆ Yang et al. (2017) transfer from CoNLL 2000/PTB-POS 91.2 91.26 +0.06 Chiu and Nichols (2016) with gazetteers 90.91 91.62 +0.71 Collobert et al. (2011) with gazetteers 88.67 89.59 +0.92 Luo et al. (2015) joint with entity linking 89.9 91.2 +1.3 Ours no LM vs TagLM unlabeled data only 90.87 91.93 +1.06
F1 F1 Model External resources Without With ∆ Yang et al. (2017) transfer from CoNLL 2003/PTB-POS 94.66 95.41 +0.75 Hashimoto et al. (2016) jointly trained with PTB-POS 95.02 95.77 +0.75 Søgaard and Goldberg (2016) jointly trained with PTB-POS 95.28 95.56 +0.28 Ours no LM vs TagLM unlabeled data only 95.00 96.37 +1.37
Table 4: Improvements in test set F1 in CoNLL 2000 Chunking when including additional labeled data (except the case of TagLM where we do not use additional labeled data).
Use LM embeddings at F1± std input to the first RNN layer 91.55± 0.21 output of the first RNN layer 91.93± 0.19 output of the second RNN layer 91.72± 0.13
Table 5: Comparison of CoNLL-2003 test set F1 when the LM embeddings are included at different layers in the baseline tagger.
Table 5 shows that the second alternative performs best. We speculate that the second RNN layer in the sequence tagging model is able to capture interactions between task specific context as expressed in the first RNN layer and general context as expressed in the LM embeddings in a way that improves overall system performance. These results are consistent with Søgaard and Goldberg (2016) who found that chunking performance was sensitive to the level at which additional POS supervision was added.
Does it matter which language model to use? In this experiment, we compare six different configurations of the forward and backward language models (including the baseline model which does not use any language models). The results are reported in Table 6.
We find that adding backward LM embeddings consistently outperforms forward-only LM em-
beddings, with F1 improvements between 0.22 and 0.27%, even with the relatively small backward LSTM-2048-512 LM.
LM size is important, and replacing the forward LSTM-2048-512 with CNN-BIG-LSTM (test perplexities of 47.7 to 30.0 on 1B Word Benchmark) improves F1 by 0.26 - 0.31%, about as much as adding backward LM. Accordingly, we hypothesize (but have not tested) that replacing the backward LSTM-2048-512with a backward LM analogous to the CNN-BIG-LSTM would further improve performance.
To highlight the importance of including language models trained on a large scale data, we also experimented with training a language model on just the CoNLL 2003 training and development data. Due to the much smaller size of this data set, we decreased the model size to 512 hidden units with a 256 dimension projection and normalized tokens in the same manner as input to the sequence tagging model (lower-cased, with all digits replaced with 0). The test set perplexities for the forward and backward models (measured on the CoNLL 2003 test data) were 106.9 and 104.2, respectively. Including embeddings from these language models decreased performance slightly compared to the baseline system without any LM. This result supports the hypothesis that adding language models help because they learn composi-
7
601
602
603
604
605
606
607
608
609
610
611
612
613
614
615
616
617
618
619
620
621
622
623
624
625
626
627
628
629
630
631
632
633
634
635
636
637
638
639
640
641
642
643
644
645
646
647
648
649
650
651
652
653
654
655
656
657
658
659
660
661
662
663
664
665
666
667
668
669
670
671
672
673
674
675
676
677
678
679
680
681
682
683
684
685
686
687
688
689
690
691
692
693
694
695
696
697
698
699
Forward language model Backward language model LM perplexity F1± std Fwd Bwd — — N/A N/A 90.87± 0.13 LSTM-512-256∗ LSTM-512-256∗ 106.9 104.2 90.79± 0.15 LSTM-2048-512 — 47.7 N/A 91.40± 0.18 LSTM-2048-512 LSTM-2048-512 47.7 47.3 91.62± 0.23 CNN-BIG-LSTM — 30.0 N/A 91.66± 0.13 CNN-BIG-LSTM LSTM-2048-512 30.0 47.3 91.93± 0.19
Table 6: Comparison of CoNLL-2003 test set F1 for different language model combinations. All language models were trained and evaluated on the 1B Word Benchmark, except LSTM-512-256∗ which was trained and evaluated on the standard splits of the NER CoNLL 2003 dataset.
tion functions (i.e., the RNN parameters in the language model) from much larger data compared to the composition functions in the baseline tagger, which are only learned from labeled data.
Importance of task specific RNN. To understand the importance of including a task specific sequence RNN we ran an experiment that removed the task specific sequence RNN and used only the LM embeddings with a dense layer and CRF to predict output tags. In this setup, performance was very low, 88.17 F1, well below our baseline. This result confirms that the RNNs in the baseline tagger encode essential information which are not encoded in the LM embeddings. This is unsurprising since the RNNs in the baseline tagger are trained on labeled examples, unlike the RNN in the language model which is only trained on unlabeled examples.
Dataset size. A priori, we expect the addition of LM embeddings to be most beneficial in cases where the task specific annotated datasets are small. To test this hypothesis, we replicated the setup from Yang et al. (2017) that samples 1% of the CoNLL 2003 training set and compared the performance of TagLM to our baseline without LM. In this scenario, test F1 increased 3.35% (from 67.66 to 71.01%) compared to an increase of 1.06% F1 for a similar comparison with the full training dataset. The analogous increases in Yang et al. (2017) are 3.97% for cross-lingual transfer from CoNLL 2002 Spanish NER and 6.28% F1 for transfer from PTB POS tags. However, they found only a 0.06% F1 increase when using the full training data and transferring from both CoNLL 2000 chunks and PTB POS tags. Taken together, this suggests that for very small labeled training sets, transferring from other tasks yields
a large improvement, but this improvement almost disappears when the training data is large. On the other hand, our approach is less dependent on the training set size and significantly improves performance even with larger training sets.
Number of parameters. Our TagLM formulation increases the number of parameters in the second RNN layer R2 due to the increase in the input dimension h1 if all other hyperparameters are held constant. To confirm that this did not have a material impact on the results, we ran two additional experiments. In the first, we trained a system without a LM but increased the second RNN layer hidden dimension so that number of parameters was the same as in TagLM. In this case, performance decreased slightly (by 0.15% F1) compared to the baseline model, indicating that solely increasing parameters does not improve performance. In the second experiment, we decreased the hidden dimension of the second RNN layer in TagLM to give it the same number of parameters as the baseline no LM model. In this case, test F1 increased slightly to 92.00 ± 0.11 indicating that the additional parameters in TagLM are slightly hurting performance.4
Does the LM transfer across domains? One artifact of our evaluation framework is that both the labeled data in the chunking and NER tasks and the unlabeled text in the 1 Billion Word Benchmark used to train the bidirectional LMs are derived from news articles. To test the sensitivity to the LM training domain, we also applied TagLM with a LM trained on news articles to the SemEval 2017 Shared Task 10, ScienceIE.5 Scien-
4A similar experiment for the Chunking task did not improve F1 so this conclusion is task dependent.
5https://scienceie.github.io/
8
701
702
703
704
705
706
707
708
709
710
711
712
713
714
715
716
717
718
719
720
721
722
723
724
725
726
727
728
729
730
731
732
733
734
735
736
737
738
739
740
741
742
743
744
745
746
747
748
749
750
751
752
753
754
755
756
757
758
759
760
761
762
763
764
765
766
767
768
769
770
771
772
773
774
775
776
777
778
779
780
781
782
783
784
785
786
787
788
789
790
791
792
793
794
795
796
797
798
799
ceIE requires end-to-end joint entity and relationship extraction from scientific publications across three diverse fields (computer science, material sciences, and physics) and defines three broad entity types (Task, Material and Process). For this task, TagLM increased F1 on the development set by 4.12% (from 49.93 to to 54.05%) for entity extraction over our baseline without LM embeddings.6 We conclude that LM embeddings can improve the performance of a sequence tagger even when the data comes from a different domain.
4 Related work
Unlabeled data. TagLM was inspired by the widespread use of pre-trained word embeddings in supervised sequence tagging models. Besides pre-trained word embeddings, our method is most closely related to Li and McCallum (2005). Instead of using a LM, Li and McCallum (2005) uses a probabilistic generative model to infer contextsensitive latent variables for each token, which are then used as extra features in a supervised CRF tagger (Lafferty et al., 2001). Other semisupervised learning methods for structured prediction problems include co-training (Blum and Mitchell, 1998; Pierce and Cardie, 2001), expectation maximization (Nigam et al., 2000; Mohit and Hwa, 2005), structural learning (Ando and Zhang, 2005) and maximum discriminant functions (Suzuki et al., 2007; Suzuki and Isozaki, 2008). It is easy to combine TagLM with any of the above methods by including LM embeddings as additional features in the discriminative components of the model (except for expectation maximization). A detailed discussion of semisupervised learning methods in NLP can be found in (Søgaard, 2013).
LM embeddings are related to a class of methods (e.g., Le and Mikolov, 2014; Kiros et al., 2015; Hill et al., 2016) for learning sentence and document encoders from unlabeled data, which can be used for text classification and textual entailment among other tasks.
Neural language models. LMs have always been a critical component in statistical machine translation systems (Koehn, 2009). Recently, neural LMs (Bengio et al., 2003; Mikolov et al., 2010) have also been integrated in neural machine translation systems (e.g., Kalchbrenner and Blunsom,
6Test data set performance withheld to preserve anonymity and will be updated before publication.
2013; Devlin et al., 2014) to score candidate translations. In contrast, TagLM uses neural LMs to encode words in the input sequence.
Unlike forward LMs, bidirectional LMs have received little prior attention. Most similar to our formulation, Peris and Casacuberta (2015) used a bidirectional neural LM in a statistical machine translation system for instance selection. They tied the input token embeddings and softmax weights in the forward and backward directions, unlike our approach which uses two distinct models without any shared parameters. Frinken et al. (2012) also used a bidirectional n-gram LM for handwriting recognition.
Interpreting RNN states. Recently, there has been some interest in interpreting the activations of RNNs. Linzen et al. (2016) showed that single LSTM units can learn to predict singular-plural distinctions. Karpathy et al. (2015) visualized character level LSTM states and showed that individual cells capture long-range dependencies such as line lengths, quotes and brackets. Our work complements these studies by showing that LM states are useful for downstream tasks as a way of interpreting what they learn.
Other sequence tagging models. Current state of the art results in sequence tagging problems are based on bidirectional RNN models. However, many other sequence tagging models have been proposed in the literature for this class of problems (e.g., Lafferty et al., 2001; Collins, 2002). LM embeddings could also be used as additional features in other models, although it is not clear whether the model complexity would be sufficient to effectively make use of them.
5 Conclusion
In this paper, we proposed a simple and general semi-supervised method using pre-trained neural language models to augment token representations in sequence tagging models. Our method significantly outperforms current state of the art models in two popular datasets for NER and Chunking. Our analysis shows that adding a backward LM in addition to traditional forward LMs consistently improves performance. The proposed method is robust even when the LM is trained on unlabeled data from a different domain, or when the baseline model is trained on a large number of labeled examples.
9
801
802
803
804
805
806
807
808
809
810
811
812
813
814
815
816
817
818
819
820
821
822
823
824
825
826
827
828
829
830
831
832
833
834
835
836
837
838
839
840
841
842
843
844
845
846
847
848
849
850
851
852
853
854
855
856
857
858
859
860
861
862
863
864
865
866
867
868
869
870
871
872
873
874
875
876
877
878
879
880
881
882
883
884
885
886
887
888
889
890
891
892
893
894
895
896
897
898
899

The paper proposes an approach where pre-trained word embeddings and pre-trained neural language model embeddings are leveraged (i.e., concatenated) to improve the performance in English chunking and NER on the respective CoNLL benchmarks, and on an out-of-domain English NER test set. The method records state-of-the-art scores for the two tasks.  - Strengths:  For the most part, the paper is well-written and easy to follow. The method is extensively documented. The discussion is broad and thorough.  - Weaknesses:  Sequence tagging does not equal chunking and NER. I am surprised not to see POS tagging included in the experiment, while more sequence tagging tasks would be welcome: grammatical error detection, supersense tagging, CCG supertagging, etc. This way, the paper is on chunking and NER for English, not for sequence tagging in general, as it lacks both the multilingual component and the breadth of tasks.  While I welcomed the extensive description of the method, I do think that figures 1 and 2 overlap and that only one would have sufficed.  Related to that, the method itself is rather straightforward and simple. While this is by all means not a bad thing, it seems that this contribution could have been better suited for a short paper. Since I do enjoy the more extensive discussion section, I do not necessarily see it as a flaw, but the core of the method itself does not strike me as particularly exciting. It's more of a  focused contribution  (short paper description from the call) than  substantial  work (long paper).  - General Discussion:  Bottomline, the paper concatenates two embeddings, and sees improvements in English chunking and NER.  As such, does it warrant publication as an ACL long paper? I am ambivalent, so I will let my score reflect that, even if I slightly lean towards a negative answer. Why? Mainly because I would have preferred to see more breadth: a) more sequence tagging tasks and b) more languages.  Also, we do not know how well this method scales to low(er)-resource scenarios. What if the pre-trained embeddings are not available? What if they were not as sizeable as they are? The experiments do include a notion of that, but still far above the low-resource range. Could they not have been learned in a multi-task learning setup in your model? That would have been more substantial in my view.  For these reasons, I vote borderline, but with a low originality score. The idea of introducing context via the embeddings is nice in itself, but this particular instantiation of it leaves a lot to ask for.


Naturalizing a Programming Language via Interactive Learning
Our goal is to create a convenient language interface for performing well-specified but complex actions such as analyzing data, manipulating text, and querying databases. However, existing natural language interfaces for such tasks are quite primitive compared to the power one wields with a programming language. To bridge this gap, we seed the system with a core programming language and allow users to “naturalize” the core language incrementally by defining alternative syntax and increasingly complex concepts in terms of compositions of simpler ones. In a voxel world, we show that a community of users can simultaneously teach one system a diverse language and use it to build 240 complex voxel structures. Over the course of three days, these builders went from using only the core language to using the full naturalized language in 74.7% of the last 10K utterances.
1 Introduction
In tasks such as analyzing and plotting data, querying databases, manipulating texts, or controlling the Internet of Things, people need computers to perform well-specified but complex actions. To accomplish this, one route is to use a programming language, but this is inaccessible to most and can be tedious even for programmers. Another route is to convert natural language into a formal language, which has been the subject of work in semantic parsing (Zettlemoyer and Collins, 2005; Artzi and Zettlemoyer, 2013; Kushman and Barzilay, 2013; Quirk et al., 2015; Pasupat and Liang, 2015). However, the expressivity of semantic
Cubes: initial – select left 6 – select front 8 – black 10x10x10 frame – black 10x10x10 frame – move front 10 – red cube size 6 – move bot 2 – blue cube size 6 – green cube size 4 – (some steps are omitted)
Monsters, Inc: initial – move forward – add green monster – go down 8 – go right and front – add brown floor – add girl – go back and down – add door – add black column 30 – go up 9 – finish door – (some steps for moving are omitted) Deer: initial – bird’s eye view – deer head; up; left 2; back 2; { left antler }; right 2; {right antler} – down 4; front 2; left 3; deer body; down 6; {deer leg front}; back 7; {deer leg back}; left 4; {deer leg back}; front 7; {deer leg front} – (last step censored)
Figure 1: Users interactively naturalize the language in Voxelurn (http://anonymous. url). Three of the 230 structures are shown.
parsers is still quite primitive compared to the power one wields with a programming language.
In this paper, we propose bridging this gap with a new interactive language learning process which we call naturalization. We seed a system with a core programming language, always available to the user. As users instruct the system to per-
2
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
form actions, they augment the language by defining new constructions — e.g., the user can explicitly tell the computer that ‘X’ means ‘Y’. Through this process, the user gradually teaches the system to understand the language that they want to use, rather than the core language that users are forced to use initially. This process accommodates both the user preference and the computer action space, which tends to result in a combination of English, short forms, and precise commands suitable for the computer.
Compared to interactive language learning with weak denotational supervision (Wang et al., 2016), our use of definitions is critical for increasing the scope of what can be learned. Definitions equate a novel utterance to a sequence of utterances the system already understands. For example, ‘go left 6 and front’ might be defined as ‘repeat 6 [go left]; go front’, which eventually can be traced back to the expression ‘repeat 6 [select left of this]; select front of this’ in the core language. Unlike in programming, the user uses concrete values rather than variables, which makes creating our definitions more accessible. The onus is now on the system to produce the correct generalization, so we propose a grammar induction algorithm tailored for the definitions setting. Compared to standard machine learning, say from demonstrations, definitions provide a much more powerful learning signal: the system is told directly that ‘a 3 by 4 red square’ is ‘3 red columns of height 4’, and does not have to induce this from observing many structures of different sizes.
We implemented a system called Voxelurn, which is a command interface for a voxel world initially equipped with a programming language supporting conditionals, loops, and variable scoping. We recruited 70 users from Amazon Mechanical Turk to build 300 voxel structures using our system. All users teach the system at once, so that what is learned immediately generalizes across users. It is thus a community of users that evolves the language to becomes more efficient over time. We show that the community defined many new utterances—shorter and alternative expressions, different syntax, more expressive operations, and also complex concepts such as ‘green monster, yellow plate 3 x 3’. As the system learns, the expressiveness of the naturalized language grows, and users increasingly prefer to use the naturalized language over the core language
where 74.7% of of the last 10k accepted utterances are in the naturalized language.
2 Voxelurn
World. A world state in Voxelurn contains a set of voxels, where each voxel has relations ‘row’, ‘col’(column), ‘height’, and ‘color’. There are two domain-specific actions, ‘add’ and ‘move’, one domain-specific relation ‘direction’. In addition, the state contains a selection, which specifies a set of positions that subsequent actions are performed with respect to. Generally, we view the world as a set of similar objects equiped with some relations — events on a calendar, cells of a spreadsheet, and lines of text are other examples.
Core language. The core language for manipulating states in Voxelurn is a programming language that the system is born understanding (see Table 1 for an overview). We protect the core language from being redefined so it is always precise and usable.1
The language has expressive control primitives—‘if’, ‘foreach’, ‘repeat’, etc. Actions operate on sets, which are represented using basic lambda DCS expressions (Liang, 2013). Besides standard set operations like union, intersection and complement, Lambda DCS leverages the tree dependency structure common in natural language: for the relation ‘color’, ‘has color red’ refers to the set of voxels that has color red, and its reverse ‘color of this’ refers to the set of colors of the current selected voxels. Tree-structured
1We found that not doing so resulted in ambiguities that propagated uncontrollably, once ‘red’ means all colors.
3
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
Rule(s) Example(s) Description
A→ A; A select left; add red perform actions sequentially A→ repeat N A repeat 3-1 add red top repeat action N times
A→ if S A if has color red [select origin] action if S is non-empty A→ while S A while not has color red [select left of this] action while S is non-empty
A→ foreach S A foreach this [remove has row row of this] action for each item in S A→ [A ] [select left or right; add red; add red top] group actions for precedence A→ {A } {select left; add red} scope only selection
A→ isolate A isolate [add red top; select has color red] scope voxels and selection
A→ select S select all and not origin set the selection A→ remove S remove has color red remove voxels
A→ update R S update color [color of left of this] change property of selection S this current selection S all | none | origin all voxels, empty set, (0, 0)
R of S | has R S has color red or yellow | has row [col of this] lambda DCS joins not S | S and S | S or S this or left and not has color red set operations
N | N+N | N -N 1,. . . ,10 | 1+2 | row of this + 1 numbers and arithmetic argmax R S | argmin R S argmax col has color red superlatives
R color | row | col | height | top | left | · · · voxel relations C red | orange | green | blue | black | · · · color values D top | bot | front | back | left | right direction values
S→ very D of S very top of very bot of has color green syntax sugar for argmax A→ add C [D] | move D add red | add yellow bot | move left add voxel, move selection
Table 1: Grammar of the core programming language, which includes actions (A), relations (R), and sets of values (S). The grammar rules are grouped into four categories. From top to bottom: domain-general action compositions, actions using sets, lambda DCS expressions for sets, and domain-specific relations and actions.
joins can be chained without using any variables, so that ‘has color [yellow or color of has row 1]’ refers to all voxels with color yellow or those colors found in voxels in the first row.
In addition to expressivity, the core language interpolates well with natural language. We avoid explicit variables by using a selection, which serves as the default argument for most actions.2 For example, ‘select has color red; add yellow top; remove’ adds yellow on top of red voxels and then removes the red voxels.
To enable the building of more complex structures in a more modular way, we introduce a notion of scoping. Suppose one is operating on one of the palm trees in Figure 2. The user might want to use ‘select all’ to select only the voxels in that tree rather than all of the voxels in the scene. In general, an action A can be viewed as taking a set of voxels v and a selection s, and producing an updated set of voxels v′ and a modified selection s′.
2The selection is like the turtle in LOGO, but can be a set.
There are two constructs that alter the flow: First, ‘{A}’ takes (v, s) and returns (v′, s), thus restoring the selection. This allows one A to use the selection as a temporary variable without affecting the rest of the program. Second, ‘isolate [A]’ takes (v, s), calls A with (s, s) (restricting the set of voxels to just the selection) and returns (v′′, s), where v′′ consists of voxels in v′ and voxels in v that occupy empty locations in v′. This allows A to only focus on the selection (e.g., one of the palm trees). Although scoping can be explicitly controlled via ‘[ ]’, ‘isolate’, and ‘{ }’, it is an unnatural concept for non-programmers. Therefore, we let each expression generate three possible scoping interpretations, and let the model learn which one is intended based on the context. This is an example where we have preemptively started to naturalize the core language to reduce the burden on users.
4
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
3 Learning interactively from definitions
The goal of the user is to build a structure in Voxelurn. In Wang et al. (2016), the user provided interactive supervision to the system by selecting from a list of candidates proposed by the system. This is practical when there are less than 50 candidates, but is completely infeasible for a complex action space such as Voxelurn. Roughly, 10 possible colors over the 3× 3× 4 box containing the palm tree in Figure 2 yields 1036 distinct denotations, and many more programs. Obtaining the the structures in Figure 1 in this manner seems even more far-fetched.
To scale to these more complex structures, we allow the user to provide definitions in addition to the usual interactive supervision of Wang et al. (2016). Each definition consists of a head utterance and a body, which is a sequence of utterances that the system understands. There are two main use cases of definitions: The first is paraphrasing, which helps naturalizes the core language (e.g., defining ‘add brown top 3 times’ as ‘repeat 3 add brown top’). The second is for the user to build up more complex concepts; in Figure 2, ‘add yellow palm tree’ is defined as a sequence of steps for building the palm tree. Once the system understands the head, it can be used in the body of other definitions; see Figure 3 for the full definition tree of ‘add palm tree’, after which one can build an entire forest of them.
def: add palm tree def: brown trunk height 3
def: add brown top 3 times repeat 3 [add brown top]
def: go to top of tree select very top of has color brown def: add leaves here def: select all sides
select left or right or front or back add green
Figure 3: Defining ‘add palm tree’, tracing back to the core language (utterances without def:).
The interactive definition process is described in Figure 4. When the user types an utterance x, the system generates candidate parses. If the user selects one, then we simply execute the resulting program. If there are no parses or the user rejects all of the existing ones, the user is asked to provide the definition body for x. Any utterances
in the body not yet understood can be defined recursively. Alternatively, the user can carry out a sequence of steps and mark these as the body of some head specified post-hoc.
When constructing the definition body, users can type utterances with multiple parses; e.g., ‘move forward’ could either modify the selection (‘select front’) or move the voxel (‘move front’). Rather than propagating this uncertainty forward, we force the user to commit to an interpretation. This demonstrates the utility of interactivity in stopping a combinatorial explosion of interpretations.
4 Model and Learning
Let us turn to how the system learns and predicts. This section contains prerequisites before we can describe definitions and grammar induction in the next section.
Semantic parsing. Our system is based on a semantic parser that maps utterances x to programs z, which can be executed on the current state s (set of voxels and selection) to produce the next state s′ = JzKs. Our system is implemented in SEMPRE (Berant et al., 2013); see Liang (2016) for a gentle exposition.
A derivation d represents the process by which the utterance x turns into a program z = prog(d). More precisely, d is a tree where each node contains a list of children derivations [d1, . . . , dn], the corresponding span of the utterance (start(d), end(d)), and grammar rule rule(d), and the grammar category cat(d).
Following Zettlemoyer and Collins (2005), we define a log-linear model over derivations d given
5
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
458
459
460
461
462
463
464
465
466
467
468
469
470
471
472
473
474
475
476
477
478
479
480
481
482
483
484
485
486
487
488
489
490
491
492
493
494
495
496
497
498
499
Feature Description Rule.ID ID of the rule Rule.Type core?, used?, used by others? Social.Author ID of author Social.Friends (ID of author, ID of user) Social.Self rule is authored by user? Span (left/right token(s), category) Scope type of scoping for each user
Table 2: Summary of features.
an utterance x produced by the user u:
pθ(d | x, u) ∝ exp(θTφ(d, x, u)), (1)
where φ(d, x, u) ∈ Rp is a feature vector and θ ∈ Rp is a parameter vector. The user u does not appear in previous work on semantic parsing, but we use it to personalize the semantic parser trained on the community.
We use a standard chart parser to construct a chart. For each chart cell, indexed by the start and end indices of a span, we construct a list of partial derivations recursively by selecting children derivations from subspans and applying a grammar rule. The resulting derivations are sorted by model score and only the top K are kept. We use chart(x) to denote the set of all partial derivations across all chart cells. The set of grammar rules starts with the set of rules for the core language (Table 1), but grows via grammar induction when users add definitions (Section 5).
Features. Derivations are scored using a weighted combination of features. There are three types of features, summarized in Table 2. Rule features fire on each rule used to construct a derivation. ID features fire on specific rules (by ID). Type features track whether a rule is part of the core language or user-induced, whether it has been used again after it was defined, if it was used by someone other than its author, and if the user and the author are the same. (5 + #rules features) Social features fire on properties of rules that capture the unique linguistic styles of different users and their interaction with one other. Author features capture the fact that some users provide better, and more generalizable definitions that tend to be used. Friends features are cross products of author ID and user ID, which captures whether rules from a particular author is systematically preferred or not by the current user, due stylistic similarities or differences. (#users+ #users× #users features)
Span features are conjunctions of the category with 2 tokens outside the span, and 1 token inside the span. These capture a weak form of context-dependence that are generally helpful. (≈ V 4 × #cats features for a vocabulary of size V ) Scoping features track how the community, as well as individual users prefer each of 3 scoping choices (none, selection only, and voxels+selection), as described in Section 2. There are 3 global indicators, and 3 indicators for each user firing every time a particular scoping choice is made. (3 + 3× #users features)
Parameter estimation. When the user types an utterance, the system generates a list of candidate next states. When the user chooses a particular next state s′ from this list, the system performs a online AdaGrad (Duchi et al., 2010) update on the paratmers θ according to the gradient of the following loss function:
− log ∑
d:Jprog(d)Ks=s′ pθ(d | x, u) + λ||θ||1,
which attempts to increase the model probability on derivations whose programs produce the next state s′.
5 Grammar Induction
Recall that the main form of supervision is via user definitions, which allows naturalization of the core language and creation of user-defined concepts. In this section, we show how to turn these definitions into new grammar rules that can be used by the system to parse new utterances.
Previous work on grammar induction for semantic parsing is given pairs of utteranceprogram pairs (x, z). Both GENLEX (Zettlemoyer and Collins, 2005) and higher-order unification (Kwiatkowski et al., 2010) algorithms overgenerate rules that liberally associate parts of x with parts of z. Though unpromsing rules are immediately pruned, many spurious rules are undoubtedly still kept. In the interactive setting, we must keep the number of candidates small to avoid a bad user experience, our precision bar for new rules is much higher.
Fortunately, the structure of definitions makes the grammar induction task easier. Rather than being given a utterance-program (x, z) pair, we are given a definition, which consists of an utterance
6
501
502
503
504
505
506
507
508
509
510
511
512
513
514
515
516
517
518
519
520
521
522
523
524
525
526
527
528
529
530
531
532
533
534
535
536
537
538
539
540
541
542
543
544
545
546
547
548
549
550
551
552
553
554
555
556
557
558
559
560
561
562
563
564
565
566
567
568
569
570
571
572
573
574
575
576
577
578
579
580
581
582
583
584
585
586
587
588
589
590
591
592
593
594
595
596
597
598
599
x (head) along with the body X = [x1, . . . , xn], which is a sequence of utterances. The body X is fully parsed into a derivation d, while the x is likely only partially parsed into a set of partial derivations chart(x).
At a high-level, we find matches—partial derivations chart(x) of the head x that also occur in the full derivation d of the body X . A grammar rule is produced by substituting any set of nonoverlapping matches by their categories. As an example, suppose the user defines
‘add red top times 3’ as ‘repeat 3 [add red top]’.
Then we would be able to induce the following two grammar rules:
A→ add C D times N : λCDN.repeat N [add C D]
A→ A times N : λAN.repeat N [A]
The first rule substitutes primitive values (‘red’, ‘top’, and ‘3’) with their respective pre-terminal categories (C, D, N ). The second rule contains compositional categories like actions (A), which requires some care. One might expect that greedily substituting the largest matches or the match that covers the largest portion of the body would work, but the following example that this is not the case:
A1 A1 A1︷ ︸︸ ︷ ︷ ︸︸ ︷ ︷ ︸︸ ︷ add red left and here = add red left; add red︸ ︷︷ ︸ ︸ ︷︷ ︸
A2 A2
(2)
Here, both the highest coverage substitution (A1: ‘add red’, which covers 4 tokens of the body), and the largest substitution available (A1: ‘add red left’) would generalize incorrectly. The correct grammar rule only substitutes the primitive values (‘red’, ‘left’).
5.1 Highest scoring abstractions
We now propose a grammar induction procedure that optimizes a more global objective and uses the learned semantic parsing model to choose substitutions. To make things more formal, let M be the set of partial derivations in the head whose programs also appear in the derivation d of the body
X:
M def = {d′ ∈ chart(x) : ∃d′′ ∈ desc(d) ∧ prog(d′) = prog(d′′)},
where desc(d) are partial derivations that are the descendants of d. Our goal is to find a packing P ⊆ M , which is a set of derivations corresponding to non-overlapping spans. We say that a packing P is maximal if there is no other derivations may be added without creating an overlap.
Letting packing(M) be the set of all maximal packings, we can now frame our problem as finding the maximal packing that has the highest score under our current semantic parsing model:
P ∗l = argmax P∈packing(M); ∑ d∈P score(d).
Finding the highest scoring packing can be done using dynamic programming on P ∗i for i = 1, . . . , l, where l is the length of x. To obtain this dynamic program, let Di be the highest scoring maximal packing containing a derivation ending exactly at position i,
Di = {di} ∪ P ∗start(d), (3)
di = argmax d∈M ;end(d)=i
score(d ∪ P ∗start(d)). (4)
Then the maximal packing of up to i can be defined recursively as
P ∗i = argmax D∈{Ds+1,Ds+2,...,Di} score(D) (5)
s(i) = max d:end(d)≤i start(d), (6)
where s = s(i) is the largest index such that Ds(i) is no longer maximal for the span (1, i) (i.e there is a d′ ∈ M on the span start(d) >= s(i) ∧ end(d) ≤ i.
Once we have a packing P ∗ = P ∗l , we can go through d′ ∈ P ∗ in order of start(d′), as in Algorithm 1.
Algorithm 1 generates one high precision rule per packing per definition. In addition to the highest scoring packing, we also use the “simple packing”, which includes only primitive values (in Voxelurn, these are colors, numbers, and directions). We also attempt to generalize compositional categories like sets and actions. Here correctness is not guaranteed—however a rule that often generalizes incorrectly will be down-weighted,
7
601
602
603
604
605
606
607
608
609
610
611
612
613
614
615
616
617
618
619
620
621
622
623
624
625
626
627
628
629
630
631
632
633
634
635
636
637
638
639
640
641
642
643
644
645
646
647
648
649
650
651
652
653
654
655
656
657
658
659
660
661
662
663
664
665
666
667
668
669
670
671
672
673
674
675
676
677
678
679
680
681
682
683
684
685
686
687
688
689
690
691
692
693
694
695
696
697
698
699
Input : x, d, P ∗ Output: rule r ← x; f ← d; for d′ ∈ P ∗ do
r ← r[span(d′)← cat(d′)] f ← λ cat(d′).f [d′ ← cat(d′)]
return rule cat(d)→ r : f Algorithm 1: Extract a rule r from a derivation d and a packing P ∗. Here, f [s← t] means substitute s with t in f , with the usual care taken about names in variable bindings.
along with the score of its packings, relative to the score of the simple packing. As a result, a different rule might be induced, even with the same inputs x and d.
5.2 Extending the chart via alignment
The previous procedure yields high precision rules, but fails to generalize enough. Suppose that ‘move up’ is defined as ‘move top’. But since ‘up’ does not parse, it also does not match anything, whereas we would like to infer that ‘up’ means ‘top’.
To handle this, we leverage a property of definitions that we haven’t been using thus far: the utterances themselves. If we align the head and body, then we would intuitively expect aligned phrases to correspond to the same derivations. We can then transplant these derivations from d to chart(x) to create some new matches. This is more constrained than the usual alignment problem (e.g., in machine translation) since we only need to consider spans of X which corresponds to derivations in desc(d).
Input : x,X, d for d′ ∈ desc(d), x′ ∈ spans(x) do
if aligned(x′, d′, (x,X)) then dt ← d′; start(dt)← start(x′); end(dt)← end(x′); chart(x)← chart(x) ∪ dt
end end
Algorithm 2: Extending the chart by alignment: If d′ is aligned with x′ based on the utterance, then we pretend that x′ should also parse to d′, and d′ is transplanted to chart(x) as if it came from x′.
Algorithm 2 provides the algorithm for extending the chart via alignments. The aligned function is implemented using the following two heuristics:
• exclusion: if all but 1 pair of short spans are matched, the unmatched pair is considered aligned.
• projectivity: if d1, d2 ∈ desc(d) ∩ chart(x), then ances(d1, d2) is considered aligned to the corresponding span in x.
With the extended chart, we can run the algorithm from Section 5.1 to induce rules. The transplanted derivations (e.g., ‘up’) might now form new matches which allows the grammar induction for induce more generalizable rules. We only perform this extension when the body consists of one utterance, which tend to be paraphrases. Multiutterance definitions tend to be about constructing new concepts, for which extension is not useful. Because low precision rules significantly degrades user experience in the interactive setting, we induce alignment based rules conservatively—only when 1 or 2 tokens were left unmatched.
6 Experiments
Setup. Our ultimate goal is to create a community of users who can build interesting structures in Voxelurn while naturalizing the core language. We created this community using Amazon Mechanical Turk (AMT) in two stages. First, we have qualifier tasks, in which an AMT worker was instructed to build an exact simple, fixed target that we provide . In addition, replicating a target ensures that the users are able to use basic constructs of the core language, which is the starting point of the naturalization process.
Next, we allowed the workers who qualified to enter the second freebuilding stage, in which they were asked to build any structure they wished in 30 minutes. This two stage process was designed to give users freedom while guarding against spam.
Incentives. To incentivize good structures, we setup a leaderboard which ranked structures based on recency and upvotes (Hacker News) Over the course of 3 days, we picked three prize categories to be released daily. The prize categories for each day were bridge, house, animal – tower, monster(s), flower(s) – ship(s), dancer(s), and castle.
Finally, to incentivize more definitions, we also track citations. When a rule is used in an accepted
8
701
702
703
704
705
706
707
708
709
710
711
712
713
714
715
716
717
718
719
720
721
722
723
724
725
726
727
728
729
730
731
732
733
734
735
736
737
738
739
740
741
742
743
744
745
746
747
748
749
750
751
752
753
754
755
756
757
758
759
760
761
762
763
764
765
766
767
768
769
770
771
772
773
774
775
776
777
778
779
780
781
782
783
784
785
786
787
788
789
790
791
792
793
794
795
796
797
798
799
Short forms left, l, mov left, go left, <, sel left br, blk, blu, brn, orangeright, left3 add row brn left 5 := add row brown left 5 Syntactic go down and right := go down; go right select orange := select has color orange add red top 4 times := repeat 4 [add red top] l white := go left and add white mov up 2 := repeat 2 [select up] go up 3 := go up 2; go up Higher level add black block width 2 length 2 height 3 :=
{repeat 3 [add black platform width 2. . . flower petals := flower petal; back; flower petals cube size 5, get into position start, 5 x 5 open green square, brownbase
Table 3: some definitions on the leaderboard. utterance by another user, the rule (and its author) gets a citaion. We pay bonuses to top users according to their h-index.
Statistics. 70 workers were granted the qualification, and 42 workers participated in the final free building experiment, who built 230 structures. Users made 64,075 utterances, of these 36,589 are accepted (so the action is performed). There were 2,495 definitions resulting in 2,817 grammar rules, compared to less than 100 core rules.
Is naturalization happening? Yes! according to Figure 5, which plots the cummulative percentage of utterances that belong to core, induced, or none (which means we did not parse it). At the end, 58% of the utterances are parsed with induced rules. To rule out that these utterances are getting rejected more often, we consider only accepted utterances in the middle plot of Figure 5, which plots the percentage of induced rules among accepted utterances for the entire community, as well as for the 5 heaviest users. At the end, 64.3% of all accepted utterances are induced, which is 77.9% in the last 10,000 accepted utterances.
We see three interpretable modes of naturalization outlined in Table 3. For very common operations, like moving the selection, people found ‘select left’ too verbose and shorterned it. For syntactic variations, where many of the definition body are not in core. The definitions for high level concepts tend long and hierarchical. The bottom plot of Figure 5 shows that users are building higher level concepts, since programs become
much longer relative to utterances over time.
0 10000 20000 30000 40000 50000 60000 70000
query #
0
10
20
30
40
50
60
70
p e rc
e n t
none induced core
0 10000 20000 30000 40000 50000 60000 70000
query #
0
20
40
60
80
100
p re
ce n t
in d u ce
d
all #1 #2 #3 #4 #5
0 10000 20000 30000 40000 50000 60000 70000
query#
0
20
40
60
80
100
le n (z
) /
le n (x
)
all #1 #2 #3 #4 #5
Figure 5: learning curves. top: percentage of all utterances belonging to each type. mid: percentage of accepted utterances belonging to induced. bot: expressiveness measured by the average length of the program per token of the utterance.
7 Discussion
We applied our methodology to Voxelurn, but it should apply to other settings: answering questions about semi-structured data (Pasupat and Liang, 2015), parsing to regular expressions (Kushman and Barzilay, 2013), or following robot instructions (Chen and Mooney, 2011; Tellex et al., 2011; Artzi and Zettlemoyer, 2013; Misra et al., 2015). In these applications, instead of learning from a training set, we could start with the programming language. User interactions can provide us with very strong, usable supervision through definitions. We hope that naturalization can lead to better language interface technologies that strikes a more usable balance between precision and naturalness.
9
801
802
803
804
805
806
807
808
809
810
811
812
813
814
815
816
817
818
819
820
821
822
823
824
825
826
827
828
829
830
831
832
833
834
835
836
837
838
839
840
841
842
843
844
845
846
847
848
849
850
851
852
853
854
855
856
857
858
859
860
861
862
863
864
865
866
867
868
869
870
871
872
873
874
875
876
877
878
879
880
881
882
883
884
885
886
887
888
889
890
891
892
893
894
895
896
897
898
899

Thanks for the response. I look forward to reading about the effect of incentives and the ambiguity of the language in the domain.  Review before author response: The paper proposes a way to build natural language interfaces by allowing a set of users to define new concepts and syntax. It's an (non-trivial) extension of S. I. Wang, P. Liang, and C. Manning. 2016. Learning language games through interaction  Questions: - What is the size of the vocabulary used  - Is it possible to position this paper with respect to previous work on inverse reinforcement learning and imitation learning ?  Strengths: - The paper is well written - It provides a compelling direction solution to the problem of dealing with a large set of possible programs while learning natural language interfaces.   Weaknesses: - The authors should discuss the effect of the incentives on the final performance ? Were other alternatives considered ?  - While the paper claims that the method can be extended to more practical domains, it is not clear to me how straightforward it is going to be. How sensitive is the method to the size of the vocabulary required in a domain ? Would increased ambiguity in natural language create new problems ? These questions are not discussed in the current experiments. - A real-world application would definitely strengthen the paper even more.