For those in laptop domain, we use pre-trained glove.840B.300d 6 .

2 http://sklearn-crfsuite.readthedocs.io/en/latest/ 3 As we use our own implementation of LSTM, the reported results are different from those in (Liu et al., 2015) 4 we list the result of RNCRF over D1 without opinion annotations for fair comparison.


To test this hypothesis, we first collect examples of the form (e, f, e , f ), where e = ¬e, rep(e) = rep(f ), and rep(e ) = rep(f ).

we find pairs of pairs of RNN representations f and f for which the natural language messages (e, e ) serve as a denotational certificate that f behaves as a negation of f .


Importantly, we essentially use IE as a black box (thus, our method could potentially work with any IE system), and focus on strategies that guide what the IE system reads for a complex information need.

we consider the common scenario where a biologist (or other modelbuilding process) queries the literature on: How does one participant (source) affect another (destination), where the connection is typically indirect?


, l m } = Y is x i 's class label, we want to build a model f (x) that can classify each test instance x to one of the m training or seen classes in Y or reject it to indicate that it does not belong to any of the m training or seen classes, i.e., unseen.

we want to build a (m + 1)-class classifier f (x) with the classes C = {l 1 , l 2 , .


We still perform topic-based classification.

given a review, the system decides what class of product the review is about.


A survey by Kuperberg (2010) of over 50 years of observations in the schizophrenia literature concludes that schizophrenia patients "may use common words in an idiosyncratic or bizarre manner."

colorful (and metaphorical) examples of bizarre speech recorded by Andreasen (1986) include patients who referred to watches as "time vessels" and to gloves as "hand shoes."


We adapt the token-level metaphor identification algorithm of Do Dinh and Gurevych (2016) to our task.

we use a multilayer perceptron (MLP) architecture with three layers.


Table 2, varies a lot across the different methods.

the agreement in the intrusion task with four key concepts is higher for clusters generated with our method (0.79), while it is very low using LDA (0.32).


As shown in experiments, Zipporah selects a high-quality parallel corpus from a large, mixed quality data pool.

for one noisy dataset, Zipporah achieves a 2.1 BLEU score improvement with using 1/5 of the data over using the entire corpus.


Tables 2, 3 and 4 show the BLEU performance of the selected subsets of the Zipporah system compared to the baseline, which selects sentence pairs at random; for comparison, we also give the BLEU performance of systems trained on Europarl.

table 4: BLEU Performance, Spanish-English for the Germen-English corpus, when selecting less than 2% of the data (10 million words), on the tED-talk dataset, Zipporah achieves a 5.5 BLEU score improvement over the baseline; by selecting less than 4% of the data (20 million words) the system gives better performance than using all data.


The manual creation of such a dataset is very time-consuming, as the annotation includes many subtasks.

an annotator would need to manually identify all concepts in the documents, while only a few of them will eventually end up in the summary.


To ensure that the automatic classification does not remove suitable propositions, we tuned the classifier to avoid false negatives.

we introduced class weights, improving precision on the negative class at the cost of a higher fraction of positive classifications.


Perhaps somewhat counterintuitively, this current paradigm treats dialog as a static supervised learning problem, rather than as the interactive agent learning problem that it naturally is.

a typical pipeline is to collect a large dataset of human-human dialog (Lowe et al., 2015;Das et al., 2017a;de Vries et al., 2017;Mostafazadeh et al., 2017), inject a machine in the middle of a dialog from the dataset, and supervise it to mimic the human response.


natural language does not emerge 'naturally' in multi-agent dialog, despite independently reported successful demonstrations in recent literature.

in a sequence of 'negative' results culminating in a 'positive' one, we find that while agents always successfully invent communication protocols and languages to achieve their goals with near-perfect accuracies, the invented languages are decidedly not compositional, interpretable, or 'natural'; and that it is possible to coax the invented languages to become more and more human-like and compositional by increasing restrictions on how two agents may communicate.


Next, given a resultant concept for each of the node/leaf, we backtrack in time and check for the first occurrence when only tuples which satisfy the corresponding concept are assigned to that particular node/leaf.

we compute the earliest time when a node/leaf is 'pure' with respect to its final learned concept.


We highlight key observations below: (a) The agents ground most of the tasks initially at around epoch 20.

q-BOT assigns Y to both (shape, style), (style, shape), (shape,color) and (color, shape), while (color, style) is mapped to Z.


(c) We see a strong correlation between improvement in performance and when agents learn a language grounding.

there is an improvement from 40% to 80% within a span of 25 epochs where most of the grounding is achieved, as seen from Fig.


Each convolutional network performs average pooling to produce its output.

the model considers non-overlapping se-quences of n posts and keeps the average feature value across all sequences.


We compare our model against several baselines using MNB and SVM classifiers (Wang and Manning, 2012).

we consider two sets of features for the classifiers.


This can be solved by the original inference algorithm.

our approach works as a metaalgorithm and developers do not need to implement a new inference algorithm.


However, since the attacker has access to the representation function r parameterized by θ r , they can generate a dataset from any corpus containing the private variables they want to recover.

it is not necessary that they have access to the original training corpus to train their classifier.


• We modify the objective function of the main classifier to incorporate a penalty when the adversarial classifier is good at reconstructing z.

the main classifier tries to update its parameters so as to confuse the duplicate attacker.


Also, we would like to evaluate whether the attacker's task is easier when the variables to predict are explicitly in the input, compared to when these information are only potentially and implicitly in the input.

this setting simulates the case where private in-formation may be used by the model to improve classification, but should not be exposed too obviously.


• As an upper bound, we trained a classifier that can optimize the hidden representations (r) for the attacker's tasks.

this baseline is trained to predict de-  mographic variables from x, as if it were the main task.


Neural methods for structure prediction have been used extensively in other areas of NLP, and we leverage these methods here.

we use a neural encoder-decoder architecture with beam search decoding, representative of several current state-of-the-art systems (Bahdanau et al., 2014;Wiseman and Rush, 2016;Vinyals et al., 2015).


The encoder operates over every (s t , e j ) ∈ S × E pair to create an encoded representation c t j of the action described by sentence s t , as applied to entity e j .

we can consider the overall action to be represented by |E| embeddings, one for each of the entities in E, encoding the action's effects on each.


We have shown how the predicted effects of actions can be improved by treating the task as a structured prediction problem, allowing commonsense knowledge to be injected to avoid an overall inconsistent or improbable set of predictions.

we have shown how two kinds of knowledge can be exploited: hard constraints to exclude impossible and nonsensical state changes, and soft constraints to encourage likely state changes.


Natural language surface forms are often used in RE in a weak-supervision setting (Mintz et al., 2009;Hoffmann et al., 2011;Riedel et al., 2013).

if entity1 and entity2 are known to be related by relation, it is assumed that every sentence observed which mentions both entity1 and entity2 is assumed to be a realization of relation: i.e.


While the task of claim verification will not tell us the absolute truth of this claim, it is expected to determine whether the claim is supported by evidence in a given text collection.

given a claim and a text corpus, evidential claim verification, demonstrated in 1 Figure 1: Illustration of evidential claim verification task.


2 Background and Related Work A large body of research evidences that human semantic memory is inherently re-constructive and associative (Collins and Loftus, 1975;Anderson and Bower, 2014).

memories are not exact static copies of reality, but are rather reconstructed from their stimuli and associated concepts each time they are retrieved.


To achieve this, we propose to reconstruct the vector of associated word from the corresponding multimodal semantic representation.

in the decoding process we change the equation 3to: and equation 4to: x t = g(W tĥ t + bt) x a = g(W aĥ a + bâ) To train the model, we add an additional objective function, which is the mean square error between the embeddings of the associated word y and their re-constructive embeddingsx asc : where y i and x i are the embeddings of a pair of associated words.


As shown clearly, our bimodal and trimodal AMA models achieve better performance than baselines in both V/A (visual or auditory, the testing data that have associated visual or auditory vectors) and ZS (zero-shot, the testing data that do not have associated visual or auditory vectors) region.

our models outperform baseline models on words with or without perceptual information.


Our multimodal models With either bimodal or trimodal inputs, the proposed AMA-M model outperforms all baseline models by a large margin.

our AMA-M model achieves an relative improvement of 4.1% on average (4.5% with trimodal inputs) over the state-of-the-art Ridge model.


We introduce an effective and efficient method that grounds (i.e., localizes) natural sentences in long, untrimmed video sequences.

a novel Temporal GroundNet (TGN) 1 is proposed to temporally capture the evolving fine-grained frame-byword interactions between video and sentence.


The task is to localize an image region described by natural language, which involves comprehending and modeling different spatial contexts, such as spatial configurations , attributes (Yu et al., 2018), and relationships between objects (Hu et al., 2017).

the task is usually formulated as a ranking problem over a set of candidate regions in a given image, where candidate spatial locations come from region proposal methods (Uijlings et al., 2013;Jie et al., 2016b,a;Ren et al., 2017) such as Edge-Box (Zitnick and Dollár, 2014).


The methods proposed in (Gao et al., 2017;Hendricks et al., 2017) learn a common embedding space shared by video segment features and sentence representations, in which their similarities are measured.

moment context network (MCN) (Hendricks et al., 2017) learns a shared embedding for video clip-level features and language features.


With the obtained video frame features V = {f t } T t=1 and word embeddings of the sentence S = {w n } N n=1 , we employ two long shortterm memory networks (LSTMs) (Hochreiter and Schmidhuber, 1997) to sequentially process the two different modalities, i.e., video and sentence, independently.

one LSTM sequentially models the video V , yielding the hidden states , while the other LSTM processes the sequential words in the sentence S, resulting in its corresponding hidden states {h s n } N n=1 .


Our proposed grounder works in one single pass without introducing overlapping sliding windows, which thus results in a fast runtime.

as shown in Figure 2, at each time step t, the grounder efficiently scores a set of K grounding candidates by considering multiple time scales (Buch et al., 2017) that end at time step t. we use different K for different datasets, which is determined by the distribution of the lengths of all ground-truth groundings in a certain dataset.


To simplify the following discussions, the lengths of K time scales are assumed to be an arithmetic sequence with the common difference δ and all the temporal candidates are sorted by increasing lengths.


Note that all grounding candidates considered at time t have a fixed ending boundary.

at each time step t, the grounder will classify each temporal candidate in consideration as a positive grounding or a negative one with respect to the given sentence.


The training samples collected in X for NSGV are video-sentence pairs.

each video V is temporally associated with a set of sentence annotations: , where M is the number of annotated sentences of the video, and S i is a sentence description of a video clip, with t b i and t e i indicating the beginning and ending time in the video.


To extract visual features, we consider both appearance and optical flow features.

we study four widely-used visual features: VGG16 (Simonyan and Zisserman, 2014), C3D (Tran et al., 2015), Inception-V4 (Szegedy et al., 2017), and optical flow .


Different CNNs are used to encode video visual information.

we consider VGG16, C3D, and Inception-V4.


The difference between NM and TGN is that there is no interaction LSTM in NM.

when calculating the attention weight for each word as in Eq.


To verify our assumption that PreCo embodies the core challenges of coreference, we evaluate a strong baseline coreference resolver on it.

we (i) estimate the room for improvement of the baseline system to show that the dataset is challenging, (ii) study the impact of training-test overlap to model performance and error analysis to show the advantages of PreCo, and (iii) quan-  titatively evaluate the mention detector to understand the bottlenecks of the coreference resolution system.


To better capture long range dependencies and synthesize the information of the sentence, we extend self-attention mechanism into the framework.

we try to improve Chinese NER task performance by incorporating shared boundary information from CWS task.


It belongs to a class of directed hypergraphs (Gallo et al., 1993), where each hyperedge e consists of a single designated parent node (head of e) and an ordered list of child nodes (tail of e).

our segmental hypergraph consists of the following 5 types of nodes: • A i encodes all such mentions that start with the i-th or a later word • E i encodes all mentions that start exactly with the i-th word • T k i represents all mentions of type k starting with the i-th word • I k i,j represents all mentions of type k that contain the j-th word and start with the i-th word • X marks the end of a mention.


As the authors have mentioned in their papers, for a given sub-structure in their model, there exist multiple ways of interpreting the combination of mentions.

in both representations, the decisions on where the beginning and the end of a mention are made locally.


(2016) to use a bidirectional LSTM that takes the character embeddings as input.

the character-level representation ch i for each word is obtained by concatenating the last hidden vectors of the forward and backward LSTMs.


Since the linear-chain CRF cannot handle overlapping structures, we only use the outermost mentions for learning.

every outer-most mention is labeled based on the BILOU tagging scheme, which was empirically shown to be better than the BIO scheme (Ratinov and Roth, 2009).


The character-level representations (+ char) make both restricted and unrestricted SH perform even better.

sH (c=n) + char achieves the best results in all datasets compared with other recent neural models (Katiyar and Cardie, 2018;Ju et al., 2018;Wang et al., 2018).


As shown in Table 2, there are over 8% such mentions in ACE and over 4% in GENIA.

sH relies on the hyperedges between I nodes to capture such interactions explicitly.


We also explore hierarchical relationships among the latent variables.

we introduce the VSL-GG-Hier model which has two Gaussian latent variables with the hierarchical structure shown in Figure 1c.


We investigate the effect of attaching the classifier to different latent variables.

for the VSL-GG-Hier model, we compare the attachment of the classifier between z and y.


The entire framework is trained using a unified objective function, which is efficient and applicable to arbitrary language pairs that exist in multi-lingual KBs.

we build a bilingual entity network from inter-language links 1 in KBs for regularizing cross-lingual entities through a variant of skipgram model (Mikolov et al., 2013c).


Instead, we build G en−zh = (E en ∪ E zh , R en ∪R zh ∪R en−zh ) by enriching the neighbors of cross-lingual linked entities.

we add edgesR en−zh between two mono-lingual ENs by letting all neighbors of e en i be neighbors of e zh i ′ , and vice versa, if ⟨e en i , e zh i ′ ⟩ ∈ R en−zh .


Aiming to facilitate transfer to resource poor languages and domains, our methods rely on as little supervision as possible.


(2017) that require several thousands translated words as a supervision signal.

except from BEs induced using comparable word alignment signals -words aligned through indirect sig-nals such as related images or through comparability of their features (e.g.


Another example in ZR18 for the same domain pair (see Figure 1 in their paper) is: "I was at first very excited with my new Zyliss salad spinner -it is easy to spin and looks great", from this sentence PBLM learns the connection between easy -an adjective that is often used to describe kitchen appliances, but not books -and great.

pBLM is able to learn the connection between witty and easy to facilitate adaptation between the domains.


PBLM can naturally feed a structure-aware task classifier.

in the PBLM-CNN ar- chitecture that we consider here (Figure 2b), 2 the PBLM's softmax layer (that computes the probabilities of each pivot to be the next unigram/bigram) is cut and a matrix whose columns are the PBLM's h t vectors is fed to the CNN.


This mismatch is somewhat reduced when training on unlabeled data from both (L s , D s ) and (L t , D t ).

we rely on the trained parameters of the models to align the input spaces when trained on unlabeled data from both (L s , D s ) and (L t , D t ).


For instance, the neighboring words of China in English (Japan, India and Taiwan) should be close to the neighboring words of Cina in Italian (Beijing, Korea, Japan) in the common semantic space.

we encourage the consistency of neighborhoods across multiple languages.


As shown in Table 3, our approaches outperform previous approaches in almost all cases 13 .

by augmenting word representation with neighboring words in the common semantic space as in Eq.


The multilingual embeddings learned by our approach consistently outperforms MultiCCA and MultiCluster.

when there are not enough annotated examples, the performance could be improved by incorporating annotated examples from other languages.


Therefore, we in this section propose a generalization of the existing refinement methods to incorporate a multilingual objective.

mAT can produce an approximately aligned embedding space.


In the following sections, we treat English as the major language and Chinese as an additional bilingual signal, while their roles can be mutually exchanged.

english and Chinese iteratively become the major language during the training procedure.


Formally, given the t-th parallel bilingual sentence C zh t , the bilingual context of w i is therefore To ensure efficiency, continuous bag-of-words (CBOW) model is applied, where it takes wordlevel input tokens and outputs sense-level identities.

given an English word embedding matrix P en , the local context can be modeled as the average of word embeddings from its context, 1 Similarly, we can model the bilingual contextual information given Chinese word embedding matrix P zh using the CBOW formulation and obtain 1 M w j ∈c i P zh j .


This module is the degraded version of bilingual sense induction module when α = 1, which occurs where no parallel bilingual signal exists.

every bilingual sense induction module will experience the degradation during the training process presented in Algorithm 1.


To ensure sense embeddings of two different languages align well, we hypothesize that the target sense identity s i not only predicts the sense identity s j of w j in C en t but also one sampled sense identity s l of w l from the parallel sentence C zh t , where s l is decoded by the Chinese monolingual sense induction module.

the bilingual skip-gram objective can be formulated using the English sense embedding matrix U en and the bilingual collocation estimation matrix V zh as: where s k iterates over all possible Chinese sense identities in the denominator.


The AR method affects only a subset of the full vocabulary V, and consequently only a (small) subspace of the original space X (see Figure 1).

it specializes the embeddings X s corresponding to V s , the vocabulary of words observed in the constraints.


Nevertheless, the perturbation underwent by the original observed embeddings can provide evidence about the general effects of specialization.

it allows to learn a global mapping function f : The parameters for this function can be trained in a supervised fashion from pairs of original and initially specialized word embeddings (x Figure 2.


However, this mapping should not just model the inherent transformation, but also ensure that the resulting vector is 'natural'.

assuming that word representations lie on a manifold, the mapping should return one of its values.


The parameters θ G are learned by minimizing the L 2 distance between the training pairs.

the loss is a contrastive margin-based ranking loss with negative sampling (MM) as proposed by Weston et al.


This creates a bilingual synthetic dictionary that allows to further refine the coarse initial mapping.

the optimal parameters for the linear mapping minimizing the L 2 -distance between source-target pairs are provided by the closed-form Procrustes solution (Schönemann, 1966) based on singular value decomposition (SVD): where || • || F is the Frobenius norm.


Word embeddings are one of the most widely used resources in NLP, as they have proven to be of enormous importance for modeling linguistic phenomena in both supervised and unsupervised settings.

the representation of words in cross-lingual vector spaces (henceforth, crosslingual word embeddings) is quickly gaining in popularity.


Our experimental results show that the proposed additional transformation does not only benefit cross-lingual evaluation tasks, but, perhaps surprisingly, also monolingual ones.

we perform an extensive set of experiments on standard benchmarks for bilingual dictionary induction and monolingual and cross-lingual word similarity, as well as on an extrinsic task: cross-lingual hypernym discovery.


Empirically it was found that such meta-embeddings can often outperform the individual word embeddings from which they were obtained.

it was recently argued that word vector averaging can be a highly effective approach for learning such metaembeddings (Coates and Bollegala, 2018).


Our approach for improving cross-lingual embeddings consists of three main steps, where the first two steps are the same as in existing methods.

given two monolingual corpora, a word vector space is first learned independently for each language.


First, we leverage the same bilingual dictionary that was used to obtain the initial alignment (Section 3.1).

let D = {(w, w )} be the given bilingual dictionary, where w ∈ V and w ∈ V , with V and V representing the vocabulary of the first and second language, respectively.


As a complement of this analysis we show some qualitative results which give us further insights on the transformations of the vector space after our average approximation.

we analyze the reasons behind the higher quality displayed by our bilingual embeddings in monolingual settings.


It was shown in Table 4 that the performance of our model gradually increased alongside the size of the training data in the target language until surpassing VecMap in the most informed configuration (i.e., EN+2k).

our model seems to show a higher presence of generic words in the output hypernyms, which may be explained by these being closer in the space.


What can we learn about language by observing the editing process that we cannot readily learn by observing only the final edited text?

the contributions of this paper are: • A new corpus (WikiAtomicEdits) of 26M atomic insertions and 17M atomic deletions covering 8 languages ( §3 and §4): http: //goo.gl/language/wiki-atomic-edits.


This formulation is desirable because it exposes a relationship between the surface form and the semantics of natural language while remaining agnostic about the underlying semantic representation.

the difference in "meaning" between s and e(s) is exactly the "meaning" of p (in context), regardless of how that meaning is represented.


Second, while insertions permit a clean set of assumptions about the relationship between the original sentence and the edited sentence ( §2), it is more difficult to make generalizations about deletions.

it is difficult to say whether the original sentence should not communicate the information in the deleted phrase (i.e.


We then manually inspect the sample and record whether or not the annotators' choices of different insertion points give rise to sentences with different semantic meaning or simply to sentences with different discourse structure.

we consider three categories for the observed disagreements: 1) the sentences are meaning equivalent from a truth-conditional perspective, 2) the sentences contain significant differences in meaning from a truth-conditional perspective, or 3) the sentences contain minor differences or ambiguities in meaning (but would likely be considered equivalent from the point of view of most readers).


Table 7 shows our results for English (Spanish and German are given in the Supplementary Material).

we see that many words which are inserted at a significantly higher-than-baseline rate reflect "refinement"-type edits.


We find a significant difference 10 (p < 0.01) between the types of edits on which the General LM makes correct predictions and the types on which it makes incorrect predictions.

the General LM appears to be especially good at predicting location for fluency/discourse edits, and especially poor at predicting the location of refinement edits.


In addition to reporting standard LM perplexity, we compute two measures of performance, which are intended to provide an intuitive picture of how well each model captures the nature of the information that is introduced by the human editors.

we compute Exact Match as the proportion of sentences for which the model produced the gold phrase (i.e.


Table 10 provides a few qualitative examples of how the phrases generated by the Edits model differ from those generated by the General model.

we see that the Edits model proposes phrases which better capture the discourse function of the human edit: e.g.


In order to evaluate how cross-lingual structural variation hinders the design of effective generalpurpose algorithms, we propose the task of language modeling (LM) as a testbed.

we opt for a full-vocabulary setup where no word encountered at training time is treated as an unknown symbol, in order to a) ensure a fair comparison across languages with different word frequency rates and b) avoid setting an arbitrary threshold on vocabulary size (Cotterell et al., 2018).


Through this large-scale multilingual analysis, we shed new light on the current limitations of standard LM models and offer support to further developments in multilingual NLP.

we demonstrate that the previous fixedvocabulary assumption in fact ignores the limitations of language modeling for morphologically rich languages.


(5), all uttering the sentence "I will guard the doors and I will not open (them)".

3 (2) tôi consider how tense and person are expressed on verbs.


Words are regarded as the smallest meaningful unit of speech or writing that can stand by themselves in human languages, but not the smallest indivisible semantic unit of meaning.

the meaning of a word can be represented as a set of semantic components.


Since some sememes only appear few times in HowNet, which are expected to be unimportant, we filter out those low-frequency sememes.

the frequency threshold is 5, and the final number of distinct sememes used in our experiments is 1, 400.


Cross-lingual NER attempts to address this challenge by transferring knowledge from a high-resource source language with abundant entity labels to a low-resource target language with few or no labels.

in this paper we attempt to tackle the extreme scenario of unsupervised transfer, where no labeled data is available in the target language.


To address the above issues, we propose a new lexical mapping approach that combines the advantages of both discrete dictionary-based methods and continuous embedding-based methods.

we first project embeddings of different languages into the shared BWE space, then learn discrete word translations by looking for nearest neighbors in this projected space, and finally train a model on the translated data.


Instead of using a common distance metric such as cosine similarity, we adopt the cross-domain similarity local scaling (CSLS) metric (Lample et al., 2018), which is designed to address the hubness problem common to the shared embedding space (Dinu and Baroni, 2014).

denotes the mean cosine similarity between x i and its K neighbors y t .


(2016), and use bidirectional LSTMs for both the character level and word level neural networks.

given an input sequence of words (w 1 , w 2 , ..., w n ), and each word's corresponding character sequence, the model first produces a representation for each word, x i , by concatenating its character representation with its word embedding.


Here, we conduct ablation studies over these two variables.

we consider the following three variants.


Then, these words and their contexts in the training data are memorized with a contextual memory which is finally accessed dynamically during decoding to solve the translation problem of the troublesome words.

we first decode all the source sentences of the bilingual training data with baseline NMT and define the troublesome source words according to the distance between the predicted words and the gold words.


For this criterion, we utilize the predicted probability gap between the gold target word and the top one.

the gap can be calculated by: is the top one in the probability distribution at the i th decoding step.


For a troublesome word, we now introduce how to build a contextual memory M to store its translation knowledge.

the contextual memory contains five elements: each of them is described as follows: • s m is a troublesome source word.


We can see that our method is still effective on EN-DE translation.

when the translation unit is word, the proposed method improves the baseline by 1.13 BLEU points.


In order to bypass the expensive human annotation and achieve the first step towards the ultimate goal of building a universal dialog system, we set out to build a cross-lingual state tracking framework.

we assume that there exists a source language with dialog belief tracking annotations while the target languages have no annotated dialog data of any form.


In this paper, we are interested in learning a cross-lingual DST.

we assume that the DST for the source language has access to a human-annotated training dataset D while the DSTs for the target languages do not have access to annotated data in other languages except for testing data.


Furthermore, we assume that all the languages share a common multi-lingual database, whose column/row names and entry values are stored via multiple languages (see the database in Figure 1).

the ontology of dialog among different languages is known with a one-to-one mapping between them (e.g., greek=griechisch=greco, food=essen=cibo).


(2017) showed that targeting specific words during data augmentation improves the generation of these words in the right context.

adding synthetic data to the training data has an impact on the prediction probabilities of individual words.


However, all targeted sampling variants in turn outperform random sampling.

the best performing model for GermanÑEnglish, MEANPREDLOSS, uses the mean of prediction loss for the target vocabulary to oversample sentences including these tokens.


In this paper, we propose a multi-domain NMT model with word-level domain context discrimination.

we first jointly model NMT with monolingual attention-based domain classification tasks.


Here we describe only the modeling procedure of the domain classifier, while it is also applicable to the adversarial domain classifier.

e r (x) is defined as follows: where , and v a and W a are the relevant attention parameters.


Since domain-specific and domain-shared contexts have different effects on NMT, and thus should be distinguished and separately captured by NMT model.

we first leverage the sentence representations E r (x) and E s (x) to generate two gating vectors, g r i and g s i , for annotation h i in the following way: where W * gr , W * gs , b gr and b gs denote the relevant matrices and bias, respectively.


With these two vectors, we construct domain-specific and domain-shared annotations h r i and h s i from h i : The upper half of Figure 2 illustrates the architecture of our decoder.

with the attention weights of target words from the domain classifier, we employ word-level cost weighting strategy to refine model training.


Given a mixed-domain training corpus D = {(x, y, d)}, we train the proposed model accord-ing to the following objective function: where J nmt ( * ), J s dc ( * ), J t dc ( * ) and J s * adc ( * ) are the objective functions of NMT, source-side domain classifier, target-side domain classifier, and source-side adversarial domain classifier, respectively, θ={θ nmt , θ s dc , θ t dc , θ s1 adc , θ s2 adc }, and λ is the hyper-parameter for adversarial learning.

to ensure encoding accuracy of domain-shared contexts, we follow  to adopt an alternative two-phase strategy in training, where we alternatively optimize J (D; θ) with θ s1 adc and {θ-θ s1 adc } respectively fixed at a time.


We consider a version of the task that only relies on monolingual corpora in the tradition of Rapp (1995) and Fung (1995).

the goal is to produce a bilingual lexicon primarily from unannotated raw text in each of the respective languages.


In the previous section, we have formulated the induction of a bilingual lexicon as the search for an edge set E, which we treat as a latent variable that we marginalize out in equation 2.


Indeed, we only need to change the edge-set prior p(m) to allow for edge sets other than those that are matchings.

a matching enforces a one-to-one alignment between types in the respective lexicons.


by enforcing more reliable translation pairs for subsequent iterations.

15 the hubness of a target word measures how often it occurs in the neighborhood of the query terms.


Another support for the plausibility of unsupervised word translation came with the realization that when words are represented as vectors that encode co-occurrences, the mapping between two languages is well captured by an affine transformation (Mikolov et al., 2013b).

not only that one can expect the most frequent words to be shared, one can also expect the representations of these words to be similar up to a linear transformation.


We use the reconstruction cost as an unsupervised metrics for measuring convergence of MBC-ICP.

we measure how closely every x ∈ X and y ∈ Y is reconstructed by a transformed word from the other domain.


In the future, we plan to investigate better methods for training the SAT to further shrink the performance gap between the SAT and the Transformer.

we believe that the following two directions are worth study.


(2018) show that sampling synthetic sources is more effective than beam search.

they sample multiple sources for each target whereas we draw only a single sample, opting to train on a larger number of target sentences instead.


Adding noise to input sentences has been very benefi-cial for the autoencoder setups of (Lample et al., 2018a;Hill et al., 2016) which is inspired by denoising autoencoders (Vincent et al., 2008).

we transform source sentences with three types of noise: deleting words with probability 0.1, replacing words by a filler token with probability 0.1, and swapping words which is implemented as a random permutation over the tokens, drawn from the uniform distribution but restricted to swapping words no further than three positions apart.


We report the cross entropy loss averaged over all tokens and separate the loss over the synthetic data and the real bitext data.

we choose the setup with 24M synthetic sentences.


We found it beneficial to adjust the ratio of bitext to synthetic data observed during training.

we tuned the rate at which we sample data from the bitext compared to synthetic data.


Generating synthetic sources by sampling or by adding noise to beam outputs leads to higher accuracy than argmax inference which is typically used.

sampling and noised beam outperforms pure beam by 1.7 BLEU on average on newstest2013-2017 for WMT English-German translation.


1 Transliteration is the process of transducing names from one writing system to another (e.g., ओबामा in Devanagari to Obama in Latin script) while preserving their pronunciation (Knight and Graehl, 1998;Karimi et al., 2011).

back-transliteration from foreign languages to English has applications in multilingual knowledge acquisition tasks including named entity recognition (Darwish, 2013) and information retrieval (Virga and Khudanpur, 2003).


We show that a weak generation model can be iteratively improved using constrained discovery.

our work uses a weak generation model to discover new training pairs, using constraints to drive the bootstrapping.


Most of the prior methods can be characterized as a series of linear transformations.

(Artetxe et al., 2018a) propose a framework to differentiate prior methods in terms of which transformations they perform: embedding normalization, whitening, re-weighting, de-whitening, and dimensionality reduction.


The difference is that the locally trained map is only trained using training data from the neighborhood, whereas the global map is trained using training data from the neighborhood but also from all other neighborhoods and more (˜10000 word pairs).

the training data for global maps is a superset of the local training data.


Although the introduction of RNNs as a decoder has resulted in substantial improvements in terms of translation quality, simultaneously it imposes a serious restriction on the capability of encoder-decoder framework caused by the structure of RNNs.

when the RNN decoder generates the t-th word y t in decoding phase, only y <t can be utilized, while the possible words y >t are directly neglected.


Towards the effect of the decoding depth set {2,3,4,5}, our multi-pass decoders obtain the approximate results, but the whole curve of BLEU is on an upward trend.

the multi-pass decoder with decoding depth 5 achieves the best performance with 38.64 BLEU, while the one with decoding depth 3 performs the worst among the decoding depth set with 38.55 BLEU.


As shown in Table  2, the proposed adaptive multi-pass decoder obtains an improvement about 0.41 to 0.5 BLEU on average over the {2,3,4,5}-pass decoder, which demonstrates the effectiveness of the policy network.

the adaptive multi-pass decoder outperforms the multi-pass decoder with a fixed decoding depth by 0.69, 0.71, 0.68 and 0.45 BLEU scores on NIST03, NIST04, NIST05 and NIST06 datasets at most.


These results indirectly proves that our policy network has the capability of choosing the proper decoding depth.

when the source sentence is difficult to be translated such as some long sentences, more decoding passes are consumed to improve the translation quality, while in simple cases such as short source sentences, one-pass decoding is adequate.


Compared with the dominant NMT model, two attention models were involved to compute the context vectors.

an attention model is utilized to calculate the context vector for each machine system, while the other attention model obtains the context vector over the all context vectors of machine systems.


Because we would like to focus our data collection on noisy comments, we devise criteria that allow us to distinguish potentially noisy comments from clean ones.

we compile a contrast corpus composed of clean text that we can compare to, and find potentially noisy text that differs greatly from the contrast corpus.


Once enough data has been collected, we isolate 15, 000 comments in each language by the follow-  ing procedure: • Remove all duplicates.

this handles comments that might have been scraped twice or automatic comments from bots.


We run a series of tests to count the number of occurrences of some of the types of noise described in Section 2.

we pass our data through spell checkers to count spelling and grammar errors.


The results are recorded in the last row of each section in Table 5.

for the languages with a segmental writing system, English and French, spelling errors are the dominant type of noise, followed by grammar error.


• en ↔ ja: We concatenate the respective train, validation and test sets of the three corpora mentioned in 3.2.

we detokenize the Japanese part of each dataset to make sure that any tokenization we perform will be uniform (in practice we remove ASCII spaces).


To better understand the types of errors made by our model, we count the n-grams that are overand under-generated with respect to the reference translation.

we compare the count ratios of all 1-to 3-grams in the output and in the reference and look for the ones with the highest (over-generated) and lowest (under-generated) ratio.


Specifically, we compare the count ratios of all 1-to 3-grams in the output and in the reference and look for the ones with the highest (over-generated) and lowest (under-generated) ratio.

we find that in English, the model undergenerates the contracted form of the negative ("do not"/"don't") or of auxiliaries (""/"I'm").


Recently there has been a growing interest in developing natural language representations that can be transferred across tasks (Vendrov et al., 2016;Wieting et al., 2016;Conneau et al., 2017, inter alia).

sNLI (Bowman et al., 2015) and MultiNLI (Williams et al., 2017) encourage architectures that first encode the hypothesis and the premise independently before a comparator neu-1 nlp.cs.washington.edu/piqa ral network is applied.


the output size is 2k) to model both R θ and K θ .

r θ (D, j) is the j-th output of BiLSTM on top of D, and we similarly define K θ with unshared parameters.


We therefore formulate the estimation of θ z as a reinforcement learning problem via a policy gradient method.


(2018) extended this approach by implementing a neural re-ranking of the candidate document, yet keeping the fixed number of n selected documents unchanged.

the interplay between both modules for document retrieval and machine comprehension has not yet been studied.


For this purpose, we run a series of experiments in order to obtain a better understanding of the interplay between document retrieval and machine comprehension modules.

we specifically compare the recall of document retrieval to the end-toend performance of the complete QA system; see Fig.


As a naïve baseline, we propose a simple threshold-based heuristic.

n i is determined such that the cumulative confidence score reaches a fixed threshold θ ∈ (0, 1].


That is, n i is determined such that the cumulative confidence score reaches a fixed threshold θ ∈ (0, 1].

formally, the number n i of retrieved documents is given by the heuristic fills up documents until surpassing a certain confidence threshold.


For GloVe, recommended parameters in (Pennington et al., 2014) are used.

we set α = 3 4 , x max = 100, initial learning rate as 0.05, 100 iterations.


Although pattern-based approaches such as LRA have achieved promising results in some semantic relational tasks (Turney, 2008a,b), they have a crucial problem that a co-occurrence of all semantically related word pairs cannot be observed because of Zipf's law, which states that the frequency distribution of words has a long tail.

most words occur very rarely (Hanks, 2009).


Our Bag-of-Substring (BoS) word vector generation model views a word as a bag of its substrings, or character n-grams.

we maintain a vector lookup table for each possible substrings (or character n-grams) of length between l min and l max .


In this paper, we propose an alternative statistical dependency-based approach to unsupervised word translation.

we propose to search for the cross-lingual word pairing that maximizes statistical dependency in terms of squared loss mutual information (SMI) (Yamada et al., 2015;Suzuki and Sugiyama, 2010).


The SMI is an fdivergence (Ali and Silvey, 1966).

it is a non-negative measure and is zero only if the random variables are independent.


Although each model uses auxiliary representations from the other model, but the learning of the model focuses on its own parameters.

gradient calculation is disabled along the dashed lines in Fig.


These tweets were sampled from data corresponding to 500 optimist and 500 pessimist users, and were manually annotated using Amazon Mechanical Turk.

each tweet was manually annotated by five independent annotators using a score between −3 (very pessimistic) and 3 (very optimistic).


In our second experiment, we investigate the correlation between sentiment and optimism / pessimism, and argue that sentiment analyzers, that are trained to predict sentiment (Liu, 2012;Pang and Lee, 2008), fail to detect optimism and pessimism.

we train several sentiment classifiers on the large SST and TSA sentiment datasets (described in Section 2) and evaluate the performance of these classifiers on the optimism/pessimism categories from the OPT dataset.


Figure 2 shows the normalized number of examples from optimism and pessimism categories classified as positive, negative and neutral, using the CNN model trained on TSA.

we show how many tweets from the set of optimistic (or pessimistic) tweets in the OPT dataset are predicted as negative, neutral or positive by the TSA sentiment classifier.


So human readers can "imagine" the article before they read it and implicitly base their behavior on their expectation."

(Hardt and Rambow, 2017) readers are able to anticipate the contents of an article in advance from a headline, because of the linguistic and world knowledge that they bring to bear when assessing the headline.


As prior work, we concentrate on direct disagreement and agreement between quote-response (Q-R) pairs.

in the proposed model, the size of hidden units is 128 and all word embeddings are initialized by GloVe (Pennington et al., 2014) of 300d.


We obtain T by minimizing the least-square difference in T captures the retrofitting operation, and allows us to modify the test subjects' representations as if age and gender were known, despite the absence of class information.

by applying T to the matrix of the author embeddings in the unlabeled test set D test , we obtain a retrofitted version D test that preserves the transformation learned on the training data.


As a subjective factor, emotion performs an elemental difference between humans and machines.

machines that could understand emotion would be more responsive to human needs.


Within this work, we investigate how shifts in themes across time affects performance of the intention classifications for the dialogue.

we focus on the patient intent classification task and propose a method that adopts the temporal factor by domain adaptation to improve performance of the classifiers.


We evaluate our approach on a dataset of college alcoholism (Carey et al., 2009;Borsari et al., 2012), containing transcripts of MI conducted with U.S. college students.

we first explore the theme shift and give a brief analysis by topic modeling (Blei et al., 2003).


In this paper, we focus on classifying patient behavior on the utterance-level.

we classify patient behavior on collapsed MISC  annotation codes into with three categories: "CT": Change talk indicates utterances that reflect motivating factors related to change; "ST": Sustain talk indicate the patient has no intentions to change; "FN": Follow neutral means there is no indication of patient inclination.


We feed four types of information to the model: topic-and word-level data of the utterance (content), preceding interventionist verbal behavior (context) and prior MISC annotations of utterances (MISC).

we empirically extracted previous 5 utterances as context and 10 previous codes as MISC 2 , where we set "unk" as the default.


Example 1 shows one such argument alongside a speech snippet, demonstrating the unique nature of this domain and data.

the argument against the motion is implied from the speech, but is not explicitly mentioned in it.


Given a motion and a speech, the annotators were instructed to listen to it once, preferably without pausing, and select which ones of iDebate argument titles were mentioned in it, or None if none of them was.

they were instructed to answer positively if it would be correct to say that "the speaker argued that arg ..." where arg is the argument's title.


To explore this issue in our dataset, we assessed the correlation between several features of the argument title and the label.

we computed the Spearman's correlation (Spearman, 1904) between the label and the title's length, occurrences of named entities or negation words in the title, and the correlation between the labels and the titles' 100 most frequent words, stopwords excluded.


For each method, we select a threshold that maximizes accuracy over dev, and apply it to test.

an argument is considered to be mentioned in the speech if its similarity score is above the threshold.


The explicit-to-implicit (Exp2Imp for short) relation transformation is used.

we propose to introduce active learning into the data expansion process, with the aim to reduce redundancy and reinforce the use of informative instances.


However, its small size, inherent repetitiveness, and synthetic nature limit its broader applicability.

we see it as a viable benchmark for evaluating models, but not for training them.


To extract a full sentence C and its candidate split into S = (S 1 , S 2 ), we   ineni et al., 2002) to ensure similarity between the original and the split versions.

we discard pairs where BLEU(C, S 1 ) or BLEU(C, S 2 ) is less than δ (an empirically chosen threshold).


4 In Set 2, annotators are encouraged to split only in cases where it simplifies the original sentence.

simplicity is implicit in Set 1 and explicit in Set 2.


The high correlations we obtain between the metrics indicate that this may be the case.

bLEU-1ref obtains correlations of 0.86 (p = 7 × 10 −3 ) without splits and of 0.52 (p = 0.04) where splitting is involved.


In this paper, we propose Sequence to Sequence with Prototype Memory Network (S2SPMN) to exploit the relevant information provided by the large dialogue corpus to enhance response generation.

we devise two simple approaches in S2SPMN to select the relevant information (named prototypes) from the dialogue corpus.


To this end, we are interested in exploiting the relevant responses in the training set as soft prototypes to assist the response generation.

in this paper, we propose Sequence to Sequence with Prototype Memory Network (named S2SPMN).


Static PMN: For static PMN(SPMN), we randomly select m responses before training starts and the entire PMN remains unchanged during the training process.

we use the same prototypes for all the post-response pairs.


It is clear that our models (SPMN500, SPMN1000, DPMN100) generate much more informative and valid responses and much less meaningless or "safe" responses than baseline models (S2SA, TAS2S).

sPMN500, sPMN1000 and DPMN100 all  Table 2: Human Annotation outperform s2sA and TAs2s by producing more informative and valid responses.


Table 1 shows examples of bad-endings generated by a self-critical based RL algorithm (model details refer to Section 4).

46.44% generated captions end with phrases as "with a", "on a", "of a", etc.


The second drawback is that RL-based text generation is sample-inefficient due to the large action space.

the search space is of size O(|V| T ), where V is a set of words, T is the sentence length, and | • | denotes the cardinality of a set.


In this work, to tackle these two issues, we propose a simple yet effective solution by introducing coherent language constraints on local action selections in RL.

we first obtain wordlevel n-gram (Kneser and Ney, 1995) model from the training set and then use it as an effective prior.


We directly compare with our counterparts that have the same structures but no n-gram modules.

they are encoder-decoder based self-critical (ED-SC), and the one with attention (Att-SC).


After obtaining the optimized θ * LM , we can use it to constrain the action space similar to the N-gram language model.

given previous t − 1 sampled words from current caption model, we compute p θ * LM (w t |w 0 , w 1 , ..., w t−1 ), which is the probability of the next word over the entire vocabulary.


In this work, we consider an approach for training paragraph captioning models that focuses on increasing the diversity of the output paragraph.

we note that self-critical sequence training (SCST) (Ranzato et al., 2015;Rennie et al., 2016), a technique which uses policy gradient methods to directly optimize a target metric, has been successfully employed in standard captioning, but not in paragraph captioning.


In this paper, we examine it with respect to automatic summary evaluation, which has become most common for system development and evaluation, thanks to its speed and low cost.

we use several variants of the ROUGE metric (Lin, 2004), which is almost exclusively utilized as an automatic evaluation metric class for summarization.


We see in the tables that evaluating system summaries of all lengths against references of a single length often performs on par with the standard configuration.

the single fixed set of 50-word reference summaries performs overall as well as the standard approach, and, although not substantially, is the most effective configuration within the data analyzed.


In particular, the single fixed set of 50-word reference summaries performs overall as well as the standard approach, and, although not substantially, is the most effective configuration within the data analyzed.

in this dataset, the 50-word reference summaries provide a "test sample" for evaluating the longer system summaries, which is as effective as the same length references used by the standard method.


We model this problem as an instance of sequence labeling.

a document is viewed as a sequence of sentences and the model is expected to predict a True or False label for each sentence, where True indicates that the sentence should be included in the summary.


We built a summarizer that only copy input words for outputs with trivial modifications upon the pointer-generator model (See et al., 2017).

it implements a sequence-to-sequence model that uses the soft attention distribution to produce an output sequence whose elements are all from the input sequence, similar to what a pointer network (Vinyals et al., 2015) does.


In recent years, neural networks have been widely used to grade student essays automatically and achieve state-of-the-art performance.

a distributed representation is learned for an essay with variant neural networks and a linear layer is then used to produce the final score.


Our LSTM processes one word at a timestamp.

given the word embedding sequence {x 1 , x 2 , ..., x n } for the essay, the hidden states of the LSTM are calculated as follows: the average value over all hidden states of each LSTM layer are computed, and we concatenate the mean states of the two layers together as the embedding vector of the essay.


A packed evaluation strategy is used for QWK computation and the scoring of each essay is treated as a single action.

dilated LSTM is used to encode an essay, and a softmax layer is utilized for essay grading.


In addition to lexical features, we also extract syntactic features that can inform the model on any anomaly in the structure of the query.

we annotate the query with POS-tags using SyntaxNet POS tagger (Alberti et al., 2015) and extract POS-1, 2, 3-grams.


Compared to WETC C , in calculating C we do not perform top-N operation in W , but directly use W for weighted sum.

we use W -weighted topic vector construction in Step-2 and W -weighted sum of the cosine similarities between word vectors and topic vectors in Step-4.


1 The source code and corpora generated by this research are available at https://github.com/ abhijith-athreya/ASDUS.

titles are concise summaries of the following prose text.


A potential weakness of MLE is the mismatch between b p(X, Y ) and the true data distribution p(X, Y ).

b p(X, Y ) is usually a bootstrap distribution defined only on the observed training pairs, while p(X, Y ) has a much larger support, i.e.


To account for variance, we run each experiment multiple times and report the median BLEU.

each experiment without SwitchOut is run for 4 times, while each experiment with SwitchOut is run for 9 times due to its inherently higher variance.


nearest neighbor), then bucket the instances by the distance to their nearest neighbor and measure the gain in BLEU afforded by SwitchOut for each bucket.

we use (negative) word error rate (WER) as the similarity measure, and plot the bucket-by-bucket performance gain for each group in Figure 2.


Sentence encoder We use a CNN architecture with one convolution layer followed by one max pooling layer to encode a sentence.

the RNFs are applied to every possible window of m words in the sentence Then a max-over-time pooling operation (Collobert et al., 2011) is used to produce a fixed size sentence representation: v = max{C}.


The evaluation results on sentiment classification and answer sentence selection are shown in Table 1 and Table 2  Best published results (Lei et al., 2017) 53.2 89.9 MAP score on WikiQA.

cNN-RNF-LSTM achieves 53.4% and 90.0% accuracies on the fine-grained and binary sentiment classification tasks respectively, which match the state-ofthe-art results on the Stanford Sentiment Treebank.


Key phrase hit rate We examine linear filters and RNFs on the ability to detect a key phrase in a sentence.

we define the key phrases for a sentence to be the set of phrases that are labeled with the same sentiment as the original sentence.


We then employ a novel graph-to-sequence (Graph2Seq) model (Xu et al., 2018), which consists of a graph encoder and a sequence decoder, to learn the representation of the syntactic graph (see §3).

the graph encoder learns the representation of each node by aggregating information from its K-hop neighbors.


Given the syntactic graph G = (V, E), we take the embedding generation process for node v ∈ V as an example to explain the node embedding generation algorithm 1 : (1) We first transform node v's text attribute to a feature vector, a v , by looking up the embedding matrix W e ; (2) The neighbors of v are categorized into forward neighbors N (v) and backward neighbors N (v) according to the edge direction.

n (v) returns the nodes that v directs to and n (v) returns the nodes that direct to v; (3) We aggregate the forward representations of , where k∈{1, ..., K} is the iteration index.


In particular, N (v) returns the nodes that v directs to and N (v) returns the nodes that direct to v; (3) We aggregate the forward representations of , where k∈{1, ..., K} is the iteration index.

this aggregator feeds each neighbor's vector to a fully-connected neural network and applies an element-wise max-pooling operation to capture different aspects of the neighbor set.


The decoder is an RNN which predicts the next token y i given all the previous words y <i = y 1 , ..., y i−1 , the RNN hidden state s i for time-step i and the context vector c i that captures the attention of the encoder side.

the context vector c i depends on a set of node representations (h 1 ,...,h V ) to which the encoder maps the input graph.


Different from previous works, we evaluate the robustness of our model by creating adversarial examples with the hope to investigate the impact of introducing more syntactic information on robustness.

we create two types of adversarial examples and conduct experiments on the ATIS dataset.


This observation motivates us to represent the SQL query as a graph.

we use the following method to transform the SQL query to a graph: 1 SELECT Clause.


In addition, we consider different, semantically equivalent, sources of information.

we compare the paraphrases obtained when generating text from data, from text or from text and data.


Universal Decompositional Semantics (UDS) (White et al., 2016) is a contemporary semantic representation of text (Abend and Rappoport, 2017) that forgoes traditional inventories of semantic categories in favor of bundles of simple, interpretable properties.

uDS includes a practical implementation of Dowty's theory of thematic proto-roles (Dowty, 1991): arguments are labeled with properties typical of Dowty's proto-agent (AWARENESS, VOLITION ...) and proto-patient (CHANGED STATE ...).


Please refer to Appendix B for details on the datasets used in this section.


Instead of training the transferred model by ourselves, in this paper, we adopt the ELMo word representations , which are derived from a bidirectional language model (BiLM) trained on one billion word benchmark corpus (Chelba et al., 2014).

this BiLM has one character convolution layer and two biLSTM layers, and correspondingly there are three internal representations for each word x t , which are denoted as {h LM t,l } 3 l=1 .


Since they are only asked to distinguish between pictures of different concepts, they have no incentive to keep different instances of a concept distinct (if the agents are never asked to distinguish one dog from another, they might eventually become unable to tell dogs apart).

we might be assisting to the inception of a form of categorical perception (Goldstone and Hendrickson, 2010), whereby the agents lose sensitivity to within-category differences.


We evaluate our method with different benchmarks, and it is demonstrated that our method improves the precision of the predicted relations.

we show that capsule networks improve multiple entity pairs relation extraction 1 .


The results indicate the dynamic routing is contributing to improve the performance.

in the iteration algorithm, the b j|i = b j|i +û j|i • v j .


The entity has type labels 1 stored in the KB, and all labels will be assigned to this entity mention.

these are noisy global labels without considering the specific context of the entity mention.


The result shows that the proposed PubSE model consistently outperforms all the baselines with a statistically significant margin, and the advantage is up to 11.8%.

the use of the webpage-level sub-module helps PubSE to handle multi-line publication strings, which yields a significant performance gain.


Katiyar and Cardie (2018) used recurrent neural networks to extract features for a hypergraph which encodes all nested mentions based on the BILOU tagging scheme.

given a sequence of words {x 0 , x 1 , .


c w i denotes the representation learned by a character-level model using a bidirectional LSTM.


Formally, when a word x i is shifted to the stack at time step k, its representation is computed as: Finally, the state of the system p k is the concatenation of the states of buffer, stack and action history: We employ the greedy strategy to maximize the log-likelihood of the local action classifier in (1).

let z ik denote the k-th action for the i-th sentence, the loss function with 2 norm is: where λ is the 2 coefficient.


• If the top element of the stack is labeled, then unary actions are not valid.


Seeing that explanations in addition to IF does not outperform an IF baseline, we next measure whether explanations help a user not already familiar with Vicki via IF.

we evaluate if explanations help against a No IF baseline by providing an explanation only in the test phase, and no IF (see Fig 4; No IF + Explanation).


This work targets these shortcomings by designing a model and training scheme targeted towards entity tracking.

we introduce Figure 1: A LAMBADA example where the final word "julie" (with reference chain in brackets) is the answer, y, to be predicted from the preceding context x.


We found it useful to have two hyperparameters per auxiliary task governing the number of distinct named entity word types and tokens used in defining the losses L 1 and L 2 .

per document these hyperparameters control in a top-tobottom order the number of distinct named entity word types we attempt to predict, as well as the number of tokens of each type considered.


Figure 1 shows an example of a sequence of three questions.

we collect bonus questions from http://quizdb.org for the tournaments in 2008-2018.


In the future, we would like to invest in building better sequential question answering models that push the accuracy beyond the presented baselines.

we will look at how to better model the interaction between the reader and the relation embedding model and how to improve the relation embedding model itself by adopting ideas from the relation extraction literature (Miwa and Bansal, 2016;Peng et al., 2017;Ammar et al., 2017).


Therefore, Equation (4) becomes: Each row in the RBF kernel matrixK is now interpreted as a feature vector, going from step S2 to step S3 in Figure 2.

each sample z i is represented by a feature vector that contains the similarity between the respective sample z i and all the samples in Z (step S3 in Figure 2).


The overall architecture differs significantly from previous work.

we design two novel neural units that take target aspects into account.


A sentence of length n can be represented as a matrix A convolution filter w ∈ R h×k with width h is applied to the word matrix to get high-level representative features.

for a word window v i:i+h−1 ∈ R h×k , a feature c i is generated by where represents element-wise product, b ∈ R is a bias term and f is a non-linear function.


In this paper, we target at improving the performance of multi-label emotion classification with the help of sentiment classification.

we propose a new transfer learning architecture to divide the sentence representation into two different feature spaces, which are expected to respectively capture the general sentiment words and the other important emotion-specific words via a dual attention mechanism.


In recent years, the number of user-generated comments on social media platforms has grown exponentially.

social platforms such as Twitter allow users to easily share their personal opinions, attitudes and emotions about any topic through short posts.


For emotion classification, since emotion words are relatively more important for final predic-tions, we adopt the widely used attention mechanism (Bahdanau et al., 2014) to select the key words for sentence representation.

we first take the final hidden state h n as a sentence summary vector z, and then obtain the attention weight α i for each hidden state h j as follows: where W h , W z ∈ R a×d and v ∈ R a are learnable parameters.


This corresponds to predicting one of the four classes N/A, positive, negative and neutral for each aspect.

we obtain a hidden representation of an input document X in the following manner: where embed ∈ { word2vec, glove, fasttext } and DO = dropout (Hinton et al., 2012).


Overall, all models predicted gender of people's names substantially above chance (50%).

the multivariate models performed better than the single-feature model though the difference is small, suggesting that gender information is likely encoded in a restricted set of features.


We used a conservative criterion in determining gender of brand names.

we took majority vote from prediction of the three classifiers for any given brand name as opposed to relying on prediction from a single classifier.


First, we evaluate ConvLexLSTM performance in an ablation experiment to determine the role played by each component for emotion detection.

we compare ConvLexLSTM with ConvLSTM (a model that has the same architecture as ConvLexLSTM, but does not use any external lexicon), CNN, LSTM, and support vec-  tor machine (SVM) with the (concatenated) features from the seven lexicons (described above).


1 Sequence modeling in deep learning has largely focused on autoregressive modeling.


The proposed approach could instead be viewed as learning a conditional denoising autoencoder which is known to capture the gradient of the logdensity.

we implicitly learn to find a direction Y in the output space that maximizes the underlying true, data-generating distribution log P (Y |X).


Then, the goal of learning is to maximize the log-probability of the original reference Y ⇤ given the corrupted version.

to minimize Once this cost J DAE is minimized, we can recursively perform the maximum-a-posterior inference, i.e.,Ŷ = arg max Y log p ✓ (Y |X), to findŶ that (approximately) maximizes log p(Y |X).


l of iterative refinement, the model is trained to maximize the log-likelihood of target Y given the weighted mixture of generated samples from the previous iterationŶ l 1 and a corrupted targetỸ .

the corrupted version of target is "infused" We compose three transformer blocks ("Encoder", "Decoder 1" and "Decoder 2") to implement the proposed non-autoregressive sequence model.


A slightly more sophisticated way is to jointly consider the ASR/MT and language model in a beam search decoder (Amodei et al., 2016).

at each time step, the decoder appends every symbol in the vocabulary to each sequence in the current candidate set.


However, a potential problem with this formulation is that the second term (corresponding to the inferior hypotheses) may dominate the optimization.

the training is almost always driven by the x i,j 's, but does not effectively enhance the discrimination.


We define x i,0 x i and assume that the candidates {x i,j } K j=1 are sorted such that for ASR, and for MT.


We apply the LMs trained under different criteria to rescore the beams in various ASR systems.

we are interested in knowing which of the two training mechanisms is better: minimizing PPL (e.g., the RNNLM in Section 3.1), or fitting to the WER metric by the proposed methods.


Although the attention-only model had somewhat better perplexity on the validation set, when the models were tested specifically on challenging subject-verb agreement dependencies, the attention-only model made three times as many errors as the LSTM.

the LSTM learned more robust syntactic representations, but this advantage was not reflected in its average perplexity on the corpus, since syntactically challenging sentences are relatively infrequent.


While evaluation on naturally occurring examples is appealing, this approach has its limitations (see Section 2).

syntactically challenging examples are sparsely represented in a corpus, their identifica-tion requires a clean parsed corpus, and naturally occurring sentences are difficult to control for confounds.


By contrast, in cases where there are multiple nouns in the sentence, identifying which of them is the subject of a given verb requires understanding the structure of the sentence.

the relevant subject is not necessarily the first noun of the sentence: (4) Agreement in a sentential complement: a.


Crucially for the purposes of the present work, the scope of negation is structurally defined.

the negative noun phrase needs to c-command the NPI: the syntactic non-terminal node that dominates the negative noun phrase must also dominate the NPI.


Perplexity: The perplexity of the n-gram model on the Wikipedia test data was 157.5, much higher than the perplexity of the single-task RNN (78.65) and the multi-task RNN (61.10).

perplexity tracked accuracy on our syntactic data set -an unsatisfying outcome given our goal of dissociating perplexity and our syntactic evaluation method, but an expected one given that each model was conditioned on richer information than the previous one.


Let's recall the WFSA B (Figure 1 and Example 5) using the real semiring hR, +, •, 0, 1i.

equation 6 is recovered by parameterizing B's weight functions µ and with Denote the resulting WFSA by B i , and we have: Running a single layer RNN in example 6 over any nonempty input string x 2 ⌃ + , the ith dimension of its hidden state at time step t equals the score assigned by B i to x :t : the ith dimension of the RNN in example 6 can be seen as a WFSA structurally equivalent to B.


The recurrences of single-layer T-GRU, T-LSTM, and QRNN are rational.

a single-layer d-dimensional QRNN using K-window convolutions can be recovered by a set of d WFSAs, each with O(2 |⌃| K 1 ) states.


Another notable modification is the additional state q 3 , which is used to create a "shortcut" to reach q 2 , together with an "-transition.

starting from q 0 , a path can now take the "-transition and reach q 3 , and then take a transition with weight µ 2 to reach q 2 .


Furthermore, we try to retain the effectiveness of the pruned model.

we modify the L 1 regularization for encouraging the selection weights to be not only sparse but binary, which protects the retained layer connections from shrinkage.


Recently, the dense connectivity and residual connectivity have been proposed to handle these problems (He et al., 2016;Huang et al., 2017).

dense connectivity refers to adding direct connections from any layer to all its subsequent layers.


After that, we try to further retain the effectiveness of the compressed model.

we choose to prepare the LMs for the pruned inputs, thus making them more robust to pruning.


this dropout is only applied to the input of recurrent layers, which aims to imitate the pruned input without totally removing any layers.


We refer to their output as To ensure the model to predict valid label sequences, we append a first-order conditional random field (CRF) layer to the model (Lample et al., 2016).

the model defines the generation probability of is the set of all generic label sequences for U and φ(y t−1 , y t , u t ) is the potential function.


Similarly, LMs encode a specific context as the hidden states of RNNs, and use them to predict future contexts.

we recognize the cell states of LSTMs are more like to be the sentence embedding (Radford et al., 2017), since they are only passed to the next time stamps.


Our implementations are based on the PyTorch 0.3.1 5 , and all experiments are conducted on the CoNLL03 dataset with the Nvidia GTX 1080 GPU.

due to the limited size of CoNLL03 test set, we measure such speeds on the training set.


We further studied the layer selection patterns.

we use the same setting of LD-Net (9) in Table 3, conduct model pruning using for 50 times, and summarize the statics in Figure 6.


Many attempts have been paid to develop better neural language models (Zilly et al., 2017;Inan et al., 2016;Godin et al., 2017;Melis et al., 2017).

with extensive corpora, language models can be well trained to generate high-quality sentences from scratch (Józefowicz et al., 2016;Grave et al., 2017;Li et al., 2018;Shazeer et al., 2017).


Intuitively, if events inside a document are organized around the core content, a model capturing their relations well should easily identify the intruder(s).

we take a bag of unordered events {O 1 , O 2 , .


To observe the ranking among the salient origins, we compute a separate AUC score between the intruders and the salient origins, denoted as SA-AUC.

sA-AUC is the AUC score on the list with non-salient origins removed.


Feature Based: LeToR outperforms the baselines significantly on all metrics.

its P@1 value outperforms the Frequency baseline the most (4.64%), indicating a much better estimation on the most salient event.


This paper uses the learned word embeddings as the source of basic features.

we use the Skip-gram model (Mikolov et al., 2013) to learn word embeddings on the NYT corpus.


In this section, we introduce an invertibility condition for our neural projector to tackle the optimization challenge.

we constrain our neural projector with two requirements: (1) dim(x) = dim(e) and (2) f −1 φ exists.


Figure 2 shows the available transitions in the top-down algorithm.

the Shift transition moves the first (leftmost) word in the buffer to the stack; the Non-Terminal-X transition pushes onto the stack the non-terminal node X that should be on top of a coming constituent, and the Reduce transition pops the topmost stack nodes until the first non-terminal node appears (which is also popped) and combines them into a constituent with this non-terminal node as their parent, pushing this new constituent into the stack.


In that way, we easily simulate test time conditions, when the parser greedily chooses the highest-scoring transition, even when it is not an optimal one, placing the parser in an incorrect state.

we run experiments on development sets for each benchmark/algorithm with three different error exploration probabilities and choose the one that achieves the best F-score.


Non-surjectivity Our encoding, as defined formally in Section 2.1, is injective but not surjective, i.e., not every sequence of |w| − 1 labels of the form (n i , c i ) corresponds to a tree in T |w| .

there are two situations where a label sequence formally has no tree, and thus Φ −1 |w| is not formally defined and we have to use extra heuristics or processing to define it: • Sequences with conflicting nonterminals.


To represent a word w i , we are using information of the word itself and also contextual information from w [i−1:i+1] .

2 : • We extract the word form (lowercased), the PoS tag and its prefix of length 2, from .


As an engineering matter, we now make the strong assumption that the unordered dependency trees are similar across languages.

we suppose that different languages use similar underlying syntactic/semantic graphs, but differ in how they realize this graph structure on the surface.


To do this, first we estimate a bigram language modelq from the actual distribution q of POS sequences observed in u.

second, let p θ denote the distribution of POs sequences that we expect to see in B , that is, POs sequences obtained by 2 the 93% of nodes tagged with NOUN, PROPN, PRON or VERB in Universal Dependencies format.


These node bigram probabilities can be obtained by enumerating all possible orderings π, a matter we return to below.

it is now easy to compute c a : c a inherits all non-boundary bigrams st that fall within its child constituents (via c within a ).


= 5040 candidates per node, again using SJT.

8 in this paper we run exact algorithms ( §3), but only on a subset of B.


Unlike Wang and Eisner (2017), we exclude duplicated languages in development and testing.

13 there are 3 duplicated sets: {grc, grc proiel}, {la, la proiel, la itt}, and {fi, fi ftb}.


Similar to word accuracy, the more relevant to the question the predicted words, the better the VQA performance.

when the word-question relevance exceeds 0.8, the predicted words are highly pertinent to the question, boosting the VQA accuracy to 76.15%.


Each recipe includes an arbitrary number of steps containing both textual and visual elements.

each step of a recipe is accompanied by a 'title', a 'description' and a set of illustrative 'images' that are aligned with the title and the description.


Given a task, we first randomly select a set of steps from each recipe and construct our questions and answers from these steps according to the task at hand.

we employ the modality that the comprehension task is built upon to generate the candidate answers and use the remaining content as the necessary context for our questions.


employ a different strategy and select the distractors from the relevant modalities (titles, descriptions or images), which are not too far or too close from the correct answer.

we employ the following simple heuristic.


In our implementation, we used a uni-directional stacked LSTM architecture with 3 layers, in which we feed the context of the question to the network in a sequential manner.

we preserve the temporal order of the steps of the recipe while feeding it to the neural model, by mimicking the most common reading strategy -reading from top to bottom.


We aim to provide a dataset that merges the best qualities from all of the previous datasets as well as focus on multimodal compositionality.

we collect a new large-scale dataset that is built on natural video content with rich dynamics and realistic social interactions, where questionanswer pairs are written by people observing both videos and their accompanying dialogues, encouraging the questions to require both vision and language understanding to answer.


We consider temporal adjectives, adverbs, and prepositions found both by closely analyzing moment-localization datasets and consulting lists containing words which belong to different parts of speech.

we rely on the preposition project (Litkowski and Hargraves, 2005) † to scrape relevant temporal words.


We first compare different moment localization models on TEMPO -TL (Table 4).

our model performs well on "before" and "after" words.


When considering all sentence types, MLLC has the best performance across all metrics.

our model has the strongest performance for all sentence types considering the mIoU metric.


Clearly, the score variances for CARD-660 are significantly lower than those for the two crowdsourced datasets, i.e., SimVerb-3500 and RW.

6 for the majority of pairs in CARD-660 the annotation variance is lower than the other two datasets' first quartile (bottom of the blue square  which splits the lower 25% of the data from the top 75%).


In the remainder of this paper, we provide two sets of experiments to showcase the challenging nature of our dataset.

in Section 5.1 we report the performance of common pretrained word embeddings on CARD-660, and in Section 5.2 we provide experimental results for state-ofthe-art rare word representation techniques.


Table 4 provides correlation performance results for different embedding sets on the RW and CARD-660 datasets.

we considered different variants of Word2vec 7 (Mikolov et al.,7 https://code.google.com/archive/p/word2vec/ 2013) and Glove 8 (Pennington et al., 2014), two commonly-used word embeddings that are trained on massively large text corpora; Dependencybased embeddings 9 (Levy and Goldberg, 2014) which extends the Skip-gram model to handle dependency-based contexts; LexVec 10 (Salle et al., 2016) which improves the Skip-gram model to better handle frequent words; and ConceptNet Numberbatch 11 (Speer et al., 2017) which exploits lexical knowledge from multiple resources, such as Wiktionary and WordNet, and was the best performing system in SemEval 2017 Task 2.


Table 1 shows that the red words are more important than the blue words when distinguishing the sense of the target word.

we should pay more attention to the words which can "overlap" between the context and the gloss when generating the representations of context and gloss.


, e Tx ] as the word-level representations of the sentence.

context's word-level representations are denoted as [e c 1 , e c 2 , .


For the non-hierarchical model CAN in Figure 1, the final score µ and ν are generated only by the outputs of the one level co-attention layer.

for the word-level co-attention model CAN w : For the sentence-level co-attention model For the hierarchical co-attention model HCAN in Figure 2, the outputs of the word and sentence level layer are merged together to generate the final results.


Both of them train a dedicated classifier for each word individually.

each target word has its own parameters.


In order to ablate the co-attention mechanism, we replace the co-attention function CoAt in Equation 1 with a function Avg which simply calculates the average of input representation vectors.

in function Avg, the outputs c = j C :j and g = j G :j .


We define the SOCO probability of words w and v as where w : v denotes a SOCO and V is the vocabulary.

p (w : v) is the probability that two based on first-order word vectors, such as in (Schütze, 1998).


This allows us to discover word senses in an unsupervised manner, while preserving the simplicity of the skip-gram approach.

unlike previous probabilistic models for multisense word embeddings, disambiguated skip-gram can be seen as a simple feed-forward neural network amendable to end-to-end training with back-propagation.


Furthermore, disambiguated skipgram admits an effective pruning strategy for the learned word sense embeddings.

we control the granularity of the learned representations by penalizing the entropy of the probability distributions learned by the disambiguation model.


We will also use notation y w for a vector y (from some set of vectors indexed by the vocabulary words) corresponding to the word w. Note that in this case we disregard the position of w in X and use just the word as the index.

if some word w occurs multiple times in X, all occurrences share a single vector y w .


That said, we can prune the sense embedding vectors by considering their probabilities according to the learned disambiguation model.

after training we estimate the marginal probability: for each word d ∈ D and each sense index j ∈ {1, .


10) by adding to it an entropy S of the probability distribution p (z = j | w, C w ): where: The hyper-parameter γ, which we further call entropy cost, controls the strength of the regularization and, in turn, the granularity of the learned sense representations.

γ > 0 encourages the model to learn more coarse-grained sense representations, whereas γ < 0 increases the granularity of the learned senses.


(2016) demonstrated that contextual word similarity experiments do not reflect the quality of multi-sense representations.

the best performance in contextual word similarity task is often achieved by the baseline skip-gram model, which does not recognize word senses.


This is a consequence of the fact that disambiguated skip-gram discovers word senses using only the information about the contexts in which these words occur.

it does not employ any supervision from an external knowledge base.


In this evaluation we follow the experimental setup from (Bartunov et al., 2016), allowing for direct comparison with the results reported therein.

we evaluated disambiguated skip-gram on the datasets from the SemEval-2007 Task 2 competition (SE-2007), SemEval-2010 Task 14 competition (SE-2010), SemEval-2013 Task 13 competition (SE-2013) and the Wikipedia Word-sense Induction (WWSI) dataset introduced by Bartunov et al.


We use a simple procedure for resolving word senses.

we average all sense embedding vectors of all context words and select the sense z w of the center word w whose embedding vector is most similar to the average vector: where:c The intuition behind this procedure is that we expect the average to preserve a shared component in the embedding vectors, namely embeddings for senses related to the sense of the center word, and cancel out embeddings of unrelated senses.


12) indeed allows for pruning senses learned by disambiguated skip-gram.

when the entropy cost increases, disambiguated skip-gram allocates more of the marginal probability mass (Eq.


This translates to coarser sense representations.

the model with no entropy cost learned an average of 4.7 senses per word with marginal probability p ≥ 0.05 (Tab.


However, parallel to this work there has been a significant progress in gradientbased training of models with discrete latent variables.

proposed an unbiased low-variance gradient estimator, called REBAR, that is applicable to models with categorical latent variables.


By controlling how similar the source documents are, we can flexibly adjust the difficulty of the task.

as we show below, we can generate data that exhibits challenging inference problems by exploiting the Wikipedia category structure.


After manually reviewing 20 samples from each *-HARD dataset, we found that NYT-HARD more frequently contains salads that are impossible for humans to pick apart while WIKI-HARD more frequently contains salads that are possible, though challenging.

in 9 out of 20 WIKI-HARD salads we found that access to world knowledge and inference would be beneficial.


First, whether two sentences are from the same narrative cannot be determined globally: there aren't two "globally-contrasted" 10 narratives (or bag-ofwords based topics) from which sentences are sampled.

sentences are always (pairwise) compared in the context of the document mixture from which they are drawn.


We train sentence encoders with dynamic metaembeddings using two well-known and often-used embedding types: FastText (Mikolov et al., 2018;Bojanowski et al., 2016) and GloVe (Pennington et al., 2014).

we make use of the 300-dimensional embeddings trained on a similar WebCrawl corpus, and compare three scenarios: when used individually, when naively concatenated or in the dynamic meta-embedding setting (unweighted, context-independent DME and contextualized CDME).


Despite the success of these prior efforts in yielding multimodal embeddings and applying them to downstream NLP tasks, there are still several deficiencies.

the visual features (as such) are not guaranteed to be suitable for the word embedding task since they are typically optimized independently for another objective (e.g., image classification).


Since x w is real valued, we let p θ (x w |z w ) be a Gaussian parameterized by a generative neural network (or decoder).

for tractability purposes, Σ θ is restricted to be diagonal.


Interestingly, if a non-visual word w 1 shares a similar context c with a visual word w 2 , then the factor z w 2 will affect e w 1 via v c .

our formulation makes it possible to implicitly propagate perceptual information from one word to another through shared contexts.


(2015) combines two objectives, we use a joint probabilistic model integrating both visual and text information in a principled way.

our model seeks latent factors that are good at explaining the word-context co-occurrences.


Among the RNN variants, LSTMs have been shown to provide state-of-the-art performance for several benchmark language modeling tasks (Józefowicz et al., 2016;Merity et al., 2018;Melis et al., 2018).

the LSTM with projection introduced by Sak et al.


In this section, we examine the intrinsic properties of contextual vectors learned with biLMs, focusing on those that are independent of the architecture details.

we seek to understand how familiar types of linguistic information such as syntactic or coreferent relationships are represented throughout the depth of the network.


The penalty is only applied to some words because Indeed, some words are defined but are not used in definitions and some words are used in definitions but not defined.


We focus on intrinsic evaluation methods.

we study how different models trade off similarity and relatedness.


In this section, we create odd-man-out solvers from collections of word embeddings.

we experiment with two types of embeddings: (1) traditional word embeddings, which map words to a single vector representation (Subsection 5.2), and (2) sense embeddings, which map words to a set of vectors, each pertaining to a different sense of the word (Subsection 5.3).


In this framework, given n puzzle options, we find the subset of n − 1 words of maximal similarity (given some similarity score), and return the excluded word as the odd-man-out.

define an embedding as a function that maps every word to a set (possibly empty) of real vectors.


This paper presents an end-to-end neural network framework for simile sentence recognition.

we make following contributions: • We build a dataset consisting of 11.3k sentences containing a frequently used comparator "oe" for simile recognition in Chinese, which can support data-driven approaches.


The factorization layer tries to decompose the alignment vector (constructed from the variations of a − b, a b and [a; b]), learning higher-order feature interactions between each compared alignment.

it models the second-order (pairwise) interactions between each feature in every alignment vector using factorized parameters, allowing more expressive comparison to be made over traditional fully-connected layers (FC).


In this paper, we propose the Convolutional Interaction Network (CIN), a general model to capture the interaction between two sentences, which can be an alternative to the attention mechanism for NLI.

cIN encodes one sentence with the filters dynamically generated based on another sentence.


In this paper, we propose a new interaction module, called Convolutional Interaction Network (CIN), which can serve as an alternative module of attention mechanism.

cIN utilizes convolutional neural network to extract the valued features (or representations) from sentences.


CIN utilizes the dynamic convolutional filters to model the interaction between two sentences.

each sentence is convolved by dynamical filters generated based on another sentence.


We propose to integrate this knowledge in a way similar to how attention is used in the ESIM model.

we calculate the attention matrix e ∈ R n×m between all pairs of tokens a i and b j in the inputs sentences, where n is the length of the hypothesis and m is the length of the premise.


In STAMP, the probability of a token to be generated is calculated as Equation 1, where p z (•) is the probability of the channel z t to be chosen, and p w (•) is the probability distribution of generating a word y t from the selected channel.

(1) the encoder takes a question as the input, uses bidirectional RNN with GRU cells to compute the hidden states, and feeds the concatenation of both ends as the initial state of the decoder.


In this work, we incorporate stochastic latent variables (Cao and Clark, 2017;Serban et al., 2017) to the sequence-to-sequence model in order to increase question diversity.

we introduce a latent variable z ∼ p(z), which is a standard Gaussian distribution N (0, I n ) in our case, and calculate the likelihood of a target sentence y as follows: We maximize the evidence lower bound (ELBO), which decomposes the loss into two parts, including (1) the KL divergence between the posterior distribution and the prior distribution, and (2) a cross-entropy loss between the generated question and the ground truth.


However, such syntaxbased training objective deviates from the goal of synthesizing semantically equivalent programs.

these syntax-based approaches fail to handle the problem of Program Aliasing (Bunel et al., 2018), i.e., a semantically equivalent program may have many syntactically different forms.


A sequence-to-sequence learning model learns the distribution of regular expressions R given an input NL sentence S: By default, the sequence-to-sequence learning model uses maximum likelihood estimation (MLE) for training, i.e., maximizing the likelihood of mapping the input sequence to the output sequence for each pair in the training set.

the optimal parameters θ * are obtained as follow: Gradient descent is used to search out optimal parameters θ * .


To address this problem, we use the Monte Carlo estimate as the approximation of the expected value.


To generate distinct string examples, we aim to generate paths to cover as many transitions as possible.

we mask all transitions that have been covered by previously generated paths.


We delexicalize several categories of content words and keep function words, which describe the utterance structure, in their lexicalized form.

any verb 1 whose lemma is not "be" or "do" is delexicalized.


Therefore, we adopt a "distillation approach", where we train a supervised model over abstract examples to mimic the predictions of an unsupervised model that has access to the full lexicalized examples.

we use a standard unsupervised word aligner (Dyer et al., 2013), which takes all lexicalized examples {( i=1 in all D domains and produces an Alignment matrix A * for every example, where A * ij = 1 iff token i in the logical form is aligned to token j in the utterance.


Moreover, we can use similar constraints to prevent logical forms that are highly unlikely according to our prior knowledge.

we define once(z) to be true iff each date, named entity, and number in the logical form z appear exactly once.


This domain contains mostly spatial language, different from other domains in OVERNIGHT.

prepositions, which we did not lexicalize map to relations in the KB (e.g., "below" and "above" map to the relations Below and Above).


To deal with these issues, we use a greedy search that keeps the consistency among spans and can return multiple spans for adjunct labels.

we greedily select higher scoring labeled spans subject to two constraints.


While these neural span features cannot be used in BIO-based SRL models, they can easily be incorporated into spanbased models.

taking a span representation h s as input, the labeling function f label returns the score for the span s = (i, j) with a label r. we use the following labeling function, where W ∈ R |R|×2d hidden has a row vector associated with each label r, and W[r] denotes the r-th row vector.


We use the same greedy search algorithm used for our base model (Section 2.4).

during training, we update only the parameters of the ensemble model, i.e., we fix the parameters of each trained model m. As the loss function, we use the cross-entropy (Eq.


The span-based model confused A0 and A1 arguments the most.

the model confused them for ergative verbs.


To investigate a relation between the span representations and predicted labels, we qualitatively analyze nearest neighbors of each span representation with its predicted label.

for each predicted span in the development set, we collect 10 nearest neighbor spans with their gold labels from the training set.