There is no need for a carefully curated ontology and the labels are highly interpretable.

we differ from QA-SRL in focusing on all words in the sentence rather than just verbs, and allowing free form questions instead of using templates.


Although QAMR questions' free-form nature is crucial for our approach, it means that predicates are not explicitly marked.

with a simple predicate-finding heuristic, we can align QAMR to PropBank, NomBank, and QA-SRL and show high coverage of predicate-argument structure, including more than 90% of non-discourse relationships.


More broadly, non-expert annotation has been used extensively to gather question-answer pairs over natural language texts, for example in reading comprehension (Rajpurkar et al., 2016;Richardson et al., 2013;Nguyen et al., 2016) and visual question answering (Antol et al., 2015).

while these treat question answering as an end task, we regard it as a representation of predicateargument structure, and focus annotators on a smaller selection of text (a few target words in a single sentence, rather than a paragraph) aiming to achieve high coverage.


Because the target words shown to a single worker are close to each other-and often share a constituent-it restricts the set of QA pairs they write to relate to a certain part of the sentence.

asking that the target word appears either in the question or the answer makes it flexible enough so that the worker is almost never stuck with no reasonable question to write.


Incentives and payment Base pay for the generation stage was 20c, with a bonus of 3(k + 1) cents for each question beyond the number required (so, the first extra question would reward them 6c, the next 9c, and so on).

their bonuses were only calculated based on the number of questions considered valid by annotators.


We show that an AddSentDiverse-based adversariallytrained model beats an AddSent-trained model across 3 different adversarial test sets, showing an average improvement of 24.22% in F1 score, demonstrating a general increase in robustness.

even with our diversified adversarial training data, the model is still not fully resilient to AddSent-style attacks, e.g., its antonymy-style semantic perturbations.


First, as shown, the AddSent-trained model is not able to perform well on test sets where the distractors are not inserted at the end, e.g., the AddSentRandom adversarial test set.

5 it can be seen that retraining with AddSentDiverse boosts performance of the model significantly across all adversarial datasets, indicating a general increase in robustness.


Members of such communities of interest are often identifiable via their social network connections, and shared social connections are clearly important in recommendations.

shared connections often reflect a subset of a person's interests, and there may be users of interest where any shared connections are distant.


Subsequently, aspect- (Yang et al., 2016) and argument-based (Liu et al., 2017a) features are demonstrated to improve the prediction performance.

these methods rely on sufficient labeled data and may not perform ideally for domains with limited data.


On top of that, specific shared framework use both a shared NN and domain-specific NNs to derive shared and domain-specific features (Ganin et al., 2016;Taigman et al., 2017;Yu et al., 2018).

it may not be ideal to separate shared and specific features, recent studies (Ganin et al., 2016;Taigman et al., 2017;Liu et al., 2017b) consider the adversarial networks to learn more robust shared features across domains.


Now, as the schema is fixed, simple rule based techniques (Konstas and Lapata, 2013) or template based solutions could be employed.

due to the vast space of selection (which attributes to use in the summary based on the current value it takes) and generation (how to express these selected attributes in natural language) choices possible, such approaches are not scalable in terms of the number of templates as they demand hand-crafted rules for both selection and generation.


So, attention over attributes can be static.

while generating a word by the decoder, there can be a preference of focusing on a record type say, temperature, over some other type say, windSpeed.


Specific variants of the task exist for certain domains, such as narrative (Mani, 2004) and emailthread summarization (Rambow et al., 2004).

summaries for query-based summarization only cover parts of the text that are about the topic specified by a query (Rahman and Borah, 2015).


2017developed a dataset of aligned extractive and abstractive summaries by creating separate tasks for summarization, alignment, and classification of changes.

maintaining accuracy when the complexity of the given text increases has remained an open question.


It has been shown (Mikolov et al., 2013), that the presence of multi-word phrases intermixed with unigram words increases the performance and accruacy of the embedding models trained using techniques such as Word2Vec.

in our framework we take a different approach in detecting meaningful and cohesive chunk of phrases while preparing the text samples for training.


Nonetheless, all these approaches focus on task of biographies generation, and only in Englishthe language with the most language resources and knowledge bases available.

with these works, we explore the generation of sentences in an open-domain, multilingual context.


They have a high correlation with human judgments and are capable of identifying good summaries.

we need to test whether the combination of the three θ's is well behaved under optimization.


BEwT-E transforms basic elements to help in matching.

it requires complex transformation rules, which are difficult to apply to languages other than English.


The function-word pair "to the" bears almost no meaning but is frequently found since function words appear in sentences quite often.

a dependency triple holds the syntactic information that the dependency of "to" is not "the" but "store".


These models are computationally efficient and scalable to long documents.

because they make independent decisions about coreference links, they are susceptible to predicting clusters that are locally consistent but globally inconsistent.


A direct replacement of s a (i, j) with s c (i, j) results in a performance loss of over 3 F1 in our experiments.

s c (i, j) is much more efficient to compute.


(2016) adopted a deep neural network language model.

a neural approach is usually a black-box and it is hard to interpret its reasoning for the final classification decisions.


Based on error analysis of the POS-based CNN/RNN model's classification result, we found that almost all AD-positive results are classified correctly as AD-positive.

there is more error in classifying non-AD samples, which could be due to the fact that DementiaBank includes patients with probable and possible AD, each exhibiting various degrees of symptoms.


The initial "and" is highlighted as well, corroborating the results of the activation clusters in that starting with a coordinating conjunction is a trait of Alzheimer's speech.

in Figure 3 (middle), the "uh" filler word is not highlighted, showing that most control patients do not use filler words as heavily as Alzheimer's patients.


They classify characters based on their speech and actions into an introvert or extrovert class.

we focus on attributes of characters and actions beyond such coarse traits, and when these attributes are expressed as complex descriptions.


We find that both models capture interesting ties between character attributes and actions.

there is much scope for improved models of generation.


Across five languages, both models greatly outperformed the baseline on unseen tokens, with the soft attention model doing a bit better for smaller data sizes, and the hard attention model doing a bit better for larger ones.

these improvements did not translate into clearly better POS tagging downstream.


This success can be largely attributed to the feature sharing by fusing some layers among tasks.

most existing approaches just fully or proportionally share the features without distinguishing the helpfulness of them.


More models adopt this thought to enhance the performance (Liu et al., 2015(Liu et al., , 2016.

under the scheme of proportional addition (Ruder et al., 2017;Misra et al., 2016), all the features are shared with the same weight between every pair of tasks.


So, in the one-dimensional case, the transformations that a word vector can do are restricted to translation.

when word vectors have more than two dimensions, they can do almost any affine transformation.


In ATIS dataset, the proposed slot-gated model with only intent attention achieves slightly better performance with fewer parameters (from 284K to 251K).

it does not achieve better results in Snips dataset.


The deep spatial contextual long-term recurrent convolutional network (DSCLRCN) of Liu and Han (2016)  The SM heatmaps are very focused, which is a consequence of that model being trained on SALICON, which contains focused human attention maps.

our evaluation uses rank correlation, rather than correlation on absolute attention scores, and is therefore unaffected by this issue.


This is likely due to the fact that SM was trained on the SALICON dataset; a higher correlation can probably be achieved by fine-tuning the salience model on the PASCAL Actions Fixation data.

this would no longer be fair comparison with our verb prediction model, which was not trained on fixation data (it only uses image description datasets at training time, see Section 3).


As a sanity check, both architectures outperform a language-agnostic model on an accuracy-based colorization metric.

we are more interested in whether modifications to the caption properly manifest themselves in output colorizations (e.g., switching one color with another); crowdsourced evaluations confirm that our models properly localize and color objects based on captions ( Figure 1).


RSA has the advantage of being a fairly simple and straightforward method for comparing the representations across two embedding spaces.

rSA has the disadvantage that it produces one aggregate number for a pair of correlation matrices, and does not show which concepts contributed most to a high or low score.


GT+P results show that inter-modality constraints help in improving the results (about 2% F1) which indicates some of the visual relations successfully confirmed and boosted their corresponding relations in the text modality.

this improvement is limited which is expected considering the low performance of Preposition Classifier.


The alignment classifiers in the AC model also slightly improve the BM+C.

as it is visible in AC+P results, when we have both noisy alignments and noisy visual relations the results drop slightly compared to AC.


On one hand, the quality improvement attributed to eye movement signals on lower-level tasks implies that such signals do contain linguistic information.

it is not clear whether these signals can also provide significant improvement for tasks dealing with higher-level semantics.


The impact of the eye-tracking features varies between the different combinations of datasets.

the inclusion of eye-tracking features improves the results for all combinations, except for the models trained on the ZuCo corpus  and tested on the GECO corpus.


Thus, it is under-standable that in this scenario the original gaze features and the gaze features aggregated only on the individual datasets result in better models.

when evaluating the NER models in a cross-corpus scenario, the type-aggregated features lead to significant improvements.


A full description of the SR-number units is beyond our scope.

we note that 10 SRnumber units in the second layer of the network were identified, which had efferent weights with a similar segregated structure as that of the LR units (Figure 1d).


As the results in Figure 2(c) show, instances in the highest queue match well with current training data (and hence the current model), and, therefore, are less informative.

instances in the designated queues show considerably smaller similarity with training instances in both positive and negative classes, and, therefore, do not match well with training data.


the corresponding performance on Churn dataset decreases from 69.90 to 68.56, i.e.


All networks show the basic garden path effect (p < .001 in JRNN, GRNN, and RNNG; p < 0.01 in TinyLSTM).

the garden path effect in TinyLSTM is much smaller than the other models: RC reduction causes an additional .3 bits of surprisal at the disambiguating verb, as compared to 2.8 bits in the RNNG, 1.9 in JRNN, and 3.6 in GRNN (TinyLSTM's garden path effect is significantly smaller than each other model at p < 0.001).


For example, the classic phonemic restoration effect (Warren, 1970) shows that adults infer missing phonemes from context with such ease that they often fail to notice when acoustic cues to phone identity are erased.

infants first learning their phonemic categories have not yet acquired reliable top-down lexical and phonotactic models and must rely more heavily on bottom-up perceptual information.


Disfluencies in spontaneous speech are known to be associated with prosodic disruptions.

most algorithms for disfluency detection use only word transcripts.


For example, duration cues are expected to be useful -disfluencies are often associated with duration lengthening related to hesitation.

duration varies with phonetic context, word function, prosodic phrase structure, speaking rate, etc.


Multilingual transfer in ASR often relies on using bottle-neck features (Vesely et al., 2012;Vu et al., 2012;Karafiát et al., 2018) and adapting an acoustic model trained on one language to effectively recognize the sounds of other languages (Schultz and Waibel, 2001;Le and Besacier, 2005;Stolcke et al., 2006;Tóth et al., 2008;Plahl et al., 2011;Thomas et al., 2012;Imseng et al., 2014;Do et al., 2014;Heigold et al., 2013;Scharenborg et al., 2017).

while most work uses less than 10 languages for model training, we include up to 100 languages in training.


Results in Table 3 show that in all cases multilingual pretraining of QUE+CYR with the auxiliary objectives outperformed its counterpart without the objectives (which frequently undeperformed the monolingual model), and in all but one case this led to an improvement over the monolingual baseline.

7 To gain insight into how the auxiliary objectives change the representation of speech learnt by the 7 this doesn't hold in the language adaptation scenario, where the auxiliary objectives help QUE+CYR only slightly; see Section 8. models, we applied 2D t-SNE dimensionality reduction (Van Der Maaten and Hinton, 2008).


Multilingual pretraining of languages from a different language family and script benefitted from an explicit phoneme objective and adversarial objective when there was sufficient diversity in the pretraining languages.

a change in orthography was conflated with a change in language family, geographic location, and phono- 14.8 (-6.3%) Avg.


The PHON/INV sets for the first two of these languages emphasized Quechuan languages, and this corroborates the indication in Section 6 that the auxiliary objectives may not help so much when pretraining languages are similar.

the Indonesian PHON/INV included Afro-Asiatic and Niger-Congo languages, as well an Indo-European language and Huave, a language isolate from Mexico, yet it was not improved by auxiliary objectives.


languages, while the PHON/INV had a diverse array of Austronesian, Indo European, Afro-Asiatic, Sino-Tibetan and Mayan languages.

on average, these results suggest that geographical proximity is a decent guide to pretraining language selection.


Using the auxiliary objectives overcame the difference, resulting in a -1.6% average relative WER change.

pretraining with GEO+phn+adv yielded an average relative delta of -7.4% over the monolingual model.


An approximation to this is whether one of the translations listed in a bilingual dictionary appears in the translated or interpreted outputs respectively, and as a first pass we identify all source terms with the corresponding targetlanguage translations.

we found that this automatic method did not suffice to identify many terms due to lack of dictionary coverage and also to non-literal translations.


• Word frequency: We anticipate that interpreters often leave rarer source words untranslated because they are probably more difficult to recall from memory.

we would expect loan words, words adopted from a foreign language with little or no modification, to be easier to recognize and translate for an interpreter.


Recently, there have been some efforts to solve video captioning with audio input (Hori et al., 2017(Hori et al., , 2018.

the audio input merely serves as auxiliary features for video captioning, and as a result, it only marginally improves the performance (e.g.


To that end, we tailor the AudioSet dataset (Gemmeke et al., 2017) that comprises 1,789,621 human-labeled 10 second YouTube excerpts with an ontology of 527 audio event categories.

an immediate collection of captions from these audios pose several difficulties: (i) too many audio clips, (ii) inconsistent level of abstraction among the classes, (iii) distribution bias of some labels and (iv) noisy labels that are only noticeable from visual cues.


In order to inject the global semantics, we perform an element-wise addition of h ctxt to the mid-level feature along the time axis, and feed them into the bi-directional LSTM one at a time, producing a hidden state ← → h a2 t ∈ R D 2 at each step: In many captioning models Yu et al., 2017;Laokulrat et al., 2018;Long et al., 2018), semantic attention has been independently used from temporal/spatial attention.

it can be troublesome because there may exist some discrepancies between the two attentions i.e.


Editors apparently had difficulty generating humor when they were severely constrained in choosing a word to replace, or when they had too many choices for replacement.

edited headlines with a higher proportion of replaceable words were generally funnier, as shown in Table 3.


A goal of natural language processing is to develop techniques that enable machines to process naturally occurring language.

not all language is clear and, as humans, we may not always understand each other (Grice, 1975); in cases of gaps or mismatches in knowledge, we tend to ask questions (Graesser et al., 2008).


Theoretically, the discriminator should be trained using (context, true question, true answer) triples as positive instances and (context, generated question, generated answer) triples as the negative instances.

we find that training a discriminator using such positive instances makes it very strong since the generator would have to not only generate real looking questions but also generate real looking answers to fool the discriminator.


This is not surprising given that these criteria are not very subjective.

the agreement on usefulness and specificity is quite moderate since these judgments can be very subjective.


The decoder has the same architecture as the encoder, stacking L identical blocks of multi-head attention with feed-forward networks for the target hidden states.

the decoder block has an extra attention layer over the encoder's hidden states.


It is an evident improvment compared to the un-pre-trained ones.

the denoising auto-encoder improves the single model from 54.67 to 58.8 (+4.13).


Moreover, our parser can deal with any ε-free and simple MCFG provided by an external tool, making it more flexible than discodop and rparse.

we are not able to compete with ddctf-dop in terms of accuracy, since discontinuous data-oriented parsing is a more accurate formalism (van Cranenburgh and Bod, 2013).


Thus, languages can share grammatical features without necessarily being genetically related, either by contact or by mere chance.

the tree model is still a good starting point in practice and language families align well with grammatical similarity as recent works on typological analysis of UD treebanks have shown (Chen and Gerdes, 2017;Schluter and Agić, 2017).


Amongst Indo-European languages, there are one or two Armenian languages as well as one or two Albanian languages (depending on the criteria for being a language), while there are tens of Slavic languages and Romance languages.

there is no reason to believe that Slavic or Romance languages are better witnesses of proto-Indo-European than Armenian or Albanian.


The analysis is similar for Komi which on top of sharing morphology with its relatives also share the writing system which provides it an extra gain.

this might word in the opposite direction as well, as we can see with Faroese, Breton and Sanskrit.


we score each possible combinations independently: the score of a single combination only depends on the two constituents that are combined, regardless of the rest of the set.


Our approach does not decrease incrementality of the parser in this way.

having a special mechanism for right adjunction makes parser both more incremental and more accurate.


Hence, z can largely determine the reconstructed x.

when an auto-regressive decoder is used in a text VAE (Bowman et al., 2015), there are two paths from x to its reconstruction, as shown in Figure 1(b).


But, starting from the 2nd cycle, much more divided clusters are shown when learning on top of the 1st cycle results.

< 1 leads to some holes between different clusters, making q(z) violate the constraint of p(z).


Lee (1997)'s reduction shows that there can be a no O(|G|n 3− ) for some constant > 0 (sub-cubic-time) algorithm for CFG-parsing without implying a significant breakthrough in BMM, which is widely believed not to be possible.

the construction required the grammar size |G| = Θ(n 6 ) to be dependent on the the input size n, which, as Lee (1997) points out, is unrealistic in most applications.


The number of paths may be exponential in the size of T , or infinite if T is cyclic.

in contrast to WFSTs, we abandon this combinatorial structure in favor of neural nets when defining the probability distribution over a.


In hopes of improving the inductive bias of the learner, we partitioned the hidden state vector into three sub-vectors: The mark scoring function f θ (s t−1 , ω t ) was as before, but we restricted the form of g θ , the state update function.

s a t encodes all past marks and depends on the full hidden state so far: we make s x t encode only the sequence of past input marks, ignoring all others.


On the other hand, for the task of finding the best string among a pool candidates, we do not need to compute (or approximate) the denominator in equation 13, since As in the case for paths, the language L(Y ) is usually infinitely large.

given an output candidate y k ∈ L ⊆ L(Y ), we can approximate the summation in equation (14) using importance sampling: Algorithm 2 Training procedure for G θ .


The model is not required to learn transduction rules that conform to our linguistic knowledge.

we expect that a well-performing one would tend to pick up rules that resemble what we know.


At the heart of building such task-oriented bots lies the challenge of accurately capturing the user's intent (e.g., find cafes in Chicago), and then extracting its entities to service the request (e.g term= "cafes", location="Chicago").

its success relies heavily on obtaining both, large and high quality corpora of training samples showing mappings between sample utterances and intents.


To characterize the types of paraphrasing issues, two authors of this paper investigated the crowd-sourced paraphrases, and recognized 5 primary categories of paraphrasing issues.

we only considered paraphrase-level issues related to the validity of a paraphrase without considering dataset-level quality issues such as lexical diversity (Negri et al., 2012) and bias (Henderson et al., 2018).


Paraphrase detection and textual semantic similarity (STS) methods are designed to measure how two pieces of text are semantically similar.

they do not differentiate between different types of errors (e.g., Cheating, Answering, Semantic Errors) in our settings.


Other datasets utilize search engine logs to collect their questions (Berant et al., 2013), which creates a bias towards simpler questions that search engines can already answer reasonably well.

comQA questions come from WikiAnswers, a community QA website where users pose questions to be answered by other users.


Over the past five years, many datasets were introduced for this setting.

as Table 1 shows, they are either small in size (Free917, and ComplexQuestions), composed of synthetically generated questions (Sim-pleQuestions, GraphQuestions, LC-QuAD and ComplexWebQuestions), or are structurally simple (WebQuestions).


(2013) is worse than the others due to the incompleteness of its underlying extractions and the complexity of ComQA questions that require higher-order relations and reasoning.

the system answered some complex questions, which KB-QA systems failed to answer.


The answer to such a question is more readily available in Web text, compared to a KB, where more sophisticated reasoning is required to handle the superlative.

a slightly modified question such as "What is the fourth highest mountain in the state of Washington?"


Finally, once questions are formulated using a variety of rich and sophisticated representations in natural languages, such factoid QA tasks may serve as an excellent testbed to study many natural language understanding problems, e.g., examine the recently emerging research efforts to combine neural models with the traditional symbolic processing methods (Liang et al., 2017;Mou et al., 2017).

machine learning approaches for NLP are data hungry since they require large amounts of real-world data to train the models for the best possible performance.


Note that one question-answer pair may generate several matched triples when multiple related predicates are found since each question may contain multiple entities and each subject node may link to an object node through different predicates.

this procedure becomes inefficient since there is an enormous number of object nodes to process for some popular subject nodes, such as United States (m.09c7w0), leading to a tremendous number of Freebase queries.


Among these matches, FreebaseQA contains 28,348 unique questions in total, with 20,358, 3,994 and 3,996 in the train, dev and eval sets respectively.

another important factor to consider is the linguistic sophistication of the data.


Answerable Questions Redefined Petrochuk and Zettlemoyer (2018) set an upper bound of 83.4% for the accuracy on the SimpleQuestions dataset.

our models are able to do better than the upper bound.


Open-domain question answering (QA) has been extensively studied in recent years.

many existing works have followed the 'search-and-answer' strategy and achieved strong performance (Chen et al., 2017;Kwon et al., 2018;Wang et al., 2018b) spanning multiple QA datasets such as TriviaQA (Joshi et al., 2017), SQuAD (Rajpurkar et al., 2016), mS-macro (Nguyen et al., 2016), ARC   open-domain QA tasks become inherently more difficult when (1) dealing with questions with little available evidence; (2) solving questions where the answer type is free-form text (e.g.


However, this assumption ignores the difficulty of retrieving question-related evidence from a large volume of open-domain resources, especially when considering complex questions which require reasoning or commonsense knowledge.

aRC does not provide passages known to contain the correct answer.


Instead, the task of identifying relevant passages is left to the solver.

questions in ARC have multiple answer choices that provide indirect information that can help solve the question.


we might retrieve sentences similar to the question or the answer choice, but would struggle to find evidence explaining why the answer choice is correct).

a reformulated query consisting of essential terms in the question and Choice 4 can help retrieve evidence explaining why Choice 4 is a correct answer.


For the first question, there exists evidence that can justify the answer candidate (C).

the model chooses (D) which has more words overlapping with its evidence.


At each hop, it predicts a relation path using the reasoning module, and also optimizes it using intermediate results.

uHop has demonstrated the ability to process large-scale knowledge graphs in experiments conducted on Freebase (Bordes et al., 2015).


If the process should continue, i.e., stop is false, loss is defined as where score s r is the score of the question paired with the gold relation r in the next hop and sr is the score of the question paired with the extracted relationr.

if the process should terminate, we optimize the model by The model thus learns to infer sr is greater than s r , resulting in the termination of relation extraction.


Coref-GRU (Dhingra et al., 2018) uses coreferences among tokens in documents.

it is still limited by the longdistance relation propagation capability of RNNs.


In addition, bidirectional attention (Seo et al., 2016) shows its superiority to vanilla mutual attention because it provides complementary information to each other for both contexts and queries.

little work exploits the attention between graphs and queries.


Both MHQA-GRN and Entity-GCN utilize graph networks to resolve relations among entities in documents.

the lack of attention and complementary features limits their performance.


For example, 'text' carries semantic information of the spoken sentence, whereas 'acoustic' information reveals the emphasis (pitch, voice quality) on each word.

the 'visual' information (image or video frame) extracts the gesture and posture of the speaker.


System (Poria et al., 2017b) uses contextual information for the prediction but without any attention mechanism.

(Zadeh et al., 2018a) uses multiattention blocks but did not account for contextual information.


(2018) proposed an inter-modal attention framework for the multi-modal sentiment analysis.

the key differences with our current work are as follows: a) Ghosal et al.


The underlying assumption of this task is that the entire text has an overall polarity.

the users' comments may contain different aspects, such as: "This book is a hardcover version, but the price is a bit high."


The results of the BERT-single model on aspect detection are better than Dmu-Entnet, but the accuracy of sentiment classification is much lower than that of both SenticLstm and Dmu-Entnet, with a difference of 3.8 and 5.5 respectively.

bERT-pair outperforms other models on aspect detection and sentiment analysis by a substantial margin, obtaining 9.4 macro-average F1 and 2.6 accuracies improvement over Dmu-Entnet.


Directly fine-tuning the pre-trained BERT on TABSA does not achieve performance growth.

when we separate the target and the aspect to form an auxiliary sentence and transform the TABSA into a sentence pair classification task, the scenario is similar to QA and NLI, and then the advantage of the pre-trained BERT model can be fully utilized.


Rules R3/R4/R5 are less effective on their own.

as a whole, they can still improve the overall performance.


This means this rule may introduce some noises into the objective function.

"-R3" can result in worse accuracy for TripAdvisor, which means it is still complementary to the other rules for this dataset.


It should be good news that Ross has his paper published and Rachel is glad to see related reports about it.

the transcripts do not reveal very strong emotions compared to what the characters might act in the TV show.


Negation rules have been found to work effectively across different domains and rarely need finetuning (Taboada et al., 2011).

rule-based approaches entail several drawbacks, as the list of negations must be pre-defined and the selection criterion according to which rule a rule is chosen is usually random or determined via cross validation.


Multilingual Word Embeddings BWE methods can be extended to the case of multiple languages by simply mapping all the languages to the vector space of a selected language.

directly learning multilingual word embeddings (MWE) in a shared space has been shown to improve performance (Ammar et al., 2016;Duong et al., 2017;Chen and Cardie, 2018;Alaux et al., 2018).


This method can be generalized to the cross-lingual setting by training monolingual sentimental embeddings on both languages then aligning them in a common space.

it requires sentiment resources in the target language thus is impractical for low-resource languages.


In the unsupervised case, such a dictionary can be induced from the monolingual embeddings S and T (Artetxe et al., 2018).

the quality of this dictionary is usually not good, which in turn degrades the quality of the projection matrices learned from this dictionary.


(3) for the task Y s(l) and l th layer except that it does not treat the NMT parameters θ as constant.

it always occurs on the training data that Y s To alleviate this inconsistency issue for better approximating the full-coverage method, we leverage the above structural property by adding another regularization term.


Since layer 1 is not selected as the regularized layer, no significant gap is observed.

since layer 1 is close to the loss directly imposed on layer 2, improvements about 5% and 8% are obtained.


Since UD attaches subordinating and coordinating conjunctions to the subsequent conjunct, this results in them being positioned in the same conjunct they relate (e.g., After will be included in the first conjunct in "After arriving home, John went to sleep"; and will be included in the second conjunct in "John and Mary").

uCCA places conjunctions as siblings to their conjuncts (e.g., "[After] [arriving home], [John went to sleep]" and "[John] [and] [Mary]").


Units relating a Scene to the speech event or to the speaker's opinion are Ground (e.g., "no, Warwick in New Jersey" and "Please visit my website").

discourse elements that relate one Scene to another are Linkers (e.g., anyway).


Our results thus suggest that encoding syntax more directly, perhaps using syntactic scaffolding (Swayamdipta et al., 2018) or guided attention (Strubell et al., 2018), may assist in predicting unit boundaries.

tUPA often succeeds at making distinctions that are not even encoded in UD.


Low perplexity signifies less uncertainty over which words can be used to continue a sequence, quantifying the ability of a language model to predict gold or reference texts (Brown et al., 1992;Young et al., 2006).

style transfer outputs are not necessarily gold standard, and the correlation between perplexity and human judgments of those outputs is unknown in the style transfer setting.


For style transfer intensity, kappas for relative scoring do not show improvement over the previously used approach of absolute scoring of x .

we observe the opposite for the aspect of naturalness.


(2002), only focus on pairs of unigrams (single words).

the concept of semantic relatedness applies more generally to any unit of text.


For example, apples and bananas (co-hyponyms of fruit) are both edible, they grow on trees, they have seeds, etc.

semantically related concepts may not have many properties in common, but there exists some relationship between them which lends them the property of being semantically close.


on how to get food delivered, or on the risks of caffeine intake.

given how the narrative is constructed, we can intuit that the more likely goal of the narrator is to get advice on how to overcome the effects of sleep deprivation so that they can be alert for the upcoming programming lesson.


To find plausible alternative answer options for each candidate cloze test instance, one direct approach could be to find questions that are semantically related to the ground-truth question.

there are two underlying problems with this approach.


The Penn Discourse Treebank (PDTB) is one of the most well-known examples (Miltsakaki et al., 2004;Prasad et al., 2008).

although multimodal corpora increasingly include discourse relations between linguistic and nonlinguistic contributions, particularly for utterances and other events in dialogue (Cuayáhuitl et al., 2015;Hunter et al., 2015), to date there has existed no dataset describing the coherence of text-image presentations.


At each user turn, dialogue state tracking (DST) aims to estimate user's goal by processing the current utterance.

in many turns, users implicitly refer to the previous goal, necessitating the use of relevant dialogue history.


2017discussed on similar limitations of current DST task and introduced a new task of frame tracking that explicitly tracks every slot-values that were introduced during the dialogue.

that significantly complicates the task by maintaining multiple redundant frames that are often left unreferenced.


The parameters added due to using relevant context are the parameters for encoding the antecedent referential user utterance and the previous system utterance as well as the past utterance and past slot-value scorers.

we also observe high variance in the joint goal accuracy.


We leverage these annotations to obtain the ground truth visual groundings (A V ) for the referents in our questions.

each of the caption and question templates has referring phrase annotations in them, thus giving the ground truth textual groundings (A T ).


In our second experiment, we train and test a classifier on the original Waseem-dataset in 10fold crossvalidation.

we remove either of the two types of biased words from the dataset: (i) We remove 25 topic words from the 100 most correlating words that we thought bear no relation towards abusive language (e.g.


It also provides a perfect solution to the problem of removing the gender direction from non-gendered words.

as we show in this work, while the gender-direction is a great indicator of bias, it is only an indicator and not the complete manifestation of this bias.


For example, in the binary gender case, the extremes of bias subspace reflect extreme male and female terms.

this is not possible when projecting multiple classes into a linear space.


In line with Gonen and Goldberg's (2019) findings, simply removing the bias component is insufficient to remove multiclass "cluster bias".

increasing the size of the bias subspace reduces the correlation of the two variables (Table 4 in the Appendix).


such lexica has emerged (Emerson and Declerck, 2014;Altrabsheh et al., 2017), borrowing ideas from crowdsourcing (Raykar et al., 2010;Hovy et al., 2013).

this is a non-trivial task, because lexica can use binary, categorical, or continuous scales to quantify polarity-in addition to different interpretations for each-and thus cannot easily be combined.


(2016) use distributions of parts-ofspeech, sentiment, and exaggerations.

to these approaches, our model uses only word embeddings as input representations.


These datasets are frequently used for the task of predicting titles from abstracts or short stories.

no keyphrases are provided; they do not serve to our purpose.


Recall that in the original Pyramid, SCUs are exhaustively collected; then, coreferring SCUs between reference summaries are merged and weighted by the number of reference summaries from which they originate.

our method enables using a sample of SCUs for evaluation, out of the SCUs collected in this phase (we have sampled, for uniformity, 32 SCUs per topic).


Then, nodes having the same label form a group of the partitioning.

to the local search, CW does not directly optimize the objective function proposed by , however, we empirically found that it yields partitionings that score very well with regard to that objective.


Our BiLSTM implementation yields lower performance than : our model achieves a score of 79.1, compared to their reported score of 80.3.

our BDH model yields a score of 80.8, already achieving state-of-the-art performance.


7 The original AllenNLP library uses a byte representation.

the multilingual fork assigns a unique character id to each unicode character, thereby avoiding the need to recognize mutibyte representations.


Given a configuration with word w i on top of the stack, as the pointer network just returns a position p from a given sentence, they proceed as follows to determine which transition should be applied: • If p = i, then the pointed word w p is considered as a child of w i ; so the parser chooses a Shift-Attach-p transition to move w p from the buffer to the stack and build an arc w i → w p .

• if p = i, then w i is considered to have found all its children, and a Reduce transition is applied to pop the stack.


These parsers are conceptually simple, not needing traditional parsing algorithms or auxiliary structures.

experiments on the PTB and a sample of UD treebanks show that they provide a good speed-accuracy tradeoff, with results competitive with more complex approaches.


This is the case of tasks such as PoS tagging, chunking or named-entity recognition, for which different approaches obtain accurate results (Brill, 1995;Ramshaw and Marcus, 1999;Reimers and Gurevych, 2017).

previous work on dependency parsing as sequence labeling is vague and reports results that are significantly lower than those provided by transition-, graph-based or sequence-tosequence models Kiperwasser and Goldberg, 2016;Dozat and Manning, 2017;Zhang et al., 2017a).


We tested four different encodings, training a standard BiLSTM-based architecture.

to previous work, our results on the PTB and a subset of UD treebanks show that this paradigm can obtain competitive results, despite not using any parsing algorithm nor external structures to parse sentences.


They are based on character-level language models which treat text as distributions over characters and are capable of generating embeddings for any string of characters within any textual context.

such purely character-based approaches struggle to produce meaningful embeddings if a rare string is used in a underspecified context.


we also find no significant improvements on the WNUT-17 task on emerging entities.


Our pooling operation makes the conceptual simplification that all previous instances of a word are equally important.

we may find more recent mentions of a word -such as words within the same document or news cycle -to be more important for creating embeddings than mentions that belong to other documents or news cycles.


Supervised approaches to named entity recognition (NER) are largely developed based on the assumption that the training data is fully annotated with named entity information.

in practice, annotated data can often be imperfect with one typical issue being the training data may contain incomplete annotations.


The former approach may even lead to sub-entity level annotations (e.g., "radio" is annotated as part of an entity).

we argue such assumptions can be largely unrealistic.


In practice, their approach worked for the task of citation parsing, where the q distribution may not deviate much from the uniform distribution (Figure 3(c)) in such a task.

in the task of NER, we find such a simple treatment to the q distribution often leads to sub-optimal results (as we can see in the experiments later) as the q distribution is highly skewed due to the large amount of O labels.


Since existing work explicitly exploited annotated trigger words in their approaches, they can directly model this observation.

in our case, annotated triggers are unavailable.


On one hand, v att is calculated by the dot product of s att and t 1 , which is designed to capture local features (specifically, features about hidden trigger words).

the last output of the L-STM layer, h n , encodes global information of the whole sentence, thus v global = h n • t T 2 is expected to capture global features of a sentence.


In this task, we focus on named entity recognition at discourse level (DiscNER).

to traditional sentence-level NER (SentNER), where sentences are processed independently, in DiscNER, long-range dependencies and constraints across sentences have a crucial role in the tagging process.


For benchmarks annotated with the knowledge base (KB) guided distant supervision, this assumption is often valid since all types are from KB ontologies and naturally follow tree-like structures.

since knowledge bases are inherently incomplete (Min et al., 2013), existing KBs only include a limited set of entity types.


More sophisticated features based on different kinds of similarity measures have also been considered, such as the surface similarity based on Dice coefficient and the WuPalmer WordNet similarity between argument heads (Mc-Conky et al., 2012;Cybulska and Vossen, 2013;Krause et al., 2016).

these features are computed using either the outputs of event argument extractors and entity coreference resolvers (Ahn, 2006;Ng, 2014, 2015; or semantic parsers (Bejan and Harabagiu, 2014;Yang et al., 2015;Peng et al., 2016) and therefore suffer from serious error propagation issues (see Lu and Ng (2018)).


Several previous works proposed joint models to address this problem (Lee et al., 2012;, while others utilized iterative methods to propagate argument information Choubey and Huang, 2017) in order to alleviate this issue.

all of these methods still rely on argument extractors to identify arguments and their roles.


The event pair (m 1 , m a 2 ) has a pair of implicit compatible arguments in the REASON-argument role and is likely to be coreferent.

altering the target argument to contentious citizenship amendment bill (m b 2 ) would yield an pair of implicit incompatible arguments, and the resulting event pair would become non-coreferent.


Ideally, in order to make the memory best represents a previous task, we hope to choose diverse samples that best approximate the distribution of task data.

distribution approximation itself is a hard problem and will be inefficient due to its combinatorial optimization nature.


These approaches could successfully prevent forgetting.

they do not suit many lifelong settings in NLP.


In OTyper (Yuan and Downey, 2018), each type is represented by averaging the embedding of the words constitutes the type label.

protoLE (Ma et al., 2016) represents each type by a prototype that consists of manually selected entity mentions, where the type embedding is obtained by averaging the prototype mentions' word embeddings.In contrast, our work differs from OTyper and protoLE by constructing the type representations based on the Wikipedia descriptions of the types, which not only carry more information about the type but also can be easily adapted to other tasks such as event typing and text classification.


Short descriptions are less informative and carry less shared semantics with the type's mentions.

overly long descriptions could also be confusing as it might share a significant number of common words with the descriptions of other types.


However, its input is not the meaning vector but the form vector:f The motivator has the same architecture as the discriminator, and the same loss function.

while the adversarial loss forces the encoder E to produce a meaning vector m with no information about the form f , the motivational loss encourages E to encode this information in the form vector by minimizing L M : The overall training procedure follows the methods for training GANs (Goodfellow et al., 2014;Arjovsky et al., 2017) and consists of two stages: training the discriminator D and the motivator M , and training the encoder E and the generator G. to Arjovsky et al.


If the true post-modifier, the one that is actually used in the context, is rated the highest compared to the rest, then we assume the post-modifier is indeed specific to the context.

if the crowd workers rate multiple other post-modifiers as good fits for the context, then the true post-modifier is not context specific.


People tend to prefer generated post-modifiers over the ones written by professional journalists when they are shorter and to use more general terms without elaborating too much about the entity.

longer   and more detailed human written post-modifiers are preferred when they are especially relevant to the rest of the sentence.


It continues by adding to this hypothesis the second word, small.

suppose it then extends this hypothesis with cat.


Given a perfect rewriter that always generates semantically equivalent paraphrases and a perfect NLI model robust to perturbations, we would expect no change in predictions between the original development set and the rewritten ones.

this is not what we observe; Table 4 shows that rewriting leads to a greater percentage of newly incorrect predictions than newly correct predictions.


(2016  It is shown that augmenting at evaluation time (aggregation by voting) result in stable improvement (around +2~3% MAP and +2~6% MRR for both scenarios that either augments the training data or not)-this shows that increasing the paraphrastic diversity of the answer candidates could potentially make the system more robust.

augmenting the training set does not yield such improvements-we speculate that this may introduce some noise to the training data.


As for the crea., without any lexical constraint, the normal model always generates the sentences which are similar to the relatively high-frequency sentences in the corpus and results in the lack of novelty.

the target words are used in its less used literal senses in the uncommon sense model and the results seem to be kind of creative.


It is one of the key challenges in natural language understanding, and has drawn increasing attention in recent years (Levesque et al., 2011;Roemmele et al., 2011;Zhang et al., 2017;Rashkin et al., 2018a,b;Zellers et al., 2018;Trinh and Le, 2018).

due to the lack of labeled training data or comprehensive hand-crafted knowledge bases, commonsense reasoning tasks such as Winograd Schema Challenge (Levesque et al., 2011) are still far from being solved.


Because DAGs do not contain cycles, they must always have at least one root and one leaf, but they can have multiple roots and multiple leaves.

our results apply in different ways to single-rooted and multi-rooted DAG languages, so, given a label set Σ, we distinguish between the set of all connected DAGs with a single root, G 1 Σ ; and those with one or more roots, G * Σ .


Standard word representation models are based on the distributional hypothesis (Harris, 1954) and induce representations from large unlabeled corpora using word co-occurrence statistics (Mikolov et al., 2013;Pennington et al., 2014;Levy and Goldberg, 2014).

as pointed out by recent work (Bojanowski et al., 2017;Vania and Lopez, 2017;Pinter et al., 2017;Chaudhary et al., 2018;Zhao et al., 2018), mapping a finite set of word types into corresponding word representations limits the capacity of these models to learn beyond distributional information, which leads to several fundamental limitations.


f Θ is a composition function taking R w as input and outputting a single vector w as the word embedding of w. For the distributional "word-level" training, similar to prior work (Bojanowski et al., 2017), we adopt the standard skip-gram with negative sampling (SGNS) (Mikolov et al., 2013) with bag-ofwords contexts.

we note that other distributional models can also be used under the same framework.


(2017), and remains a strong baseline for many tasks.

addition treats each subword with the same importance, ignoring semantic composition and interactions among the word's constituent subwords.


This result is quite intuitive: sms is trained according to the readily available gold standard morphological segmentations.

sms is less useful for entity typing, where almost all best-performing configurations are based on morf (see also Figure 4).


(2017) proposes KGE-LDA to incorporate embeddings of KGs into topic models to extract better topic representations for documents and shows promising performance.

kGE-LDA forces words and entities to have identical latent representations, which is a rather restrictive assumption that prevents the topic model from recovering correct underlying latent structures of the data, especially in scenarios where only partial kGs are available.


For example, for the second column of 20 Newsgroups, topic words from both TMKGE and KGE-LDA are related to computers.

it can be noted that words from TMKGE focus more on the core words of computer science.


(2018) introduced a rank-based similarity measure for word embeddings, called APSynP, and demonstrated its efficacy on outlier detection tasks.

the results on the word-level similarity benchmarks were mixed, which, interestingly enough, could have been predicted in advance by our analysis.


We show that the addition of MIL regularizers for generating explanations using thresholded attention improved precision and recall hypothesis explanations.

similar improvements were not realized for the premise sentence.


(3) On the more difficult WN18RR and FB15k-237, ConvR consistently outperforms most of the baselines, except for MRR score of ConvKB on FB15k-237.

on WN18RR ConvR outperforms Con-vKB on all known metrices, especially MRR.


After that, (Nguyen et al., 2018) propose ConvKB that explores the global relationships among same dimensional entries of the entity and relation embeddings.

neither of them models the interactions between various positions of entities and relations.


As shown in the above example, we can see it is necessary to include temporal information in DS.

in fusion module, most existing work focused on denoising using methods such as attention or reinforcement learning.


TempMEM also follows the encoding-fusion framework (Zeng et al., 2015;Lin et al., 2016;Luo et al., 2017).

we make two crucial modifications to the original framework.


2 Hyper-Parameter Settings For WIKI-TIME experiments, we construct query over each appeared time spot in the mention set.

for NYT-10 experiments, we adopt a single query without temporal encoding to compare results with other baseline methods since the dataset only contains one label for each mention set.


This classifier is trained using a cross entropy loss with a training dataset whose examples are from seen classes only.

the zero-shot classifier is a binary classifier with a sigmoid output.


hand, the combination of [v w ; v c ], which included semantic embeddings of both words and the class label, increased the accuracy of predicting unseen classes clearly.

the zero-shot classifier fed by the combination of all three types of inputs ] achieved the highest accuracy in all settings.


Standard word embedding algorithms, such as word2vec and Glove, make a restrictive assumption that words are likely to be semantically related only if they co-occur locally within a window of fixed size.

this restrictive assumption may not capture the semantic association between words that co-occur frequently but non-locally within documents.


The co-occurrences in both word2vec and glove are essentially local in nature.

our proposed algorithm leverages both local and non-local co-occurrences.


In our method, we use a similar graph-based construction to train vector representations of a node (each node a word).

we use a stratified sampling approach within a maximum distance (hopcount) of 2, instead of allowing the random walk to proceed along in a combined depth-first and breadth-first manner, as in (Grover and Leskovec, 2016).


Among the baseline approaches, both node2vec and SGNS-LDA work well on the concept categorization task.

the performance improvements are inconsistent across datasets, e.g.


Similarly, it can be seen that the results tend to improve with higher values of β, which confirms that direct associations between words in the word-node graph are more important than transitive ones (2nd plot from the left and the rightmost plot of Figure 1).

second-order transitive associations are still important because the results tend to decrease for β close to 1.


Furthermore, we claim that polysemous words change their meaning in different topic domains; this is reflected in rela-tive shifts of their distributional representations in different topic-based DSMs.

semantic anchors should have consistent semantic relationships regardless of the domain they reside in.


In particular, given the wordpair (w, w ), and their provided contexts (c, c ) we define: Following the notation used in 3.2, K is the number of topics returned by the trained LDA model, x j is the word embedding trained on the subcorpus corresponding to the j-th topic after being projected to the unified vector space, p(j|w, c) denotes the posterior probability of topic j returned by LDA given as input the context c of word w, d denotes the cosine similarity between the two input representations and finallyx (w) = u argmax 1≤j≤K p(j|w,c) (w) is the vector representation of word w that corresponds to the topic with the maximum posterior for c. Intuitively, a higher score in MaxSimC indicates the existence of more robust multi-topic word representations.

avgSimC provides a topic-based smoothed result across different embeddings.


The "pairs in resource" assessment is particularly antithetical to the spirit of RW, which is often employed to assess word vector coverage, and we admit that FN only contains 6.3% of the RW word pairs.

we   would argue that there is an important difference between concluding that a semantic resource does not yield gains in retrofitting vs. concluding that the resource improves the quality of the vectors it covers.


The broad success of CWRs indicates that they encode useful, transferable features of language.

their linguistic knowledge and transferability are not yet well understood.


Our results confirm that task-trained contextualization is important when the end task requires specific information that may not be captured by the pretraining task ( §4).

such end-taskspecific contextualization can come from either fine-tuning CWRs or using fixed output features as inputs to a task-trained contextualizer; Peters et al.


However, the settings that achieve the highest performance for individual target tasks often involve transferring between related tasks (not shown in  the syntactic dependency arc classification (EWT) task, we see the largest gains from pretraining on the task itself, but with a different dataset (PTB).

pretraining on syntactic dependency arc prediction (PTB), CCG supertagging, chunking, the ancestor prediction tasks, and semantic dependency arc classification all give better performance than bidirectional language model pretraining.


Prior work on combining language modeling and unsupervised tree learning typically embed soft, tree-like structures as hidden layers of a deep network (Cho et al., 2014;Chung et al., 2017;Shen et al., 2018Shen et al., , 2019.

buys and blunsom (2018) make Markov assumptions and perform exact marginalization over latent dependency 22 Many prior works that induce trees directly from words often employ additional heuristics based on punctuation (Seginer, 2007;Ponvert et al., 2011;Spitkovsky et al., 2013;Parikh et al., 2014), as punctuation (e.g.


Standard linguistic theories propose that natural language is structured as nested constituents organised in the form of a tree (Partee et al., 1990).

most popular models, such as the Long Sort-Term Memory network (LSTM) (Hochreiter and Schmidhuber, 1997), process text without imposing a grammatical structure.


As shown in Fig.1, our model has a little loss in accuracy as the length increases to ten times the maximum length seen during training.

we notice that final representations produced by the parser are very similar to each other.


Traditionally, supervised parsers trained on datasets such as the Penn Treebank (Marcus et al., 1993) are used to obtain syntactic trees.

the treebanks used to train these supervised parsers are typically small and restricted to the newswire domain.


It contains Wikipedia articles from a wide range of topics.

the CoNLL 2003 corpus consists of news articles.


Given an initial value c 0 c(0) ≥ 0 and a slope parameter r, we define: In this case, new training examples are constantly being introduced during the training process, with a constant rate r (as a proportion of the total number of available training examples).

note that we can also define r = (1 − c 0 )/T , where T denotes the time after which the learner is fully competent, which results in: Root: In the case of the linear form, the same number of new and more difficult, examples are added to the training set, at all times t. as the training data grows in size, it gets less likely that any single data example will be sampled in a training batch.


This is contrary to other results in the machine translation community (e.g., Vaswani et al., 2017), but could be explained by the fact that we are not using any learning rate schedule for training Transformers.

they never manage to outperform Transformers in terms of test BLEU score of the final model.


Back-translation has been dominantly used in these approaches, where pseudo sentence pairs are generated to train the translation systems with a reconstruction loss.

it is inefficient because the generated pseudo sentence pairs are usually of low quality.


Recently, neural-based methods (Chu et al., 2016;Grover and Mitra, 2017;Grégoire and Langlais, 2018) aim to select potential parallel sentences from monolingual corpora in the same domain.

these neural models need to be trained on a large parallel dataset first, which is not applicable to language pairs with limited supervision.


Most existing methods in comparable corpora mining introduce two encoders to represent sentences of two languages separately, and then use another network to measure the similarity (Chu et al., 2016;Grover and Mitra, 2017;Grégoire and Langlais, 2018).

owing to the shared encoders and decoders in language modeling, the semantic spaces of two languages are already strongly connected in our scenario.


These results validate the effectiveness of our approach and indicate that the proposed extract-edit learning framework can learn a better mapping and alignment between language spaces than back-translation.

if extracting only top-1 target sentence in our approach, the performances are not always improved (e.g., en → de, de → en, and en → ro).


(2018b) make encouraging progress on unsupervised NMT structure mainly based on initialization, denoising language modeling, and back-translation.

all these unsupervised models are based on the back-translation learning framework to generate pseudo language pairs for training.


Solving the problem could significantly improve data efficiency-a single multilingual model would be able to generalize and translate between any of the O(k 2 ) language pairs after being trained only on O(k) parallel corpora.

performance on zero-shot tasks is often unstable and significantly lags behind the supervised directions.


(2016) also show that resulting models seem to exhibit some degree of zero-shot generalization enabled by parameter sharing.

since we lack data for zero-shot directions, composite likelihood (3) misses the terms that correspond to the zero-shot models, and hence has no statistical guarantees for performance on zero-shot tasks.


(2017) would require training O(k 2 ) models to encompass all the pairs.

we use a single multilingual architecture which has more limited model capacity (although in theory, our approach is also compatible with using separate models for each direction).


While our approach gives small gains over these baselines, we believe the dataset's pecularities make it not reliable for evaluating zero-shot generalization.

on our proposed preprocessed IWSLT17 that eliminates the overlap and reduces the number of supervised directions (8), there is a considerable gap between the supervised and zeroshot performance of Basic.


Note that, the number of recurrence step T is allowed to be unequal to the length of input sequence J.

to RNN which recurs over the individual symbols of the input sequences, ARN recurrently revises its representations of all symbols in the sequence with an attention model.


A classic solution employs reinforcement learning (RL) to learn a dialog policy that models the optimal action distribution conditioned on the dialog state (Williams and .

since there are infinite human language possibilities, an enduring challenge has been to define what the action space is.


From Table 2, it appears that the word-level RL baseline performs better than Lite-Cat in terms of rewards.

figure 2 shows that the two LaRL models achieve strong task rewards with a much smaller performance drop in language quality (PPL), whereas the word-level  model can only increase its task rewards by deviating significantly from natural language.


(2016a) encourage diversity by re-ranking the beam search results according to their mutual information with the conversation context.

as beam search itself often produces lists of nearly identical sequences, this method can require a large beam width (e.g.


Gaussian) z instead of a point encoding.

it suffers from the vanishing latent variable problem (Bowman et al., 2016;Zhao et al., 2017) when applied to text generation tasks.


Accordingly we refer this approach as SPACEFU-SION with path regularization.

to previous multi-task conversation model (Luan et al., 2017), where S2S and AE are trained alternately, our approach trains S2S and AE at the same time by minimizing the loss function of Equation 3.


End-to-end neural networks trained for these task-oriented dialogs are expected to be immune to any changes in the KB.

existing approaches breakdown when asked to handle such changes.


(2018) showed how the reported high performance of multi-prototype techniques in this dataset was not due to an accurate sense representation, but rather to a subsampling effect, which had not been controlled for in similarity datasets.

a context-insensitive word embedding model would perform no better than a random baseline on our dataset.


Isidora, therefore, is the city of his dreams… Figure 1: Calvino labels the thematically-similar cities in the top row as cities & the dead.

although the bottom two cities share a theme of desire, he assigns them to different groups.


The two cities appear largely dissimilar: Isaura is a city with a thousand wells dug by its inhabitants, while Armilla is an "unfinished" city without walls, ceilings, or floors.

both cities' descriptions mention supernatural beings living underground.


All these prior approaches were evaluated on existing datasets.

we perform studies on PAWS, a new dataset that emphasizes the importance of capturing structural information in representation learning.


Our preliminary results showed that the distribution of paraphrase to nonparaphrases from this method is highly imbalanced (about 1:4 ratio).

we seek to create a balanced dataset, so we use an additional strategy based on back translation-which has the opposite label distribution and also produces greater diversity of paraphrases while still maintaining a high BOW overlap.


GEC models have been previously evaluated based on a single commonly applied corpus: the CoNLL-2014 benchmark.

the evaluation remains incomplete because the task difficulty varies depending on the test corpus and conditions such as the proficiency levels of the writers and essay topics.


This field pre-viously focused on improving parsing accuracy on Penn Treebank (Marcus et al., 1993).

robustness was largely improved by evaluation using multiple corpora including Ontonotes (Hovy et al., 2006) and Google Web Treebank (Petrov and McDonald, 2012).


Transformer performs best on CoNLL-2014.

it exhibits thirdbest performance among FCE, KJ, and ICNALE; LSTM outperforms the other models by a large margin of up to 5.3 F 0.5 points.


Similar to (Joty and Hoque, 2016), we could use h i to classify sentence x i into one of the speech act types using a Softmax output layer.

in that case, we would disregard the discourse-level dependencies between sentences in a conversation.


In this case, since the out-ofdomain labeled dataset (MRDA) is much larger, it overwhelms the model inducing features that are not relevant for the task in the target domain.

when we provide the models with some labeled in-domain examples in the semisupervised (50%) setting, we observe about 11% absolute gains in QC3 and BC3 over the corresponding Merge baselines, and 7 -8% gains over the corresponding Fine-tune baselines.


continuing the classification with the positively categorized instances of the previous step.

the independent classification shows already that the amount of data is insufficient.


The SemEval task was to estimate the intensity of a given tweet and its corresponding emotion.

in this study, we utilize the labeled dataset only to classify the tweets into four emotion categories and use the training, development and test sets provided in this dataset in our experiments.


In this approach, adversarial examples are constructed through an optimization process that uses gradient descent to search for input examples that maximally change the predictions of a model.

developing attacks with only black-box access to a model (no access to gradients) is still under-explored in NLP.


Our model identifies semantically-relevant parts of documents and locally integrates their representations through clustering and autoencoding.

to averaging, our model prevents larger semantically-relevant parts of inputs to dominate final representations.


As offensive content has become pervasive in social media, there has been much research in identifying potentially offensive messages.

previous work on this topic did not consider the problem as a whole, but rather focused on detecting very specific types of offensive content, e.g., hate speech, cyberbulling, or cyber-aggression.


For instance, in the following sentence "... , which binds to the enhancer A located in the promoter of the mouse MHC class I gene H-2Kb, ...", when determining the trigger type of binds, we need to carefully select its contextual words, such as H-2Kb, which indicates the object of binds.

binds and H-2Kb are sepa-  [trigger] [argument] The framework of the KB-driven Tree-LSTM model.


We can see that, without using KB information, the Tree-LSTM mistakenly predict the argument role of E1 as None.

by incorporating KB concept embeddings, especially the information from the function description positive regulation of transcription, DNA-templated for Tax, our approach successfully promotes the probability of E1 being predicted as the Theme of E2.


Moreover, the additional data should be similar enough to existing training data to be helpful.

crafting new features requires creativity and collaboration with subject matter experts, and the implementation can be time consuming.


Generative Adversarial Networks The idea of aligning representations by making them indistinguishable is inspired by GAN (Goodfellow et al., 2014), where a generator produces fake images (or other data) that are as similar to real data as possible.

our model does not have a generator component as GANs do.


The first two rows present the average lengths of each utterance of counselors and clients in terms of words and characters, showing there is a small difference between the counselor utterance and the client utterance.

the average number of utterances in a single counseling session differs; on average, clients write more utterances than counselors.


Modern neural architectures for NLP are highly effective when provided a large amount of labelled training data (Zhang et al., 2015;Conneau et al., 2017;Bowman et al., 2015).

a large labelled data set is not always readily accessible due to the high cost of expertise needed for labelling or even due to legal barriers.


This produces dramatic improvements over a range of NLP tasks where appropriate unlabelled data is available (Peters et al., 2017(Peters et al., , 2018Akbik et al., 2018;Devlin et al., 2019).

there is still a lack of systematic study on how to select appropriate data to pretrain word vectors or LMs.


The question in domain adaptation is usually framed as 'Given a source and a target, how to transfer?'.

the question we address is 'Given a specific target, which source to choose from?'.


On one hand, it is expected that PubMed is similar to CRAFT and JNLPBA, since they are all sampled journal articles about biology and health, thus being similar in terms of both field and tenor.

although ScienceIE does not have the same field as PubMed (computer science, material and physics versus biology and health), they are similar because they share a similar tenor (scholarly publications).


The field of CADEC is therefore more similar to PubMed which includes journal articles in health discipline and MIMIC which contains clinical notes.

cADEc is written by patients, and can be considered as 'drug reviews'.


A practical approach for collecting a suf-ficiently large corpus would be to use crowdsourcing platforms like Amazon Mechanical Turk (MTurk).

crowd workers in general are likely to provide noisy annotations (Abad and Moschitti, 2016;Plank et al., 2014;Alonso et al., 2015), an issue exacerbated by the technical nature of specialized content.


The model with best precision is different for Patient, Intervention and Outcome labels.

re-weighting by difficulty does consistently yield the best recall for all three extraction types, with the most notable improvement for I and O, where recall improved by 10 percentage points.


Obtaining expert annotations for the one thousand most difficult instances greatly improved the model performance.

the choice of how many difficult instances to annotate was an uninformed choice.


Neural approaches seem to be stronger for languages with complex case systems and agglutinative morphology.

edit-tree approaches excel on more synthetic languages (e.g.


This is most likely due to the fact that, from an edit-tree approach perspective, a large number of trees creates a large number of classes, which leads to higher class imbalance and more sparsity.

edit-tree based approaches do outperform representation learning methods for languages with lower number of trees, which leads to the intuition that the edit-tree formalism does provide a useful inductive bias to the task of lemmatization and it should not be discarded in future work.


Comparing encoder representations to decoder representations, it is interesting to see that in several cases the decoder side representations performed better than the encoder side ones, even though the former were trained using a uni-directional LSTM.

since there is no difference in the general trends between the encoder-and the decoder-side representations, below we focus on the encoder-side only.


For English, the subword (BPE and Morfessor) and the character representations yielded comparable results.


9 We can see that combinations involving characters (B+C, W+C in the table) yield larger improvement compared to combining wordand BPE-based representations (W+B).

combining all three performed best for all languages and for all tasks.


We can see that in most cases, the subword-based systems perform better than the word-based and the character-based ones.

this is not true in the case of using their representations as features for a core NLP task as in our experiments above.


Lemmatization has previously been shown to improve recall for information retrieval (Kanis and Skorkovská, 2010;Monz and De Rijke, 2001), to aid machine translation (Fraser et al., 2012;Chahuneau et al., 2013) and is a core part of modern parsing systems (Björkelund et al., 2010;Zeman et al., 2018).

the task is quite nuanced as the proper choice of the lemma is context dependent.


A key feature of our model is its simplicity: Our contribution is to show how to stitch existing models together into a joint model, explaining how to train and decode the model.

despite the model's simplicity, it still achieves a significant improvement over the state of the art on our target task: lemmatization.


Principles are universally true for all languages.

languages are also governed by parameters.


The likelihood of this model is now convex in the parameter embeddings.

to the full matrix factorisation setting, here, all language-specific knowledge must come from an external source, namely, the unlabelled text.


Brown clustering is a predictable, bottom-up, agglomerative, hard clustering algorithm that for the same hyper-parameter k, generates the same clusters and therefore only one data sample.

2 the Exchange algorithm is an iterative clustering algorithm that has a complete and valid cluster partitioning at the end of each iteration.


czech is highly inflected accounting for gender, case, number and person.


Note that the choice of word types w T and w S to calculate CVL is arbitrary.

cVL is only meaningful when the two word types are semantically related, such as word translations, because those word pairs are where the knowledge transfer takes place.


This term usually cannot be estimated in practice since the labels for target documents are unavailable.

we can still calculate this term for the purpose of analysis.


In the middle panel of Figure 2, CVL over all word pairs from topic words is decreasing as sampling proceeds and becomes stable by the end of sampling.

the correlations between CNPMI and CVL are constantly decreasing.


The larger the crosslingual entropy, the harder it is to get a low CVL because it needs larger monolingual entropy to decrease the bound, as shown in Section 2.4.

the inner product of word pairs shows an opposite pattern of CVL, indicating a negative correlation (Lemma 1).


In Figure 2 we  see the correlation between CNPMI and CVL is around −0.4 at the end of sampling, so there are fewer clear patterns for CNPMI in Figure 4.

we also notice that the word pairs with higher CNPMI scores often appear at the bottom where crosslingual entropy is low while the monolingual entropy is high.


McDonald and Nivre (2007) and Mc-Donald and Nivre (2011) have shown that historybased features enhance transition-based parsers as long as they do not suffer from error propagation.

nivre (2006) has also shown that lookahead features are absolutely crucial given the greedy left-to-right parsing strategy.


We hypothesise that arc depth might impact the usefulness of composition, since more depth means more matrix multiplications with the composition function.

we find no correlation between average arc depth of the treebanks and usefulness of composition.


Early work focused on supervised methods maximizes the similarity of the embeddings of words that exist in a manually-created dictionary, according to some similarity metric (Mikolov et al., 2013;Faruqui and Dyer, 2014;Jawanpuria et al., 2018;.

recently proposed unsupervised methods frame this problem as minimization of some form of distance between the whole set of discrete word vectors in the chosen vocabulary, e.g.


Also, DeMa-BME demonstrates notably better performance on distant language pairs (en-ru, enja and en-zh) over other unsupervised methods, which often achieve good performance on etymologically close languages but fail to converge on the distant language pairs.

when the dictionary is initialized with identical strings for SL-unsup, we obtain decent results on these languages.


Most previous work in this field focuses on extracting phrases from target posts  or selecting candidates from a pre-defined list Zhang et al., 2017).

hashtags usually appear in neither the target posts nor the given candidate list.


Topic models are also widely applied to induce topic words as hashtags (Krestel et al., 2009;Ding et al., 2012;Godin et al., 2013;Gong et al., 2015;.

these models are usually unable to produce phrase-level hashtags, which can be achieved by ours via generating hashtag word sequences with a decoder.


We then explore three shielding methods-visual character embeddings, adversarial training, and rule-based recovery-which substantially improve the robustness of the models.

the shielding methods still fall behind performances achieved in non-attack scenarios, which demonstrates the difficulty of dealing with visual attacks.


Recently, some NLP systems have exploited visual features to capture visual relationships among characters in compositional writing systems such as Chinese or Korean (Liu et al., 2017).

in more general cases, current neural NLP systems have no built-in notion of visual character similarity.


Numbers, especially, are more difficult to recover.

even at 80% disturbance level, humans can, on average, correctly recover at least 93% of all characters in the input text in all conditions.


VIPER with DCES and p = 0.1 achieves a success rate of 24.1%-i.e., roughly one fourth of the toxic comments receive a lower TL.


To prevent introducing bias in this manual filtering step, we define each relation descriptor directly as its five nearest neighboring words in the input vocabulary.

since the full vocabulary would contain many uncommon words which could hinder the interpretation of relation descriptors, we limit both models to choose descriptor words from the most frequently occurring 500 words in their own processed input vocabulary (i.e., verb predicates only for LARN and all words for RMN).


LEAD-1 achieves ROUGE (1/2/L)scores of 27.5/9.6/23.7 respectively.

our selector achieves scores of 30.2/12.2/26.45 which presents an improvement of over 10% in each category.


Generating text is a core part of many NLP tasks such as image captioning (Lin et al., 2014), opendomain dialogue , story generation (Roemmele, 2016), and summarization (Nallapati et al., 2016).

proper evaluation of natural language generation has proven difficult (Liu et al., 2016;Novikova et al., 2017;Chaganty et al., 2018).


First, we show that human evaluation alone is insufficient to discriminate model generations from the references, leading to inflated estimates of model performance.

hUSE is able to reveal deficiencies of current models.


In the spirit of the Turing Test, we could consider using the error rate of a human discriminator f hum instead, often considered the gold standard for evaluation.

while humans might have knowledge of p ref , they do not have full knowledge of p model and thus would have difficulties determining which sentences a model cannot generate.


This is analogous to learning the discriminator in a Generative Adversarial Network (GAN) (Goodfellow et al., 2014) or learning an evaluation metric from human judgments (Lowe et al., 2017).

as (x, y) are high-dimensional objects, training a good classifier is extremely difficult (and perhaps not significantly easier than solving the original generation problem).


Table 1 shows that single-sentence language models are nearly indistinguishable, with HUSE = 0.86 and implied discriminator error of 43%.

both summarization and dialogue are highly distinguishable (HUSE ≈ 0.5) with relatively low quality when sampled from t = 1.0.


Examining the samples in this top-right region reveals that these are news stories with short headlines such as "Nadal pulls out of Sydney International" which can be reliably generated even at t = 1.0.

the model frequently generates low quality samples that can easily be distinguished such as "two new vaccines in the poor countries were effective against go-it-alone study says" (Table 2).


This results in two incomparable evaluation metrics, which prevent us from reasoning about tradeoffs between diversity and quality.

hUSE allows us to make precise statements about the tradeoffs between model quality and diversity because it is a single metric which decomposes into diversity and quality terms.


Task success combined with dialogue cost can be linked to human judgments like user satisfaction via the PARADISE framework (Walker et al., 1997).

in chitchat tasks, which we study in this work, automatic metrics and their relation to human ratings are less well-understood.


Our most engaging model, which controls both repetition and question-asking -marked 'Question (CT)' in Figure 3 (left) -matches the engagingness of the winning entry in the ConvAI2 competition, as both models achieve a raw score 7 of 3.1 (Dinan et al., 2019).

the Con-vAI2 winner, Lost in Conversation, was trained on approximately 12× as much data as our model.


8 For full details of these scores, see Appendix F and H. For time and budget reasons, we did not evaluate any models controlling both question-asking and specificity.

we expect it is possible to obtain further improvements by doing so.


Globally normalized neural sequence models are considered superior to their locally normalized equivalents because they may ameliorate the effects of label bias.

when considering high-capacity neural parametrizations that condition on the whole input sequence, both model classes are theoretically equivalent in terms of the distributions they are capable of representing.


When conditioned on the the full input sequence and the entire prediction history, both locally normalized and globally normalized conditional models should have same expressive power under a highcapacity neural parametrization in theory, as they can both model same set of distributions over all finite length output sequences (Smith and Johnson, 2007).

locally normalized models are constrained in how they respond to search errors during training since the scores at each decoding step must sum to one.


(2016) prove that this class of locally normalized models that relies on the structural assumption of access to only left-to-right partial input at each step, is strictly less expressive than its globally normalized counterpart.

the standard sequence-to-sequence models used most often in practice and presented in this paper actually condition the decoder on a summary representation of the entire input sequence, x, computed by a neural encoder.


This illustrates a mechanism by which search-aware training of globally normalized models in a large search spaces might  Soft Beam Computation Soft-k-argmax be more effective.

as discussed earlier, if we can perform exact search then this label bias ceases to exist because both the models have the same expressive power with a search-agnostic optimization scheme.


For puns, both the local and global surprisal should be positive because they are unusual sentences by nature.

the global surprisal should be lower than the local surprisal due to topic words hinting at the pun word.


SURGEN sometimes generates creative puns that are rated even funnier than human-written puns (example 1).

nEURALJOInTDECODER at best generates ambiguous sentences (example 2 and 3) and sometimes the sentences are ungrammatical (example 1) or hard to understand (example 4).


Given sentence vector s i as input, SUMO computes: r i , e ij = TMT(r i ,ẽ ij ) (11) Iterative Structure Refinement SUMO essentially reduces summarization to a rooted-tree parsing problem.

accurately predicting a tree in one shot is problematic.


t 0 1 2 3 4 w t -Bernie Sanders for president c t 2 1 0 0 0 Unlike the controlled-length scenario, at test time we do not know the number of new content words to generate.

the count for most FTFYs is between 1 and 5, inclusive, so we can exhaustively search this range during decoding.


In a programming subreddit, a sarcastic FTFY might be this is a strange feature.

in a Pokémon subreddit, an FTFY might be this is a strange dinosaur in an argument over whether Armaldo is a bug or a dinosaur.


5It is not possible to treat the missing annotations as "total disagreement" because per Krippendorff (1995), α U has no concept of this; there is no lowest disagreement score.

are lower than the agreement among extensively trained annotators reported by Stab and Gurevych (2014) (α U = 0.7726, 0.6033, 0.7594).6 they are broadly comparable to interannotator agreement scores reported in similar (and in some cases, even simpler) discourse-level argument annotation studies with expert-trained annotators, such as Aharoni et al.


Furthermore, we observe that a main event may trigger several consequent events which themselves are causally related.

causal relations involving only non-main events are less likely to show transitivity.


(2009) proposed a matrix factorization method based on SVD to learn latent representations of users and items from the rating matrix between users and items.

since the numbers of users and items in online platforms are usually huge, and the rating matrix between users and items is usually very sparse, it is quite diffi- cult for those rating based recommendation methods to learn accurate user and item representations (Zheng et al., 2017;Tay et al., 2018).


These methods usually concatenate the reviews from the same user or the same item into a long document.

different reviews usually have different informativeness in representing users and items.


ESA is usually trained on Wikipedia, since the authors of the original ESA paper suggest that the articles of the training corpus should represent disjoint concepts, which is only guaranteed for encyclopedias.

stein and Anerka (Gottron et al., 2011) challenged this hypothesis and demonstrated that promising results can be obtained by applying EsA on other types of corpora like the popular Reuters newspaper corpus as well.


In our scenario with rather short text snippets and keyword lists, this was not much of an issue.

for large documents, such a comprehensive comparison could become soon infeasible.


With these in mind, we make two important observations about the existing keyphrase extraction techniques: • In the supervised setting, word importance is captured in metrics and engineered features, as is local random walk scores.

the structure of the graph formed by the text is not exploited.


Enhancements to the model introduce much sophisticated local information aggregation between the node pairs as in Graph Attention Networks (GAT) Veličković et al..

we note that such prior methods fall inherently into the classification paradigm, and hence focus on only local aggregation; i.e., to pull in the most significant feature from its neighbors.


This state-of-the-art technique exploits a complementary idea of sequential semantic modeling focused on generating keyphrases rather than merely extracting them.

their model does not address the common scenario of keyphrase extraction from long documents but only for short excerpts (namely, the abstract).


This attention is normalized in GAT as: to smooth the gradients.

to GCN, GAT replaces theÃ with a learnedÃ where each entry α ij is a normalized score computed on each node pair by the gradient (Fig.


In this way, such a node channels more to its neighbor, exerting comparatively steeper gradients to less essential nodes, hence giving a larger chance for Node h 1 to be considered a keyphrase.

without Glocal's scaling mechanism, such modeling is not captured and essentially unlearnable.


It lacks only 0.15 of a MAP point on test set as compared to the highest scoring LSP-AP model.

as LSP does not optimize the ranking measure directly, it may result unstable.


It is also true that in this work, by fixing the weighting schema v, we limited our study to one particular case of a structural ranking representation.

finding an appropriate structural feature space, e.g., to the extent enabled by tuning the positional weights v j for the particular application, can be potentially beneficial.


(2015) proposed a neural implementation of multi-instance learning to leverage multiple sentences which mention an entity pair in distantly supervised relation extraction.

their model picks only one sentence to represent an entity pair, which wastes the information in the neglected sentences.


We can consider it as a sub-domain of sentiment analysis.

the goal of sentiment analysis is to classify the polarity of a tweet sentiment based on its contents, whereas identification of stance is dependent on the specific target.


The NMT achieved especially high performance in terms of fluency.

it tends to generate more omission errors than statistical machine translations generate.


Near 0.0, the distance between T and R is small.

in the automatic MT evaluation metrics, the score is close to 1.0 when the evaluation for the translation is generally high.


This word-level loss ensures efficient and scalable training of seq2seq models.

this word-level training objective suffers from a few crucial limitations, namely the label bias, the exposure bias, and the loss-evaluation mismatch (Lafferty et al., 2001; Figure 1: The BLEU score of BSO decreases after beam size 3 as results of increasing length ratio 1 in German→English translation.


The beam search restarts with gold when gold falls off the beam (at step 5).

without using locally-normalized score does not mean that we should stop using the probabilistic value function.


After convergence, Z m l is added to the multilingual space X m : In supervised settings this approach would be impractical as it requires bilingual dictionaries D k,l for all language pairs k, l, and not only with the hub language.

within an unsupervised framework this constraint is lifted.


We observe that value dropping is crucial for SHS to succeed for English-Finnish.


While simple to motivate, this may not always perform well because neural methods benefit from randomization in the minibatches and multiple epochs.

the probabilistic curriculum (Bengio et al., 2009) works by dividing the training procedure into distinct phases.


We can then conclude that perplexity selection may not be an appropriate way to determine the optimal amount of unlabeled-domain data to use for NMT models.

if computational resources are limited, according to the experiment results ( Figure  2) in our work, we recommend 1024k as the first choice for cutoffs on ranked unlabeled-domain data, for NMT domain adaptation models trained with curriculum learning strategy.


From Table 4 we note that the Baseline + FT w/ EP-100k-TBT model already produces a reasonable translation for the input sentence.

if we further fine-tune the model using only 10k MTNT data, we note that the model still struggles with generation of *very*.


With more gains arising from continued research on new neural network architectures and accompanying train-ing techniques (Vaswani et al., 2017;Gehring et al., 2017;Chen et al., 2018), NMT researchers, both in industry and academia, have doubled down on their ability to train high capacity models on large corpora with gradient based optimization.

despite huge improvements in overall translation quality NMT has shown some glaring weaknesses, including idiom processing, and rare word or phrase translation (Koehn and Knowles, 2017;Isabelle et al., 2017;Lee et al., 2018) tasks that should be easy if the model could retain learned information from individual training examples.


Existing approaches have proposed using off the shelf search engines for the retrieval stage.

our objective differs from traditional information retrieval, since the goal of retrieval in semiparametric NMT is to find neighbors which might improve translation performance, which might not correlate with maximizing sentence similarity.


Average and ensemble techniques on checkpoints can lead to further performance improvement.

as these methods do not affect the training process, the system performance is restricted to the checkpoints generated in the original training procedure.


VBSIX applies the same traversal in execution space (lines 10-21).

since each vertex in the execution space represents an execution result and not a particular prefix, we need to modify the scoring function.


In SCENE, each component has only a slight advantage over beam-search, and therefore both are required to achieve significant improvement.

in ALCHEMY and TANGRAM most of the gain is due to the value network.


(2018) assumed ac-cess to the full annotated logical form.

we trained from longer sequences, keeping the logical form and intermediate states latent.


We find that when there are many differences between the current and target world, the value network correctly estimates low expected reward in 87.0% of the cases.

when there is just one mismatch between the current and target world, the value network tends to ignore it and erroneously predicts high reward in 78.9% of the cases.


This case is simpler than restrictive advice, since the human operator just has to provide the direction to adjust the predictions, rather than the precise region of the coordinates.


model would have made a prediction ('x') close to the true block (square).

the advice region (blue) was incorrect (due to the true block being close to the edge of it) and this led to a significantly worse prediction (circle).


This task is inherently discriminative, i.e., there is only a single most relevant clip pertaining to Figure 1: Clip Extraction task for the given query 'the biker jumps to another ramp near the camera' a given query in the corresponding video.

most prior works (Hendricks et al., 2017(Hendricks et al., , 2018Chen et al., 2018;) explore this as a ranking task over a fixed number of moments by uniformly sampling clips within a video.


(2019) address this through a query-guided segment proposal network (QSPN).

the similarity metric used by these approaches is difficult to learn as it is sensitive to the choice of negative samples (Yu et al., 2018) and it still does not consider the discriminative nature of the task.


Furthermore, if we add LSTM span predictors along with the video LSTM (ExCL 2-{b, c}) we obtain an additional boost in performance.

without a recurrent visual encoder, a recurrent span predictor is essential to capture both uni-modal and cross-modal interactions (ExCL 1-{b, c}).


unimodal visual model for German verb sense disambiguation, but we find the opposite for Spanish unimodal verb sense disambiguation.

the early fusion multimodal model outperforms the best unimodal model for both German and Spanish.


We also observe that most uncertainty comes from morphological categories such as noun number, noun definiteness (which is expressed morphologically in Bulgarian), and verb tense, all of which are inherent (Booij, 1996) 8 and typically cannot be predicted from sentential context if they do not participate in agreement.

9 aspect, although being closely related to tense, is well-predicted since it is mainly expressed as a separate lexeme.