In the original dataset, there are 200 words categorized into 17 classes.

we remove words that do not rank in the top 20,000 by frequency in any decade in our training data to ensure that the synthetic words do not lack context words at a given time.


The statistic in M [w, c] is usually related to pointwise mutual information (Levy et al., 2015a): , and thus larger co-occurrence count #(w, c).

the derivation has two flaws: (1) c could contain negative values and (2) lower #(w, c) could still lead to larger P M I(w, c) as long as the #(w) is small enough.


At this point, the gradients coming from negative sampling (the second term) decrease the same amount of embedding values for both x and y.

the embedding of hypernym y would get higher or equal positive gradients from the first term than x in every dimension because #(x, c) ≤ #(y, c).


In theory, word embeddings obtained by these joint models could be as good as representations produced by models which finetune input vector space.

their performance falls behind that of fine-tuning methods (Wieting et al., 2015).


Lower absolute scores for Italian and German compared to the ones reported for English are due to multiple factors, as discussed previously by : 1) the AR model uses less linguistic constraints for DE and IT; 2) distributional word vectors are induced from smaller corpora; 3) linguistic phenomena (e.g., cases and compounding in DE) contribute to data sparsity and also make the DST task more challenging.

it is important to stress the consistent gains over the vector space specialised by the state-of-the-art ATTRACT-REPEL model across all three test languages.


Fixed-length context windows S running over the corpus are used in word embedding methods as in C-BOW (Mikolov et al., 2013b,a) and GloVe (Pennington et al., 2014).

here we have k = |V| and each cost function f S : R k → R only depends on a single row of its input, describing the observed target word for the given fixed-length context S. for sentence embeddings which are the focus of our paper here, S will be entire sentences or documents (therefore variable length).


(2017) who also use additive compositionality to obtain sentence embeddings.

in contrast to our   Table 4: Comparison of the performance of the unsupervised and semi-supervised sentence embeddings by (Arora et al., 2017) with our models.


An important assumption of such work is that different tree structures lead to different semantic representations even for the same sentence.

they all resort to ex-ternal syntactic resources, such as parse trees or Treebank annotations (Marcus et al., 1993), which limits their broader applications.


A desirable solution would be to automatically and dynamically induce the tree structures for target-specific sentence representations.

the challenge is that the absence of external supervisions makes it difficult to evaluate the quality of the tree structures and train the parameters.


By comparing the decoded outputs with the ground-truth labels, the prediction errors can back-propagate to update parameters of the encoder-decoder network.

it is no more applicable in our settings, as we do not have any explicit supervisions from external syntactic resources.


Sentiment-based method gives the highest F1 score on the positive class.

its performance is not consistent on the negative class, which suggests that it tends to misclassify the sentence as positive.


(2014) cascades parse errors to later stages, which will hurt the performances on downstream tasks.

the adapted tree structures in Dong et al.


Literature survey suggests a wide range of research on sentiment analysis (at the document or sentence level) is being carried out in recent years (Turney, 2002;Kim and Hovy, 2004;Jagtap and Pawar, 2013;Poria et al., 2016;Kaljahi and Foster, 2016;Gupta et al., 2015).

most of these researches are focused on resource-rich language like English.


For a missing word representation, the literature suggests two possible solutions: a) zero vector (Bahdanau et al., 2017) or b) random vector (Dhingra et al., 2017).

in both the cases the resultant vector could be completely out of context and often does not fit well with others.


In a model that takes into account only the simple polar word score, the sentence would have high relevance towards the positive sentiment.

the sequence information of the phrase "far from liking and recommending" dictates the negative sentiment of the sentence.


"I'm far from liking and recommending this phone to anyone."

to A3, architecture A2 does not rely on the sequence information of the extracted features and let the network to learn on its own.


Another limitation of our work is that 7.2M sentences is not a big number in terms of word embedding computation.

the underlying method performs considerably better compared to the state-of-the-art systems, even with all these constraints.


SMT error & corpus size), the proposed method with bilingual embeddings (76.29%) performs considerably at par against the monolin-gual embeddings created from a very large corpus of 53M (77.74%).

the monolingual WE computed using the same amount of corpus (i.e.


First, we follow Katiyar and Cardie (2016) which set aside 132 documents for development and used the remaining 350 documents for 10-fold CV.

in the 10-fold CV setting, the size of the tests sets is 3 times smaller than the dev set size ( Table 2, row 3), and, consequently, results in high-variance estimates on the test sets.


As we have seen so far, many holders that are subjects or A0 roles, and targets that are A1 roles, are properly labeled by both models.

a considerable amount of such holders and targets are not correctly predicted (Table 7-8, col. 2, rows 2-3).


Finally, for example 7 our models make plausible predictions.

the gold holder is always the entity from the coreference cluster that is the closest to the opinion.


More closely related is our prior work (Vyas and Carpuat, 2016) where we used lexical context based embeddings to detect cross-lingual lexical entailment.

the focus of this work is on hypernymy, a more well-defined relation than entailment.


• reverse noising: For reverse noising, we simply train a reverse model from Y → X using our parallel noisy-clean corpus and run standard beam search to generate noisy tar-getsỸ from clean inputs Y .

we find vanilla reverse noising tends to be too conservative.


We observe that token noising, despite matching the frequency of errors, fails to generate realistic errors (Figure 4).

reverse noising yields significantly more convincing errors, but the edit distance between synthesized examples is significantly lower than in real data ( Figure 5).


To test this, we plot MAP for our best self-training model and various QA baselines as we vary the proportion of labeled training set in Figure 3.

we keep the unlabeled text fixed (10K Wikipedia paragraphs).


We observe that all the baselines significantly drop in performance as we reduce the proportion of labeled training set.

the drop happens at a much slower rate for our selftrained model.


, use a search engine and a QA model to answer the simple questions, from which we compute the final answer a. Neural models trained over large datasets led to great progress in RC, nearing human-level performance .

analysis of models revealed (Jia and Liang, 2017;Chen et al., 2016) that they mostly excel at matching questions to local contexts, but struggle with questions that require reasoning.


The first four utilities all return a value by performing the named math operation on its two input arguments.

sum(function,condition) returns the sum of the values of FOL function instances which can be unified with the first argument (i.e., function) and satisfy the second argument (i.e., condition).


The studies include age, gender prediction (Marquardt et al., 2014;Sap et al., 2014), psychological well being (Dodds et al., 2011;Choudhury et al., 2013), and a host of other behavioural, psychological and medical phenomena .

a few works exist which analyze these factors using socio-economic characteristics of the Twitter users.


Surprisingly, we found that average intelligent users are past-oriented but considering the sentiment dimension they seem to be more future-positive.

this should be validated with further investigation.


Interestingly, we found a negative correlation with the future-positive.

we found a positive correlation (0.3614) with the future-neutral which suggests that the users with much above intelligence are futuristic and they express a neutral view.


We found similar patterns of relative goodness of measures: WW was consistently better in scoring similarity between two words and WC was better in measuring the thematic relatedness.

the asymmetry between WC and CW did not come out clearly in these experiments and the overall performance of the GloVe model in the similarity task was much lower than Skipgram.


Indeed, a given sentence-level operation could both change the original meaning by adding or removing information (affecting the P score) and increase simplicity (S).

the identity transformation perfectly preserves the meaning of the original sentence without making it simpler.


One problem with soft attention is that it considers all entity vectors when constructing the topic vector.

not all entities are important and necessary when generating summaries.


A quick read of the original article tells us that the main topic of the article is all about the two political parties arguing over the deal with Iran.

the entity "nuclear" appeared a lot in the article, which makes the soft model wrongly focus on the "nuclear" entity.


This is a major advantage of DUC compared to other datasets, especially when evaluating with ROUGE (Lin, 2004b,a), which was designed to be used with multiple references.

dUC datasets are small, which makes it difficult to use them as training data.


For instance, a summary might contain many individual words from the article and therefore have a high coverage.

if arranged in a new order, the words of the summary could still be used to convey ideas not present in the article.


We observe that publications with lower compression ratio (top-left of the figure) exhibit higher diversity along both dimensions of extractiveness.

as the median compression ratio increases, the distributions become more con-  centrated, indicating that summarization strategies become more rigid.


Semantic Parsing Results SP results are summarized in Table 2.

the neural models, especially those with biasing and copying, strongly outperform all other models and are competitive with related work.


Consistent with recent findings (Dong and Lapata, 2016), we show that relatively simple neural sequence models are competitive with, and in some cases outperform, traditional grammar-based SP methods on bench-mark SP tasks.

this result is not observed in our technical documentation task, in part because this problem is much harder for neural learners given the sparseness of the target data and lack of redundancy.


In this way, we can extract accurate relation mentions for triples with high precision.

if a relation mention doesn't contain any hyponym/synonym words of the relation, our method would be unable to identify it.


Pairs of unprovable sub-goals and plausible single premises are identified by means of a variable unification routine and then linguistic relations between their logical predicates are checked using lexical knowledge such as Word-Net and VerbOcean (Chklovski and Pantel, 2004).

this mechanism is limited to capturing word-to-word relations within a sentence pair.


To increase their coverage of phrasal knowledge, the system combines a resolution strategy to align clauses and literals in a sentence pair and a statistical classifier to identify their semantic relation.

this strategy only considers one possible set of alignments between fragments of a sentence pair, possibly causing inaccuracies when there are repetitions of content words and meta-predicates.


The SNLI dataset (Bowman et al., 2015) contains inference problems requiring phrasal knowledge.

it is not concerned with logically challenging expressions; the semantic relationships between a premise and a hypothesis are often limited to synonym/hyponym lexical substitution, replacements of short phrases, or exact word matching.


In contrast, memory networks are exactly designed to remember previous information.

given the large size of documents and paragraphs, basic memory networks do not handle well irrelevant and noisy information, which we confirmed in our experiments.


In particular, gradient tree boosting has been shown to be highly competitive for ED in recent work (Yang and Chang, 2015;Yamada et al., 2016).

although achieving appealing results, existing gradient-tree-boosting-based ED systems typically operate on each individual mention, without attempting to jointly resolve entity mentions in a document together.


Joint entity disambiguation has been shown to significantly boost performance when used in conjunction with other machine learning techniques (Ratinov et al., 2011;Hoffart et al., 2011).

how to train a global gradient tree boosting model that produces coherent entity assignments for all the mentions in a document is still an open question.


Their experiments on the OAEI benchmarks show that their techniques, even when combined with classical NLP techniques, could not outperform the state-of-theart.

we refine pre-trained word embeddings with the intention of leveraging a new word vector set that is tailored to the ontology matching task.


Schema.org is a collaborative, community activity with a mission to create, maintain, and promote schemas for structured data on the Internet, on web pages, in email messages, and beyond.

dBpedia is a crowd-sourced community effort to extract structured information from Wikipedia and make this information available on the Web.


This operation clearly reduces the model complexity, and can learn intragroup features efficiently.

it fails to capture dependency cross different groups.


Vlachos and Riedel (2014) constructed a dataset for claim verification consisting of 106 claims, selecting data from fact-checking websites such as PolitiFact, taking advantage of the labelled claims available there.

in order to develop claim verification components we typically require the justification for each verdict, including the sources used.


A differently motivated but closely related dataset is the one developed by Angeli and Manning (2014) to evaluate natural logic inference for common sense reasoning, as it evaluated sim-ple claims such as "not all birds can fly" against textual sources -including Wikipedia -which were processed with an Open Information Extraction system (Mausam et al., 2012).

the claims were small in number (1,378) and limited in variety as they were derived from eight binary ConceptNet relations (Tandon et al., 2011).


The RTE component must correctly classify a claim as NOTENOUGHINFO when the evidence retrieved is not relevant or informative.

the instances labeled as NOTENOUGHINFO have no evidence annotated, thus cannot be used to train RTE for this class.


We observe that with fewer than 6000 training instances, the accuracy of DA is unstable.

with more data, its accuracy increases with respect to the log of the number of training instances and exceeds that of MLP.


We also experimented with PPMI and smoothed PPMI with α = 0.75 (Levy et al., 2015) that are commonly used in word embedding.

the learned textual relation embedding turned out to be not very helpful for relation extraction.


We compare our models to that of Gerber and Chai (2012).

their original logistic regression model used many features based on gold annotation from FrameNet, PropBank and Nom-Bank.


The TempRel extraction task has a strong dependency on prior knowledge, as shown in our earlier examples.

very limited attention has been paid to generating such a resource and to make use of it; to our knowledge, the TEMPROB proposed in this work is completely new.


They have been used for a wide variety of tasks, such as textual entailment (Berant et al., 2011), question answering (Fader et al., 2014), and knowledge base population (Angeli et al., 2015).

perhaps due to limited data, existing methods use semisupervised approaches (Banko et al., 2007;Wu and Weld, 2010), or rule-based algorithms (Fader et al., 2011;Mausam et al., 2012;Del Corro and Gemulla, 2013).


Like QA-SRL, QAMR represents predicate-argument structure with a set of question-answer pairs about a sentence, where each answer is a span from the sentence.

while QA-SRL restricts questions to fit into a particular verb-centric template, QAMR is more general, allowing any natural language question that begins with a wh-word and contains at least one word from the sentence.


Hence, the written forms of the two languages should be very similar, which makes the language embeddings based on language modelling highly similar to one another.

when the embeddings are fine-tuned on a task taking orthography as well as phonology into account, this is no longer the case.


Figure 1 shows that the language embeddings of Norwegian Bokmål and Danish diverge from each other, which is especially striking when comparing to the converging with the typologically much more distant languages Tagalog and Finnish.

the absolute difference between Norwegian Bokmål and both Tagalog/Finnish is still greater than that of Norwegian Bokmål and Danish even after 3,000 iterations.


Just as A * approximates h with a "heuristic"ĥ, the next section will approximate H t using a neural estimateĤ t (equations (5)-(6)).

the specific form of our approximation is inspired by cases where H t can be computed exactly.


Reduce probabilities are computed conditioned on positions k and j, which are accessible through the dynamic program deduction rule.

the shift probabilities cannot be computed at the shift rule for deducing [j − 1, j], as it does not have access there to the top of the stack.


We observe the same pattern for the embedding space mapping approach for noise reduction against the narrow window embeddings.

combining the extension with the embedding space mapping methods, along with the LSTMbased character embeddings, results in the best performing system.


The orange points in Figure 4 show the performance of this experiment with different context sizes k. We observe that including shuffled distant words is substantially better than truncating them completely.

shuffling does cause performance to degrade relative to the base parser even when the unshuffled win-  dow is moderately large, indicating that the LSTM is propagating information that depends on the order of words in far-away positions.


More advanced methods, such as GloVe, set non-uniform weights for observed entries to reflect their confidence.

the time complexity of their algorithm is proportional to number of nonzero weights (|(i, j) | C ij = 0|), thus they have to set zero weights for all the unobserved entries (C ij = 0 for Ω − ), or try to incorporate a small set of unobserved entries by negative sampling.


As the corpus size grows, the performance of all models improves, and the PU-learning model consistently outperforms other methods in all the tasks.

with the size of the corpus increases, the difference becomes smaller.


We chose that lexicon since we have extra information available for its entries that we want to examine, namely polar intensity ( §4.1.1) and sentiment views ( §4.1.2).

since we noted that the Subjectivity Lexicon misses some prototypical abusive words (e.g.


Among the three intensity types, the most effective one is the person-based intensity (INT person ).

it can be effectively combined with the remaining types.


These approaches are motivated from an information extraction perspective, for instance in aiding tasks such as knowledge base population.

1 it has not been studied whether such sophisticated author commitment analysis can go beyond what is expressed in language and reveal the underlying social contexts in which language is exchanged.


O' Barr (1982) analyzed courtroom interactions and identified hedges and hesitations as some of the linguistic markers of "powerless" speech.

there has not been any computational work which has looked into how power relations relate to the level of commitment expressed in text.


As discussed earlier, this is expected since superiors issue more requests (as found by (Prabhakaran and Rambow, 2014)), the propositional heads of which would be tagged as NA by the belief tagger.


VRB Verbosity (e.g., message count) PST Positional (e.g., thread initiator?)

tHR thread structure (e.g., reply rate) DIA Dialog act tagging (e.g., request count) ODP Overt displays of power LEX Lexical ngrams (lemma, POS, mixed ngrams) None of the features used in POWERPREDIC-tOR use information from the parse trees of sentences in the text in order to accurately obtain the belief labels, deep dependency parse based features are critical (Prabhakaran et al., 2010).


Automatic evaluations, such as those based on word deletions, are frequently used since they enable rapid iterations and are easy to reproduce.

it is unclear to what extent they correspond with human-based evaluations.


machine Translation and Language model are globally appeared in the input document collections over time and captured in the topics by RNN-RSm and DTm.


As the only large human-annotated corpus for NLI currently available, the Stanford NLI Corpus (SNLI;Bowman et al., 2015) has enabled a good deal of progress on NLU, serving as a major benchmark for machine learning work on sentence understanding and spurring work on core representation learning techniques for NLU, such as attention (Wang and Jiang, 2016;Parikh et al., 2016), memory (Munkhdalai and Yu, 2017), and the use of parse structure (Mou et al., 2016b;Bowman et al., 2016;Chen et al., 2017).

sNLI falls short of providing a sufficient testing ground for machine learning models in two ways.


One of the mainstream approaches to this task is to exploit the lexico-syntactic paths connecting two target words, which reflect the semantic relations of word pairs.

this method requires that the considered words co-occur in a sentence.


Thus, in the neural path-based method, paths(w 1 , w 2 ) for these word pairs is padded with an empty path, like UNK-lemma/UNK-POS/UNK-dep/UNK-dir.

this process makes path-based classifiers unable to distinguish between semanticallyrelated pairs with no co-occurrences and those that have no semantic relation.


We test this property on HyperLex , a gold standard dataset for evaluating how well word representation models capture graded LE, grounded in the notions of concept (proto)typicality (Rosch, 1973;Medin et al., 1984) and category vagueness (Kamp and Partee, 1995;Hampton, 2007)  As shown by the high inter-annotator agreement on HyperLex (0.85), humans are able to consistently reason about graded LE.

10 current state-of-the-art representation architectures are far from this ceiling.


its numbers on BLESS (0.88), WBLESS (0.75), and BIBLESS (0.57) are far from the topperforming LEAR vectors (0.96, 0.92, 0.88).


WORD2GAUSS (Vilnis and McCallum, 2015) represents words as multivariate K-dimensional Gaussians rather than points in the embedding space: it is therefore naturally asymmetric and was used in LE tasks before, but its performance on Hy-perLex indicates that it cannot effectively capture the subtleties required to model graded LE.

note that the comparison is not strictly fair as WORD2GAUSS does not leverage any external knowledge.


It is therefore evident that this method is not informative in terms of the crosslingual properties of AMR.

its simplicity makes it a compelling engineering solution for parsing other languages.


Given only this sentence, or this sentence and a strict surface syntax representation that does not indicate elided predicates, this is a challenging task.

given a dependency graph that reconstructs the elided predicate for each conjunct, the problem becomes much easier and methods developed to extract information from dependency trees of clauses with canonical structures are much more likely to extract the correct information from a gapped clause.


If AMR and DG were very similar in how they represent information, such correspondences would probably hold between subgraphs consisting of a single edge, as in figure 1 cat nmod:poss →my ∼ cat poss →I.

aMR by design abstracts away from syntax and it should not be assumed that all mappings will be so clean.


11 Such an attitude reflects decades of work in the syntax-semantics interface (Partee, 2014) and the utility of dependency syntax for other forms of semantics (e.g., Oepen et al., 2014;Reddy et al., 2016;Stanovsky et al., 2016;White et al., 2016;Zhang et al., 2017;Hershcovich et al., 2017).

this assumption has not been empirically tested, and as Bender et al.


One measure of similarity between AMR and DG graphs is the configuration of the most complex subgraph alignment between them.

configuration a:b is higher than c:d if a + b > c + d. all configurations involving 0 are lower than those which do not.


In our graph-based parsing setting, we do not have a notion of parse history or partial derivations that directly connect intransitive and transitive verbs.

syntactic analogies still hold to a considerable degree in the vector representations of supertags induced by our joint models, with average rank of the correct answer nearly the same as that obtained in the transition-based parser.


Copulas A copula is usually treated as a dependent to the predicate both in our TAG grammar (adjunction) and UDR.

we found two situations where they differ from each other.


For example, it gives (be, legislation, cop) in "... on how much social legislation there should be."

our TAG grammar analyzes that "there" is attached to "be" with label 0.


Compared to KN, the 5-gram LSTM can generalize to unseen ngrams thanks to its embedding layer and recurrent connections.

it cannot discover longdistance dependency patterns that span more than five words.


Compared to many existing works that apply either metric-based or optimization-based meta-learning to image domain with low inter-task variance, we consider a more realistic setting, where tasks are diverse.

it imposes tremendous difficulties to existing state-of-the-art metric-based algorithms since a single metric is insufficient to capture complex task variations in natural language domain.


Due to such a simplified setting, almost all previous works employ a common meta-model (metric-/optimization-based) for all few-shot tasks.

this setting is far from the realistic scenarios in many real-world applications of few-shot text classification.


Moreover, these few-shot tasks are usually constructed by sampling from one huge dataset, thus all the tasks are guaranteed to be related to each other.

in real-world applications, the few-shot learning tasks could be diverse: there are different tasks with varying number of class labels and they are not guaranteed to be related to each other.


For example, in early stages where 10% or 20% of the information is available it is necessary to model very short length documents, which tend to produce sparse and low discriminative representations.

late stages require to exploit as much evidence as possible to make accurate predictions.


In this paper, we thus propose Multinomial Adversarial Networks (henceforth, MANs) for the task of multi-domain text classification.

to standard adversarial networks (Goodfellow et al., 2014), which serve as a tool for minimizing the divergence between two distributions (Nowozin et al., 2016), MANs represent a family of theoretically sound adversarial networks that, in contrast, leverage a multinomial discriminator to directly minimize the divergence among multiple probability distributions.


Denote the annotated corpus in a labeled domain d i 2 L as X i ; and (x, y) ⇠ X i is a sample drawn from the labeled data in domain d i , where x is the input and y is the task label.

for any domain d i 0 2 , denote the unlabeled corpus as U i 0 .


To make fair comparisons, the previous experiments follow the standard settings in the literature, where the widely adopted Amazon review dataset is used.


The prior art of MDTC (Wu and Huang, 2015) decomposes the text classifier into a general one and a set of domain-specific ones.

the general classifier is learned by parameter sharing and domain-specific knowledge may sneak into it.


A representation learning method should encode the review structure (e.g.

the role of the terms at first and ) in order to uncover the sentiment.


Co-training methods exploit predicted labels on the unlabeled data and select samples based on prediction confidence to augment the training.

the selection of samples in existing co-training methods is based on a predetermined policy, which ignores the sampling bias between the unlabeled and the labeled subsets, and fails to explore the data space.


Concretely, we introduce a joint formulation of a Q-learning agent and two co-training classifiers.

to previous predetermined data sampling methods of co-training, we design a Q-agent to automatically learn a data selection policy to select high-quality unlabeled examples.


As for our cases, when the two classifiers are initialized with different labeled seeding sets, they can be very unstable.

after enough iterations with the properly selected unlabeled data, the performance would be stable generally.


Usually, the more substantial labeled training datasets will lead to more stable models.

the problem is that the AGs News and DBpedia have 4 and 14 classes separately, while the Clickbait dataset only has 2 classes.


In both cases, the segmentation model is trained only in monolingual data, which may result in units that are not suitable for translation.

there have been multiple efforts to build models operating purely at the character level (Ling et al., 2015a;Yang et al., 2016;Lee et al., 2017).


While overlarge embedding sizes hurt accuracy because of overfitting issues, smaller sizes are not Test Set tst2011 tst2012 tst2013 tst2014 total RNN (Gulcehre et al., 2015) 18  preferable because of insufficient representation power.

our dense models show that with better model design, the embedding information can be well concentrated on fewer dimensions, e.g., 64.


SliceNet-Full matches our result, and SliceNet-Super outperforms by 0.58 BLEU score.

both models have 2.2x more parameters than our model.


// Encore quelques bugs/#insectes... Machine translation (MT) systems typically translate sentences independently of each other.

certain textual elements cannot be correctly translated without linguistic context, which may appear outside the current sentence.


But it is remarkable that the only failure to beat the baseline in terms of BLEU is when the algorithm is tasked with placing four random constraints (before BPE) with a beam size of 5.

dBA never has any trouble placing phrasal constraints (dashed lines).


While there is no explicit training towards the metric, BLEU, modeling in machine translation assumes that better model scores correlate with better BLEU scores.

a general repeated observation from the NMT literature is the disconnect between model score and BLEU score.


Neural machine translation (NMT) (Bahdanau et al., 2014;Sennrich et al., 2016a;Wang et al., 2017b) is now the state-of-the-art in machine translation, due to its ability to be trained end-toend on large parallel corpora and capture complex parameterized functions that generalize across a variety of syntactic and semantic phenomena.

it has also been noted that compared to alternatives such as phrase-based translation (Koehn et al., 2003), NMT has trouble with lowfrequency words or phrases (Arthur et al., 2016;Kaiser et al., 2017), and also generalizing across domains (Koehn and Knowles, 2017).


As we can see, the search engine retrieval time is negligible and the increase of NMT decoding time caused by our method is also small.

collecting translation pieces needed considerable time, although our implementation was in Python and could potentially be significantly faster in a more efficient programming language.


They use a hierarchical long-short term memory (LSTM) architecture for the discriminator.

to their approach, we apply the CNN-based discriminator for the machine translation task.


With the increasing of N , the translation performance of the model gets improved.

with N set larger than 20, we get little improvement than the model with N set as 20 and the training time exceeds our expectation.


Neural Machine Translation (NMT) with attentional encoder-decoder architectures Bahdanau et al., 2015) has revolutionised machine translation, and achieved state-of-the-art for several language pairs.

nMT is notorious for its need for large amounts of bilingual data (Koehn and Knowles, 2017) to achieve reasonable translation quality.


They report the best performance with fully sharing the encoder.

our architecture uses partial sharing on deep stacked encoder and decoder components, and the results show that it is critical for NMT improvement in MTL.


One of the reasons for their effectiveness is their ability to capture relevant source-side contextual information at each time-step prediction through an attention mechanism.

the target-side context is solely based on the sequence model which, in practice, is prone to a recency bias and lacks the ability to capture effectively nonsequential dependencies among words.


Additionally, variational NMT (Zhang et al., 2016a) introduces a latent variable to model the underlying semantics of source sentences.

to these studies, we focus instead on the contextual information on the target side.


In contrast to the first attention function, which makes use of the hidden vector s t , the second one is based only on the previous word representations, therefore, it is independent of the current prediction representation.

the normalization of this function still depends on t. To compare our approach with similar studies, we adapted two representative self-attentive networks for application to NMT.


They also demonstrated improvements over an earlier context-free baseline model (Jiampojamarn et al., 2008).

they did not evaluate on ambiguous forms, nor directly compare context-sensitive and context-free versions of their own model.


As expected, there is a strong negative correlation between the percentage of unseen words and the accuracy of Lematus 20-Ch: the rank correlation is R = −0.73 (p < 0.001; we use rank correlation because it is less sensitive to outliers than is linear correlation, and the plot clearly shows several outliers.)

to our original prediction, however, Lematus 20-Ch is actually more accurate for languages with greater ambiguity (R = 0.44, p = 0.05).


We also consider the CoNLL 2003 dataset (Tjong Kim Sang and De Meulder, 2003) as it has been used as the standard dataset for NER benchmarks.

we emphasize that both datasets present significantly different challenges and, thus, some relevant aspects in CoNLL 2003 may not be that relevant in the WNUT 2017 dataset.


For the OOV problem, we use FastText to provide vectors to 2,333 words (around 13% of the vocabulary).

the ablation experiment shows a small improvement, which suggests that those words did not substantially contribute to the meaning of the context.


Tying weights in NLM: Reusing embeddings in word-level neural language models is a technique which was used earlier (Bengio et al., 2001;Mnih and Hinton, 2007) and studied in more details recently (Inan et al., 2017;Press and Wolf, 2017).

not much work has been done on reusing parameters in subword-aware or subwordlevel language models.


2015, this can significantly reduce the total number of parameters for large models trained on huge datasets (1B tokens) with large vocabularies (800K tokens).

we do not expect significant reductions on smaller data sets (1-2M tokens) with smaller vocabularies (10-30K tokens), which we use in our main experiments.


If an embedding model has n layers, there are 2 n ways to reuse them, as each layer can either be tied or untied at input and output.

there are two particular configurations for each of the embedding models that do not interest us: (i) when neither of the layers is reused, or (ii) when only the very first embedding layer is reused.


This can be partially seen from Table 3, where the OOV words are better handled by CharCNN in terms of in-vocabulary nearest neighbors.

to fully validate the abovementioned expectation we conduct additional experiments: we train two models, CharCNN and MorphSum, on PTB and then we evaluate them on the test set of Wikitext-2 (245K words, 10K wordtypes).


one can predict the mean decrease in PPL from the TTR of a text with a simple linear regression:    The empirical perplexities in Table 2 are way above the current state-of-the-art on the same datasets (Melis et al., 2018).


In fact, the best model exhaustively generates all candidate strings, where the first part is a prefix of C 1 and the second part is a suffix of C 2 and scores them to pick the best candidate while using a backward model learned to generate the components given the blend.

we propose a more straightforward model that explicitly incorporates inherent linguistic constraints entirely obviating the need for decoding using exhaustive candidate generation yet yielding competitive performance.


In the absence of such information, just clipping to one syllable yields better performance.

when this information is exactly known (CLIPPHONE(O)), we note an improvement as expected (µ = 2.79).


We differ from all of these works in several ways.

to the blending model proposed by Gangal et al.


The most straightforward solution is to treat language as a sequence of characters (Sutskever et al., 2011).

models that operate at two levels-a character level and a word levelhave better performance (Chung et al., 2016).


To make training and inference with our model  tractable, we have assumed independence between previous adjacent events and the next word generation given the previous surface word forms ( §2.1).

thus, the posterior probability over the analysis is only determined by the left contextsubsequent decisions are independent of the process used to generate a word at time t. since disambiguating information may be present in either direction, we introduce a model variant that conditions on information in both directions.


When the model went to deeper flat NER layers, the performance dropped gradually as the number of gold entities decreased.

the performance for predictions on each flat Layer P (%) R (%) F (%) EP (%) ER (%) EF (%) #Predictions #Gold Entities  0.00 0.00 0.00 0.00 10.00 0.00 0 10 Layer 5 0.00 0.00 0.00 0.00 0.00 0.00 0 1 NER layer was different in terms of extended evaluation metrics.


Such models use simple reading mechanisms to encode the premise and hypothesis independently.

such a complex task require explicit modeling of dependency relationships between the premise and the hypothesis during the encoding and inference processes to prevent the network from the loss of relevant, contextual information.


The main idea is to represent the entities and relations in a vector space, and one can use machine learning technique to learn the continuous representation of the knowledge graph in the latent space.

even steady progress has been made in developing novel algorithms for knowledge graph embedding, there is still a common challenge in this line of research.


In a typical RL setting, each action performed by the agent will change its state, and the agent will perform a series of actions (called an epoch) until it reaches certain states or the number of actions reaches a certain limit.

in the analogy above, actions does not affect the state, and after each action we restart with another unrelated state, so each epoch consists of only one action.


Hence, it is assumed that the context of the situation is explicitly expressed in words.

language understanding involves implicit knowledge, which is not mentioned but still seems obvious to humans, e.g., 'people can sit back on a bench, but companies cannot', 'companies are in cities'.


Same as SimpleFrameId, our system is based on pretrained embeddings to build the input representation out of the predicate context and the predicate itself.

different to SimpleFrameId, our representation of the predicate context is multimodal: beyond textual embeddings we also use IMAG-INED and visual embeddings.


Regarding the setting with lexicon, the null hypothesis cannot be rejected at a significance level of α = 0.05 (p = 0.2181).

concerning accuracy scores without using the lexicon, the null hypothesis is rejected at a significance level of α = 0.05 (p < 0.0001).


This work builds on our initial study of semantic divergences (Carpuat et al., 2017), where we provide a framework for evaluating the impact of meaning mismatches in parallel segments on MT via data selection: we show that filtering out the most divergent segments in a training corpus improves translation quality.

we previously detect mismatches using a cross-lingual entailment classifier, which is based on surface features only, and requires manually annotated training examples (Negri et al., 2012.


The generation of the output sequence is conditioned on the previous words and the input.

when certain sequences are very common, the language modelling conditional probability will prevail over the input conditioning.


This is a reasonable approach, under the assumption that we are modeling a single underlying distribution in the data.

in many applications and for many natural language datasets, there exist multiple underlying distributions, characterizing a variety of language styles.


a sequence of {field, value} pairs and use a standard seq2seq model for this task.

such a model is too generic and does not exploit the specific characteristics of this task as explained below.


We observe that our final model gives the best performance -though the bifocal attention model performs poorly as compared to the basic seq2seq model on French.

the overall performance for French and German are much smaller than those for English.


For example, most English descriptions start with name followed by date of birth but this is not the case in French.

this is only a qualitative observation and it is hard to quantify this characteristic Model BLEU-4 NIST-4 ROUGE-4 (Mei et al., 2016) 10    If the proposed model indeed works well then we should see attention weights that are consistent with the stay on and never look back behavior.


some question-answer pairs are correct and some are wrong.

this kind of dataset is hard to obtain in most situations because of the lack of manual annotation efforts.


The application of Generative Adversarial Network (GAN) (Goodfellow et al., 2014; in this scenario regards every generated question-answer pair as a negative instance.

generative Domain-Adaptive Nets (gDAN)  regards every generated question-answer pair appended with special domain tag as a positive instance.


GCN (competitive) is analogous to (Goodfellow et al., 2014), where all the generated questions are regarded as negative instances (with label as zero).

gCN (collaborative) is analogous to , where the generated questions are regard as positive instances.


Our main observation from Table 1 is that simply regarding all the generated questions as negative instances ("competitive") could not bring performance boost.

regarding the generated questions as positive ones ("collaborative") improves the QA model.


Several researchers adopted this architecture for the reading comprehension (RC) style QA tasks, because it can extract contextual information from each sentence and use it in finding the answer (Xiong et al., 2016;Kumar et al., 2016).

none of this research is applied to the QA pair ranking task directly.


The word with the highest probability is then selected as the answer.

in multiple-choice QA, C is in the form of open, natural language sentences instead of a single word.


Both EntNet and QRN find a final answer by decoding the final vector(s) into a vocabulary entry via softmax classification.

many of the best performing factoid QA systems, e.g., (Seo et al., 2017a;Clark and Gardner, 2017), return an answer by finding a span of the original paragraph using attention-based span prediction, a method suitable when there is a large vocabulary.


These words are closely related to the category of dataset "Office Product", which implies both models can get a good interpretation of user reviews.

when we carefully compare the two tables, there exists some differences.


The hidden layers and word embeddings are opaque and difficult from which to draw conclusions.

a generative model that represents words explicitly as probability distributions allows for easier post-analysis.


We empirically set the cutoff based on dev set performance to optimize F1.

a detection system may desire to optimize recall at the expense of precision, thus choosing a lower λ and forcing the system to predict attacks more often.


This seems particularly appropriate when dealing with uncooperative annotators, as might be encountered, for example, in crowdsourcing (Snow et al., 2008;Zhang et al., 2016).

with a team of trained annotators, we believe that honest disagreements could contain valuable information better not ignored.


The aspects which correlate most strongly with the final recommendation are substance (which concerns the amount of work rather than its quality) and clarity.

soundness/correctness and originality are least correlated with the final recommendation.


26 Surprisingly, however, most of the reviews are not made publicly available.

we collected and organized PeerRead such that it is easy for other researchers to use it for research purposes, replicate experiments and make a fair comparison to previous results.


We focus on the task of abstractive summarization of a long document.

to extractive summarization, where a summary is composed of a subset of sentences or words lifted from the input text as is, abstractive summarization requires the generative ability to rephrase and restructure sentences to compose a coherent and concise summary.


Although it was able to capture the statistics of the player correctly (e.g., 15 penalties, 16 attempts), it still missed the player who scored the only goal in the game (i.e., kevin mirallas).

multi-agent model was able to generate a concise summary with several key facts.


On the other hand multi-agent model was able to generate a concise summary with several key facts.

similar to single agent model, it missed to capture the player who scored the only goal in the game.


As a result, encoding word orders, as is done by the encoders except AvgEmb, might bring noise to keyphrase extraction on Weibo dataset.

avgEmb is the worst encoder on Twitter dataset, as word order is crucial in English.


The generated texts are merely lists of bi-grams and not meaningful sentences which cannot be considered to be summaries.

they achieve superhuman ROUGE recall scores.


Extractive summarization uses sentence-level features (Yang et al., 2017) that have been leveraged for producing query-focused or topic-based summaries.

for RNN-based frameworks, such tuned summary generation is non-obvious due to the absence of explicit content based features.


Our contribution is a modified encoder to encode the article in a topic-sensitive manner.

for the sake of completeness, we shall provide an overview of the entire network, and will reuse notations from their work to a large extent.


Alternate to the the proposed architecture, one can append the topic vector to the initial hidden state of the encoder, or the initial state of the decoder.

in our experiments, these approaches did not produce the desired tuning.


Our last baseline is a topic-signature based approach which also works by extracting sentences from the article which are aligned to the target topic.

the selection of sentences is based on topic signatures as described by Lin and Hovy (2000) and Conroy et al.


Both methods outperformed Seq2Seq without syntactic features in terms of translation quality.

both methods fail to provide an entire parse tree until the decoding phase is finished.


(2017) consider higherorder dependency relationships in Seq2Seq by incorporating a graph convolution technique (Kipf and Welling, 2016) into the encoder.

the dependency information of the graph convolution technique is still given in pipeline manner.


This approach is known to have three advantages: its applicability to many useful submodular objective functions, the efficiency of the greedy algorithm, and the provable performance guarantee.

when it comes to compressive summarization, we are currently missing a counterpart of the extractive method based on submodularity.


Thanks to the submodularity of their objective function, their method enjoys a 1 2 (1−e −1 )-approximation guarantee.

because of the costly DP procedure, their method is less scalable than the standard greedy methods such as the extractive method (Lin and Bilmes, 2010) and ours.


Relation statements, which are strings intended for human readers, are similar to the 3-tuples, "relations", from prior work on information extraction (Banko et al., 2007).

in this work, we show that the assumptions underlying the extraction of 3-tuples for machines ( §3) leads to poor performance in summarizing mention sets for people ( §5).


Our compression-based method achieves higher yield than off-the-shelf relation extractors.

because all sentences in a mention set include (t 1 ) and (t 2 ), it is always possible to generate a very large candidate set by simply extracting all spans between (t 1 ) and (t 2 ) from the mention set, regardless if such relation statements are coherent.


A good summary for this mention set should describe a central event from this time period: when General Cedras fled to the Dominican Republic.

jean-Bertrand Aristide -United States are mentioned together in 67 months in the corpus, covering a number of important events spread across decades (figure 3c).


Then, the recall is defined as and measures how much of the desired content was returned by the system.

precision is defined as and measures how much of the returned content was actually desirable.


A possible explanation is that there are short sentences in the input documents which are considerably redundant to other high precision sentences.

overall the trend in the results (increasing evaluation scores with increasing α, which means increasing impact of ROUGE precision) substantiate the general hypothesis of this paper, namely that sentence selection measures should target precision instead of recall.


The word drop probability is normally set to 0.25, since using a higher probability may degrade the model performance (Bowman et al., 2016).

we observe that these tricks do not solve the degeneracy for the VHRED in conversation modeling.


As z conv is independent of the conditional structure, it does not suffer from the data sparsity problem.

the expressive power of hierarchical RNN decoders makes the model still prone to ignore latent variables z conv and z utt t .


This indicates that the text analyzed by both models encodes some information about egregiousness.

for the recall and hence the F1-score, the EGR model relatively improved the text-based model by 41% and 18%, respectively.


In addition, many enterprises have started to use conversational chat platforms such as Slack 2 to enhance team collaboration.

multiple conversations may 1 Facebook Messenger: https://www.messenger.


Messages in the same conversation may have higher similarity scores (Shen et al., 2006;Mayfield et al., 2012) or similar context messages (Wang and Oard, 2009).

similarity thresholds for determining new topics vary depending on context.


Unsupervised approaches (Wang and Oard, 2009) estimate the relationship between messages through unsupervised similarity functions such cosine similarity, and assign messages to conversations based on a predefined threshold.

supervised methods exploit a set of user annotations (Elsner and Charniak, 2008;Mayfield et al., 2012;Shen et al., 2006;Du et al., 2017;Mehri and Carenini, 2017) to adapt to different datasets.


Many studies also focus on similar tasks aside from conversation disentanglement, such as entailment prediction (Mueller and Thyagarajan, 2016;Wang and Jiang, 2017) and question-answering (Severyn and Moschitti, 2015;Amiri et al., 2016;Yin et al., 2016).

most of their models are complicated and require a larger amount of labeled training data; limited conversational data can lead to unsatisfactory performance as shown in Section 4.


In order to model the relation between the query entity pair, we assume that there exists an underlying latent variable (paths connecting two nodes) in the KG, which carries the equivalent semantics of their relations.

due to the intractability of connections in large KGs, we propose to use variation inference to maximize the evidence lower bound.


Large-scaled knowledge graph supports a lot of downstream natural language processing tasks like question answering, response generation, etc.

there are large amount of important facts missing in existing KG, which has significantly limited the capability of KG's application.


Since our model only deals with the relation classification problem (e s , ?, e d ) with e d as input, so it's hard for us to directly compare with MINERVA (Das et al., 2018).

here we compare with chain-RNN (Das et al., 2016) and CNN Path-Reasoner model, the results are demonstrated as Table 4.


A TLINK denotes a temporal relation between mentions, i.e., events, time expressions and document creation time (DCT) (Setzer, 2002).

annotating TLINKs is a painful work, because annotation candidates are quadratic to the number of mentions in a document.


For solving this, many dense annotation schemata are proposed to force annotators to annotate more or even complete graph pairs.

dense annotation is time-consuming and unstable human judgments on "salient" pairs are not improved at all.


Theoretically, a TORDER between two mentions with any distance in a document can be automatically computed.

it is important to make the new data in a comparable manner to the existing data.


"), and exchanging them is syntactically valid.

this naive swapping of attribute markers can result in ungrammatical outputs.


Instead, we train DELETEONLY to reconstruct the sentences in the training corpus given their content and original attribute value by maximizing: (3) For DELETEANDRETRIEVE, we could similarly learn an auto-encoder that reconstructs x from c(x, v src ) and a(x, v src ).

this results in a trivial solution: because a(x, v src ) and c(x, v src ) were known to come from the same sentence, the model merely learns to stitch the two sequences together without any smoothing.


In contrast, TEMPLATE-BASED is good at preserving the content because the content words are guaranteed to be kept.

it makes grammatical mistakes due to the unsmoothed combination of content and attribute words.


Manual inspection shows that on AMAZON, some product genres are associated with either mostly positive or mostly negative reviews.

our systems produce, for example, negative reviews about products that are mostly discussed positively in the training set.


Many of the examples exhibit complex transformations while preserving both the input semantics and grammaticality, even when the target syntax is very different from that of the source (e.g., when converting a declarative to question).

the failure cases demonstrate that not every template results in a valid paraphrase, as nonsensical outputs are sometimes generated when trying to squeeze the input semantics into an unsuitable target form.


Building a TSA model that can automatically determine the sentiment of a tweet has received significant attention over the past several years.

since most state-of-the-art TSA models use machine learning to tune their parameters, their performance -and relevance to a real-world implementation setting -is highly dependent on the dataset on which they are trained.


Higher coverage may be needed still to identify and understand these annotator disagreements.

the differences between these two subsets would be masked using the 3x coverage commonly found in other datasets.


Another related direction is to train on disparate annotations of the same task Peng et al., 2017).

the different nature of our tasks requires a modelling of their label spaces.


Sentence-level and text-level architectures are either adapted to sequential input data (typical for RNN, LSTM, GRNN and related architectures) or spatially arranged input data (as with CNN architectures).

for word embeddings (the default input for word emotion induction) there does not seem to be any meaningful order of their components.


the COMMON model was trained on much more data than the embeddings Sedoc et al.


All of this previous work only identifies affective events and their polarities.

our work aims to identify the reason for the affective polarity of an event.


Goals could be very specific to a character in a particular narrative story.

but many types of goals originate from universal needs and desires shared by most people (Max-Neef et al., 1991).


Our gold standard data set is relatively small, so supervised learning that relies entirely on manually labeled data may not have sufficient coverage to perform well across the human need categories.

the AffectEvent dataset contains a very large set of events that were extracted from the same blog corpus, but not manually labeled with affective polarity.


For example, the words "abandon" and "damage" belong to the Affect category (corresponding to our Emotion category) in LIWC.

based on our definition the event "my house was damaged" actually belongs to the Finance category.


A solution to this task will represent a substantial step towards automatic warrant reconstruction.

we present experiments with several neural attention and language models which reveal that current approaches based on the words and phrases in arguments and warrants do not suffice to solve the task.


Some argue that the distinction of warrants from premises is clear only in Toulmin's examples but fails in practice, i.e., it is hard to tell whether the reason of a given argument is a premise or a warrant (van Eemeren et al., 1987, p. 205).

freeman (2011) provides alternative views on modeling an argument.


(2017) also experimented with reconstructing implicit knowledge in short German argumentative essays.

to our work, they used expert annotators who iteratively converged to a single proposition.


In an analysis of this dataset Sugawara and Aizawa (2016) found, though, that only 6.2% of the questions require causal reasoning, 1.2% logical reasoning, and 0% analogy.

these reasoning types often make up the core of argumentation (Walton, 2007a).


Since their logical formalism builds upon an enhanced version of Aristotle's syllogisms, its applicability to natural language argumentation remains limited (see our discussion above).

to our data source, a few synthetic datasets for general natural language reasoning have been recently introduced, such as answers to questions over a described physical world (Weston et al., 2016) or an evaluation set of 100 questions in the Winograd Schema Challenge (Levesque et al., 2012).


As a result, trying to give a plausible reasoning for the opposite claim ¬C either leads to nonsense or to a proposition that resembles a rebuttal rather than a warrant (Toulmin, 1958).

if both W and AW are available, they usually capture the core of a reason's relevance and reveal the implicit presuppositions (examples follow further below).


We perform further analysis to see the effect of joint-structured model on the sentence-level task under sparsely-labeled conditions in Section 5.4.

for the document-level task,  the joint model (Joint) performs better than Joint doc and all the baseline approaches.


Our results show that the models are robust mainly where the semantics of the new instances do not change significantly with respect to the sampled instances and thus the class labels remain unaltered; i.e., the models are insensitive to our transformation to input data.

when the class labels change, the models significantly drop accuracy.


Furthermore, recent works on adversarial perturbation have tackled this problem (Goodfellow et al., 2015;Feinman et al., 2017).

most previous approaches require either annotations generated by each individual annotator (Guan et al., 2017), or both task-specific and instance-type (genuine or adversarial) labels for training (Hendrik Metzen et al., 2017;Zheng et al., 2016), or noise-free data (Xiao et al., 2015).


The two frameworks, however, differ in their views on the theory of learning as we describe below: Curriculum learning is inspired by the learning principle that humans can learn more effectively when training starts with easier concepts and gradually proceeds with more difficult ones (Bengio et al., 2009).

leitner system is inspired by spaced repetition (Dempster, 1989;Cepeda et al., 2006), the learning principle that effective and efficient learning can be achieved by working more on difficult concepts and less on easier ones.


Annotators are instructed to label a post as "relevant" if it describes a patient's experience (including sign and symptoms, treatments, etc.,) with respect to the cancer.

"irrelevant" posts are defined as generic texts (such as scientific papers, news, etc.,) that discuss cancer in general without describing a real patient experience.


This may decrease the training performance of Lit, especially with greater amount of noise in data.

this training strategy increases the spotting performance of Lit as spurious instances seem to occur in lower queues of Leitner more frequently, see Figure 4.


By contrast, without in-cremental commitment to each structure, competing material slows down the process, because it would lead to combinatory explosion.


A commonly implied assumption is that lemma retrievals shouldn't interfere with phonological processes (e.g., Schriefers et al., 1990), though it is difficult to know if a speech slowdown is due to a phonological or semantic interference due to our experimental setup.

since in our experiment, effects are still observed at large distances from the target words, either phonological forms can be retrieved in a non-incremental way (possibly even before lemmas for other words are retrieved), or the retrieval of the lemma does interfere with phonological encoding in some way; for instance, by activating related phonological forms.


It is relatively smaller than the other corpora (Section 2).

it is the largest, if not the only, corpus for the evaluation of passage completion on multiparty dialog that still gives enough instances to develop meaningful models using deep learning.


It should be noted that character anonymization process makes it harder to for people to find the answer.

it also possible that some participants of the evaluation may enter the answer randomly (i.e the results may not truly reflect human performance).


Although these cues are usually parts of sentences in long utterances, since Bi-LSTM is based on only words, it still is able to locate them correctly.

our model encodes each utterance and then feeds encoded vectors to LSTMs, so the high level representation of the cues are mixed with other information, which hinders the model's ability to find the exact string matches.


We then experimented this model on our original length dataset.

even after an extensive hyperparameter turning on the development set, this model did not achieve results comparable to those of either Bi-LSTM or our models, so we did not make a further analysis on this model.


These ranking-based retrieval strategies have been well-applied as an important approach to dialogue systems, yet the set of scripted responses are limited and are short at generalization.

statistical machine translation (SMT) systems have been applied to dialogue systems (Ritter et al., 2011), taking user's query as a source language sentence and the chatbot's response as a target language sentence.


The domain adaptation from document comprehension to dialog generation is feasible by taking the rich context of the speakers as a "document", current user's query as a "question" and the chatbot's response as an "answer".

there are still several major challenges for this domain adaptation.


Existing evaluation metrics of dialog agents measure the quality of the generated sentences only by referring to the existing responses, which obeys the same principle with NMT models' metrics.

one essential difference between NRG and NMT lies in the fact that, a large group of responses can be considered as relevant to a given query in conversations, while the number of references to a translation result is quite limited for NMT models.


From this benchmarking table, it can be observed that the attention mechanism is helpful for decoders to improve the relevance of the generated responses, since the Attention-Seq2Seq performs better than the basic Seq2Seq on the dataset, in terms of all the three metrics.

the relative gain of the attention layer is limited, indicating that modeling relation of query and response by attention module is not able to directly solve the learning paradigm of conversations.


Finally, similar to our work, memory networks (Prakash et al., 2017) have recently been used for diagnosis coding.

we would like to note two significant differences between the memory network from Prakash et al.


We use a CNN to encode each document following what is now a fairly standard approach consisting of an embedding layer, a convolution layer, a max-pooling layer, and an output layer (Collobert et al., 2011;Kim, 2014).

in our architecture, the CNN additionally aids in getting interme-diate representations for the multi-head matching network component (Section 3.2).


For popular concepts, lists of relevant concept stocks can be found from analyst reports from financial websites.


Considering the previous N-1 words, N-gram language models predict the next word.

this leads to the loss of longterm dependencies.


Many binarization methods have been proposed (Courbariaux et al., 2015Rastegari et al., 2016;Xiang et al., 2017).

only a few (Hou et al., 2016;Edel and Köppe, 2016) are related to recurrent neural network.


(Hou et al., 2016) implements a character level binarized language model with a vocabulary size of 87.

they did not do a comprehensive study on binarized large vocabulary LSTM language models.


At run-time, the input embedding and the output embedding are binarized matrices.

at train-time, float versions of the embeddings, which are used for calculating the binarized version of embeddings, are still maintained.


Batch normalization is hard to apply to a recurrent neural network, due to the dependency over entire sequences.

the structure of the batch normalization is quite useful.


The BELM model outperforms the baseline model on the MEN task, although it doesnt perform as well as the baseline model on the WS-353 task.

the MEN dataset contains many more word pairs, which makes the results on this dataset more convincing.


Models with hops in the range of 3 − 10 outperform the single layer variant.

each added hop contributes a new set of parameters for memory representation, leading to an increase in total parameters of the model and making it susceptible to overfitting.


In order to avoid error propagation and effectively utilize contexts, prior work leveraged history for contextual SLU.

most previous models only paid attention to the related content in history utterances, ignoring their temporal information.


Previous work proposed an end-to-end time-aware attention network to leverage both contextual and temporal information for spoken language understanding and achieved the significant improvement, showing that the temporal attention can guide the attention effectively .

the time-aware attention function is an inflexible hand-crafted setting, which is a fixed function of time for assessing the attention.


The assumption underlying these uses is that reading rate is a property of the reader (or con-trolled by the reader).

variation in reading rate across different passages for the same readers has also been reported (Foulke, 1968;Tauroza and Allison, 1990;Ardoin et al., 2005;Compton et al., 2004;Beigman Klebanov et al., 2017).


In general, the usage of high-specificity color words increases in more difficult conditions, as expected.

we see that Chinese speakers use them significantly less than English speakers.


Perplexity is a common intrinsic evaluation metric for generation models.

3 for comparing monolingual and bilingual models, we found perplexity to be unhelpful, owing largely to its vocabulary-dependent definition.


Specifically, if we fix the vocabulary in advance to include tokens from both languages, then the monolingual model performs unreasonably poorly, and bilingual training helps immensely.

this is an unfair comparison: the monolingual model's high perplexity is dominated by low probabilities assigned to rare tokens in the opposite-language data that it did not see.


Intuitively, we expect adding more training data on the same task will improve the model, regardless of language.

we find that the effect of dataset size is not so straightforward.


Preliminary experiments involving augmentation of the data by duplicating and deleting constituents show no gains, suggesting that the improvement depends on certain kinds of regularities in the English data that are not provided by artificial manipulations.

more investigation is needed to thoroughly assess the role of general-purpose regularization in our observations.


Adding a hard constraint on the output vocabulary would make this equivalent to a simple form of multitask learning (Caruana, 1997;Collobert and Weston, 2008).

allowing the model to use tokens from either language at any time is simpler and results in better modeling of mixedlanguage data, which is more common in non-English environments.


This is the first approach we are aware of to frame a general learning problem as optimization over a space of natural language strings.

many closely related ideas have been explored in the literature.


The advantage is that no firm decisions need to be made about object categories.

such approaches are hard to interpret and are dataset dependent (Vinyals et al., 2017).


There is also recent work applying attentionbased models (Xu et al., 2015) on explicit object proposals (Anderson et al., 2018;Li et al., 2017), which may capture object-level information from the attention mechanism.

attention-based models require object information in the form of vectors, whereas our models use information of objects as categorical variables which allow for easy manipulation but are not compatible with the standard attention-based models.


just because there is only one person in the image does not mean that it is less important than the ten cars depicted.

object size correlates with object importance in IC, i.e.


We have also tried computing the correlation with f (t c ) (frequency of the category being mentioned regardless of whether or not it is depicted).

we found the word matching process too noisy as it is not constrained or grounded on the image (e.g.


Of course, the categories are not mutually exclusive and object co-occurrence may also play a role.


Previous work suggests that incorporating visual features for less concrete concepts can be harmful in word similarity tasks (Hill and Korhonen, 2014b;Kiela and Bottou, 2014;.

it is less clear if this intuition applies to more practical tasks (e.g., retrieval), or if this problem can be overcome simply by applying the "right" machine learning algorithm.


Other work removed the dependency on a true character list by determining all names through coreference resolution.

this work also depended on the availability of scripts (Ramanathan et al., 2014).


The second person reference represents a multi-instance constraint that suggests that the mentioned name is one of the characters that are present in the scene, which increases the probability of this character to be one of the speakers of the surrounding segments.

the third person reference represents a negative constraint, as it suggests that the speaker does not exist in the scene, which lowers the character probability of the character being one of the speakers of the next or the previous subtitle segments.


One recent ensembling approach to VQA (Fukui et al., 2016) combined multiple models that use multimodal compact bilinear pooling with attention and achieved state-of-the-art accuracy on the VQA 2016 challenge.

their ensemble uses simple softmax averaging to combine outputs from multiple systems.


These advantages have motivated recent work on explainable AI systems, particularly in computer vision (Antol et al., 2015;Goyal et al., 2016;.

there has been no prior work on using explanations for ensembling multiple models or improving performance on a challenging task.


Evaluation on either split requires submitting the output to the competition's online server.

1 there are fewer restrictions on the number of submissions that can be made to the test-dev compared to the test-standard.


Pre-trained word representations (Mikolov et al., 2013;Pennington et al., 2014) are a key component in many neural language understanding models.

learning high quality representations can be challenging.


1 Due to their ability to capture syntactic and semantic information of words from large scale unlabeled text, pretrained word vectors (Turian et al., 2010;Mikolov et al., 2013;Pennington et al., 2014) are a standard component of most state-ofthe-art NLP architectures, including for question answering , textual entailment (Chen et al., 2017) and semantic role labeling .

these approaches for learning word vectors only allow a single contextindependent representation for each word.


(2017) pretrain encoder-decoder pairs using language models and sequence autoencoders and then fine tune with task specific supervision.

after pretraining the biLM with unlabeled data, we fix the weights and add additional taskspecific model capacity, allowing us to leverage large, rich and universal biLM representations for cases where downstream training data size dictates a smaller supervised model.


As a result, the biLM provides three layers of representations for each input token, including those outside the training set due to the purely character input.

traditional word embedding methods only provide one layer of representation for tokens in a fixed vocabulary.


They are spread across several parts of speech (e.g., "played", "playing" as verbs, and "player", "game" as nouns) but concentrated in the sportsrelated senses of "play".

the bottom two rows show nearest neighbor sentences from the SemCor dataset (see below) using the biLM's context representation of "play" in the source sentence.


Similar to WSD, the biLM representations are competitive with carefully tuned, task specific biLSTMs (Ling et al., 2015;Ma and Hovy, 2016).

unlike WSD, accuracies using the first biLM layer are higher than the top layer, consistent with results from deep biL-STMs in multi-task training (Søgaard and Goldberg, 2016;Hashimoto et al., 2017) and MT (Belinkov et al., 2017).


2 Zettlemoyer and Collins (2009) addressed the problem with lambda calculus, using a semantic parser trained separately with context-independent data.

we generate executable formal queries and require only interaction query annotations for training.


At turn i, this model has access to the complete prefix of the interactionĪ[: i − 1] and the current requestx i .

the concatenationbased encoder (Section 4.2) has access only to information from the previous h utterances.


Formally, we modify Equation 1 to encodex i : where h I i−1 is the discourse state following utter- E is modified analogously.

to the concatenation-based model, the recurrence processes a single utterance.


The position embedding is also used to compute the context vector c k : The discourse state and attention over previous utterances allow the model to consider the interaction history when generating queries.

we observe that context-dependent reasoning often requires generating sequences that were generated in previous turns.


Using the types only, while ignoring the indices, avoids learning biases that arise from the arbitrary ordering of the tokens in the training data.

it does not allow distinguishing between entries with the same type for generation decisions; for example, the common case where multiple cities are mentioned in an interaction.


The relatively high performance of FULL-0 shows that substituting segment copying with attention maintains and even improves the system effectiveness.

the best performance is provided with FULL, which combines both.


In general, the model resolves references well.

it fails to recover constraints mentioned in the past following a user focus state change (Grosz and Sidner, 1986).


We used a subset of 50 randomly selected text segments from the test set described in §4.

for the human evaluation, we only used the final 60 words 6 of the story segments to keep the amount of reading and context manageable for Turkers.


When asked to explain why they selected the sentence they did, a few Turkers attributed their choices to connections between pronouns in EN-GEN's suggestions to characters mentioned in the story excerpt.

a more frequent occurrence was Turkers citing a mismatch in entities as their reason for rejecting an option.


To this end, a specific architecture with 886 hidden units can simulate any Turing machine in real-time (i.e., each Turing machine step is simulated in a single time step).

their RNN encodes the whole input in its internal state, performs the actual computation of the Turing machine when reading the terminating token, and then encodes the output (provided an output is produced) in a particular hidden unit.


Not all probabilistic context-free grammars are consistent; necessary and sufficient conditions for consistency are given by Booth and Thompson (1973).

probabilistic context-free grammars obtained by training on a finite corpus using popular methods (such as expectation-maximization) are guaranteed to be consistent (Nederhof and Satta, 2006).


Our results show the non-existence of (efficient) algorithms for interesting problems that researchers using RNN in natural language processing tasks may have hoped to find.

the non-existence of such efficient or exact algorithms gives evidence for the necessity of approximation, greedy or heuristic algorithms to solve those problems in practice.


10 This demonstrates that given sufficient alternative signal, systems often do ignore gender biased cues.

winoBias provides an analysis of system bias in an adversarial setup, showing, when examples are challenging, systems are likely to make gender biased predictions.


Prior work (Carstens et al., 2014;Rajendran et al., 2016a) in identifying arguments in online reviews has considered sentence-level statements to be arguments based on abstract argumentation models.

to extract arguments at a finer level based on the idea of structured arguments is a harder task, requiring us to manually annotate argument components such that they can be used by supervised learning techniques.


In our earlier work (Rajendran et al., 2016b), we propose an approach for reconstructing structures similar to enthymemes in opinions that are present in online reviews.

the annotated dataset used in our approach was small and not useful for deep learning models.


The accuracy of the LSTM model in predicting the labels of annotated opinions improves with the size of the automatically labelled dataset.

the accuracy of the reserved method decreases in performance after 20 iterations 2 .


This supports our conjecture that the GRU layer has difficulty learning the kind of coreference-based reasoning required in this dataset, and that the bias towards coreferent recency helps with that.

perhaps surprisingly, given enough data both models perform comparably.


Many simple NLG models are based on recurrent neural networks (RNN) and sequence-to-sequence (seq2seq) model, which basically contains a encoder-decoder structure; these NLG models generate sentences from scratch by jointly optimizing sentence planning and surface realization using a simple cross entropy loss training criterion.

the simple encoderdecoder architecture usually suffers from generating complex and long sentences, because the decoder has to learn all grammar and diction knowledge.


These systems are well-suited to unconstrained translation, as often the phrase table entries are good translations of source phrases.

when rhythm and rhyme constraints are applied to PBMT, translation options become extremely limited, to the extent that it is often impossible to generate any translation that obeys the poetic constraints (Greene et al., 2010).


To illustrate the annotation, consider an excerpt from a response to the prompt "It is better to have broad knowledge of many academic subjects than to specialize in one specific subject"; metaphors are italicized: I ultimately agree with the fact that it is better to be specialized on a specific subject than to spread energy on different subjects.

i say ultimately, because being and staying focused on one subject means always to discard other subjects.


In some cases, the difference could be attributed to data being indomain; the sets marked with a plus in Table 4 are taken from the same testing programs as the training data for Arg, although the specific prompts are different.

arg does better across the board, including data completely unrelated to the annotation campaign.


A recent survey outlined eight categories of features used in hate speech detection (Schmidt and Wiegand, 2017): simple surface (Warner and Hirschberg, 2012;Waseem and Hovy, 2016), word generalization (Warner and Figure 1: Our hate speech classifier.

to existing methods that focus on a single target Tweet as input (center), we incorporate intra-user (right) and interuser (left) representations to enhance performance.


The former requires the user history to be labeled instances.

labeling user history requires significant human effort.


This is shown in the middle branch in Figure 1.

this method is likely to fail when the target tweet is noisy or the critical words for making predictions are out of vocabulary.


This inference problem is solved using gradient descent.

the energy surface is non-convex, which prevents gradient descent inference from finding the exact structure y min that globally minimizes the energy function.


In general, if SPEN ranks every pair of output structures identical to the score function, the optimum points of the score function match those of SPEN.

forcing the ranking constraint for every pair of output structures is not tractable, so we need to approximate it by sampling some candidate pairs.


For ∆LATER this correctly leads to a 0-prediction.

cOMPARE predicts strong change, because due to polysemy there is a high probability to sample distantly related use pairs in the cOM-PARE group.


To improve original word embedding models, there are various studies leveraging external knowledge to update word embeddings with post processing (Faruqui et al., 2015;Kiela et al., 2015; or supervised objectives (Yu and Dredze, 2014;Nguyen et al., 2016).

these approaches are limited by reliable semantic resources, which are hard to obtain or annotate.


Example 11 shows the advantage of combining the vector representation with orthographic distance, i.e., our model could find translations of sleddogs that have similar meaning, while in examples 12 and 13 orthographic distance helped to pick the correct translation which is the closest in terms of edit distance.

in example 14 orthographic distance caused an error because the incorrect prediction is too close to the source word in orthographic distance.


While antonymy represents words which are strongly associated but highly dissimilar to each other, synonymy refers to words that are highly similar in meaning.

antonyms and synonyms often occur in similar context, as they are interchangeable in their substitution.


These representations, when used as model inputs, have been shown to lead to faster learning and better results in a wide variety of settings (Erhan et al., 2009(Erhan et al., , 2010Cases et al., 2017).

many domains require more specialized representations but lack sufficient data to train them from scratch.


As a result, existing implementations of GloVe use an inner loop to compute this cost and associated derivatives.

since f (0) = 0, the second bracket is irrelevant whenever X ij = 0, and so replacing log X ij with  (for any k) does not affect the objective and reveals that the cost function can be readily vectorized as where M = W W + b1 + 1b − g(X).


Van de Cruys et al., 2013; Dima and Hinrichs, 2015).

dima (2016) recently showed that similar performance is achieved by representing the NC as a concatenation of its constituent embeddings, and argued that it stems from memorizing prototypical words for each relation.


These vectors were used to classify new NCs based on the nearest neighbor in the VSM.

the model was only tested on a small dataset and performed similarly to previous methods.


For the two intermediate clinical groups, aMCD and mMCD, the use of local average information from a small window including only the previous word (ψ 1 ) also produces good results.

there is no consensus regarding the source of switch identification, as for aMCD both semantic similarity and association strength were effective, and for mMCD it was semantic relatedness that provided a better characterization of the groups.


5 This also works for a euclidean distance similarity metric.

ing in the sum in (5) also appear in (6): the word analogy assumption is now most certainly broken: suppose that π permutes only two vectors, a 2 and a 3 and leaves all other vectors as is: There are two conventional practices in evaluating word embeddings that we aim to show are problematic: normalisation and the exclusion of premise vectors in prediction.


Their models are largely based on memory networks , originally developed for reasoning-focused machine reading comprehension tasks.

to memory networks, where each input sentence/word occupies a memory slot and is then accessed via attention independently, recent advances in machine reading suggest that processing inputs sequentially is beneficial to overall performance (Seo et al., 2017;Henaff et al., 2017).


Systems that rely on neural machine translation (Yuan and Briscoe, 2016;Xie et al., 2016;Schmaltz et al., 2017;Ji et al., 2017) are not yet able to achieve as high performance as SMT systems according to automatic evaluation metrics (see Table 1 for comparison on the CoNLL-2014 test set).

it has been shown that the neural approach can produce more fluent output, which might be desirable by human evaluators (Napoles et al., 2017).


As opposed to pipelining, rescoring improves precision at the expense of recall and is more effective for the CoNLL data resulting in up to 54.  produces similar results as rescoring with the NMT ensemble.

the best result for rescoring is lower than for pipelining on that test set.


The main purpose of our analysis is to look for structure in an array of entrainment measures.

we first check whether similarity is significantly greater for partners than non-partners for our lexical measures since PPL and KLD have not previously been used for lexical entrainment and Nenkova et al.


The automatic placement of new concepts in a taxonomy has also been investigated as a shared task in SemEval 2016 (Jurgens and Pilehvar, 2016).

to the best of our knowledge, there is no work that applies generative paraphrasing to expand a taxonomy.


A smaller increase in coverage can be achieved if we consider noisy paraphrases as well, 8.1% for Moses and 3.2% for seq2seq.

these additional sentences may contain significantly more noise.


We can see that in our dataset nouns are indeed preferred by our non-expert annotators.

when looking at a smaller amount of annotations, the number of annotated verbs and adjectives increases.


These KBs are useful resources in many applications such as semantic searching and ranking (Kasneci et al., 2008;Schuhmacher and Ponzetto, 2014;Xiong et al., 2017), question answering (Zhang et al., 2016;Hao et al., 2017) and machine reading (Yang and Mitchell, 2017).

the KBs are still incomplete, i.e., missing a lot of valid triples (Socher et al., 2013;West et al., 2014).


Results in Table 2 show that both MT and MTDS can significantly improve NOM detection over the baseline, and adaptive data selection in MTDS further improves over the MT model.

there is no gain at all for NAM detection for both languages.


For NLP, bootstrapping is a popular approach to semi-supervised learning due its relative simplicity coupled with reasonable performance (Abney, 2007).

a crucial limitation of bootstrapping, which is typically iterative, is that, as learning advances, the task often drifts semantically into a related but different space, e.g., from learning women names into learning flower names (McIntosh, 2010;Yangarber, 2003).


Recently, many existing systems on structured prediction focus on increasing the level of structural dependencies within the model.

the theoretical and experimental study of Sun (2014a) suggests that complex structures are tend to increase the overfitting risk, and can potentially be harm- At that time, I [Person] came to West Dongting with my parents' [Person] dream of building a state farm.


Many tasks cannot achieve a satisfying result on Chinese literature text compared to other corpus.

understanding Chinese literature text is of great importance to Chinese literature research.


The punctuation is a natural break point of the sentence, which makes subtrees more like the traditional dependency trees in the aspect of integrity.

the original dependency trees cannot be sufficiently regularized.


AutoSlog takes a fundamentally syntaxdriven approach to identifying patterns, which suggests the discovered patterns (and associated performance boost) is due to exploiting syntax.

the performance gains could also be due to additional contextual information that bigrams and larger n-grams provide over unigrams alone, rather than their syntactic properties.


Compiling, updating and translating them has traditionally been left mostly to domain experts and professional lexicographers.

the last two decades have witnessed a growing interest in automating the construction of lexicographic resources.


In the early days of DE, rulebased approaches leveraged linguistic cues observed in definitional data (Rebeyrolle and Tanguy, 2000;Klavans and Muresan, 2001;Malaisé et al., 2004;Saggion and Gaizauskas, 2004;Storrer and Wellinghoff, 2006).

in order to deal with problems like language dependence and domain specificity, machine learning was incorporated in more recent contributions (Del Gaudio et al., 2013), which focused on encoding informative lexico-syntactic patterns in feature vectors (Cui et al., 2005;Fahmi and Bouma, 2006;Westerhout and Monachesi, 2007;Borg et al., 2009), both in supervised and semi-supervised settings (Reiplinger et al., 2012;.


Examples where syntactic cues are leveraged include medical acronym expansion (Pustejovsky et al., 2001), hyponym-hypernym extraction and detection (Hearst, 1992;Shwartz et al., 2016), and definition extraction either from the web (Saggion and Gaizauskas, 2004), scholarly articles (Reiplinger et al., 2012), and more recently from Wikipedia-like definitions (Boella et al., 2014).

the interplay between syntactic information and the generalization potential of neural networks remains unexplored in definition modeling, although intuitively it seems reasonable to assume that a syntax-informed architecture should have more tools at its disposal for discriminating between definitional and non-definitional knowledge.


Among our proposed systems, the overall best performance in Wikipedia definitions is obtained by the CNN l configuration.

incorporating a BLSTM layer contributes towards the best performing model on the NLP-specific WCL W00 dataset (C-BLSTM100 d ).


Thus, the loss calculation technique of the Covington dynamic oracle is not directly applicable to the 2-Planar parser.

if we statically choose a canonical plane assignment and we calculate loss with respect to that assignment (i.e., creating a correct arc in the non-canonical plane incurs loss), then the Covington technique, based on counting individually unreachable arcs and then correcting for the presence of cycles, works for the 2-Planar parser.


Indeed, if Switch transitions are always allowed, their cost is zero because they can always be undone and thus never affect the reachability of any arcs.

when consecutive Switch transitions are banned to ensure parser termination, choosing to Switch can have consequences as, in the resulting configuration, the parser will be forced to take one of the other four transitions, which may lead to suboptimal outcomes compared to not having switched.


The latter seems to work better on treebanks with less non-projectivity such as the English, Chinese and Japanese datasets, and worse on those with higher amounts like Turkish, Dutch or Basque.

some cases like Czech or Catalan go against this trend.


to compute the marginal in the denominator.

posterior probability of all the parameters of interest (here, Ψ = {τ, v, θ}) can be computed from samples drawn using a Markov chain Monte Carlo (MCMC) method.


With our method, intransitive verb run in H works as a soft constraint on the verb in T and corrects its structure successfully.

there are some cases where using only surface forms as a cue forces the assignment of categories which is  We show the results on JSeM in Table 2.


In the above experiments, our method worked well, mainly due to the fact that the sentences in these datasets have comparably simple structure.

in other datasets, there are naturally more complex cases as in Table 3 (d), where we want different syntactic analyses for occurences of words with the same surface form.


Because of their efficiency and ease of implementation, transition-based parsers are the most common systems for dependency parsing.

efficiency comes at a price, namely a loss in expressivity: while graph-based parsers are able to produce any tree spanning the input sentence, many transition-based systems are restricted to projective trees.


In this paper, we focus on the foiled captions classification task (Section 2), and propose the use of explicit object detections as salient image cues for solving the task.

to methods from previous work that make use of word based information extracted from captions (Heuer et al., 2016;Yao et al., 2016;Wu et al., 2018), we use explicit object category information directly extracted from the images.


The query-todocument attention module in BiDAF added the attended-context vector to the document representation instead of the query representation.

the inverse attention from the objects to the words is important in our task because the representation of the object depends on its corresponding words.


Compared to the modular approach, direct modeling is advantageous in that it focuses on learning translations that are beneficial for retrieval, rather than translations that preserve sentence meaning/structure in bitext.

there exist no large-scale CLIR dataset that can support direct modeling approaches in a wide variety of languages.


We also want to avoid broadcasting relative position representations.

both issues can be resolved by splitting the computation of eq.


These three datasets are notable in containing shorter documents, which benefit the most from shared learning.

as discussed earlier, multi-domain data can introduce noise to the shared representation, causing the performance to drop over TCL, Wikipedia2 and EMEA.


Using continuation of training we are able to improve the performance by up to 1.2% absolute.

if we look at the translation quality on the out-ofdomain data set we see the catastrophic forgetting effect, with a drop of 1.7 BLEU points.


A naive approach to decoding with constraints would be to use a large beam size and select from the set of complete hypotheses the best that satis-fies all constraints.

this is infeasible in practice because it would require searching a potentially very large space to ensure that even hypotheses with low model score due to the inclusion of a constraint would be part of the set of outputs.


Decoding with gold constraints yields large BLEU gains over LNMT for all language pairs.

the length ratio on the dev set increases significantly.


As indicated by Table 5, the representations perform noticeably better than a majority baseline.

our results are not competitive with state-of-the-art systems trained specifically for Multi-NLI (Nangia et al., 2017).


the Spanish experiment is hardly within the low resource settings as it has about 24M tokens of parallel data.


1 Pre-trained word embeddings have proven to be highly useful in neural network models for NLP tasks such as sequence tagging (Lample et al., 2016;Ma and Hovy, 2016) and text classification (Kim, 2014).

it is much less common to use such pre-training in NMT (Wu et al., 2016), largely because the large-scale training corpora used for tasks such as WMT 2 tend to be several orders of magnitude larger than the annotated data available for other tasks, such as the Penn Treebank (Marcus et al., 1993).


One might imagine that most word types are unambiguous.

many inflectional systems are replete with a form of ambiguity termed syncretism-a systematic merger of morphological slots.


The model we present in §3 will resolve both syncretism and inter-paradigmatic ambiguity.

our exposition focuses on the former, as it is cross-linguistically more common.


It is straightforward to use a gradient-based optimizer, and we do.

could also be maximized by an intuitive EM algorithm: at each iteration, the E-step uses the current model parameters to partition each count c(f ) among possible analyses, as in (2.3), and then the M step improves the parameters by following the gradient of supervised regularized log-likelihood as if it had observed those fractional counts.