The importance of reasoning about quantities has been recognized and studied from multiple perspectives.

the rules were specific to the queries used, and do not extend well to unrestricted English.


An important problem in this context is cross-document coreference resolution (CCR): computing equivalence classes of textual mentions denoting the same entity, within and across documents.

all mentions pertaining to John Smith within a document refer to the same person.


Probabilistic graphical models like Markov Logic networks (Richardson & Domingos, 2006;Domingos et al., 2007;Domingos & Lowd, 2009) or factor graphs (Loeliger, 2008;Koller & Friedman, 2009) take into consideration constraints such as transitivity, while spectral clustering methods (Luxburg, 2007) implicitly consider transitivity in the underlying eigenspace decomposition, but suffer from high computational complexity.

this is also done for the cooccurring mention groups, leading to the extended scope of the original mention group considered.


BIC is a Bayesian variant of the Minimum Description Length (MDL) principle (Grunwald, 2007), assuming the points in a cluster to be Gaussian distributed.

simpler methods also perform fairly well.


Finally, our third single system baseline is the model of Toutanova et al.

similar to Tromble and Eisner (2006), for all algorithms, we first use the local solution without constraints and only apply the constraints in the case of a violation.


Another popular approach has been to apply a reranking model, which can incorporate soft structural constraints in the form of features, on top of the k-best output of local classifiers (Toutanova et al., 2008;Johansson and Nugues, 2008).

since there are no overlaps, we will not need to include incompatible edges.


(2013) report an accuracy of 92.8% on the test set; this represents almost a 30% relative error reduction.

second, even if we restrict the constraints to core reference roles, the lack of ordering between the spans in the constraint means that we would have to represent all subsets of these constraints are rarely violated in practice.


This allows word distributions to depend on both factors.

our experiments focused on understanding how SPRITE compares to commonly used models with similar structures, and how the different variants compare under different metrics.


We trained 100-dimensional skip-gram vectors (Mikolov et al., 2013) on English Wikipedia (tokenized/lowercased, resulting in 1.8B tokens of text) using window size 10, hierarchical softmax, and no downsampling.

if the current word is a local context word, we sample a new topic/sense pair for it again conditioned on all other latent variable values.


We perform inference using collapsed Gibbs sampling ( §4.2), then estimate the sense distribution for each instance as the solution to the WSI task.

in Table 2, we present results for these systems and compare them to our basic (i.e., without any data enrichment) sense-topic model with S = 3 (row 9).


Compared with LDA with full context (FULL) in row 6, performance is slightly improved, perhaps due to the fact that longer contexts induce more accurate topics.

their assumption is that words with similar distributions have similar meanings.


Length We also investigated whether solution length is a predictor of complexity (e.g., simple solutions may vary in length and amount of detail from complex ones).

τ ranges between −1 and +1, where +1 indicates equivalent rankings, −1 completely reverse rankings, and 0 independent rankings.


The Position and Permutation performed significantly better (p < 0.05) compared to Random, Length and SynSem baselines.

we perform Laplace smoothing to avoid zero probability transitions between states: This HMM formulation allows us to use efficient dynamic programming to compute the likelihood of a sequence of solutions.


The sampling sequence starts with a random initialization to the hidden variables.

we can compute the expected complexity of the solution set for problem i, using the inferred distribution over levels Table 5 shows the complexity of different problems as predicted by the position model (with 10 levels).


In our current design, the CPU portions are those that cannot be easily parallelized on the GPU or those that require too much memory to fit on the GPU.

gPUs are poor at "irregular" computations that involve conditionals, pointer manipulation, and complex execution sequences.


3 Munroe summarizes his results with 954 idealized colors-RGB values that best exemplify high frequency color labels.

the graphical model in Figure 4 formalizes our approach.


The most likely label of this posterior is the maximum likelihood estimate (MLE).

similar histogram models have been developed by Chuang et al.


In any utterance, yellowish green fits only those Hue values that are above a minimum threshold τ Lower and below a maximum threshold τ Upper .

the model describes this variability with probability density functions.


Similar to the diverse condition in Table 1, it is important that the extractor can correctly predict on diverse sentences that are dissimilar to each other.

it allows them to scale but makes them unable to output canonicalized relations.


We demonstrate the potential for web links to both complement and completely replace Wikipedia derived data in entity linking.

for example, we observe links to Apple the fruit where the surrounding context indicates an intention to link Apple Inc instead.


We define name probability as the conditional probability of a name referring to an entity: where M n,e is the set of mentions with name n that refer to entity e and M n, * is all mentions with name n. We use existing conditional probability estimates from the DBpedia Lexicalizations data set (Mendes et al., 2012).

we also present detailed experiments comparing popularity, context, and coherence components across settings.


Wikipedia models generally perform better.

references to the inventor appear with words like 'engineer', 'ac', 'electrical'.


But it is not well-equipped to capture semantic relatedness at the word level.

as is common in unsupervised segmentation (Poon et al., 2009;Sirts and Goldwater, 2013), we included the test words (without their segmentations) with the training words during parameter learning.


The prediction of the dependency topology and labels through P l means that the full RDLM has the highest perplexity of all models.

for unknown words, we back-off to a special unk token for the sequence models and P l , and to the pre-terminal symbol for the other dependency models.


Linguistic Space We construct distributional vectors from text through the method recently proposed by , to which we feed a corpus of 2.8 billion words obtained by concatenating English Wikipedia, ukWaC and BNC.

hit@k measures the percentage of images for which at least one gold attribute exists among the top k retrieved attributes.


For example, by chaining even small numbers of direct associations, such as breakfast -pancakes, pancakes -hashbrowns, hashbrowns -potato, and potato -field, a model that does not control for semantic drift may be tempted to answer questions about breakfast venues with answers that discuss wheat fields or soccer fields.

the translation table for t (q|a) is computed using GIZA++ (Och and Ney, 2003).


Other scenarios showed similar trends in our preliminary experiments.

embedding algorithms suggest some natural hyperparameters that can be tuned; many of which were already tuned to some extent by the algorithms' designers.


For all models, we normalize the embeddings so that the L-2 norm equals 1, which is important in measuring semantic similarity via inner product.

for unsupervised training on large scale raw texts (language modeling) we train fCT so that phrase embeddings -as composed in Section 2 -predict contextual words, an extension of the skip-gram objective (Mikolov et al., 2013b) to phrases.


The performance of RNN with 25dimension embeddings is too low so it is omitted.

pre-trained Word Embeddings For methods that require pre-trained lexical embeddings (FCT with pre-training, SUM (Section 5), and the FCT and RNN models in Section 6) we always use embeddings 2 trained with the skip-gram model of word2vec.


We evaluate the perplexity of language models that include lexical embeddings and our composed phrase embeddings from FCT using the LM objective.

table  5 summarizes these datasets and shows examples of inputs and outputs for each task.


While the model trained by HS is efficient to evaluate perplexities, NCE training requires summation over all words in the vocabulary in the denominator of the softmax to compute perplexity, an impracticality for large vocabulary.

as discussed in Section 3.3, the pipeline method, although commonly used in deep learning literatures, does not suit NLP applications well because of the sparsity in word embeddings.


For each of the N words w i in phrase p we construct the embedding: where e w i is the embedding for word i; and refers to point-wise product.

we report NCE loss with a fixed set of samples for NCE trained models.


We use candidate sets of size 1k/10k/100k from the most frequent N words in NYT and report mean reciprocal rank (MRR).

we propose to Table 1: Feature templates for word w i in phrase p. t(w): POS tag; c(w): word cluster (when w is a function word, i.e.


POS-tagging, morphological analysis, and linked NE phrases were used to detect other mentions of NEs that appear without links in text.

in the next section we take into consideration various classifier combination methods in order to aggregate the best decisions of SSL and DL classifiers, and to improve overall performance.


We used an algorithm adapted from Althobaiti et al.

if the base classifiers agree on the NE type of a certain word, then it is annotated by an agreed NE type.


This analysis allows us to avoid reasoning about entity triples (quadruples, etc.

our evaluation methodology is inspired by information retrieval evaluations (Manning et al., 2008).


The MAP numbers are somewhat low because almost half of the test questions have no correct answers and all models get an average precision of 0 on these questions.

the most obvious limitation is the restriction to existentially quantified conjunctions of predicates.


6 In terms of overall MAP, Freebase outperforms our approach by a fair margin.

from a machine learning perspective, training a probabilistic database via matrix factorization is easier than training a semantic parser, as there are no difficult inference problems.


The second-layer hidden variable vector h 2 are used as the LFR of this sample.

the features in F s and F t are are ordered the same as the order they appeared in training data.


Our DBN model uses the second-layer hidden variable vector h 2 to represent this sample.

to reduce the number of argument candidates, we adopt the pruning strategy in Zhao et al., (2009), which is adapted from the strategy in Xue and Palmer (2004).


Below is a sentence from an article entitled "Photolithography" in Simple Wikipedia: Microphototolithography is the use of photolithography to transfer geometric shapes on a photomask to the surface of a semiconductor wafer for making integrated circuits.

in the next section, we will review the evaluation methodology used in recent research, discuss its shortcomings and propose alternative evaluations.


We make our argument not as a criticism of others or ourselves, but as an effort to refocus research directions in the future (Eisenstein, 2013).

although discourse is known to affect readability, the relation between discourse and text simplification is still under-studied with the use of statistical methods (Williams et al., 2003;Siddharthan, 2006;Siddharthan and Katsos, 2010).


We also introduce a new comparative approach to simplification corpus analysis.

we also thank action editor Rada Mihalcea and three anonymous reviewers for their thoughtful comments, and Ani Nenkova, Alan Ritter and Maxine Eskenazi for valuable discussions.


For example with T = 40, our w2v-LDA and glove-LDA obtain F 1 scores at 40.0% and 38.9% which are 4.5% and 3.4% higher than F 1 score at 35.5% obtained by the LDA model, respectively.


We see that λ = 1.0 gives the highest NPMI score.

given a topic t represented by its top-N topic words w 1 , w 2 , ..., w N , the NPMI score for t is: where the probabilities in equation 12are derived from a 10-word sliding window over an external corpus.


in the TAC KBP, in addition to determining if a mention has no entity in the KB to link, all the mentions that represent the same real world entities must be clustered together.


(2014) provide an easy-to-use evaluation toolkit on the AIDA data set.

in this paper, we examine 9 EL data sets and discuss the inconsistencies among them.


"+BOTH": an average of two scores is used for re-ranking.

annotations in an early TAC-KBP dataset (2009) select the whole span as the mention.


For example, in "... while returning from Freeport to Portland.

neither the city nor the country can actually make a proposal.


In such models, the distributed representations and compositional operators can be fine-tuned by backpropagating supervision from task-specific labels, enabling accurate and fast models for a wide range of language technologies (Socher et al., 2011;Socher et al., 2013;.

prior work has not applied these ideas to the classification of implicit relations in the PDTB, and does not consider the role of entities.


Dependency syntax may be a better alternative .

we had the data re-annotated by two authors of this paper who are native English speakers.


Some systems adopt logic-based representations but use distributional evidence for predicate disambiguation (Lewis and Steedman, 2013) or to weight probabilistic inference rules Beltagy et al., 2013).

formal semantics captures fundamental aspects of meaning in set-theoretic terms: Entailment, for example, is captured as the inclusion relation between the sets (of the relevant type) denoted by words or other linguistic expressions, e.g., sets of possible worlds that two propositions hold of (Chierchia and McConnell-Ginet, 2000, 299).


All these measures provide a score that is higher when a significant part of the candidate antecedent feaspective, we are interested in testing general methods of composition that are also good for other tasks (e.g., modeling sentence similarity), rather than developing ad-hoc composition rules specifically for entailment.

we compute the output value of h as the conjunction across all dimensions in w. Concretely, h Θ (p, q) is obtained as follows.


Spoken Term Discovery Spoken term discovery is the problem of using unsupervised pattern discovery methods to find previously unknown keywords in speech.

figure 4 shows an analysis of stored lexical units for each lecture, plotting grams revealed two likely reasons for this.


We formulate the noisy-channel model as a PCFG and encode the substitute, split, and delete operations as grammar rules.

to reduce the inference load on the HMM, we exploit acoustic cues in the feature space to constrain phonetic boundaries to occur at a subset of all possible locations (Lee and Glass, 2012).


The flexibility of the framework, toolkit and analysis methods presented in this paper helps researchers to devise, analyze and compare representations for coreference resolution.

to create training graphs, we employ a slight modification of the closest pair heuristic (Soon et al., 2001), which worked best in preliminary experiments.


Furthermore, in principle we can take the structure into account.

via latent antecedents, the model can avoid learning from the most unreliable pairs.


The average is the metric for ranking the systems in the CoNLL shared tasks on coreference resolution (Pradhan et al., 2011;Pradhan et al., 2012).

they do not develop a unified framework for comparing approaches, and their analysis is not qualitative.


10 Driven by the results of Section 6.1, we turn to explore a methodology for identification of translationese in a mixed-domain setup.

excellent indomain classification results on the one hand and poor cross-domain predictive performance on the other, imply that the model describing the relation in a certain domain is inapplicable to a different (even seemingly similar) domain due to significant differences in the distribution of the underlying data.


The two-phase approach outperforms the flat one in most cases: the latter attempts to cluster data instances by domain and translation status simultaneously, and is therefore potentially more error-prone.

it has been suggested that the accuracy of translation detection deteriorates when the classifier is evaluated outside the domain it was trained on.


After submitting this paper, we developed a penalized expectation propagation method (Cotterell and Eisner, 2015).

this edit sequence corresponds to a character-wise alignment of u to s. Our features for modeling the contextual probability of each edit are loosely inspired by constraints from Harmonic Grammar and Optimality theory (Smolensky and Legendre, 2006).


We currently model S θ (s | u) as the probability that a left-to-right stochastic contextual edit process ( Figure 2) would edit u into s. This probability is a sum over all edit sequences that produce s from u-that is, all s-to-u alignments.

with certain fancier approaches to modeling S θ , which we leave to future work, this effect could be mitigated while preserving the transducer property.


Further, the other metrics show that it places most of  its probability mass on that form, 12 and the rest on highly similar forms.

reduplication is not regular if unbounded, but we can adopt morphological doubling theory (Inkelas and Zoll, 2005) and model it by having U concatenate two copies of the same morph.


As shown in Table 2, a country that fills the SUPPLIER role is more likely to also fill the role of a SELLER than that of a BUYER.

country names, for example, can be observed as fillers for different roles depending on the text genre and its perspective.


To identify particularly reliable indicators for discourse salience, we inspected held-out development data.

we change the argument labeling procedure from predicate-specific to frame-specific roles and implement I/O methods to read and generate FrameNet XML files.


Both are composed of a set of source sentences, their machine translated outputs and the corresponding post-editing time.

both cases are evidence of model overfitting.


Examples include other tree kernels like the Partial Tree Kernel (Moschitti, 2006a) and string kernels like the ones defined on character ngrams (Lodhi et al., 2002) or word sequences (Cancedda et al., 2003).

interest in model selection procedures for kernelbased methods has been growing in the last years.


The function ∆ can be defined recursively, where pr(n) is the grammar production at node n and preterm(n) returns true if n is a pre-terminal node.

we can employ any kind of valid kernel in this procedure as long as its gradients can be computed.


For all NLP experiments the grid is fixed for all hyperparameters (including γ, the lengthscale value in the RBF kernel), with its corresponding values shown on

the GP optimization can also benefit from multiple cores by running each kernel computation inside the Gram matrix in parallel.


In the remainder of this section we describe the general setup of the experiments.


In some other settings, exact inference is NPhard.

5equates to a summation over exponentially many projective parse trees.


For certain choices of higher-order factors, polynomial time is possible via dynamic programming (McDonald et al., 2005;Carreras, 2007;Koo and Collins, 2010).

the annealed risk objective requires an annealing schedule: over the course of training, we linearly anneal from initial temperature t = 0.1 to t = 0.0001, updating t at each step of stochastic optimization.


Here we describe inside-outside and the accompanying backpropagation algorithm over a hypergraph.

non-projective parsing becomes NP-hard with even second-order factors (McDonald and Pereira, 2006).


In fact, the model parameters are not sparse in practice.

the mention prior alone does surprisingly well on this task, but well below the previous best results, as might be expected.


Since the supervised model is trained only on Wikipedia, the mention prior component dominates, and the system incorrectly infers that George Harrison refers to the Beatle.

while both these approaches can be implemented using a distributed processing framework such as map-reduce (Dean and Ghemawat, 2008), the latter where we only infer the missing entity labels scales better than the standard EM approach.


The posterior is intractable due to a combinatorial number of possible link configurations.

we assume that each document in the corpus consists of a set of mentions -text spans -that describe event actions, their participants, times, and locations.


Theorem 1 For noncrossing dependency graphs, Maximum Subgraph can be solved in time O.jV j 3 /.

in this paper we therefore address maximum acyclic subgraph parsing under the restriction that the subgraph should be noncrossing, which informally means that its arcs can be drawn on the half-plane above the sentence in such a way that no two arcs cross (and without changing the order of the words).


Unfortunately, however, this setting also did not allow for a gain in accuracy, likely due to to the low recall (15%) of the matching between paraphrasing grammar and semantic parsing rules.

in this section we briefly outline two other (unsuccessful) attempts to do so: creation of a pipelined paraphrasing/semantic parsing system, and addition of features from a large paraphrase database.


In this work, we construct the tri-synchronous grammar by transforming the basic SCFG for semantic parsing G into a 3-SCFG.

in this paper we introduced a method for constructing a semantic parser for ambiguous input that paraphrases the ambiguous input into a more explicit form, and verifies the correctness using a language model.


In order to reduce the space of possible equation trees, ALGES reorders Qsets {s 1 , .

on Monday, 375 students went on a trip to the zoo.


We have now showed that its possible to automatically construct large lexicons from smaller seed lexicons.

a typical lexicon contains all possible attributes that can be displayed by a word.


One key question is which of the features in our graph are important for projecting morphological attribute-values.

we will model attributes propagation/transformation as a function of the features shared on the edges between words.


Second, some crossword questions mirror definitions, in that they refer to fundamental properties of concepts (a twelve-sided shape) or request a category member (a city in Egypt).

this is likely to come at the cost of a greater memory footprint, since the model requires access to its database of dictionaries at query time.


The normal distributions are parametrized to encourage smooth change in multinomial parameters, over time (see Section 3.2 for details), and the extent of change is controlled through a precision parameter κ.

in this paper we present a dynamic Bayesian model of diachronic meaning change.


With the exception of AMBRA, all other participating systems used external resources (such as Wikipedia and Google n-grams); it is thus fair to assume they had access to at least as much training data as our SCAN model.

a hypothetical system would then have to decide amongst the following classes: {1700-1702, 1703-1705, ..., 1961-1963, ..., 2012-2014} {1699-1706, 1707-1713, ..., 1959-1965, ..., 2008-2014} {1696-1708, 1709-1721, ..., 1956-1968, ..., 2008-2020} The first set of classes correspond to fine-grained intervals of 2-years, the second set to medium-grained intervals of 6-years and the third set to coarsegrained intervals of 12-years.


We tokenized, lemmatized, and part of speech tagged DATE using the NLTK (Bird et al., 2009).

we define this prior as an intrinsic Gaussian Markov Random Field (iGMRF; Rue and Held 2005), which allows us to model the change of adjacent parameters as drawn from a normal distribution, e.g.


Identifying different viewpoints is related to the well-studied area of subjectivity detection, which aims at exposing opinion, evaluation, and speculation in text (Wiebe et al., 2004) and attributing it to specific people (Awadallah et al., 2011;Abu-Jbara et al., 2012).

while some of the perspective words are neutral, mostly literal and occur in both English and Spanish (e.g.


These differences are mined from multilingual, non-parallel datasets of Twitter and news data.

the goal of a topic model is to characterize observed data in terms of a much smaller set of unobserved, semantically coherent topics.


To characterize these different edit types, we manually reviewed a sample of 100 rewrites and categorized the types of changes that were made.

note that we are not able to make a meaningful comparison against against any of the previously published statistical models for formality detection.


Note that, while the basic LM perplexity correlates very weakly with formality overall, the Email genre actually exhibits a trend opposite of that which we expected: in Email, sentences which look less like Gigaword text (higher perplexity) tend to be more formal.

the effect of prior post formality on current post formality becomes stronger later in a thread compared to at the beginning of a thread.


This is expected because our system is designed for pure monoalphabetic substitution ciphers.

we define a word's alphagram distance with respect to an ordering of the alphabet as the number of letter pairs that are in the wrong order.


For each such τ 1 , Z, τ 2 , it searches for a τ 1 , Z , τ 2 where Z is only a subset of H i .

because c appears in no forbidden tier substrings in R, it is freely distributed in L = L( T, S ).


For a TSL 2 grammar G = T, S , T ⊆ Σ will be referred to as the tier, S the allowed tier substrings, and R = fac T-2 (Σ * ) − S as the forbidden tier substrings.

these are words which may contain bs interspersed with as and cs provided that no a precedes another without an intervening c. For example, bbabbcbba ∈ L but bbabbbbabb ∈ L, because E t (bbabbbbabb) = aa and aa ∈ fac 2 ( aa ) but aa ∈ S. Like the class of SL k languages, the class of tSL languages (for fixed t and k) is a string extension language class.


We use the posteriors to re-estimate LM parameters as follows To obtain better parameter estimates for word predictions and avoid overfitting, we use smoothing in the M-step.

though they also consider divergences between distributions of latent variable vectors, they use these divergences at learning time to bias models to induce representations maximally invariant across domains.


Sennrich (2012a) proposed to cluster training data in an unsupervised fashion to build mixture models that yield good performance on multiple test domains.

we observe consistent improvements over a baseline which does not explicitly reward domain invariance.


As the result suggests, using our induction framework tends to yield slightly better translation results in terms of METEOR and especially BLEU.

• The translation improvement is observed also for training with a development set of mixed domains (even for the mixed-domain minus in-domain setting when excluding the Legal data from the mixed development set).


Though they also consider divergences between distributions of latent variable vectors, they use these divergences at learning time to bias models to induce representations maximally invariant across domains.

a translation rule with source and target phrases having two similar distributions over the latent subdomains is likely safer to use.


In addition we assume the following entries: It can be verified that (BIND f (nsubj (dobj founded f ) Jobs)) has semantics λu.

eQ(z)(z ) is true iff z and z are equal (refer to the same entity).


In contrast-partly due to the lack of a strong type system-dependency structures are easy to annotate and have become a widely used form of syntactic analysis for many languages.

an important constraint on the lambda calculus system is as follows: all natural language con-stituents have a lambda-calculus expression of type Ind × Event → Bool.


It is designed specifically for UMLS and it does not disambiguate two candidates if they are classified into the same semantic category.

in the end, all the selected concept candidates are ranked according to s j i , and a list of ranked concepts is returned for each mention.


The first step of constructing our training examples is to cluster concepts in the KBs by the cross reference attributes and also extract all pairs of concepts that have "has participant" relations.

if two concepts in different KBs are determined to be the same, we can assume that one is the "gold label" for the other, and extract textual and relational features between them, making this pair an approximation of the real grounding instance.


A more recent focus has been on learning representations using weaker forms of supervision that require minimal amounts of manual annotation effort (Clarke et al., 2010;Liang et al., 2011;Krishnamurthy and Mitchell, 2012;Artzi and Zettlemoyer, 2013;Berant et al., 2013;Kushman et al., 2014).


One answer is that error tags can be informative and useful to provide feedback to language learners, especially for specific closed-class error types fluency evaluation.

in this section, we investigate the impact that different reference sets have on the system ranking found by different evaluation metrics.


This is difficult for NER because of the absence of upper-case spelling, which is not untypical in social media, for example.

binary values denote the presence or absence of a feature (e.g., a particular token); real-valued ones typically denote frequencies of observed features.


We retrained this model by using the same corpus-specific training data that we use for J-NERD .

this makes the dataset not only cleaner but also more demanding, as metonymous mentions are among the most difficult cases.


For tractability, probabilistic graphical models are typically constrained by making conditional independence assumptions, thus imposing structure and locality on X ∪ Y.

typically, NER would pass only the mentions "David", "manu", and "la" to the NED stage, which then is prone to many errors like mapping the first two mentions to any prominent people with first names David and Manu, and mapping the third one to the city of Los Angeles.


Template f 14 (type i , tok i ) generates a binary feature function if the background corpus contains the pattern dep i = deptype(arg1 , arg2 ) where the current token tok i is either arg1 or arg2 , and tok i is labeled with NER label type i .

f 8 (David Beckham, "David") may be lower than f 8 (David Bowie, "David"), for example, as this still active pop star is more frequently and prominently mentioned than the retired football player.


Previous work on this task (Banko et al., 2007), as well as on its generalization, called unsupervised semantic parsing (Poon and Domingos, 2009;Titov and Klementiev, 2011), groups patterns between entity pairs (e.g., wrote a review, wrote a critique and reviewed) and uses these clusters as relations.

this is not surprising as the argument independence assumption is very strong, and the general motivation we provided in Section 2 does not really apply to the selectional preference model.


In order to examine the influence of the decoder on the model performance, we performed additional experiments in a more controlled setting.

in the following section, we formally describe the problem.


Supervised methods for RE have been successful when small restricted sets of relations are considered.

the most-frequent trigger for three induced relations are presented in table 2 Cluster 66 instead groups together expressions such as leads or president (of), so it can vaguely be described as a LEADERSHIP relation, but it also contains the relation triggered by the word professor (in).


Generative models with rich features have also been considered in the past (Berg-Kirkpatrick et al., 2010).

though weakly-supervised approaches, such as distantly supervised methods and bootstrapping (Mintz et al., 2009;Agichtein and Gravano, 2000), reduce the amount of necessary supervision, they still require examples for every relation considered.


We present a method for unsupervised opendomain relation discovery.

both classes of approaches assume a predefined inventory of relations and a manually constructed resource.


In addition, we use sentence lengths, WordCnt (count of the number of nonstopwords in the question that also occur in the answer) and WgtWordCnt (reweight the counts by the IDF values of the question words).

(2013) turn their attention to improving the shallow semantic component, lexical semantics, by performing semantic matching based on a latent word-alignment structure (cf.


Additionally, a variety of work exists in the general field of stemmer evaluation, though much of it centers on the information retrieval community.

the relative effects of these treatments on coherence are magnified as the number of topics increases; while no ArXiv treatment differs significantly in coherence at 10 topics, at 200, the four strongest treatments (Lovins, Paice-Husk, five-truncation and four-truncation) are significantly worse.


This difference indicates our predictions have worse recall for longer dependencies such as subordinate clauses, while being more accurate in local, phrasal contexts.

annotation projection has been explored in the context of cross-lingual dependency parsing since Hwa et al.


These texts have the advantage of being translated both conservatively and into hundreds of languages (massively multi-parallel).

evaluation All our datasets-projected, training, and test sets-contain only the following CoNLL-X features: ID, FORM, CPOSTAG, and HeAD.


The average gold edge length is 3.6-which is significantly higher at p < 0.05 (Student's t-test).

for this baseline, we parse a target sentence using multiple single-source delexicalized parsers.


(2015) suggest a kernel-based approach to implicitly consider all possible feature combinations over sets of core-features.

regardless of the details of the parsing framework being used, a crucial step in parser design is choosing the right feature function for the underlying statistical model.


In the graphbased parser, we jointly train a structured-prediction model on top of a BiLSTM, propagating errors from the structured objective all the way back to the BiLSTM feature-encoder.

when training in this way the parser sees only configurations that result from following correct actions, and as a result tends to suffer from error propagation at test time.


More fundamentally, the SNM model parameterization and method of estimation are completely original, as far as we know.

we are not the first to highlight this effectiveness; previous such results were reported in Singh and Klakow (2013).


First we will investigate more properties related to word order.

from these data, we confirm that the DLM ratio and Entropy measures capture different word order properties as they are not correlated (Spearman correlation r = 0.32, p > 0.1).


1 (a) where the bias parameter is not included for simplicity.

the depth of 16 is not very deep compared to the models in computer vision (He et al., 2016).


Our result justifies via a generative model why this should be satisfied even for low dimensional word vectors.

full details appear in the ArXiv version of this paper (Arora et al., 2015).


be- 5 Note that this interpretation has been disputed; e.g., it is argued in Levy and Goldberg (2014a) that (4.1) can be understood using only the classical connection between inner product and word similarity, using which the objective (4.1) is slightly improved to a different objective called 3COSMUL.

it also helps explain why lowdimensional semantic embeddings contain linear algebraic structure that allows solution of word analogies, as shown by Mikolov et al.


The model discussed thus far conditions on the POS tags of words in the input sentence.

the embedding of a non-English word which is not aligned to any English words is defined as the average embedding of words with a unit edit distance in the same language (e.g., 'playz' is the average of 'plays' and 'play').


In order to expose the parser to more errors, we employ a cost augmentation scheme: we sometimes follow incorrect actions also if they score below correct actions.

this approach gives a good vector representation for unknown words but at the expense of ignoring many of the words from the training corpus.


The parser sees only states that result from following correct actions.

the functions are usually restricted to having a fixed maximum arity (usually two) (Socher et al., 2010;Tai et al., 2015;Socher, 2014).


In stage 2, the texter introduces the main issue, while the counselor asks for clarifications and expresses empathy for the situation.

in addition to the person directly experiencing a mental illness, family, friends, and communities are also affected (insel, 2008).


In all cases we find a statistically significant (p < 0.01; Mann-Whitney U-test) increase in the likelihood of the texter using a LIWC marker if the counselor used it in the previous message (~4-5% change).

we evaluate our model with 10-fold cross-validation and compare models using the area under the ROC curve (AUC).


To gain insights into the "specialization hypothesis" we make use the counselor annotation of the main issue (depression, self-harm, etc.).

our model performs well with only a small set of linguistic features, demonstrating they provide a substantial amount of the predictive power.


Each of these graphs is "smooth" because it incrementally adds new languages as n or m increases.

the embedded distances are reasonably faithful to the symmetrized dissimilarities: metric MDS achieves a low value of 0.20 on its "stress" objective, and we find that Kendall's tau = 0.76, meaning that if one pair of languages is displayed as farther apart than another, then in over 7/8 of cases, that pair is in fact more dissimilar.


These "unearthly" languages are intended to be at least sim-ilar to possible human languages.

we could train a system to extract features from the surface of a language that are predictive of its deeper structure.


): the morphology is correct, but the numeric value is wrong.

but not all ambiguities can be handled by normal form constraints.


The second was a TensorFlow-based RNN with an attention mechanism (Mnih et al., 2014), using an overall architecture similar to that used in a system for end-to-end speech recognition (Chan et al., 2016).

this approach can be implemented with a backtracking recursive descent parser enforcing the aforementioned normal form constraints and propagating the top-down arithmetic constraints.


Above, we focused solely on cardinal numbers, and specifically their citation forms.

while the former approach is certainly more appealing given current trends in NLP, only the latter is feasible for lowresource languages which most need an automated approach to text normalization.


The COMSENSE approach models the decision as interactions between high-level categories of entities, actions and utterances.

most interestingly, it does so with a larger margin when tested over the out-of-domain dataset, demonstrating that it is more resistant to overfitting compared to other models.


Similarly, we associated with the quote topic PROFANITY a list of profanity words.

the high level categories assigned to the NRG vertices are not observed, and as a result we view it as a weakly supervised learning problem, where the category assignments correspond to latent variable assignments.


Our time-aware method was able to assign higher weights to the new taxonomic relations than the older relations due to their recent timestamps, even though the frequencies of these new relations are fewer than that of the older relations.

secondly, if we normalize the score, the information on the number of evidence sentences, which is important for the LsP method to recognize true taxonomic relationships, will be lost.


This effect is likely due to the network graph becoming increasingly dense with more nodes (Figure 1, bottom right).

although it is not surprising that the network graph visualization better communicates common phrases in the corpus as edges are drawn between these phrases, this suggests other approaches to drawing edges.


This was because averaging z-transformed values (and back transforming) tends to yield less biased estimates of the population value than averaging the raw coefficients (Silver and Dunlap, 1987).

the image-based model is built using a deep convolutional neural network approach, similar in nature to those recently used to study neural representations of visual stimuli (see Kriegeskorte (2015), although note this is the first application to study word elicited neural activation known to the authors).


Also, they do not include selectional preference or common-sense knowledge effects in their analysis.

in order to choose which inference rules can be applied to yield "soap", we can inspect Figure 4.


The guessing experiment provides a basis to estimate human expectation of already mentioned DRs (the number of clicks on the respective NPs in text).

this feature captures the distance l t (d) between the position t and the last occurrence of the candidate DR d. As a distance measure, we use the number of sentences from the last mention and exponentiate this number to make the dependence more extreme; only very recent DRs will receive a noticeable weight: exp(−l t (d)).


(2008) used the concept of spine for TAG (Schabes, 1992;Vijay-Shanker and Joshi, 1988), which is similar to our constituent hierarchy.

the weights of contribution β ijk are computed using the attention mechanism (Bahdanau et al., 2015).


(2011) run a baseline parser for a few future steps, and use the output actions to guide the current action.

wang and Xue (2014) and  do joint POS tagging and parsing.


Each constituent hierarchy is generated bottom-up recurrently.

it ends a constituent hierarchy, with a constituent S on the top level, dominating a VP (starting from the word "like"), and then an NP (starting from the noun phrase "this book"), and then a PP (starting from the word "in"), and finally an NP (starting from the word "the").


Galley and Manning (2010) describe a decoding algorithm for phrase-based systems where phrases can have discontinuities in both the source and target languages.

for any H j , define τ (σ(H j ), ∆) to be the equivalence class resulting from adding the edges in ∆ to the data structure σ(H j ).


In this setup, the distribution over next symbols given a context u is drawn hierarchically from a Dirichlet process whose base measure is another Dirichlet process associated with context tl(u), and so on, with all draws ultimately backing off into some unconditioned distribution over all possible next symbols.

for all but one lexicon, we find that the autosegmental models do not significantly outperform the N-gram models on artificial data.


And-node C is a class node bundling together the oral features of vowel segments.

we have presented a model where forms are built out of parts by structure-building operations.


For example, "2 + Context Gate (both)" in Row 3 denotes integrating "Context Gate (both)" into the baseline in Row 2 (i.e., GroundHog (vanilla)).

over GroundHog-Coverage (GRU) We finally tested on a stronger baseline, which employs a coverage mechanism to indicate whether or not a source word has already been translated (Tu et al., 2016).


In neural machine translation (NMT), generation of a target word depends on both source and target contexts.

they employ a gating scalar to scale only the source context.


After identifying drug, gene and mutation mentions in the text, co-occurring triples with known interactions were chosen as positive examples.

to overcome these challenges, we explore a general relation extraction framework based on graph LStMs.


(2015) focuses specifically on causal language -language used to appeal to psychological notions of cause and effect.

to generalize across these, we crafted a set of scripts for tSurgeon (Levy and Andrew, 2006), a tree-transformation utility built on tRegex.


Accordingly, on  Each syntactic pattern inherently encodes the positions in the tree of the cause and effect heads.

two of the reasons for this are familiar issues in NLP.


We say that a system predicts "rightward" according to whetherp(→| r, L) > 0.5.

3 we ask: given that u i and u j are nearby tag tokens in the test corpus u, are they likely to be linked?


Figure 5c shows more complex interactions between syntax and sentiment for deciding the head word.

theis and Bethge (2015) proposed Spatial LStMs to handle image data.


We consider three baseline methods, namely left branching (L), right branching (R) and averaging (A).

this parameter set is not very large compared to the modern memory capacity, even for a computer with 16GB RAM.


(2013b), Con-Tree and DepTree denote constituency Tree LSTMs and dependency Tree LSTMs, respectively.

x t is used to estimate the input (i t ), output 1 In this paper, we work on binary trees only, which is a common form for CKY and shift-reduce parsing.


While there is prior research in topic modeling that provides topic-specific indices when modeling the link structure, these do not extend to individual indices, and most previous citation-based indices are defined for each individual but without considering topics.


Once the topic variables are learned using the vanilla LDA, authority and citation variables are then inferred consecutively.

several variants of topic modeling consider the relationship between topics and citations in academic corpora.


π i←ja ∈ {0, 1} is an element of π i←j which measures the influence of author a on the citation from j to i and sums up to one ( a∈A i π i←ja = 1) over all authors of publication i.

our model combines the merits of both topic-specific and individual-specific indices to provide topical authority information for academic researchers.


Using this approximation, we can represent with the time complexity of O(S E K 2 ), where D − a is the number of rows with negative links in Ψ a .

if topic distributions of paper i and j are similar and if η values of the cited paper's authors are high, the citation formation probability increases.


We consider debate as a process of argument exchange.

empirical evidence has also led to an increasing awareness of the dangers of style and rhetoric in biasing participants towards the most skillful, charismatic, or numerous speakers (Noelle-Neumann, 1974;Sunstein, 1999).


We have two sets of AMIE features: typed and untyped.

this allows us to distinguish different contextual uses of a verb without introducing a proliferation of verb sense symbols.


For other domains like medicine, legal it might require domain experts to encode this knowledge.

in our domain, we use a manually constructed inventory of 45 types (animal, artifact, body part, measuring instrument, etc.).


We summarize these features in Table 1, where ⊕ denotes the binary operator which defines features as a combination of word forms at different (not necessarily contiguous) positions of a sentence.

we also il-lustrate the advantageous generalization property of our model as it retained 89.8% of its original average POS tagging accuracy when trained on only 1.2% of the total accessible training sentences.


Similar to our POS tagging experiments, using polyglot SC vectors tend to perform best for NER as well.

we then use the previously described set of features in a linear chain CRF (Lafferty et al., 2001) using CRFsuite (Okazaki, 2007) with its default settings for hyperparameters, i.e., the coefficients of 1.0 and 0.001 for 1 and 2 regularization, respectively.


for j := 1 to minibatch size : for t := 0 to T −1 : Intervene: Evaluate each action at s t forā t ∈ A(s t ) : 11: 12: Improve: Train with dataset aggregation 13: Finalize: Pick the best policy over all iterations encountered when running the new policy π i+1 (or when parsing test sentences).

in fact, a user does not even have to quantify their global preferences by writing down such a function.


First, the number of fully projected trees in some languages is so low such that the density-driven approach cannot start with a good initialization to fill in partial dependencies.

we use the more efficient algorithm from Stratos et al.


Twitter's "following" relation is a relatively low-cost form of social engagement, and it is less public than retweeting or mentioning another user.

for each basis model k, each author a has an instance weight α a,k .


But personalization requires annotated data for each individual user-something that may be possible in interactive settings such as information retrieval, but is not typically feasible in natural language processing.

3 we use the Twitter API to crawl the friends of the SemEval users (individuals that they follow) and the most recent 3,200 tweets in their timelines.


This allows us to embed vector spaces of multiple languages into a single vector space, exploiting information from high-resource languages to improve the word representations of lower-resource ones.

their performance has not surpassed that of fine-tuning methods.


These include: 1) the ATTRACT-REPEL source code; 2) bilingual word vector collections combining English with 51 other languages; 3) Hebrew and Croatian intrinsic evaluation datasets; and 4) Italian and German Dialogue State Tracking datasets collected for this work.

ppDB relies on large, high-quality parallel corpora such as Europarl (Koehn, 2005).


In these dialogues, the speaker is trying to identify their (privately assigned) target color for the listener.

this prediction results in the blends L a and L e preferring the true target, allowing the speaker's perspective to override the listener's.


However, if the target were the cyan, the speaker would have many good options.

the base listener interprets this as referring to the cyan with 91% probability, perhaps due to the extreme saturation of the cyan maximally activating certain parts of the neural network.


Understanding model behavior We find that much of the gain in model performance comes from the first two rounds of training.

our proposed approach attempts to combine the advantages of both approaches, by defining an objective that incorporates both levels of linguistic properties over the entire forest representation, and adopting an alternating training regime for optimization.


A commonly used log-linear formulation enables these models to consider a rich set of features ranging from orthographic patterns to semantic relatedness.

data To obtain gold information about morphological clusters, we use CELEX (Baayen et al., 1993).


Minimized number of roots relatively to vocabulary size Similarly, the number of roots, and consequently the number of morphological families is markedly smaller than the size of the vocabulary.

these models generally bypass global constraints (Narasimhan et al., 2015) or require performing inference over very large spaces (Poon et al., 2009).


Now let us consider how to derive our ILP formulation using the notations above.

they have barely trained word vectors, contributing to the low recall value.


(2016) report that characterlevel decoding is only 14% slower than subwordlevel decoding.

(1) Training/decoding latency For the decoder, although the sequence to be generated is much longer, each character-level softmax operation costs considerably less compared to a word-or subword-level softmax.


This gives a source BPE vocabulary of size 20k−24k for each language.

a character-level model has a much better chance recovering the original word or sentence.


The minibatch size was fixed at 100, and the learning rate was controlled by RMSprop.

similar to QANTA, the method adopts the LR classifier with the derived representations as features.


Table 5: Accuracies of the proposed method and the state-of-the-art methods for the factoid QA task.

this simple method enables us to easily train the distributed representations of words and entities simultaneously.


Figure 1: Syntactic trees of the sentence "The little boy likes red tomatoes.".

such binarization requires a set of language-specific rules, which hampers adaptation of parsing to other languages.


We find that the bottom-up parser and the top-down parser have similar results under the greedy setting, and the in-order parser outperforms both of them.

the top-down and the bottom-up parsers incorrectly recognize the "This has both made investors uneasy" as a complete sentence.


Our focus here is on replicating an advantage of normalized features found in cognitive models of categorization.

if the cognitive model were extended to infant discrimination tasks, a similar strategy could be used to evaluate theories of human representation learning with respect to children's discrimination abilities throughout the learning process.


Speaker normalization has previously been found to improve phonetic categorization (Cole et al., 2010;Nearey, 1978), increase a phonetic categorization model's match with human behavior (Apfelbaum and McMurray, 2015;McMurray and Jongman, 2011), and improve the performance of speech recognizers (Povey and Saon, 2006;Wegmann et al., 1996).

the following section describes the method we use for evaluating these features against human discrimination data.


The acoustic values are assumed to lie in a continuous feature space, such as MFCCs.

mFCCs have the advantage that they are computed deterministically from the speech signal, and thus are not subject to the error inherent in automatic formant tracking.


If there is a path from p to q, q is reachable from p. Item Types As shown in Figure 1, our items start and end on words, fully covering the spaces in between.


We estimate monolingual phrase embeddings via the element-wise addition of the word embeddings composing the phrase.

in phrase-based statistical machine translation (SMT), translation models are estimated over a large amount of parallel data.


1), we evaluated whether our induced phrase table improves the translation of in-domain texts over the vanilla SMT system which used only one phrase table trained from general-domain parallel data.

as science domain monolingual data for the Science translation task, we used the English side of the aSPEC parallel corpus (Nakazawa et al., 2016).


(2013a) with the only exception that we deal with not only words but also phrases.

for each pair of domain and translation direction, we have four word embedding spaces: those with 300 or 800 dimensions for source and target languages.


these methods heavily rely on document-level information (Zhao and Vogel, 2002;Utiyama and Isahara, 2003;Fung and Cheung, 2004;Munteanu and Marcu, 2005) to reduce their search space by scoring only sentence pairs extracted from each pair of documents.


We apply our method on Chinese-English, because the annotated alignment type training data is provided in this language pair.

in alignment models, a hidden variable a = {a 1 , a 2 , .


Similarly, some of the major gains in F 1 for the disease types, and major losses in F 1 for the disaster types, do not come from the most or least frequent document labels.

this framework naturally generalizes to hierarchical and semisupervised extensions with no additional modeling assumptions.


Unlike the gains in topic overlap and coherence, the F 1 score increases do not simply correlate with which document labels appeared most frequently.

these generalizations come at the cost of increasingly elaborate and unwieldy generative assumptions.


Episodes are ordered by increasing model precision.


(2015) used sentence vectors obtained by a pre-trained hierarchical CNN (Denil et al., 2014) as features under an unweighted average MIL objective.


We describe such an architecture in more detail as we use it as a point of comparison with our own model.

the acquisition of sentence-or phrase-level sentiment labels remains a laborious and expensive endeavor despite its relevance to various opinion mining applications, e.g., detecting or summarizing consumer opinions in online product reviews.


This dataset enforces that all words are composed of exactly two morphemes.

morphological segmentation is a structured prediction task that seeks to break a word up into its constituent morphemes.


(2017) proposed a neural architecture for jointly mapping instructions and visual observations (pixels) to actions in the environment.

their work stops short of exploring generalization over map layouts, which our model is designed to handle.


Yelp reviews were obtained from the 2013 Yelp Dataset Challenge.

we used the Chu-Liu-Edmonds algo- 14.4% depth 6 10.3% 4.5% Same Edges 38.7% Table 5: Descriptive statistics for dependency trees produced by our model and the Stanford parser  on the SNLI test set.


The proposed approach returns multiple relevant temporal expressions only for a small fraction of events.

the choice of the activation function is a hyperparameter and was optimized on a development set.


For multi-token expressions, we only use the first token.

using the relaxed metric, we see an accuracy of 94.9% for the begin point and 80.2% for the end point.


SGNS randomly initializes all the embeddings before training begins, and it relies on negative samples created by randomly selecting word and context pairs (Mikolov et al., 2013;Levy et al., 2015).

we predict that multiple runs of SGNS on the same corpus will not produce the same results.


Happily, this variability can be quantified by averaging results over multiple bootstrap samples.

sampling in PPMI is inspired by a similar method in the word2vec implementation of sGNs (Levy et al., 2015).


Indeed, community style matching is shown to be correlated to comment popularity in Reddit (Tran and Ostendorf, 2016).

words in comments and replies associated with overpredicted (controversial) cases are related to controversial topics (sexual, regulate, liberals), named political parties, and mentions of downvoting or indication that the comment has been edited with the word "Edit."


Time structure is represented only among a set of siblings: p(t) is the sibling predecessor in time, and s(t) is the sibling successor.

in several prior studies, authors constrained the problem to reduce the effect of those factors (Lakkaraju et al., 2013;Tan et al., 2014;Jaech et al., 2015).


Likewise, the embeddings enable predicting correctly the relative sizes of unseen objects.

we split the data into 10 parts and employ 90% of the data for training and 10% for testing.


Formally, the FUTURE layer is a recurrent neural network (the first gray layer in Figure 2) , and its state at time step t is computed by where F is the activation function for the FUTURE layer.

we attribute the former case to the lack of untranslated future modeling, and the latter one to the overloaded use of the decoder state where the language modeling of the decoder leads to the fluent but wrong predictions.


Table 4 lists the alignment performances of our proposed model.

the proposed approach almost addresses the errors in these cases.


The supervised baseline results are lower than the French-English case, which is partly due to the low coverage of WordNet translations for Arabic (see Table 5).

we will refer to this model as the subword skip-gram.


For each of the datasets described above, we generated 100-dimensional word embeddings using the subword skip-gram model (Bojanowski et al., 2017).

such alignments are not available for all languages and dialects, and while a small dictionary might be feasible to acquire, discovering word mappings with no prior knowledge whatsoever is valuable.


11 This starting tensor has a precision of about 80% and acts as a valuable resource for challenging tasks such as question answering.

advances in the latter set of methods have led to several embedding-based methods that are highly successful at KB completion for named entities (Nickel et al., 2011;Riedel et al., 2013;Dong et al., 2014;Trouillon et al., 2016;Nickel et al., 2016a).


For each relation r, the relation schema imposes a type constraint on the entities that may appear as its source or target.

first, we use taxonomy guided uncertainty sampling to propose a list L to potentially query.


(5) and (6) can be used to infer uncertain triples: if every sibling ofẽ has relationship r with an entity e , we can infer for "free" that this is the case forẽ as well.

transE is omitted due to its very low precision here, around 10%.


(2017) proposed a neural model for fine-grained entity typing and for robustly using type information to improve relation extraction, but this is targeted for Freebase style named entities.

when siblings disagree in this respect, there is more uncertainty about (ẽ, r, e ) (according to (5) and (6)), making this triple a good candidate to query.


2013, the fork and join decision, and the preterminal, top and bottom category label sub-models described in Section 3 can now be defined in terms of these sideand depth-specific grammars G s,d and depth-specific left-corner expectations E + d .

for CCL and UPPARSE, the NP agg f1 scores are not reported because they do not produce labeled constituents.


When there is no join, the top category is defined in terms of a depth-specific sub-model: 4 Decisions about the bottom categories b 1..D t (which correspond to right children in tree structures) also depend on the outcome of the fork and join variables, but are defined in terms of a side-and depth-specific sub-model in every case: 5 2 Here, again,d In a sequence model inducer like Shain et al.

the competing systems include UPPARSE (Ponvert et al., 2011) 13 , CCL (Seginer, 2007a) 14 , BMMM+DMV with undirected dependency features (Christodoulopoulos et al., 2012) 15 and UHHMM (Shain et al., 2016).


22 Table 5 shows the PARSEVAL scores of all systems.

(2016), these depth-specific models are assumed to be independent of each other and fit with a Gibbs sampler, backward sampling hidden variable sequences from forward distributions using this compiled transition model M (Carter and Kohn, 1996), then counting individual sub-model outcomes from sampled hidden variable sequences, then resampling each sub-model using these counts with Dirichlet priors over a, b, and p models and Beta priors over f and j models, then re-compiling these resampled models into a new M. note that with K category labels this model contains DK 2 + 3DK 3 separate parameters for preterminal categories and top and bottom categories of derivation fragments at every depth level, each of which can be independently learned by the Gibbs sampler.


In addition, these result populations can be statistically compared for significance, allowing for stronger claims on improvement.

the correlation coefficients across devices presented in table 9 leads us to suspect that there can be substantial changes in effectiveness when switching the backend from CPU to GPU and vice versa.


(2017) present a model which explicitly computes O(N 2 ) possible tree nodes for N words, and uses a soft gating strategy to approximately select valid combinations of these nodes that form a tree.


In Sections 3 to 5, we describe the three main stages of our approach respectively.

their method could have a relatively large variance in the document sentiment classification performance because of the domain mismatch (e.g., F 1 = 0.874 for the "Tangled" tweets and F 1 = 0.647 for the "Obama" tweets), whereas our approach would perform quite consistently over different domains.


In the second dataset, MEDHOP, the goal is to establish drug-drug interactions based on scientific findings about drugs and proteins and their interactions, found across multiple MEDLINE abstracts.

overall, although both neural RC models clearly outperform the other baselines, they still have large room for improvement compared to human performance at 74% / 85% for WIKIHoP.


When answering 100 questions, the annotator knew the answer prior to reading the documents for 9%, and produced the correct answer after reading the document sets for 74% of the cases.

here the first document states that the drug Leuprolide causes GnRh receptor-induced synaptic potentiations, which can be blocked by the protein Progonadoliberin-1.


As possible end points, we consider any other drug, apart from drug1 and those interacting with drug1 other than drug2.

for a given query-answer pair, the item entity is chosen as the starting point for the graph traversal.


Finally, we identified the selection of relevant document sets as the most promising direction for future research.

5 For FastQA we use the implementation provided by the authors, also with pre-trained GloVe embeddings, no characterembeddings, no maximum support length, hidden size 50, and batch size 64 for 50 epochs.


(2017a) are used, with pretrained GloVe (Pennington et al., 2014) embeddings.

note that WIK-IHOP inherits the train, development, and test set splits from WIKIREADInG -i.e., the full dataset creation, filtering, and sub-sampling pipeline is executed on each set individually.


Masking is consistent within one sample, but generally different for the same expression across samples.

for the open-domain setting of WIKIHOP, a reduction of the answer vocabulary to 100 random single-token mask expressions clearly helps the model in selecting a candidate span, compared to the multi-token candidate expressions in the unmasked setting.


Following the same general methodology, we next construct a second dataset for the domain of molecular biology -a field that has been undergoing exponential growth in the number of publications (Cohen and Hunter, 2004).

the question tokens are helpful to detect relevant documents, but exploiting only this information compares poorly to the other baselines.


We can utilize this relatedness by sharing the vocabulary across all related languages.

previous work has looked at the most natural setup -training on a single language pair.


(2015) for the basic encoder-decoder architecture.


We conducted 2 experiments: (a) train on ar- (pl,sk,sl) pairs and test on ar-cs pair, and;(b) train on en- (bn,kn,ta) pairs and test on en-hi pair.

larger target side data does not explain the improvement in transliteration accuracy due to multilingual transliteration.


We find that the proposed dataset meets the desiderata we set out in Section 3.1.

7 In this section, we show that NarrativeQA presents a challenging problem for current approaches to reading comprehension by evaluating several baselines based on information retrieval (IR) techniques and neural models.


Cognates are words in two languages that share both a similar meaning and a similar form.

one of the frequent English synonym alternatives {approval, acceptance} would better fit this context.


Bethard and Parker (2016) proposed an alternate scheme, Semantically Compositional Annotation of Time Expressions (SCATE), in which times are annotated as compositional time entities (Figure 1), and suggested that this should be more amenable to machine learning.

for example, next week refers to the week following the DCT, and in such a case the value of the property INTERVAL-TYPE for the operator NEXT would be DOCTIME.


and SVM ling + GloVe are statistically significant, with p = .029 for Pearson's r and p .01 for both Spearman's ρ and Kendall's τ .

the variational inference algorithm begins by initializing the parameters G,f , C, a and b at random.


To test this hypothesis, we simulate an active learning scenario, in which an agent iteratively learns a model for each fold.

we found that mean GloVe embeddings produced substantially better performance in all tests.


Note that the attention mechanism is not shown here for clarity.

we also experimented earlier with a discrete version of LFT, where we treated responses in the [0.8, 1.0] range as polite, [0.2, 0.8] as neutral, and [0.0, 0.2] as rude.


2013, e.g., positive ones such as gratitude, deference, greeting, positive lexicon, indirection, indicative modal, and negative ones such as negative lexicon, direct question, direct start, 2nd person start.

it is worth noting that in the last example, while LFT and Polite-RL seem to provide a relevant compliment, they are actually complimenting the wrong person.


0.457 without any novel features 0.474 Abu-Jbara et al.

authors often use multiple sentences to indicate a citation's purpose (Abu-Jbara and Radev, 2012;Ritchie et al., 2008;He et al., 2011;Kataria et al., 2011).


BiRNN-CRF is adopted as the fundamental segmentation framework that is complemented by an attention-based sequence-to-sequence transducer for non-segmental multiword tokens.

in addition to multiword tokens, the UD scheme also allows multitoken words, that is, words consisting of multiple tokens, such as numerical expressions like 20 000.


For these languages, we build dictionaries from the training data to look up the multiword tokens.

(2014) present a data-driven word segmentation system for Arabic based on a sequence labelling framework.


In terms of generation techniques that capture semantics, the sentence variational autoencoder (SVAE) (Bowman et al., 2016) is closest to our work in that it attempts to impose semantic structure on a latent vector space.

this was some of the best <norp> food i've had in the <gpe>.


This task is substantially simpler, since the goal is to identify a single word (such as "good:better::bad:?")

a sample from q is simply a perturbed version of f : obtained by adding von-Mises Fisher (vMF) noise, and we perturb the magnitude of f by adding uniform noise.


We thank all editors and reviewers for their helpful feedback and suggestions.

for the categorical variables we compare the mean values per category with the numerical dependent variable.


Traditionally, languages have been grouped into the four 1 Chinese, Japanese, and Thai are sourced from Wikipedia and processed with the Polyglot tokeniser since we found their preprocessing in the PW is not adequate for language modeling.

it is not straightforward how to advance these ideas to the output side of the model, as this second set of word-specific parameters is directly responsible for the next-word prediction: it has to encode a much wider range of information, such as topical and semantic knowledge about words, which cannot be easily obtained from its characters alone (Jozefowicz et al., 2016).


Empirically we find that it leads to similar results in testing our theory (Section 4) and better results in downstream WSI applications (Section 6).

, A m there will be directions corresponding to clothing, sports matches, etc., that will have high inner products with tie1, tie2, etc., respectively.


It has the advantage that it is easy to measure and is widely used as a criteria for model fit, but the limitation that it is not directly matched to most tasks that language models are directly used for.

results are reported for contextconditioned perplexity and generative model text classification accuracy, using contexts that capture a range of phenomena and dimensionalities.


Consider a sequence X for which we want to calculate its probability.

this corpus is limited by being relatively small, only containing approximately 45,000 sentences, which we found to be insufficient to effectively train lattice language models.


We are exploring techniques to store fixed embeddings dynamically, so that the non-compositional phrases can be selected as part of the end-to-end training.

compositional Representation the non-compositional embeddings above only account for a subset of all n-grams, so we additionally construct compositional embeddings for each chunk by running a BiLSTM encoder over the individual embeddings of each unit-level token within it (Dyer et al., 2016).


Then, define the literal semantics of a single-word message x to be x def = {s : s ∧ v x = 0}, where ∧ denotes element-wise and and 0 denotes the zero matrix.

in comparison to these works, we seek to predict human behavior instead of modeling artificial agents that communicate with each other.


It is notable that their method has been incorporated into Thomson Reuters Eikon TM , their commercial datato-text NLG software product for macro-economic indicators and mergers-and-acquisitions deals .

this paper focuses on the problem of choosing appropriate verbs to express the direction and magnitude of a percentage change (e.g., in stock prices).


The major problem with the Neural Network baseline is that, similar to the probabilistic model without smoothing, its verb choices would concentrate on the most frequent ones and thus have very poor diversity.

random sampling would lead to a much wider variety of verbs.


Although existing corpora have promoted research into coreference resolution, they suffer from gender bias.

parallelism+URL tests the page-context setting; all others test the snippet-context setting.


We choose Wikipedia as our base dataset given its wide use in natural language understanding tools, but are mindful of its well-known gender biases.

we saw systematic differences between genders in analysis; this is consistent with many studies that have called out differences in how men and women are discussed publicly.


The PMB, too, is constructed using an existing semantic parser, but a part of it is completely manually checked and corrected (i.e., gold standard).

for both models the f-score clearly still improves when using more training instances, which shows that there is at least the potential for additional data to improve the score.


Submission batch: 7/2018; Revision batch: 9/2018; Published 12/2018.

for (a) Standard naming $1 REF @1 $1 male "n.02" @1 $1 Name @1 "tom" $2 REF @2 $2 EQU @2 "now" $2 time "n.08" @2 $0 NOT $3 $3 REF @3 $3 Time @3 @2 $3 Experiencer @3 @1 $3 afraid "a.01" @3 $3 Stimulus @3 @4 $3 REF @4 $3 entity "n.01" @4   Figure 1.


In this model, we directly use the sense selected for each token by one of the WSD systems above, and use the embeddings of the respective sense as generated by NMT after training.

it appeared that learning sense embeddings for NMT is better than using embeddings learned separately by other methods, although such embeddings may be useful for initialization.


† indicates that a neural model's performance was found to be significantly different (p < 0.05) from the MGL.

4.1 Follow-ups to Rumelhart and McClelland (1986) Over the Years Following R&M, a cottage industry devoted to cognitively plausible connectionist models of inflection learning sprouted in the linguistics and cognitive science literature.


A total of 168 of the 4,039 verb types were marked as irregular.

nLP as a discipline has a distinct practical bent and more often concerns itself with the large-scale engineering applications of language technologies.


In brief, the main sources of improvement are twofold: Synthetic languages.

t D is the directionalities typology that was studied by Liu (2010) and used as a training target by Wang and Eisner (2017).


We will choose our universal parameter values by minimizing an estimate of their expected loss, where L train is a collection of training languages (ideally drawn IID from the distribution D of possible human languages) for which some syntactic information is available.

some other work on generalizing from source to target languages assumes the availability of source-target parallel data, or bitext.


It is important to relax the delexicalized assumption.

their method only found global typological information: it did not establish which 70% of the direct objects fell to the right of their verbs, let alone identify which nouns were in fact direct objects of which verbs.


Experiments on sentence modeling with zero-context (sentiment analysis), singlecontext (textual entailment) and multiplecontext (claim verification) demonstrate the effectiveness of ATTCONV in sentence representation learning with the incorporation of context.

we now describe light and advanced versions of ATTCONV.


We show improvements by adding knowledge from our learned entailments without changing the graphs or tuning them to this task in any way.

to our method, these works rely on supervised data and take a local learning approach.


(2013) develop a small corpus of Korean particle errors and build a classifier to perform error detection.

system combination, model ensembles, and adding a spell checker boost these numbers by 4 to 6 points (Chollampatt and Ng, 2018;.


This corpus was used in two editions of the QALB shared task Rozovskaya et al., 2015).

the classification system for Russian obtains a much lower score of 21.0.


On a different but related note, our preliminary experiments on English and Hebrew show that the Arc ZEager variant always outperforms Arc Eager.

let J p , J g be the form-based (rather than index-based) arcs of the predicted and gold representations of x.


In this work we empirically confirm this hypothesis for Modern Hebrew, an MRL with complex morphology and severe wordlevel ambiguity, in a novel transition-based framework.

for example, in the phrase w+b+silwp ewbdwt (''and in distortion of facts''), the conjunction marker w is labeled comp in gold while the parser correctly picks cc.


This survey attempted to review and summarize as much of the current research as possible, while organizing it along several prominent themes.


We argue that an important consequence of the simplicity of unlexicalized systems is that their derivations are easier to learn.

we converted the Negra corpus to labeled dependency trees with the DEPSY tool 3 in order to annotate each token with a dependency label.


The BASE templates form a minimal set that extracts Configuration: 7 indexes from a configuration, relying only on constituent boundaries.

unlexicalized parsing models renounce modeling bilexical statistics, based on the assumptions that they are too sparse to be estimated reliably.


Contrary to previously discussed phenomena, there is usually no lexical trigger (wh-word, speech verb) that makes these discontinuities easy to spot.

they only experiment with a linear classifier, and assume access to gold part-of-speech (POS) tags for most of their experiments.


• Extensive experiments on NIST Chinese-English, WMT14 English-German and WMT18 Russian-English translation tasks demonstrate that our SB-NMT model obtains significant improvements over the strong Transformer model by 3.92, 1.49, and 1.04 BLEU points, respectively.

our proposed model alleviates the under-translation problems by exploiting the combination of left-to-right and right-to-left decoding directions, reducing 30.6% of under-translation errors.


For an embedding x in the source language, its transformation to the target language space is given by W ts x.

the detailed experimental settings for this BLI task can be found in Conneau et al.


We learn the matrices B, U s , and U t corresponding to the transformations φ(•), ψ s (•), and ψ t (•), respectively.

this requires an n-way dictionary to represent n languages.


Then, we turn on coverage training, as advised by See et al.

we let the RNN generate the special vectors at time step t (i) by linearly embedding the RNN input x t ∈ R N x to an embedded inputε t ∈ R N h , and (ii) by obtaining a target memory τ t as a linear combination of the current input x t (projected in the hidden space) and the previous history h t−1 (after a linear transformation).


The novelty of our model is that it exploits the simple and fundamental multiplicative closure of rotations to generate rotational associative memory for RNNs.

the RNN is fed a sequence of letter-digit pairs followed by the separation indicator ''??''


Several studies have focused on learning joint input-label representations grounded to word semantics for unseen label prediction for images Socher et al., 2013;Norouzi et al., 2014;Zhang et al., 2016;Fu et al., 2018), called zero-shot classification.

(2015) proposed hierarchical recurrent neural networks and showed that they were superior to CNNbased models.


One potential solution is the use of implicit definitions (Rogers, 1997).

conversely, x's corresponding output position is labeled H if its predecessor is an H or if x is the last position and is an H. In this way, the ''shifting'' behavior is captured by determining the output label of each position via the predecessor of its corresponding input position.


Let |w| indicate the length of w ∈ Σ * .

since members of the same tier are never associated to each other, A(t 1 , t 2 ) will always evaluate to False.


2 Using this notation, the examples from (1) are as below in Example (2).

we then examine a variety of commonly attested tone patterns that are both ISL and not ISL (i.e., local and non-local in terms of strings) to see if they are A-ISL (i.e., local over ARs).


for any k, and for any n ≥ k, the strings H∅ n and ∅ n have the same (k − 1)-suffix: ∅ k−1 .


Then let τ be de- where True is any unary predicate that is always true for models in S. In this transduction, ϕ P b (x) is only true for input positions labeled b that do not follow another b; conversely, ϕ P c (x) is only true for input positions labeled b that do follow another b.

this allows us to use the usual conventions from strings: For an AR w, w n represents the AR consisting of n repetitions of w; w 0 is the empty AR whose tiers are both λ and thus has no association lines.


Memory Lookup: The memory lookup looks up scratch memory with a given probe, say x (of arbitrary dimension), and retrieves the memory entry having closest key embedding to x.

this, to our knowledge, is the first NPI system to be trained with only the gold answer as (very distant) supervision for inducing such complex programs.


# of questions per dialogue 1.6 / 10 Avg./Max.

concurrently, there is a growing interest in conversational reading comprehension such as coQA (Reddy et al., 2018).


Results show that GBDT++ is superior to the fine-tuned language model on the questions under the category matching (68.1% vs. 57.0%) and the latter model is more capable of answering implicit questions (e.g., under the category summary, logic, and commonsense) which require aggregation of information from multiple sentences, the understanding of the entire dialogue, or the utilization of world knowledge.

we further define four subcategories as follows.


Besides surface matching, a significant portion of questions require multiple-sentence reasoning and external knowledge (Richardson et al., 2013;Mostafazadeh et al., 2016;Khashabi et al., 2018;Ostermann et al., 2018).

we map three most common speaker abbreviations (i.e., ''M''; ''W'' and ''F'') that appear in dialogues to their eight most common corresponding mentions (i.e., ''man,'' ''boy,'' ''he,'' and ''his''; ''woman,'' ''girl,'' ''she,'' and ''her'') in questions.


Our neural sequence-to-sequence models utilize an encoder-decoder setup (Cho et al., 2014;Sutskever et al., 2014) with an attention mechanism (Bahdanau et al., 2015).

our ramp loss objectives can be formulated as follows: where y − is a fear output that is to be discouraged and y + is a hope output that is to be encouraged.


Table 2 shows the average C v scores over the produced topics given N = 5 and N = 10.

our model captures topic and discourse representations embedded in conversations.


To further understand why our model learns meaningful representations for topics and discourse, we present a case study based on the example conversation shown in Figure 1.

this shows that with sufficiently large training data, with or without using the pre-trained word embeddings do not make any difference in the topic coherence results.


We further analyze the errors in our outputs.

future work should involve a better alternative to evaluate the latent discourse without Figure 5: Visualization of the topic-discourse assignment of a twitter conversion from TWT16.


All these non-neural models train language models on the whole Gigaword corpus.

when we set the number to 18, GCN+RC+LA achieves a BLEU score of 19.4, which is significantly worse than the BLEU score obtained by DCGCN2 (23.3).


(2018) show that vanilla residual connections proposed by He et al.

the BLEU margins between DCGCN models and their best GCN models are 2.0, 2.7, 2.7, and 3.4, respectively.


Out-of-domain data are often much easier to obtain, and we can therefore conclude that the proposed approach is preferable in many practically relevant situations.

this causes equation 14 to be replaced by this extension moves the model closer to the basic two-stage model, and the inclusion of the context vectors and the block drop-out operation on the hidden decoder states ensures that the second stage decoder does not rely too strongly on the first stage outputs.


Including an auxiliary ASR task is straightforward with the two-stage model by simply computing the cross-entropy loss with respect to the softmax output of the first stage, and dropping the second stage.

as a remedy, we propose the use of endto-end trainable models with two attention mechanisms, the first establishing source speech to source text alignments, the second modeling source to target text alignment.


We did not find it beneficial to apply a scaling factor when adding this loss to the main cross-entropy loss in our experiments.

while a cascade would use the source-text outputs of the first stage as inputs into the second stage, in this model the second stage directly computes attentional context vectors over the decoder states of the first stage.


Whereas the regular English verb paradigm has 4-5 slots in our annotation, the Archi verb will have thousands (Kibrik, 1998).

concretely, we show that the more unique forms an inflectional paradigm has, the more predictable the forms must be from one another-for example, forms in a predictable paradigm might all be related by a simple change of suffix.


We reiterate that no other output has positive probability under their model, for example, swapping -a for -es or ablaut of a stem vowel.

in the implementation, we actually decrement the weight of every edge σ → σ (including when σ = empty) by the weight of empty → σ.


The number of distinct inflected forms of each word indicates the number of morphosyntactic distinctions that the language makes on the surface.

the only true assumption we make of morphology is mild: We assume it is Turing-computable.


Long-range dependencies often pose problems for SRL models; in fact, special networks like GCN  and PathLSTM (Roth and Lapata, 2016) have been proposed to explicitly percolate information from each word in the sentence to its syntactic neighbors.

the semantic role predictor consists of: • a word representation component that encapsulates predicate-specific dependency information; • a J-layer BiLStM encoder that takes as input the representation of each word in a sentence; • a classifier that takes as input the BiLStM representations of predicates and their arguments and assigns semantic roles to the latter.


We omit all these results from   Figure 4: Edit distance per slot (which we call average edit distance, or AED) for each of the 5 corpora.

consider generating a sentence from a syntactic grammar as follows: Hail the king [, Arthur Pendragon ,] [, who wields [ " Excalibur " ] ,] .


The above example 18 shows how our model handles commas in conjunctions of 2 or more phrases.

the possibleū values are characterized not by a lattice but by a cyclic WFSA (as |u i | is unbounded whenever |x i | > 0).


In our future work, we are interested in relaxing these assumptions and evaluating our proposed method on actual real-world deployments with real users.

in our proposed method, we also have a classifier C that takes as input the dialog state vector s and makes a decision on whether to use the model to respond to the user or to transfer the dialog to a human agent.


The results in Section 5.2 have shown that the transformations-based generalization strategy used by TransWeight works well across different languages and phrase types.

this shows that treating similar words similarly-and not each word as a semantic island-has a twofold benefit: (i) it leads to good generalization capabilities when the training data are scarce and (ii) gives the model the possibility to accommodate a large number of training examples without increasing the number of parameters.


In our experiments, 100 transformations yielded optimal results for all phrase sets.

in this work, we have trained TransWeight for specific phrase types.


A restricted dictionary makes the evaluation easier because many similar words are excluded.

understanding what the transformations encode requires taking a step back and contemplating again the architecture of the model.


We add sentential contexts from Wikipedia, keeping up to 10 sentences per example.

one future direction is a richer training objective that simultaneously models multiple co-occurrences of the constituent words across different texts, as is commonly done in noun compound interpretation (e.g., Ó Séaghdha and Copestake, 2009;Shwartz and Waterson, 2018;Shwartz and Dagan, 2018).


We set the number of hidden units proportional to the input length (see Section 3.1 for configuration details).

areas of study that were historically heavily dependent on feature engineering, including sentiment analysis or image processing (Manjunath and Ma, 1996;abbasi et al., 2008), have made their way towards alternatives that do not involve manually developing features, and instead favor deep learning (Wang et al., 2016).


For instance, syntax paraphrase networks (Iyyer et al., 2018) applied to Quizbowl only yield valid paraphrases 3% of the time (Appendix A).

our examples go beyond surface level ''breaks'' such as character noise (Belinkov and Bisk, 2018) or syntax changes (Iyyer et al., 2018).


Pronimial one The word one is a very common NFH anchor (61% of the occurrences in our corpus), and can be used either as a number (viii) or as a pronoun (xiii).

these rules look for common cases that are often not captured by the parser.


The sense anaphora phenomena also cover numerals, and significantly overlap with many of our NFH cases.

(2005) perform both tasks, but restrict themselves to one anaphora cases and their nounphrase antecedents.


Each row represents an example from the data.

we examined monologues (TED talks; Cettolo et al., 2012), Wikipedia (WikiText-2 and WikiText-103;Merity et al., 2016), journalistic text (PTB: Marcus et al., 1993), and product reviews (Amazon reviews 3 ) in which we found that more than 35.5%, 33.2%, 32.9%, 22.2%, and 37.5% of the numbers, respectively, are NFHs.


For consistency, we consider the pronominal usages to be NFH, with the implicit head PEOPLE.

16 the coreference-centric architecture had to be adapted to the particularities of the NFH task.


Elo provides a baseline; rather than opaque scores associated with individual users at different times, we seek to explain the probability of winning through linguistic features of past debate content.

this implies that our model generalizes relatively well to most debaters regardless of their experience.


When using only the last debate's features and ignoring all previous debates, our performance is initially very good in the years 2013 and 2014 (when our training sets are smaller and histories shorter).

online debate communities offer an opportunity to investigate these questions.


(2016) analyzed the combination of available reference sets and metrics to identify the best evaluation configuration.

the original Wiki annotations have much lower precision and recall (40.9 and 33.2, respectively).


We refer to the model that either uses dependency or contextual graphs instead of random graphs as GCN-SeA+Structure.

we also performed ablations by removing specific parts of the encoder.


(2017) developed a variant of RNN cell that computes a refined representation of the query over multiple iterations before querying the memory.

we see that Seq2seq+Attn model is not able to suggest a restaurant with a high rating whereas HRED gets the restaurant right but suggests an incorrect price range.


The RNN and GCN hidden dimensions were also chosen to be 300.

as opposed to our work, all these works ignore the underlying structure of the entityentity graph of the KB and the syntactic structure of the utterances.


using morphology, which helps discover the correct syntactic dependencies.

modifies the noun root ''masa'' (table) even though the final part of speech of ''masalı'' is an adjective.


Pappas and Popescu-Belis (2014) adopt a multiple instance regression model to assign sentiment scores to specific product aspects, using a weighted summation of predictions.

this research is based upon work supported in part by the Office of the Director of National Intelligence (ODNI), Intelligence Advanced Research Projects Activity (IARPA), via contract FA8650-17-C-9118.


Although most topic models are unsupervised, some variants can also accommodate documentlevel supervision (Mcauliffe and Blei, 2008;Lacoste-Julien et al., 2009).

aside from automatic evaluation, we also assessed model performance against human elicited domain labels for sentences and words.


Our implementation is based on fairseq, 3 and we make use of its multi-GPU support to train on 16 NVIDIA V100 GPUs with a total batch size of 128,000 tokens.

these sentence embeddings are used to initialize the decoder LStM through a linear transformation, and are also concatenated to its input embeddings at every time step.


While some proposals use a separate encoder for each language (Schwenk and Douze, 2017), sharing a single encoder for all languages also gives strong results (Schwenk, 2018).

this approach has two obvious drawbacks when trying to scale to a large number of languages.


In addition, the success of these models stems from a large amount of data they used.


Our approach exhibits small GPU memory footprints, due to reductions of the arithmetic complexity (with fewer intermediate results to keep) and trainable parameter size (with fewer parameters to store).

com/uclanlp/ELMO-C. Contextual representation We review contextual representation models from two aspects: how they are trained and how they are used in downstream tasks.


This is because when training the ELMO-C model reported in 4.2, we actually train a forward ELMO-C on two cards and train a backward ELMO-C on two other cards, which reduces the communication cost by half.

a pre-trained word embedding, which projects words with different meanings into different positions in space, is a natural choice for the projection matrix W and can help preserve much of the variance of A.


Prior studies show that the encoders trained in this way can capture generic contextual information of the input text and improve a variety of downstream tasks significantly.

it is rather a waste to allocate extensive computational resources to the softmax layer.


's models and ours learn systematic generalizations like subject-verbobject order.


(2016) and Marvin and Linzen (2018) is to evaluate whether language models' assign higher probability to the acceptable sentence in a minimal pair.

in the classifier, the sentence embedding is passed through a sigmoid output layer (optionally preceded by a single hidden layer) giving a scalar representing the probability of a positive classification (either the sentence is real or acceptable, depending on the task).


We hence focused our experiments in cross-lingual and lightly supervised parsing setups.

in this setup, to keep with the low resource language spirit, we do not learn the noise parameter (σ) but rather use fixed noise parameters for the perturbated models (see below).


In such cases it is likely that a diverse list of high-quality solutions will be valuable.

ideally, such a list should be high-quality and diverse, in order to explore candidate structures that add information over the structure returned by the argmax inference problem.


We linearly project the last representation h t using F ∈ R d model ×d model for querying W .

we extend Transformer (Vaswani et al., 2017) for supporting insertion operations, where the generation order is directly captured as relative positions through self-attention inspired by Shaw et al.


Self-Attention One of the major challenges that prevents the vanilla Transformer from generating sequences following arbitrary orders is that the absolute-position-based positional encodings are inefficient (as mentioned in Section 3.2), in that absolute positions are changed during decoding, invalidating the previous hidden states.

the SYN orders were generated according to the dependency parse obtained by a dependency parse parser from Spacy (Honnibal and Montani, 2017) following a parent-to-children left-to-right order.


We then choose an existing word y k (0 ≤ k ≤ t)) from y 0:t and insert y t+1 to its left or right.

model Variants Table 5 shows the results of the ablation studies using the machine translation task.


(2005) argued that the definition needed to be made more precise, so as to circumscribe the extent to which ''world knowledge'' should be allowed to factor into inferences, and to explicitly differentiate between distinct forms of textual inference (e.g., entailment vs. conventional implicature vs. conversational implicature).

figure 4 shows several examples of sentences for which the annotations exhibit clear bimodal distributions.


The task of RTE/NLI is fundamentally concerned with drawing conclusions about the world on the basis of limited information, but specifically in the setting when both the information and the conclusions are expressed in natural language.

we thus explore whether providing additional context will yield less-divergent human judgments.


Raters label pairs in batches of 20, meaning we have a minimum of 20 ratings per rater.

whether or not humans' judgments can be summarized by a single aggregate label or value might be a moot question, since state-of-the-art models do not, in practice, predict a single value but rather a distribution over values.