By naming graphs, it appears that an inexpressive, conjunctive concept logic, F L 0 , can be employed to handle a wide variety of more complex phenomena including Booleans and hypotheticals.

one should not assume that the inferential tractability of F L 0 carries across to a system that combines it with named graphs.


This is not directly comparable to existing work because it is a unique setup; but we note that this is likely a difficult task because of the large output space.

it is interesting to note that pre-training CUI embedding has a positive relative impact on performance.


The second application promoted chased no cat from the type et to NP − + → S, again with a polarity flip.

if we had used (i), we would have obtained NP + → S. this would have trivialized the polarity to =, and this effect would have been propagated up the tree.


Hence, our model must be able to capture lexical relationships other than polysemy, e.g., synonymy, antonymy (opposite verbs), troponymy, etc..


In order to improve existing approaches, one possibility is to make use of the readily available, partially annotated data (P as in partial) that cover more documents.

missing annotations in P are known to hurt, rather than help, existing systems.


In contrast to the scarcity of fully annotated datasets (denoted by F as in full), there are actually some partially annotated datasets as well (denoted by P as in partial); for example, Time-Bank (Pustejovsky et al., 2003) and AQUAINT (Graff, 2002) cover in total more than 250 documents.

since annotators are not required to label all the edges in these datasets, it is less labor intensive to collect P than to collect F. existing TempRel extraction methods only work on one type of datasets (i.e., either F or P), without taking advantage of both.


Given the obvious difference in F and P F ull , System 4 expectedly performed worse than System 1.

when we see that System 5 was still worse than System 1, it is surprising because the annotated edges in P are correct and should have helped.


Benchmarks test sets such as WordSim-353 (Finkelstein et al., 2002) are commonly used to evaluate word embeddings since they provide a fast and easy way to quickly evaluate a model (Nayak et al., 2016).

the instability of word embeddings is not detected by these test sets since only selected pairs of words are evaluated.


It seems logical that a very close neighbor remains stable from one model to the other.


Additionally, morpheme dictionaries have been effective in developing features for NER tasks in languages like Japanese (Sasano and Kurohashi, 2008), Turkish (Yeniterzi, 2011), Chinese (Gao et al., 2005), and Arabic (Maloney and Niv, 1998).

such morphological features have not yet been integrated into the new neural network models for NER.


However, there are no obvious unifying trends across these datasets: Among the judged and recast datasets, where humans do not generate the NLI hypothesis, we observe lower performance margins between majority and hypothesis-only models compared to the elicited data sets.

the baseline performances of these models are noticeably larger than on SNLI and Multi-NLI.


The drop between SNLI and Multi-NLI suggests that by including multiple genres, an NLI dataset may contain less biases.

adding additional genres might not be enough to mitigate biases as the hypothesis-only model still drastically outperforms the majority-baseline.


it does not have the experimental problems of using social media data directly.


Consequently, baselines accounting for this imbalance perform strongly in predicting judgments on assertions.

the distribution corresponds to what we observe in many social network sites, where e.g.


As the neural approach on directly predicting aggregated judgments, we use a single branch of the Siamese network.

since we are trying to solve a regression problem, here the network ends in a single node equipped with a linear activation function.


For the SNN, the predictions follow a similar pattern: resembling a learning curve, the performance increases rapidly with increasing n, but then plateaus from a certain number of assertions.

the number of assertions for which we observe a plateau varies significantly.


For the prediction of individual judgments reference approaches yield competitive results.

for the prediction of group judgments the best approach using judgment similarity as predicted by a SNN outperforms other approaches by a wide margin.


For example, in "I wiped the table clean", the table can change on the cleanliness scale and thus allows a RESULT construction here.

in "I wiped the table happy", the mood scale is not applicable to tables, ruling out the RESULT construction.


The examples reported in Table 3 include quite arbitrary, albeit correct, frame instances.

the examples in Table 5 are indeed highly prototypical, e.g., a shirt is a prototypical piece of clothing, while the examples in Table 4 can be placed somewhere in the middle of the prototypicality scale.


If w is cancer and c is BORN UNDER X, then PPMI(w, c) is low (as is APPMI(w, c)).

aPPMI(c, w) is quite high, allowing us to highly score cancer when expanding {taurus, aries}.


With this alternate definition of context, we produce D C→V exactly analogous to M C→V from Equation 5.

if we define D V→C analogous to M V→C and use these matrices for expansion, we run into unintended consequences since expanding {evolution} provides not what things evolution is seen with, but rather those things that cooccur with what evolution co-occurs with.


On one hand, the AMR tree structure can be used directly with the TreeLSTM architecture described in Section 2, in which only node information is utilized to encode sentences into certain fixed-length embedding vectors.

since AMR provides rich information about semantic relations between nodes, the mTreeL-STM architecture is more applicable due to its capability of modeling edges in the tree.


(2017) introduced a different set of parameters for each edge-type in their LSTM-based approach for relation extraction.

to these works, our mTreeLSTM model incorporates relation information via a multiplicative mechanism, which we have shown is more effective and uses less parameters.


In the resulting ranked list, this method separates the unterminologic elements to the one end and a lot of terms with meaning shift to the other end: altbacken 'dowdy/stale', gedämpft 'low voice/steamed', Schnee, Fond 'fund/stock', Auflauf 'crowd/casserole', Form 'shape/(baking) mould' together with other cooking-related words like Spaghetti, Pfannkuchen 'pancake', Pommes 'French fries', Ananas 'pineapple', where the latter words have a lower original shift value.

other sub-technical terms like schlagen 'beat/whip (cream)', abschrecken 'discourage/chill ', binden 'tie/thicken (sauce)' are still among the unterminologic elements, most likely because they have rather varying contexts in GEN as well.


Such character-level features have been used for other NLP tasks (e.g.

sapkota et al., 2015) contextual features can also be incorporated into this framework (e.g.


For the local hinge loss above, we treat each label associated with the input x as being independent.

for the global hinge loss, we compare scores for subsets of labels.


So far, these relations have been treated as separate tasks, where a model trains and tests on one relation at a time while ignoring the rest.

we hypothesize that each relation may contain useful information about the others, and training on only one relation inevitably neglects some relevant information.


Rangarajan Sridhar (2015) propose learning a vector space representation of words in a corpus using Word2Vec, similarly to our approach, except without Tf-Idf weights, and then fitting a mixture of gaussians on the resulting vectors using standard EM.

the dimensionality of a Word2Vec representation is typically high (50-300 in practice), where gaussian mixtures are known to perform poorly (Krishnamurthy, 2011).


(2018a) in telling workers that the data are collected for a reading comprehension task for children, in order to get linguistically simple and explicit questions.

as mentioned above, we guide workers towards asking questions about target sentences rather than a complete text.


Since the target sentences are hidden in the texts, it can be expected that some questions cannot be answered from the text anymore.

the necessary information for finding an answer might be inferred from script knowledge, so workers were explicitly told that they might need commonsense to find an answer.


However, these works have extracted visual features from complete images, and have not examined how different kinds of visual information impact performance.

we construct multimodal models that differentiate between internal visual properties of the objects and their external visual context.


For instance, some concepts may be better characterised by their own (internal) visual properties and others by the (external) visual context, in which they appear.

existing neural multimodal semantic approaches use entire images to learn visual word representations, without differentiating between these two kinds of visual information.


However, existing neural multimodal semantic approaches use entire images to learn visual word representations, without differentiating between these two kinds of visual information.

we investigate whether differentiating between internal visual properties and external visual context is beneficial compared to learning visual representations from complete images.


This leads to a number of similarity calculations quadratic in the number of nodes, which can be prohibitive in case of very large graphs.

instead of this, the training datasets for path2vec can be constructed much faster by taking the graph structure into account.


Our work is closely related to these, as our model also predicts demographic variables.

in our model the goal of demographic prediction is primarily to learn representations that will make the document classifier more robust to demographic variations, rather than the end goal being demographic prediction itself.


Path information is also found to improve performance since paths help the model learn logical rules.

mining paths from a large knowledge graph is often computationally expensive since it involves performing a traversal through the graph.


Enthymemes are distinguished from other arguments by the fact they don't hold up by themselves, but are instead accepted on the basis of identification with a topos -this doesn't include arguments that are accepted despite being unsupported.

the terms enthymeme and topos will continue to be used here: this is partly for convenience, but also because once the context indicates that an enthymemetic argument is being made (such as a recognisable sugges-tion+motivation pattern like "Let's go left here, it's a shortcut"), an unsupported 'enthymeme', once accepted, can be used to establish a potential new topos (Breitholtz, 2015).


Most work focuses on declarative conditionals, the most common form by far.

conditional clauses are also used to form conditionalised questions and directives.


The mean-teacher (MT) approach produced the best performance.

our exploration of noise was limited in the previous study and hence is the focus of the current paper.


AMR parsing converts natural language sentences into such graphs.

evaluating a parser on new data by means of comparison to manually created AMR graphs is very costly.


To sum up, our observations for the out-ofdomain data stand in some contrast to what we observe for the in-domain data.

this outcome can be plausibly explained: assuming that the out-of-domain gold parses have some unfamiliar properties, a system that has never seen such parses cannot judge well whether they are gold or not.


(2016) published SPR2 which follows a similar annotation schema.

in contrast to SPR1, the new data set contains doubly annotated data from the web domain for 14 refined properties.


Besides our approach, only REI'15 does not depend on supplementary gold syntax for the dis-played results.

all of our models outperform REI'15's feature-based linear classifier in every property (+17 pp.


The significance test results summarized in Table 4 are unambiguous: for many proto-role properties, ensembling helps to improve performance significantly (SPR1: 14 18 cases; SPR2: 7 14 cases; significance level: p < 0.05).

for few cases, ensembling resulted in significantly worse predictions (SPR1: change of location; SPR2: change of location, instigated and partitive; significance level: p < 0.05).


Only ablating the argument-marker also causes a great performance loss (SPR1: -17.4, SPR2: -6.6).

when only the predicate marker is ablated, the performance decreases only slightly (SPR1: -1.2, SPR2: -1.7).


Our results suggest that different pretraining objectives give rise to differences in function word comprehension; for instance, we see that natural language inference helps understanding negation, and CCG supertagging helps recognizing meaningful sentence boundaries.

overall, we find that the observed differences are not always straightforwardly interpretable, and further investigation is needed to determine which specific aspects of pretraining tasks yield good representations of function words.


If g P is not used, a model should not be able to successfully perform NLI.

models without g P may achieve non-trivial results, indicating the existence of biases in hypotheses (Gururangan et al., 2018;Poliak et al., 2018;Tsuchiya, 2018).


First, we confirm that with AdvCls (Figure 1a), the hypothesisonly classifier (c hypoth ) is indeed trained to perform poorly on the task, while the normal NLI classifier (c NLI ) performs much better.

retraining a classifier on frozen hypothesis representations (c Hypoth , retrained) boosts performance.


Increasing the hyper-parameters further (up to values of 5), did not lead to substantial drops, although the results are slightly less stable across configurations (Appendix A).

the AdvDat method is very sensitive to large hyper-parameters ( Figure 2b).


that data were designed for use in social science research, not natural language processing research, and thus there were several challenges in working with the data as they were collected, including: • The comments were saved in PDFs, and the metadata referenced each comment by a number that was drawn (not typed) into the PDF beside the comment.


movie script writing, instructional videos, and public safety.

translating natural language text into animation is a challenging task.


The CARDINAL system also generates animations from input texts.

neither of the tools can handle complex sentences.


This works well for simple sentences having a single verb with one subject and one (optional) object.

the sentences in a screenplay are complicated and sometimes informal.


"Ben raises his head" is not supported).

there were some cases where the NLP component can be improved.


This approach was pioneered by Yarowsky and Ngai (2001), who used the resulting partial target annotation to estimate the parameters of an HMM.

due to the automatic nature of the word alignments and the POS tags, there will be significant noise in the projected tags.


Figure 2 shows that more than two thirds of all tokens in our training data are in Wiktionary.

there is considerable variation between languages: Spanish has the highest coverage with over 90%, while Turkish, an agglutinative language with a vast number of word forms, has less than 50% coverage.


The accuracy is typically lower for these words compared to the unambiguous words.

as the number of projected token constraints is increased from zero to 100+ observations, the ambiguous words are effectively disambiguated by the token constraints.


While the accuracy of filtering is more difficult to study, due to the lack of a gold standard tagging of the bitext, Figure 5 (position 0 on the xaxis) shows that search space pruning errors are not a major issue for most languages; on average the pruning accuracy is almost 95%.

for some languages such as Chinese and Czech the correct tag is pruned from the search space for nearly 10% of all tokens.


When using Wiktionary as a pruner, the upper bound on accuracy for these languages is therefore only around 90%.

figure 5 also shows that with some manual effort we might be able to remedy many of these errors.


Three edges will always suffice as a certificate of a 1-Endpoint-Crossing violation (two vertex-disjoint edges that both cross a third).

for a property like ill-nestedness, two nodes might have a least common ancestor arbitrarily far away, and so one might need the entire graph to verify whether the sub-trees rooted at those nodes are disjoint and ill-nested.


It performs cubic time parsing for arc-factored models (Eisner, 1996;McDonald et al., 2005a) and biquadratic time for higher order models with richer sibling and grandchild features (Carreras, 2007;.

for models with general non-local features, DP is inefficient.


This style of algorithm, especially when using the weaker end state validation, is closely related to reinforcement learning approaches (Branavan et al., 2009(Branavan et al., , 2010.

we differ on optimization and objective function, where we aim for minimal loss.


For example, Huang and Harper (2010) used annotated prosodic breaks as a kind of punctuation in a supervised PCFG, while prosodic breaks learned in a semi-supervised way have been used as features for parse reranking (Kahn et al., 2005) or PCFG statesplitting (Dreyer and Shafran, 2007).

to these methods, our approach observes neither parse trees nor prosodic annotations.


The Brent corpus is distributed via CHILDES (MacWhinney, 2000) with automatic dependency annotations.

these are not hand-corrected, and rely on a different tokenization of the dataset than is present on the transcription tier.


Unlike our approach, Boonkwan and Steedman do not automatically induce an appropriate inventory of lexical category, but use an extensive questionnaire that defines prototype categories for various syntactic constructions, and requires significant manual engineering of which POS tags are mapped to what categories to generate a languagespecific lexicon.

their performance degrades significantly when only a subset of the questions are considered.


Supervised models usually regard summarization as a classification or regression problem and use various sentence features to build a classifier based on labeled negative or positive samples.

existing supervised approaches seldom exploit the intrinsic structure among sentences.


Exiting Bayesian approaches label sentences or words with topics and sentences which are closely related with query or can highly generalize documents are selected into summaries.

lDA topic model suffers from the intrinsic disadvantages that it only uses word frequency for topic modeling and can not use useful text features such as position, word order etc (Zhu and Xing, 2010).


We use the French-English parallel corpus (approximately 1.2 million sentences) from the corpus of European parliamentary proceedings (Koehn, 2005) as the data on which pivoting is performed to extract the paraphrases.

the base paraphrase system is susceptible to large amounts of noise due to the imperfect bilingual word alignments.


In section 7.2, we show that profile enrichment can be effectively coupled with other methods of lexical expansion, such as a WordNet-based expansion and an expansion that utilizes Lin's distributional thesaurus.

we find that the paraphraser-based expansion benefits the most from profile enrichment, and attains better performance on our data than the alterantive expansion methods.


Brown's conclusion -that the language development of these three children advanced in similar stages at different times -is consistent with our predictions.

when we examine  Figure 3: Automatically detected preposition omissions in un-annotated utterances from four children over time.


a ↷ b denotes that a tree b is attached to a tree a. rated in standard graph-based models.

• to joint transition-based/graphbased approaches (Zhang and Clark, 2008;Bohnet and Kuhn, 2012) which require a large beam size and make dynamic programming impractical, our two-stage approach can integrate both models with little loss of efficiency.


The usual forest reranking algorithms (Huang, 2008;Hayashi et al., 2011) remove low quality hyperedges from large forests by using inside-outside forest pruning.

producing large forests and pruning them is computationally very expensive.


The graph-based approach employs Eisner and Satta (1999)'s algorithm where spurious ambiguities are eliminated by the notion of split head automaton grammars (Alshawi, 1996).

the arc-standard transition-based parser has the spurious ambiguity problem.


UBY contains pairwise sense alignments between a subset of these resources, and this work also presented a framework for creating alignments based on the similarity of glosses (Meyer and Gurevych, 2011).

it is not clear to what extent this approach can be applied to resources which lack this kind of information (see Section 3).


OmegaWiki (OW) is a freely editable online dictionary like WKT.

there do not exist distinct language editions as OW is organized in language-independent concepts ("Defined Meanings") to which lexicalizations in various languages are attached.


(2011) use a similar approach for aligning GN and WKT.

they use word overlap as a similarity measure and do not require a machine learning component as they align to the candidate sense with the highest similarity regardless of the absolute value.


In this respect, we found significant differences between the optimal configuration for individual resource pairs.

the general observation is that short paths (λ ≤ 3) lead to a very high precision, while paths longer than 10 do not increase recall significantly any more.


WP links are also unambiguous as they lead to a distinct article.

intuitively not all links in an article are equally meaningful.


This is also reflected in the ideal λ of 10-12: Many high-confidence edges allow long paths which still express a considerable degree of relatedness.

while the results for 1:n alignments are already good, restricting oneself to 1:1 alignments gives the best overall results as the precision can then be pushed towards 0.90 without hurting recall too much.


This issue of "misleading" WP links becomes even more prominent when the links from the full articles are used as edges (LM); while the increase in recall is relatively small the precision drops substantially.

using all possible links (SR+LM) allows us to balance out precision and recall to some extent, while yielding the same F-Measure as the SR configuration.


Solving this problem would require making the graph more dense, and especially finding ways to include isolated nodes as well.

this example also shows why the hybrid approach works so well: The correct WN sense "young plant or tree grown from a seed" was recognized by the similarity-based approach with high confidence.


In real MT systems, these problems are highly interdependent, a point we emphasized in class and at the end of each assignment-for example, that alignment is an exercise in parameter estimation for translation models, that model choice is a tradeoff between expressivity and efficient inference, and that optimal search does not guarantee optimal accuracy.

presenting each problem independently and holding all else constant enables more focused exploration.


Though there is some empirical work on competitive assignments in the computer science education literature (Lawrence, 2004;Garlick and Akl, 2006;Regueras et al., 2008;Ribeiro et al., 2009), they generally measure student satisfaction and retention rather than the more difficult question of whether such assignments actually improve student learning.

it might be feasible to answer such ques-tions in large, data-rich virtual classrooms offered by MOOCs.


For example, the system cannot tell whether born in is expressing a birthplace or birthdate relation until later in the derivation, when it combines with its arguments.

all the possible logical forms are identical except for the symbols used, which means we can produce a packed logical form capturing the full distribution over logical forms.


They do not require any knowledge of lexical semantics, meaning we can evaluate the formal component of our system in isolation.

we use the same representations as in our previous experiment, even though the clusters provide no benefit on this task.


This problem has been addressed in the context of robot direction following (Kollar et al., 2010;Tellex et al., 2011) and visual attribute learning (Matuszek et al., 2012).

this work is less semantically expressive than LSP and trained using more supervision.


(2012) learns only one-argument categories ("attributes") and requires a fully supervised initial training phase.

lSP models twoargument relations and allows for weakly supervised supervised training throughout.


SDM solves the L1-Loss SSVM problem using multiple updating policies, which is similar to our approach.

there are several important differences in the detailed algorithmic design.


The first iteration of DD has to perform roughly the same work as Forest, while subsequent iterations only need to re-parse the sentence with respect to the dual upare slightly different.

note that Merlo09 is an application of the system by .


During training, for each example, Model 1 aggregates features from all governors and objects even if they are possibly irrelevant, which may lead to a much bigger model in terms of the number of active weights.

for Model 2, Algorithm 1 uses the single highest scoring prediction of the latent variables, according to the current parameters, to refine the parameters.


For example, to translate the Chinese phrase " ", we can extract a rule (R2 in Figure 9) directly from the U-tree because the phrase " " is governed by a frontier node, i.e., node "VBD+RB".

because no node governs " " in the binary parse tree, we can only obtain a rule (R1 in Figure 9) with many extra nodes and edges, such as node CD in R1.


These word segmentation papers demonstrated both the power of the AG approach and the synergistic behavior that occurs when learning multiple levels of structure simultaneously.

the bestperforming grammars were selected using the same corpus that was used for final testing, and each paper dealt with only one language.


Based on these and similar results with other baseline systems, all results reported later for unsupervised models (AG and baseline) and AG Select were obtained using training sets of 50k words.

to the above models, the semisupervised AG does not always improve with more unlabelled data (see Figure 2) and in the limit, it will match the performance of the same grammar in the unsupervised setting.


The property is restrictive, meaning that it reduces the class of trees that can be generated.

we show that the restriction to head-split trees comes at no significant loss in coverage, and it allows parsing in time O.n 6 /, an asymptotic improvement of one order of magnitude over the algorithm by Gómez-Rodríguez et al.


For example, both WordNet (Miller, 1995) and Roget's 21st Century Thesaurus (thesaurus.com) present acceptable, great, and superb as synonyms of the adjective good.

a native speaker knows that these words represent varying intensity and can in fact generally be ranked by intensity as acceptable < good < great < superb.


Unfortunately, this approach often fails because many pairs lack order-based evidence even on the Web, as explained in more detail in Section 3.

our MILP jointly uses information from all relevant word pairs and captures complex interactions and inferences to produce intensity scales.


They attempt to automatically acquire a knowledge base of near-synonym differences via an unsupervised decision-list algorithm.

their method depends on a special dictionary of synonym differences to learn the extraction patterns, while we use only a raw Web-scale corpus.


IBM Models 1, 2 and the HMM model all assume the same translation model ( | ) .

they use three different distortion models.


The traditional approach for sequencebased models uses Expectation Maximization (EM) algorithm to estimate parameters.

in our model, it is hard to find an efficient way to sum over all the possible alignments, which is required in the E-step of EM algorithm.


transition-based (Nivre, 2008) and graph-based (McDonald, 2006;Torres Martins et al., 2009) have attracted the most attention in recent years.

grammarbased approaches rely on linguistic grammars (in either dependency or constituency formalisms) to shape the search space for possible syntactic analysis.


In previous work, data-driven, discriminative approaches have been widely discussed for Chinese dependency parsing.

various PCFG-based constituent parsing methods have been applied to obtain phrase-structures as well.


λ, the performance of sub-models is improved.

since the sub-models overlap more, the diversity of base models for ensemble will decrease and the final prediction accuracy may go down.


These models have proven very useful for short and mediumrange reordering and are among the most widely used in PSMT.

their coarse classification of reordering steps makes them unsuitable to predict long-range reorderings.


The features with significantly higher values in the TYPICAL articles are: It appears that the simple expectation that VERY GOOD articles contain more visual words overall does not hold true here.

the great writing samples have a higher degree of visual content in the beginning and ends of articles.


We consider the main word as related to the article topic, so we use the conditional probability of auxiliary given main word and not the other way around.

the conditional probability has no information about the frequency of the auxiliary word.


On a set of eighty multiple-choice synonym questions from the test of English as a foreign language (TOEFL), a distributional approach recently achieved 100% accuracy (Bullinaria and Levy, 2012).

it has been difficult to extend distributional semantics beyond individual words, to word pairs, phrases, and sentences.


6 The scores ranging from 51.1% to 57.0% are not significantly different from SuperSim's score of 54.8%, according to Fisher's exact test at the 95% confidence level.

supersim answers the sAT questions in a few minutes, whereas LRA requires nine days, and supersim learns its models automatically, unlike the hand-coding of Turney (2012).


(2013) approach is perhaps the most closely related of the recent approaches for distant supervision.

there are a number of key differences: (1) They impose a hard constraint on the proportion of true positive examples for each entity pair, whereas we jointly model relation extraction and missing data in the text and KB.


Concepts that are connected on the Slip Net can "slip" to one another and are hence considered related in a metaphor.

straightforward traversal on the Slip Net can become computationally impractical and the authors did not elaborate on the implementation details.


The situation is similar with stalkcompany pair in ABG2327.

for AN 81309, manipulate-life is considered rare combination and hence has low SA and CP scores and is deemed a metaphor while in reality it is a literal use.


The oracle for the easy-first system is non-deterministic and returns a set of correct transitions.

like the static oracle, it is correct only for configurations from which the gold tree is reachable.


6 Static oracles are usually described as rules over parser configurations, i.e., "if the configuration is X take transition Y", giving the impressions they are functions from configurations to transitions.

as explained here, these rules are only correct if the sequence of transitions is followed in its entirety.


deterministic oracles are more flexible than static oracles in that they allow for spurious ambiguity: they support the possibility of different sequences of transitions leading to the gold tree.

they are still only guaranteed to be correct on a subset of the possible configurations.


However, we also see evidence that the joint model may suffer from data sparsity, as in the case of Finnish, where a model that only predicts part-of-speech tags jointly with dependency relations achieve better accuracy on some metrics.

even in this case, the joint model has the best results on the joint evaluation metrics.


For the remaining domains, unseen words have a significant effect, both in terms of BLEU scores as well as fine-grained translation distinctions.

many of these words have multiple translations, and a system must be able to correctly select which one to use in a particular context.


To the best of our knowledge, there have been only a few studies to understand how domain shifts affect translation quality (Duh et al., 2010;Bisazza et al., 2011;Haddow and Koehn, 2012).

these start from different premises than this paper, and as a result, ask related but complementary questions.


Subs: Translated movie subtitles, available through the OPUS corpora collection (Tiedemann, 2009).

to the other domains considered, subtitles consist of informal noisy text.


FLORS with n = 250 has better OOV accuracies in 5 of 6 domains.

aLL accuracy for FLORS with n = 500 is better in the majority of domains.


Surprisingly, OOV accuracies benefit a bit in four domains if we reduce l from 2 to 1 (lines 13 vs 5).

aLL accuracy consistently drops in all six domains.


OOV tagging may benefit from correct previous tags because the larger left context that is indirectly made available by left-to-right tagging compensates partially for the lack of information about the OOV word.

to standard approaches to POS tagging, the FLORS basic representation does not contain vocabulary indices.


In our experiments, we also did not find consistent improvements when we incorporated sequence constraints (Table 2, line  14).

there may be languages and applications involving long-distance relationships where local-context classification is suboptimal.


The results are in line with SANCL results: FLORS beats the baselines on ALL and OOV accuracies.

if we build the NN bias into our model by simply replacing all NNP tags with NN tags, then accuracy goes up by 4% on ALL and by almost 20% on OOV.


In terms of the basic classification setup, our POS tagger is most similar to the SVM-based approaches of Giménez and Màrquez (2004) and C&P.

we do not use a left-to-right approach when tagging sentences.


The CRF ptt and CLIPER h methods successfully labeled these two examples correctly, but failed to produce the correct label for the example in Figure 1.

a model trained with the CLIPER s method does correctly label both entities in Figure 1, demonstrating the merits of the soft projection method.


Overall, the models including phonotactic cues perform better than those that do not rely on phonotactics.

the overall gain contributed by stress to the colloc3-phon baseline is smaller, alsignificantly more accurate (p < 0.05) than the baseline models except when evaluating on the training data for the Korman and Brent corpora.


While phonotactics by itself appears to be a powerful cue, yielding a noticeable 4-5% improvement over the colloc3-nophon baseline, the learner seems to require at least around 500 utterances before the colloc3-phon model becomes clearly more accurate than the colloc3-nophon model.

even for only 100 utterances stress cues by themselves provide a 3% improvement to the colloc3-nophon model, indicating that they can be taken advantage of earlier.


For instance, state of the art systems obtain tagging accuracies over 97% for part-of-speech (POS) tagging on the English Penn Treebank.

learning accurate taggers without labeled data remains a challenge.


They also use an auto-supervision technique to smooth counts learnt from EM onto new words encountered during testing.

we do not apply any such technique for unknown words and allow them to be mapped uniformly to all possible tags in the dictionary.


It is encouraging to note that the new system achieves this performance without using any of the carefully-chosen heuristics employed by the previous method.

we do note that some of these techniques can be easily combined with our method to produce further improvements.


This is required to permit their model minimization step to be run in a feasible manner.

the new approach DMLC (method 3) scales better even when the data/label sizes are large, hence it can be run with the full data using the original model minimization formulation (rather than a two-step heuristic).


The dynamic oracles developed by Goldberg and Nivre (2013) for arc-decomposable systems are based on local properties of computations.

our novel dynamic oracle algorithms rely on arguably more complex structural properties of computations, which are computed through dynamic programming.


In order to simplify the presentation we also assume σ In the second phase we compute the loss of an input configuration using a two-dimensional array T , defined as in §4.3.

because of the way transitions are defined in the LR-spine parser, we now need to distinguish tree fragments not only on the basis of their roots, but also on the basis of their left and right spines.


We therefore regard these as parsing errors that we will train our model to avoid.

avoiding the Left-Arc transitions would require the parser to predict that the head is disfluent when it has not necessarily seen any evidence indicating that.


This could not be applied to the other two disfluency detectors, so we cannot test those differences for significance.

we note that the 20 samples for our disfluency detector ranged in accuracy from 83.3-84.6, so we doubt that 0.2% mean improvement over the Qian and Liu (2013) result is meaningful.


If it turned out that almost all of the former leftward children of disfluent words are subsequently marked as disfluent, there would be little point in returning them to the stack -we could just mark them as disfluent in the original Edit transition.

if they are almost all marked as fluent, perhaps they can just be attached as children to the first word of the buffer.


For example, AJCC Staging Codes (discussed in Section 2) are eventive only in the sense of the code being assigned to a tumor at a given moment in the patient's care.

they are of such critical importance and informative value to doctors that we have chosen to annotate them specifically so that they will show up on the patient's timeline in a clinical setting.


Because it requires domain-specific knowledge, although present in the specification, Permanence is not currently annotated.

annotators are trained on the basic idea and told about subsequent axiomatic assignment.


Likewise, if one annotator marked A BEFORE B and B BEFORE C and the other annotator did not, the inferred A BEFORE C is not counted.

if one annotator did explicitly mark A BEFORE C, then an equivalent inferred TLINK would be used to match it.


the expected output is positive (Canon 7D,sensor) and preferred(Nikon D7000, Canon 7D, textitprice).

constructing a fully annotated training corpus for this task is labor-intensive and requires strong linguistic background.


that is how the characters of Superman and Lex Luthor should be.".

our model minimizes the overlapping features by assigning them to the most likely relation candidates.


The edge co-occurrence features do not play an important role in our model since the number of co-occurred sentiment-oriented relations in the sentences with contrast conjunctions like "but" is small.

we found that allowing the co-occurrence of any sentiment-oriented relations would harm the performance of the model.


As an interesting finding, without being informed about the order of baseline and proposed method, participants reported that transcribing according to the proposed segmentation seemed harder, as they found the baseline segmentation more linguistically reasonable.

this perceived increase in difficulty did not show in efficiency numbers.


In the case of topic models, the model will be related to dynamic topic models (Blei and Lafferty, 2006) augmented by context features, and the learning procedure in §4 can be used to perform online learning of dynamic topic models.

our model captures longer-range dependencies than dynamic topic models, and can condition on nonlinguistic features or metadata.


We can further improve the bound by updating the variational parameters for timestep 1 : t − 1, i.e., µ 1:t−1 and σ 1:t−1 , as well.

this will require storing the texts from previous timesteps.


Similarly, classifying whole images into a fixed set of classes also achieves very high performance Krizhevsky et al., 2012).

similar to words, objects in images are often seen in relationships with other objects which are not adequately described by a single label.


This can be rewritten to show the similarity to the DT-RNN as h = f (W l1 c 1 + W r1 c 2 ).

there are several important differences.


Note first that in previous RNN models the parent vectors were of the same dimensionality to be recursively compatible and be used as input to the next composition.

our new model first maps single words into a hidden space and then parent nodes are composed from these hidden vectors.


As shown in Section 4.1.1, further speedup could be achieved with only a small performance degradation by considering only the textual neighborhood as source of contextual evidence.

the two major goals that we believe the aligner achieves to the greatest extent are replicability and generalizability.


The aim of EL is to discover mentions of entities within a text and to link them to the most suitable entry in a reference knowledge base.

in contrast to WSD, a mention may be partial while still being unambiguous thanks to the context.


Notwithstanding these differences, the tasks are similar in nature, in that they both involve the disambiguation of textual fragments according to a reference inventory.

the research community has so far tackled the two tasks separately, often duplicating efforts and solutions.


However, supervised approaches require huge amounts of annotated data (Zhong and Ng, 2010;Shen et al., 2013;Pilehvar and Navigli, 2014), an effort which cannot easily be repeated for new domains and languages, while unsupervised ones suffer from data sparsity and an intrinsic difficulty in their evaluation (Agirre et al., 2006;Brody and Lapata, 2009;Manandhar et al., 2010;Van de Cruys and Apidianaki, 2011;Di Marco and Navigli, 2013).

knowledge-based approaches are able to obtain good performance using readily-available structured knowledge (Agirre et al., 2010;Guo and Diab, 2010;Ponzetto and Navigli, 2010;Miller et al., 2012;Agirre et al., 2014).


In their approach they disambiguate each word in a sentence independently by exploiting the context in which it occurs.

this approach is local in that it lacks a collective notion of coherence between the selected Wikipedia pages.


For instance, given the text fragment Major League Soccer, we identify and disambiguate several different nominal and entity mentions: Major League Soccer, major league, league and soccer.

to EL, we link not only named entity mentions, such as Major League Soccer, but also nominal mentions, e.g., major league, to their corresponding meanings in the knowledge base.


We rank second on both datasets after IMS.

the differences are not statistically significant.


The crosslingual models described in the previous section do not use any corpus information from the target language: As previously discussed, our rationale is to make the methods as widely applicable as possible.

this assumption may be too cautious as more corpora and parsers continually become available.


2, it predicts min(f (w 1 ), f (w 2 )).)

wordbased DSMs perform quite well, particularly the dimensionality-reduced model (BOW PCA).


This appears to show the usual trade-off between hand-constructed knowledge and automatically acquired knowledge (Gildea and Jurafsky, 2002).

we can similarly bias our DMs towards accuracy with the aid of a simple frequency filter that only permits predictions for items where all involved lemmas occur more frequently in the German corpus than some threshold.


To retrieve tweets for a given entity (e.g., a product or an organization), a straightforward approach is to formulate a keyword query.

simple keyword matching is largely ineffective, since keywords are often ambiguous.


Compared to "Optimal", our smoothing is better on LP but not on E2E.

smoothing is still useful, as "Optimal" depends a lot on choosing the right bin size, while smoothing is less sensitive to its parameters.


While linguistic information overall effectively reflects the meaning of all concept types, we show that features encoding syntactic patterns are only valuable for the acquisition of abstract concepts.

perceptual information, whether directly encoded or propagated through the model, plays a more important role in the representation of concrete concepts.


In principle, all training edges could be used as extra evidence for testing (i.e., all blue edges may be made solid yellow).

in our experiments, we keep the training and testing sets fully disjoint.


As large amounts of labelled data are unlikely to be made available, recent work has explored using unlabelled data to improve parser lexicons (Thomforde and Steedman, 2011;Deoskar et al., 2011;Deoskar et al., 2014).

existing work has failed to improve the overall accuracy of state-of-the-art supervised parsers in-domain.


(2010)'s sequence models-but greater improvements may be possible with the combined approach we introduce here, which allows the model to both tune the embeddings and exploit sequence information.

tagging with this model was considerably slower than the neural network (again, due to the cost of decoding), so we used the neural network architecture in the remaining experiments.


This evaluation is somewhat indirect, as the parser does not make use of the supertagger probabilities for categories, but instead simply uses it to prune the search space.

we show that better pruning leads directly to better parsing accuracies.


Results show slightly lower performance than C&C in domain, but higher performance out of domain.

they are substantially lower than results using the Turian-50 embeddings.


Both systems show dramatically lower performance on these cases-the embeddings supertagger does not use POS features, but POS errors are likely to represent generally difficult examples.

the embeddings supertagger is almost 15% more accurate on this subset than the C&C supertagger, and with a relaxed beam reaches 96% accuracy, showing the advantages of avoiding a pipeline approach.


The accuracy of the embeddings supertagger converges at just 80%, suggesting that our model has overfit the labelled data.

it still outperforms the C&C supertagger by 22% with a beam allowing 2 tags per word.


(2009) to improve CCG parser performance on Wikipedia.

it caused a decrease in performance on the in-domain data, and our method achieves better performance across all domains.


(2011b) showed that the success of the sub-gradient algorithm is strongly tied to the ability of finding a good decomposition, i.e., one involving few overlapping slaves.

for generalized higher order graphical models, a lightweight decomposition is not at hand and many overlapping slaves may be involved.


Suppose c =Little York village is such a word subsequence, then among |S| 3 possible assignments of c, only the one that labels c =Little York village as a location name has non-zero weight.

the reparameterization may add ψ c to the other |S| 3 − 1 assignments, yielding many new patterns.


Approaches in this direction could be ideal for various practical applications such as image description for the visually impaired.

it is not clear whether the semantic expressiveness of these approaches can eventually scale up to the casual, but highly expressive language peo- A"cow!standing!in!the!


During learning, they aggregate sentences containing the target entities, ignoring richer contextual information.

we learn from each individual sentence taking into account all entities present, their relations, and how they interact.


Alternatively, we could achieve a non-linear response using a natively nonlinear model such as a SVM (Wang et al., 2007) or RankBoost (Sokolov et al., 2012).

mT is a structured prediction problem, in which a full hypothesis is composed of partial hypotheses.


Unlike models that rely primarily on a large number of sparse indicator features, state-of-the-art machine translation systems rely heavily on a small number of dense real-valued features.

unlike indicator features, real-valued features may benefit from non-linear transformations to allow a linear model to better fit the data.


In practice, both of these conditions are typically violated to a certain degree.

if we modify our feature set such that some lower-ranked higher-quality hypothesis can be separated from all higher-ranked lower-quality hypotheses, then we can improve translation quality.


Clearly, having alternative answers is the right approach as there are typically multiple ways to correct an error.

because the alternatives are based on the error analysis of the participating systems, the revised set may be biased .


Indeed, many teams follow this approach.

an important issue in the CoNLL task is the difference between the training and test sets, which has impact on the selection of the training set -the large Web1T has more coverage and allows for better generalization.


The difference in collection time reflects an important difference in the current resources: while crowdsourcing has established platforms with on-demand workers, no central platforms exist for games with a purpose with an analogous pool of game players.

although the current games were released in a limited fashion, later game releases to larger venues such as Facebook may attract more players and significantly decrease both collection times and overall annotation cost.


Many noun lemmas had easily-recognizable associated images, so higher-speed game play may still be accurate.

verbs were more general (e.g., "decide," "concern," and "include"), which required more abstract thinking in order to recognize an associated picture; as the game speed increased, players were not able to identify these associated pictures as easily, causing slightly decreased performance.


(2010), the inside-outside algorithm dominates the runtime of every iteration, both for sampling and variational inference.

unlike MCMC, variational inference is highly parallelizable and requires fewer synchronizations per iteration (Zhai et al., 2012).


11 tagging (Toutanova and Johnson, 2008) or dialogue modeling (Zhai and Williams, 2014).

expressing topic models in adaptor grammars is much slower than traditional topic models, for which fast online inference (Hoffman et al., 2010) is available.


Probabilistic modeling is a useful tool in understanding unstructured data or data where the structure is latent, like language.

developing these models is often a difficult process, requiring significant machine learning expertise.


In this case, an NER system based on surface statistics alone would likely predict that Freddie Mac is a PERSON.

the Wikipedia article for Freddie Mac is unambiguous, which allows us to fix this error.


Consider any k ∈ N. Observe that f 2 (aa k ) = f 2 (ab k ) = ab k and so the outputs share the same suffix of length (k − 1).

tails Similarly, consider f 3 in Figure 2.


For example, the expression "fundamentally unfair and unjust" as a whole is labeled as an opinion expression with negative polarity.

the subexpression "unjust" can be also viewed as a negative expression but it is not annotated as an opinion expression in this example (as MPQA does not consider nested opinion expressions).


The absolute number of false length answers is higher for weaker students.

the proportion of false length answers relative to all false answers is higher for stronger (0.45) than for weaker (0.32) students.


Even the reduced set of selected features comprises features from all introduced dimensions which shows that our conclusions drawn from the data analysis led to a very suitable model.

the features measuring the difficulty of the solution and the probability of the micro context seem most relevant.


Our model has been developed for the difficulty prediction of English C-tests.

it can also be generalized to other languages and to test variants of reduced redundancy testing.


The rating task benefits the most from the use of neighbor rank.

neighbor rank has very little effect for the TOEFL task in a reduced setting, where its high computational complexity is clearly not justified; the improvement on the AP clustering dataset is also fairly small.


The datasets developed in the recent dialog state tracking challenge (Henderson et al., 2014) also consist of dialogs between a user and a tourism information system.

the task is easier since only three entity types are considered (restaurant, coffeeshop and pub), a slot-filling MRL is used and the argument slots take values from fixed lists.


1 is readable and easy to annotate with.

it is not ideal for experiments, as it is difficult to compare MR expressions beyond exact match.


5, is defined as the sum of the false positive and false negative atomic predictions used to calculate precision and recall and, since it ignores the alignment between tokens and nodes, it cannot assess node prediction actions.

we can use it under DAGGER to learn a node prediction classifier together with the classifiers of the other stages.


In terms of direct comparisons with existing work, the goal of this paper is to introduce the new corpus and provide a competitive first attempt at the new semantic parsing task.

we believe it is non-trivial to apply existing approaches to the new task, since, assuming a decomposition similar to that of Sec.


A different approach is to use dialogs between a system and its users as proposed by Artzi and Zettlemoyer (2011) using the DARPA communicator corpus (Walker et al., 2002).

in that work utterances were selected to be shorter than 6 words and to include one noun phrase present in the lexicon used during learning while ignoring short but common phrases such as "yes" and "no"; thus it is unclear whether it would be applicable to our dataset.


(2013) obtain good parsing performance by building compositional representations from word vectors.

to combat the computational complexity of the full parsing scenario, they rely on a probabilistic context-free grammar to prune search space.


While this limits the contribution of the word vectors to the learned model to one dimension, attempts to use more dimensions in the SVM were unsuccessful.

12 the compositional models better capture the full dimensionality of the word vectors.


Such systems cannot be easily evaluated in our setting which allows multiple candidate heads.

our full model exploits contextual information that is not available in the RRR dataset.


From election results, number of casualties in accidents, to changes in stock prices, textual representations of quantities are extremely important in communicating accurate information.

relatively little work in Natural Language Processing has analyzed the use of quantities in text.


This local CR step may produce errors (e.g., incorrect chaining of mentions or omissions) which propagate to the later stages.

improving intra-document CR is orthogonal to our problem and thus out of the scope of this paper.


The benefit is twofold: gaining efficiency and improving accuracy, as two different entity types would not co-refer in reality.

the risk is that two differently tagged mention groups might actually refer to the same entity, with at least one tag being incorrect.


No prior methods report F1 performance figures for this large dataset.

the factor graph based approach of (Singh et al., 2010) measures the mention co-reference accuracy for a sample of 1,000 documents.


The fastest of them, the simple k-means method, had processed only about 5% of the data at this point (needing about 400 hours on extrapolation).

cROcS, using sample-based spectral clustering or graph par-titioning, needed about 19.6 hours for the 220,000 documents.


Baron and Freedman (2008) proposed a CCR method involving full clustering coupled with statistical learning of parameters.

this method does not scale to large corpora making it unsuitable for Web contents.


A popular framework for imposing these constraints has been integer linear programming (ILP), wherein the inference problem is specified declaratively (Punyakanok et al., 2008).

iLP-based inference methods often rely on generic off-the-shelf solvers that fail to exploit problem-specific structure (Martins et al., 2011).


Most previous work assumed that each instance is best labeled with a single sense, and therefore, that each instance belongs to exactly one sense cluster.

recent work Jurgens, 2013) has shown that more than one sense can be used to interpret certain instances, due to context ambiguity and sense relatedness.


One approach would be to run LDA on the instances for an ambiguous word, then simply interpret topics as induced senses (Brody and Lapata, 2009).

while sense and topic are related, they are distinct linguistic phenomena.


Without further information, the texts might discuss any of a number of possible topics.

if the sense of cold is that of cold ischemia, then the most probable topics would be those related to organ transplantation.


While not provided for the SemEval task, it is reasonable to assume this larger context in many real-world applications, such as information retrieval and machine translation of documents.

in other settings, the corpus may only have a single sentence containing the target word (e.g., search queries or machine translation of sentences).


This could be because (4) and (5) both relate to child development, and therefore LDA considers them as sharing the same topic.

topic is not the same as sense, especially when larger contexts are available.


Another approach is inspired by the observation that each local context token is treated equally in terms of its contribution to the sense.

our intuition is that certain tokens are more indicative than others.


We find that the original model provides less distinguishable senses, as it is difficult to derive separate senses from these top terms.

senses learned from the model with weighted similarities are more distinct.


(2013) demonstrated orders of magnitude speedup in exact pattern matching with suffix arrays, the algorithms at the heart of on-demand extraction (Callison-Burch et al., 2005;Zhang and Vogel, 2005).

some popular translation models use "gappy" phrases (Chiang, 2007;Simard et al., 2005;Galley and Manning, 2010), and the GPU algorithm of He et al.


Since most patterns are infrequent, allocating threads this way minimizes work.

very large corpora contain one-gap patterns for which both subpatterns are frequent.


Construction of static data structures for on-demand grammar extraction is a one-time cost given a corpus T .

under a streaming scenario where we might receive incremental changes to T as new training data become available, we need to update the data structures appropriately.


We specifically follow Gärdenfors (2000) and Jäger (2010) in assuming that color categories are convex regions in an underlying color space, and are not just determined by prototypical color values, such as in Andreas and Klein (2014).

unlike previous grounded semantics, we do not assume that words name categories unequivocally.


In any utterance, yellowish green fits only those Hue values that are above a minimum threshold τ Lower and below a maximum threshold τ Upper .

it is uncertain which thresholds a speaker will use.


the differences in the table, though numerically small, are significant (by Binomial test) at p < .02 or less.


Since the assumptions largely rely on co-occurrence, previous unsupervised approaches tend to confuse correlated but semantically different phrases during extraction.

to this, our work largely avoids these errors by exploiting the temporal negation heuristic in parallel news streams.


As in Zhang & Weld (2013), we group parallel sentences sharing the same argument pair and date in a structure called a NewsSpike.

we include all sentences mentioning the arguments (e.g.


Even the unsupervised version which uses constraints under a maximum-entropy criterion without any featurebased model, outperforms fully-supervised featurebased models in detecting challenging low frequency categories across sections.

this approach still requires substantial human effort in constraint generation.


Given the prohibitive cost of manual annotation, unsupervised and minimally supervised techniques such as clustering (Kiela et al., 2014) and topic modeling (Varga et al., 2012;O Séaghdha and Teufel, 2014) are highly important.

the performance of such approaches shows a large room for improvement.


This choice is data dependent, and the standard choice is the document level.

for scientific articles the paragraph level is a better choice, because a paragraph contains only a small subset of information structure categories while in a full article categories are more evenly distributed.


The models above all have direct anologues in web links to Wikipedia articles.

web links are a comparatively noisy source.


Web link combinations perform at 97% of Wikipedia combinations on average.

creating a KB as rich as Wikipedia represents an estimated 100 million hours of human effort (Shirky, 2010).


As expected, Web link popularity and coherence models trail Wikipedia models.

web link context models outperform wikipedia context models by 7 to 10 points.


Most state-of-the-art systems today produce morphological analysis based only on orthographic patterns.

we propose a model for unsupervised morphological analysis that integrates orthographic and semantic views of words.


However, these algorithms do not capture distributional properties of morphemes that are critical to the success of current state-of-the-art algorithms.

we utilize a single statistical framework that seamlessly combines both sources of information.


They use techniques like Contrastive Estimation, sampling and simulated annealing.

our formulation does not result in such a large search space.


We find that 56% of errors in Arabic involve a singlecharacter affix, which is much higher than the 24.6% that involve a two-letter affix.

25% of errors in English are due to single character affixesaround the same number as the 24% of errors due to two-letter affixes.


6 4 Optimizing Syntactic N-grams N-gram based metrics such as BLEU (Papineni et al., 2002) are still predominantly used to optimize the log-linear parameters of SMT systems, and (to a lesser extent) to evaluate the final translation systems.

n-gram metrics are not well suited to measure fluency phenomena with string-level gaps, and there is a danger that BLEU underestimates the modelling power of dependency language models, resulting in a suboptimal assignment of loglinear weights.


The prediction of the dependency topology and labels through P l means that the full RDLM has the highest perplexity of all models.

it also strongly prefers the human reference text over the baseline SMT output.


On the one hand, this has implications when evaluating new model components: using an objective function that cannot capture the impact of a model component can result in false negatives because the model component will not receive an appropriate weight, and the model may thus seem to be of little use, even in a human evaluation.

it is an important finding for the evaluation of objective functions: the performance of an objective function is tied to the power of the underlying model.


Building off this work, a number of LS models using either words or other syntactic and semantic structures have been proposed for QA (Echihabi and Marcu, 2003;Soricut and Brill, 2006;Riezler et al., 2007;Surdeanu et al., 2011;Yao et al., 2013;Jansen et al., 2014), or related tasks, such as semantic textual similarity (Sultan et al., 2014).

all of these models fall into the class of first-order models.


As expected, the personalized PR algorithm performs better than PR U .

their performance drops for the third-order models (rows 26 and 29), whereas our models continue to improve in their third-order configuration.


We therefore do not apply w+c to PPMI in this study.

eigenvalue Weighting (eig) As mentioned in Section 2.2, the word and context vectors derived using SVD are typically represented by (equation 1): this is not necessarily the optimal construction of W SVD for word similarity tasks.


An important factor in evaluating distributional methods is the size of corpus and vocabulary, where larger corpora tend to yield better representations.

training word vectors from larger corpora is more costly in computation time, which could be spent in tuning hyperparameters.


(2014) show a variety of experiments in which GloVe outperforms SGNS (among other methods).


Word embeddings learned by neural language models (Bengio et al., 2003;Collobert and Weston, 2008;Mikolov et al., 2013b) have been successfully applied to a range of tasks, including syntax (Collobert and Weston, 2008;Turian et al., 2010;Collobert, 2011) and semantics (Huang et al., 2012;Socher et al., 2013b;Hermann et al., 2014).

phrases are critical for capturing lexical meaning for many tasks.


We could directly use the word embeddings learned by neural language models.

our experiments show that those word embeddings are often not suited for FCT.


For longer phrases, matrix multiplication can be applied recursively according to the associated syntactic trees (Socher et al., 2010).

because the input of the model is the concatenation of word representations, matrix transformations cannot capture interactions between a word and its contexts, or between component words.


For example, Cheung and Penn (2013) apply contextualization to word compositions in a generative event extraction model.

this is an indirect way to capture interactions (the transformations are still unaware of interactions between components), and thus has not been a popular choice for composition.


Regard-ing precision, voting techniques show the highest scores.

the high precision is accompanied by a reduction in recall for both voting methods.


Furthermore, the modeltheoretic semantics provided by such parsers have the potential to improve performance on other tasks, such as information extraction and coreference resolution.

a closed predicate vocabulary has inherent limitations.


Open predicate vocabularies have the potential to capture subtle semantic distinctions and achieve high coverage.

we have yet to develop compelling approaches to compositional semantics within this paradigm.


Consequently, we evaluate our system on a new data set with unconstrained language.

we do compare our approach against manually-annotated Freebase queries on our new data set (Section 7.5).


6 In terms of overall MAP, Freebase outperforms our approach by a fair margin.

this initial impression belies a more complex reality, which is shown in Table 5.


Like our approach, this system automatically-generates query/answer pairs for training.

this system, like all Freebase semantic parsers, uses a closed predicate vocabulary consisting of only Freebase predicates.


We will refer to this syntactic parsing model as OriSynModel, and the semantic parsing model as OriSemModel.

these two models do not adapt well to the target domain.


Because of the poor performance of the OriSynModel on the target domain, the extracted features on unlabeled data contains some noise.

experiments show that our DBN model can still learn a good LFR despite the noise in the extracted features.


Text simplification often involves splitting one sentence into multiple sentences, which is likely to require discourse-level changes such as introducing explicit rhetorical rela-tions.

previous research that uses Simple-Normal Wikipedia largely focuses on sentence-level transformation, without taking large discourse structure into account.


Most state-of-the-art systems consist of all or a subset of these three components.

the popular human evaluation criteria (grammaticality, simplicity and adequacy) do not explain which components in a system are good or bad.


Simple English Wikipedia played a vital role in inspiring simplification approaches based on statistical machine translation.

it has so many drawbacks that we recommend the community to drop it as the standard benchmark set for simplification.


With small values of T , for example T ≤ 7 , using Google word vectors produces better scores than using Stanford word vectors on the small N20small dataset of normal texts and on the short text TMN and TMNtitle datasets.

the opposite pattern holds on the full N20 dataset.


Google word2vec vs. Stanford glove word vectors: On the small N20short and N20small datasets, using the Google pre-trained word vectors produces higher clustering scores than using Stanford pretrained word vectors.

on the large datasets N20, TMN and TMNtitle, using Stanford word vectors produces higher scores than using Google word vectors when using a smaller number of topics, for example T ≤ 20.


While a variety of metrics have been used for evaluation, there is little agreement on which one to use.

this detail is quite important, since the choice of metric strongly biases the results.


WikipediaAt first glance, KB:American football seems the gold-standard link.

there is another entity KB:College football, which is clearly also, if not more, appropriate.


Even without entity semantics, DISCO2 significantly outperforms these competitive models from prior work.

the surface features remain important, as the performance of DISCO2 is substantially worse when only the distributed representation is included.


These approaches essentially compute a distributional representation in advance, and then use it alongside other features.

our approach follows more recent work in which the distributed representation is driven by supervision from discourse annotations.


(2014), which also uses the notion of model from Formal Semantics to recognize entailment based on denotations of words and phrases.

since the denotations in their approach are ultimately derived from humangenerated captions of images, the method does not generalize to concepts that are not exemplified in the training database.


BDSM reaches the most consistent results with predict vectors, where it performs particularly well on BLESS, and not dramatically worse than with count vectors on LEDS.

predict vectors have a negative overall impact on SVM in 3 over 4 tasks.


SVM with count vectors is better than BDSM on LEDS-core (not significant) and LEDSdir (p<0.05).

bDSM with predict vectors is better than SVM on bLESS-coord (p<0.001) and bLESS-mero (p<0.001).


Other variants of this approach include McInnes and Goldwater (2011) who compute similarity incrementally, and Aimetti (2009) who integrates a simplified, symbolic representation of visual information associated with each utterance.

word Segmentation to spoken term discovery, models of word (or morpheme) segmentation start from unsegmented strings of symbols and attempt to identify subsequences corresponding to lexical items.


A series of studies using the framework has shown that including such additional structure can markedly improve lexicon discovery (Johnson et al., 2006;Johnson and Demuth, 2010;Johnson and Goldwater, 2009b;Johnson, 2008a;Börschinger and Johnson, 2014;Johnson, 2008b).

unsupervised Lexicon Discovery to models of spoken term discovery and word segmentation, our model addresses the problem of jointly inferring phonological and lexical structure directly from acoustic input.


While there are a few cases, such as /iy l iy/, where the model stored a sequence of phones which does not map directly onto a linguistic unit such as a syllable, morpheme, or word, most stored units do correspond to intuitively plausible linguistic constituents.

like sub-word-like units, there is variability in the scale of the linguistic structure which they capture.


For example, the sequences /ax bcl ax l/ and /ey sh ax n/ correspond to the productive suffix -able and suffix combination -ation (O'Donnell, 2015).

the model also stores lexical units which correspond to words (e.g., powerful) and multi-word collocations (e.g., the arab muslim world).


The aim of automatic coreference resolution is to predict a clustering of mentions such that each cluster contains all mentions that are used to refer to the same entity.

most coreference resolution models reduce the problem to predicting coreference between pairs of mentions, and jointly or cascadingly consolidating these predictions.


Furthermore, in principle we can take the structure into account.

we are not aware of any approaches which go beyond (variations of) Hamming loss (Hamming, 1950).


In fact, generative linguists traditionally posit that each morpheme of a language has a single representation shared across all contexts (Jakobson, 1948;Kenstowicz and Kisseberth, 1979, chapter 6).

this string is a latent variable that is never observed.


This deletes unpronounced symbols such as syllable boundaries, and translates the phonemes into an observed phonetic, articulatory, or acoustic representation.

our present paper simply merges layers 3 and 4: our layer 3 does not currently make use of any unpronounced symbols (e.g., syllable boundaries) and we observe it directly.


Our approach captures the unbounded generative capacity of language.

to Dreyer and Eisner (2009) (see section 8), we have defined a directed graphical model.


On such "loopy" graphical models, exact inference is in general uncomputable when the random variables are string-valued.

dreyer and Eisner (2009) showed how to substitute a popular approximate joint inference method, loopy belief propagation (Murphy et al., 1999).


we actually used a (0,2,1) topology, to allow features that also look at the "upcoming" input phoneme that immediately follows the edit's input (/@/ in Figure 2).


We considered an additional ablation study to determine whether the learned URs were helpful.

we did not come up with a plausible  Figure 4: Results on the CELEX datasets (1000 word types) at 4 different training set sizes N .


Their evaluation is quite different, as their aim is actually to recover underlying words from phonemically transcribed child-directed English utterances.

nothing in their model distinguishes words from morphemes-indeed, sometimes they do find morphemes instead-so their model could be used in our task.


For inference, they invert the finite-state S θ like us to reconstruct a lattice of possible UR strings.

they do this not within BP but within a block Gibbs sampler that stochastically reanalyzes utterances one at a time.


(2006) employed a combination of GPs and graph kernels for reinforcement learning.

unlike our approach, they did not attempt model selection, evaluating only a few hyperparameter values empirically.


Such parsers are traditionally trained as if the inference had been exact.

1 we train the parser such that the approximate system performs well on the final evaluation function.


For certain choices of higher-order factors, polynomial time is possible via dynamic programming (McDonald et al., 2005;Carreras, 2007;Koo and Collins, 2010).

bP will typically be asymptotically faster (for a fixed number of iterations) and faster in practice.


We define the annealed risk as the expected loss of that decoder: As T → 0 ("annealing"), the decoder almost always chooses the MBR parse, 7 so our risk approaches the loss of the actual MBR decoder that will be used at test time.

as a function of θ, it remains differentiable (though not convex) for any T > 0.


While topic modeling approaches are capable of exploiting both labeled and unlabeled data, inference, typically based on sampling, can be extremely slow when the number of entities is very large.

our inference algorithm is simple to implement, parallelizable, scalable, and easily extended to the semi-supervised setting.


For each of the test corpora, we evaluate on the exact same set of mentions as previous work that we compare against in Tables 2 and 3.

our setup differs from existing systems in two important ways.


Thus Plato detects and links both nominal and named mentions.

following most existing work, we only evaluate the resolution of gold named entity mentions to either KB or NIL.


Since the supervised model is trained only on Wikipedia, the mention prior component dominates, and the system incorrectly infers that George Harrison refers to the Beatle.

once we retrain with unlabeled data, the model learns several new relevant features for the correct entity, including Rouse Fenway Racing and Wii, and gaming.


Imagining customers as event mentions and the restaurant as a document, each mention can either refer to an antecedent mention in the document or no other mentions, starting the description of a new event.

the coreference relations may also exist across documents -the same event may be described in multiple documents.


While earlier OIE approaches relied mostly on dependencies at the level of surface text (Etzioni et al., 2008;Fader et al., 2011), more recent work has focused on deeper language understanding at the level of both syntax and semantics (Nakashole et al., 2012; and tackled challenging linguistic phenomena like synonymy and polysemy.

these issues have not yet been addressed in their entirety.


As a matter of fact, any disambiguation or entity linking strategy can be used at this stage.

a knowledge-based unified approach like Babelfy is best suited to our setting, where context is limited and exploiting definitional knowledge as much as possible is key to attaining high-coverage results (as we show in Section 6.4).


selectional preferences) constitutes a line of research by itself, rooted in earlier works like (Resnik, 1996) and focused on either class-based (Clark and Weir, 2002), or similarity-based (Erk, 2007), approaches.

these methods are used to model the semantics of verbs rather than arbitrary patterns.


More recently some strategies based on topic modeling have been proposed, either to infer latent relation semantic types from OIE relations (Ritter et al., 2010), or to directly learn an ontological structure from a starting set of relation instances (Movshovitz-Attias and Cohen, 2015).

the knowledge generated is often hard to interpret and integrate with existing knowledge bases without human intervention (Ritter et al., 2010).


A key feature of our approach is its deep syntactic-semantic analysis targeted to textual definitions.

to our competitors, where syntactic constraints are necessary in order to keep precision high when dealing with noisy data, DEFIE shows comparable (or greater) performances by exploiting a dense, noise-free definitional setting.


This guarantees we will obtain the highest scoring derivation in every parsing action.

during training we sample from a distribution over derivations, not just return the argmax.


Second, while language models for natural text benefit from the large amounts of text data available on the web, obtaining correct MRs to train a model is less trivial.

in our tri-synchronous grammar framework, in addition to the MR itself, we are generating a paraphrase that nonetheless holds some disambiguating power over the MR, as described in Section 1.


More closely related is work on language grounding, whose goal is the interpretation of a sentence in the context of a world representation (Branavan et al., 2009;Liang et al., 2009;Chen et al., 2010;Bordes et al., 2010;Feng and Lapata, 2010;Hajishirzi et al., 2011;Matuszek et al., 2012;Hajishirzi et al., 2012;Artzi and Zettlemoyer, 2013;Koncel-Kedziorski et al., 2014;Yatskar et al., 2014;Seo et al., 2014;Hixon et al., 2015).

while most previous work considered individual sentences in isolation, solving word problems often requires reasoning across the multi-sentence discourse of the problem text.


They learn a mapping from word problems to equation templates using global and local features from the problem text.

the large space of equation templates makes it challenging for this model to learn to find the best equation directly, as a sufficiently similar template may not have been observed during training.


We have now showed that its possible to automatically construct large lexicons from smaller seed lexicons.

the seed lexicons used in §4.1 have been artifically constructed from aggregating attributes of word types over the treebank.


This in and of itself is not a problem for our model, as the edge weights should learn to discount this feature.

the fact that we sample edges to make inference tractable means that more informative edges could be dropped in favor of those that are only connected via a prefix features.


One alternative method to extract morphosyntactic lexicons is via parallel data (Das and Petrov, 2011).

such methods assume that both the source and target langauges are isomorphic with respect to morphology.


None of the models or evaluations from previous academic research on reverse dictionaries is publicly available, so direct comparison is not possible.

we do compare performance with the commercial systems.


Thus, NLMs seem to generalise more 'consistenly' than OneLook on this dataset, in that they generally assign a reasonably high ranking to the correct word.

as can also be verified by querying our we demo, OneLook tends to perform either very well or poorly on a given query.


via connections with other concepts that are described or defined as cold.

the candidates produced by the OneLook and (unsupervised) W2V baseline models have nothing to do with coldness.


While successful in their tasks, all of these approaches focused on monolingual data and did not reach beyond literal language.

we present a method that detects fine-grained cross-cultural differences from multilingual data, where such differences abound, expressed both literally and figuratively.


These differences are mined from multilingual, non-parallel datasets of Twitter and news data.

to previous work, our model does not merely output a list of monolingual lexical features for manual comparison, but also automatically infers multilingual contrasts.


Therefore, a topic associated with, e.g., war will have the same distribution over the lexicon regardless of whether the document was taken from a pro-war editorial or an anti-war speech.

in reality we may expect a single topic to exhibit systematic and predictable variations in its distribution based on authorship.


(2013) developed a topic model for Debatepedia, a semi-structured resource in which arguments are explicitly enumerated.

all of these models perform their analyses on monolingual datasets.


After this change, perplexity went down immediately after the M-step and kept decreasing monotonically, rather than increasing after the M-step before decreasing.

this did not result in a significantly lower final perplexity or coherence and thus did not change the relative performance of the models.


For instance, the dictionary translated the Spanish word pito as cigarette in English.

in informal usage this word refers specifically to cannabis cigarettes, not tobacco cigarettes.


On the other hand, the loss of vowel information makes the anagram decoding step much harder.

more than three quarters of in-vocabulary tokens are still correctly recovered, including the original vowels.


When applied to a source phrase, it may give a preference towards using shorter phrases as they are inherently less domain specific.

to phrase domain specificity, phrase pair coherence reflects whether candidate target and source phrases are typically used in the same set of domains.


As the result suggests, using our induction framework tends to yield slightly better translation results in terms of METEOR and especially BLEU.

using LDA seems to lead to slightly better translation result in terms of TER.


Topics in LDA-like models encode co-occurrence patterns in bag-of-word representations of sentences.

domains in our domaininduction framework rely on ngrams and wordalignment information.


Most answer sentence ranking algorithms operate under the assumption that the degree of syntactic and/or semantic similarity between questions and answer sentences is a sufficiently strong predictor of answer sentence relevance (Wang et al., 2007;Yih et al., 2013;Yu et al., 2014;Severyn and Moschitti, 2015).

answer extraction algorithms frequently assess candidate answer phrases based primarily on their own properties relative to the question (e.g., whether the question is a who question and the phrase refers to a person), making inadequate or no use of sentence-level evidence (Yao et al., 2013a;Severyn and Moschitti, 2013).


Note that the joint model score, being a product of two probabilities, is always lower than the standalone model score.

only the relative score matters in this case, as the chunk with the highest overall score is eventually selected for extraction.


It also leads to some spurious variables that are bound by lambdas or existentials, but which do not appear as arguments of any predicate: for example in the above analysis for Disney acquired Pixar, the variables x a , y e and z e are unused.

these "spurious" variables are easily identified and discarded.


In this case acquired would be assigned the following CCG lexical entry: Note the explicit arguments corresponding to the subject and object of this transitive verb (f 1 and f 2 , respectively).

an intransitive verb such as sleeps would be assigned an entry with a single functional argument corresponding to the subject (f 1 ): the entries in our system for these two verbs are simply λx.


The advantage of our approach is that it is robust, and relatively simple, in that a strict grammar that enforces type checking is not required.

there are challenges in handling some linguistic constructions.


To generate negative candidates, we apply our candidate generation method on the name of the concept which is treated as the mention, and also uniformly sample 200 concepts from all KBs.

there is no guarantee that these candidates are really negative.


If (1) is satisfied, that is, there are multiple KBs and enough entries in them that can be aligned, then the proposed method can be applied.

the performance of this method highly depends on (2), the quality of features and how well the indirect supervision examples approximate the text at prediction time.


One can argue that traditional understanding of grammar and grammar correction encompasses the idea of native-language fluency.

the metrics commonly used in evaluating GEC undermine these arguments.


Specifically, two major GEC corpora have different taxonomies: the Cambridge Learner Corpus (CLC) (Nicholls, 2003) has 80 tags, which generally represent the word class of the error and the type of error (such as replace preposition, unnecessary pronoun, or missing determiner).

the NUS Corpus of Learner English (NUCLE) (Dahlmeier et al., 2013) has only 27 error types.


Error-coded corpora are effective training and evaluation data for targeted error correction, and statistical classifiers have been developed to handle errors involving closed-class words (Rozovskaya and Roth, 2014).

the 2014 CoNLL shared task engendered a sea change in GEC: in this shared task, systems needed to correct all errors in a sentence, of all error types, including ones more stylistic in nature (Ng et al., 2014).


The agreement between experts and non-experts is lower than the agreement between just experts or just non-experts, which may be due to the difference of these experimental settings for experts (Grundkiewicz et al., 2015) and for non-experts (this work).

this finding is not overly concerning since the correlation between the rankings is so strong.


Sagae and Tsujii (2007) propose using a maximum entropy classifier to pro-duce conditional probabilities for each action given the features of the state, and score each state using the product of the probabilities of all actions taken up to that state.

they report that searching through the resulting space for the highest scoring parse does not consistently result in improved parser accuracy over a greedy policy (i.e.


The 25 feature templates used by these classifiers are shown in Table 2.

chen and Manning (2014) use 48 feature templates, including higher-order dependency information than has been shown to improve parsing accuracy significantly (Zhang and Nivre, 2011).


However, non-deterministic oracles aim to allow a deterministic parser to recover from incorrect actions by including information in the training of the local classifiers based on the notion that there may be several correct actions at a given point, as long as a desired tree remains reachable.

our local classifier, or oracle, is trained to encode a notion of state quality or approximate global loss that is specifically designed for search.


Noise was filtered out using crowdsourcing and the resulting frame-labeled corpus showed a lexical coverage three times as high as the original FNFT.

they did not evaluate the augmented corpus as training data for semantic role classification.


(2013) which is based on text patterns that connect entities of specific types, acquired by sequence mining from the Wikipedia fulltext corpus.

to our work, those are simple surface patterns, and the task addressed here is limited to typing noun phrases that likely denote emerging entities that are not yet registered in a KB.


For example, we may have a prior of f 8 (David Beckham, "Beckham") = 0.7, as David is more popular (today) than his wife Victoria.

f 8 (David Beckham, "David") may be lower than f 8 (David Bowie, "David"), for example, as this still active pop star is more frequently and prominently mentioned than the retired football player.


We present a method for unsupervised opendomain relation discovery.

to previous (mostly generative and agglomerative clustering) approaches, our model relies on rich contextual features and makes minimal independence assumptions.


Somewhat counter-intuitively, with α = 0 (no entropy regularization) the variance is almost negligible.

given the low performance in this regime, it probably just means that we get consistently stuck in equally bad local minima.


Similarly to our method they can use feature-rich models without the need for manually annotated data.

a relation extractor trained in this way will only be able to predict relations already present in a knowledge base.


Even though this approach is not very labor-intensive (i.e., it requires only few manually labeled examples for the initial seed set), it requires some domain knowledge from the model designer.

unsupervised models are domain-agnostic and require only unlabeled text.


Thus one may question the value of having a consistent estimator for A-HMMs and Brown models in this work: if the model is wrong, what is the point of learning it accurately?

there is also ample evidence that HMMs are competitive for unsupervised POS induction when they incorporate domain-specific structures.


(2013a) extend word-to-word alignment to phraseto-phrase alignment by a semi-Markov CRF.

such approaches often require more computational resources.


We commence by demonstrating in Figure 2 that the pointwise mutual information (Church and Hanks, 1990) evaluated from co-occurrence counts has a strong linear relationship with semantic similarity judgments from survey data (Pearson's r=0.75).

2 this suggestive linear relationship does not by itself demonstrate that log cooccurrences (with normalization) can be used to define an Euclidean metric space.


unobserved) and that the topic and word distributions θ and φ are drawn from Dirichlet distributions: θ ∼ Dir(α), and φ ∼ Dir(β).

using this model, one can attempt to infer the most likely topics to generate a corpus for some preset number of topics K. the optimization problem is non-convex and intractable to solve analytically, and is thus generally solved using iterative techniques such as Gibbs sampling, expectation maximization, or variational inference.


While this does not improve the quality of the topic model by most measures, it may be suited for a particular task involving abnormally varied word forms to which the model is applied.

for stemmers encoding standard rules of spelling and grammar, such a benefit is unlikely.


While some EBC Bibles are written in dated language, we do not make any modifications to the corpus if the language is also present in WTC.

as Basque is not represented in WTC, we replace the Basque Bible from 1571 with a contemporary version from 2004, to enable the use of Basque in the parsing experiments.


(2005), it was recently elucidated by Tiedemann (2014), whose implementation we follow here.

to our approach, DCA projects trees on a source-target sentence pair basis, relying on heuristics and spurious nodes or edges to maintain the tree structure.


Our UD test languages have an average of 89% fully projective sentences.

with IBM1 for example, we only predict 55% of all sentences to be projective.


A naive approach of computing the bidirectional representation of all n elements result in O(n 2 ) computation.

it is trivial to compute the BiRNN encoding of all sequence items in linear time by pre-computing RNN F (x 1:n ) and RNN R (x n:1 ), keeping the intermediate representations, and concatenating the required elements as needed.


A common approach follows the highest scoring transition that can lead to the gold tree.

when training in this way the parser sees only configurations that result from following correct actions, and as a result tends to suffer from error propagation at test time.


A standard Elman network (Elman, 1990) with hidden layer of size H trained on a corpus of size T with vocabulary of size V has complexity where I indicates the number of iterations.

several attempts have been made to reduce training time, focusing mostly on reducing the large factors T or V : • vocabulary shortlisting (schwenk and Gauvain, 2004) • subsampling (schwenk and Gauvain, 2005;Xu et al., 2011) • class-based (Goodman, 2001b;Morin and Bengio, 2005;Mikolov et al., 2011) • noise-contrastive estimation (Gutmann and Hyvärinen, 2012;Chen et al., 2015) these techniques either come with a serious performance degradation (Le et al., 2013) or do not sufficiently speed up training.


These types of features can be easily omitted or modified to assess their contribution to parsing performance.

higher-order features, such as linear word order precedence properties, are much harder to define and to manipulate.


Referring back to the results in Table 2, we observe that MaltParser shows the same average accuracy for RB and LB-permuted data.

some languages show significantly different results between their LB and RB-permuted data, especially in their labelled accuracy scores.


Contrary to New Testament, Spanish is easier to parse when it is rearranged into the rightbranching order (88% RB vs 85% LB UAS, 80% RB vs 76% LB LAS).

the types of difficult dependencies emerging from the different branching of the LB/RB data were not similar or symmetric to that of New Testament.


In particular, our results suggest that the relative order between the children (and not only the order between heads and their children) should be taken into account (Section 4.2).

we are not aware of previous work which proposes a measure for this property and describes it typologically on a large scale.


For CoNLL-2003, we found that particle swarm produced better hyper-parameters than random search.

surprisingly for OntoNotes particle swarm was unable to produce better hyperparameters than those from the ad-hoc approach in round 1.


ber of epochs because we observed that the models did not exhibit overtraining and instead continued to slowly improve on the development set long after reaching near 100% accuracy on the training set.

despite OntoNotes being much larger than CoNLL-2003, training for more than about 18 epochs causes performance on the development set to decline steadily due to overfitting.


Neural machine translation (NMT) aims at solving machine translation (MT) problems using neural networks and has exhibited promising results in recent years.

most of the existing NMT models are shallow and there is still a performance gap between a single NMT model and the best conventional MT system.


Deep models have more parameters, and thus have a stronger ability to fit the large data set.

our experimental results suggest that deep models are less prone to the problem of over-fitting.


In fact, LDA evolved out of efforts in the 1990s to provide a generative model that "explains" the success of older vector space methods like Latent Semantic Indexing (Papadimitriou et al., 1998;Hofmann, 1999).

none of these earlier generative models has been linked to PMI models.


They showed that if there were no dimension constraint in word2vec, specifically, the "skip-gram with negative sampling (SGNS)" version of the model, then its solutions would satisfy (1.1), provided the right hand side were replaced by PMI(w, w ) − β for some scalar β.

skip-gram is a discriminative model (due to the use of negative sampling), not generative.


be- 5 Note that this interpretation has been disputed; e.g., it is argued in Levy and Goldberg (2014a) that (4.1) can be understood using only the classical connection between inner product and word similarity, using which the objective (4.1) is slightly improved to a different objective called 3COSMUL.

this "explanation" is still dogged by the issue of large termwise error pinpointed here, since inner product is only a rough approximation to word similarity.


whereF denotes the complement of event F and 1 F and 1F denote indicator functions for F andF , respectively.

when F happens, we can replace Z c by Z with a 1 ± z factor loss: The first term of the RHS of (A.2) equals to we can use E [1F ] = Pr[F ] ≤ exp(−Ω(log 2 n)) to show that the second term of RHS of (A.2) is negligible, This claim must be handled somewhat carefully since the RHS does not depend on d at all.


First, simplicity is essential in measuring the goodness of simplification.