Due to the min-formulation in the objective function, a lower weight increases the influence of the respective space.

we let our composed similarities be influenced more by the question-space similarities as in previous work (Xue et al., 2008).


Most of these approaches use word-level embeddings to encode entities and predicates, and therefore might suffer from the out-of-vocabulary (OOV) problem when they encounter unseen words during test time.

they often rely on significant data augmentation from sources such as Paralex , which contains 18 million question-paraphrase pairs scraped from WikiAnswers, to have sufficient examples for each word they encounter (Bordes et al., 2014b;Yih et al., 2014;Bordes et al., 2015).


We define the syntactic score for all sentences in s as: where Dep(s) are the dependency parse trees for all sentences in s; L Dep is a 3-gram language model over dependency triples which gives the likelihood of an arc label l being used between a pair of words (w i , w j ).

for example in figure 2, the syntactic compatibility score includes dependency likelihoods of the Syn function prefers stories s that (a) have similar dependency structure to the original story s and (b) make use of a common syntactic configuration.


The semantic coherence component expresses how well a candidate s rewrites individual words and realizes the semantic relationships that exist in the human-authored story s. Ideally, we would like to preserve enough of the semantics of s in order to produce a coherent story s , yet we are populating s with words taken from an unrelated theme.

we model the semantics of a story s in terms of the lexical semantics contributed by individual words as well as semantic relationships that exist between its elements.


Since there is no open dataset available, the lack of annotated resources has limited the research in this area.

we first present a dataset we built using SINA city news.


Verbs and verb phrases cover 93% of all cause events.

our learning algorithm mainly focus on them.


Here, we embed each user u and each product p as continuous and realvalued vectors u ∈ R du and p ∈ R dp , where d u and d p are the dimensions of user embeddings and product embeddings respectively.

the attention weight α i j for each hidden state can be defined as: where e is a score function which scores the importance of words for composing sentence representation.


The sentences that are clues to the meaning of the document vary in different users and products.

in sentence level, we also use a attention mechanism with user vector u and product vector p in word level to select informative sentences to compose the document representation.


The squash function ψ k (z) could be formalized as: where z ∈ (0, 1) is computed by logistic sigmoid function.

r k can constrain the forgetting rate in the range of ( k−1 K , k K ).


For fair comparisons, we set the total amount of neurons in our model to be same with vanilla LSTM.

the more groups we split, the less the neurons belongs to each group, which leads to a worse capacity than those who have sufficient neurons for each group.


Our proposed approach takes a set of pre-trained word representations and uses the graph structure of a semantic lexical resource in order to de-conflate the representations into those of word senses.

our approach firstly requires a set of pre-trained word representations (e.g., word embeddings).


In fact, we observed that taking a very small portion of the top-ranking words, i.e., the top 25, produces similarity scores that are on par with those generated when the full lists were considered.

we experimented with the down-sized lists which enabled us to generate very quickly sense representations for all word senses in WordNet.


The first two components were included to filter out items that are either incorrectly labeled as predicates or that the annotator could not annotate for components (iii) and (iv), and if an annotator answered no to either for a particular predicate candidate, (iii) and (iv) did not appear.

like SPR2.x, this protocol incorporates a hierarchy of questions that can be elaborated in future versions.


While WordNet is a valuable resource, the selection of a specific categorical sense under an enumerated set of prespecified options is troubling in a similar way as Dowty was concerned with thematic roles (see Kilgarriff 1997).

we follow a path similar to Sussna (1993) in asking annotators for zero or more senses that are appropriate.


This is not possible with a chart-based parser, in which complete derivations must be built first.

a shift-reduce parser is able to consider a much larger set of categories per word for a given input, achieving higher lexi-cal assignment accuracy than the C&C parser (Clark and Curran, 2007), even with the same supertagging model (Zhang and Clark, 2011;Xu et al., 2014).


However, their architecture is designed to handle only a fixed number of actions, while we observe varying number of actions.

we adopt a probabilistic model similar to Dyer et al.


The CKY parser often contains many correct parses with identical scores, up to 49K parses per sentence.

we randomly sample and keep up to 1K trees.


For example, for toys kids, we are pretty sure as a web query, its intent is to look for toys for kids, instead of the other way around.

toys should be the head of the query, and kids should be its modifier.


query aligned with the second sentence is better than the first sentence.


However, using pretrained word2vec embeddings, we also achieve 3% advantage.

it seems that context-sensitive signals plus the generalizing power of embeddings contain all the context-free signals already.


Denote the neighboring dialogues of time(r)| < δ}, we extend the optimization objective (Equation 1) into: Synopses Since a synopsis is a concise summary of each episode, we can treat plot summarization as text alignment where R i is assumed to match the content of S i .

the summarization term can be redefined by substituting D i with S i : Similarly, the contingency component can be modified to include connections from synopses to detailed descriptions.


This allows separating successive elements of the same category.

we use the IOB encoding during the annotation step.


Then, the previously identified requirements were automatically annotated (Section 5.2).

we get filtered and semantically annotated requirements in XML or JSON.


Therefore, we present a binary text classification approach that distinguishes between requirements and off-topic content.

we trained different classification algorithms and tested them using various features and parameter settings.


Previous studies have empirically shown that learning multiple related tasks simultaneously can significantly improve performance relative to learning each task independently, especially when only a few labeled data per task are available (Caruana, 1997;Bakker and Heskes, 2003;Argyriou et al., 2006).

given the limited labeled review data for each domain, we formulate the review spam detection tasks for multiple domains, e.g., hotel, restaurant, and so on, as a multi-task learning problem.


We refer to the book of Boyd and Vandenberghe (2004) for the theory behind convex optimization.

the overall expected risk (Vapnik, 1991) is the weighted sum of two components: the empirical risk and a regularization penalty term, expression referred to as "Loss+Penalty" by Hastie et al.


In that last form, we see that the loss function can be interpreted as a negative log-likelihood and the regularization penalty term as a negative log-prior over the hypothesis.

if we assume a multivariate Gaussian prior on the feature weight vector of mean vector 0 and covariance matrix Σ = σ 2 I (i. e. independent features of same prior standard deviation σ), we do obtain the L2 regularization: And similarly, if we assume a multivariate Laplacian prior on the feature weight vector (i. e. θ i ∼ Laplace(0, 1 λ )), we obtain L 1 -regularization.


This more accurately captures cohesiveness.

the K-trusses are smaller and denser subgraphs than the k-cores, and the maximal Ktruss of G better approximates its densest subgraph.


Indeed, the size of the main core (and more generally, of every level in the hierarchy) depends on the structure of the graphof-words, which by nature is uniquely tailored to the document at hand.

the distribution of the number of extracted keywords matches more closely that of the human assigned keywords .


The judgments of the four workers that selected B would be counted as inaccurate, since the feature set is the same for the judgments with A and the judgments with B.

the oracle represents the highest accuracy a model can achieve, consistent with the provided labels, using the features provided.


We found that 19% of pairs had only a vocabulary distinction, and 65% of pairs could be judged correctly either by vocabulary or syntax.

84% of pairs could be judged using vocabulary, which explains the high performance of the Age of Acquisition features.


Compared to feed-forward neural networks, recurrent neural networks are theoretically more powerful and are capable of learning more complex patterns from data.

we have mainly focused on recurrent networks in this paper.


As Table 1 shows, some differences in sentence content can be captured using POS tags, but some cannot.

it is necessary to make further syntactic and semantic analysis to distinguish true candidates from false ones.


The rationale is that a similar sentence pair tends to be an instance of non-uniform language if the different words are synonyms, rather than having other relationships such as hypernymy, hyponymy, and antonymy.

we do not deem necessary to include these relationships into our analysis.


For word-level representation, the number of translation units can increase with corpus size, especially for morphologically rich languages which leads to many OOVs.


When assigning these 100 documents to 9 other annotators, the amount of positive instances increases to 1,258 keyphrases, which means that labels for 889 keyphrase candidates, or 17% of the original negative candidates when training on annotations by a single annotator, can be considered noise and relabeled.

ratios of positive to negative data also change drastically.


During testing, we need to construct the timeline for each target entity using the events that were predicted to be anchored to it and the timestamps of the temporal expressions each event was anchored to.

we need to perform two additional tasks, event coreference and ordering.


Only 38 of the 70 systems that participated in CSSF 2015 also participated in 2014, and only 24 of the 34 systems that participated in TEDL 2015 also participated in 2014 EDL.

many KBP systems in 2015 were new and did not have past training data.


For the slot fills that were common between the runs of a given team, we compute an average confidence value, and then add any additional fills that are not common between runs.

we obtained 10 systems (one for each team) for which we have supervised data for training stacking.


Our character-based model with 1024 hidden units achieved 12.48 per-description perplexity, marginally better than 12.58 obtained with a wordbased neural network model reported in that work.

we see that modeling color names as sequences of characters is wholly feasible.


We also found that 67.5% of mistakes made by the CNN+LSTM model can be successfully predicted by checking distance of test QI pair from its k-NN training QI pairs (66.7% for the ATT model, 55.08% for the MCB model).

this analysis not only exposes a reason for mistakes made by VQA models, but also allows us to build human-like models that can predict their own oncoming failures, and potentially refuse to answer questions that are 'too different' from ones seen in past.


model (60% for the ATT model, 73% for the MCB model) which is more than the respective average accuracy on the entire VQA validation set (54.13% for the CNN+LSTM model, 57.02% for the ATT model, 60.36% for the MCB model).

producing the same response across images seems to be statistically favorable.


are asked for an image in the VQA dataset only when ground is covered with snow (because subjects were looking at the image while asking questions about it).

this analysis exposes label biases in the dataset.


We tried to test variations of this measure with our representation, but we were not able to achieve satisfying results.

we report here only the scores with the cosine.


We introduce a hyperparameter γ that discourages the model from using many higher-order features (= faster inference) and a hyperparameter λ that encourages generalization.

sweeping a range of values for γ and λ gives rise to a number of operating points along the speed-accuracy curve (triangle points in Fig.


(Any tag strings in W\W can get nonzero weight without increasing runtime.)

r(θ) would ideally measure |W|, or proportionately, |H|.


Let S be the sentence, consisting of n words A look-up table is applied to S, made up of word vectors e i ∈ R m corresponding to each word w i , where m is the word vector dimensionality.

sentence S can be represented as a matrix E S ∈ R m×n by concatenating the word vectors E S = n i=1 e w i .


However, α i is required to be positive in Dir because otherwise the normalization factor becomes divergent.

the strength of the sparsity preference is upper bounded.


Parameter can thus be seen as a smoothing factor that prevents any element in x to become too small.

with proper parameters, mDir is able to simultaneously achieve sparsity and smoothness.


However, the pseudo-Dirichlet distribution is no longer conjugate to the multinomial distribution.

its maximum a posteriori inference becomes complicated and has no time-complexity guarantee.


From this graph, we captured the compatible arguments (" û foreign businessman " and "è ' entrepreneur "), incompatible arguments ("' item " and " û foreign businessman "), ("5 ½ rule " Chinese o û è' Ü{ Ã 8 ' 5½ English translation protect foreign businessman entrepreneur legal profit six item rule and " û foreign businessman ").

" û foreign businessman " and "è' entrepreneur " should be the roles of "Ý] invest " simultaneously , ("' item ", " û foreign businessman ") and ("5 ½ rule ", " û foreign businessman ") should not be the roles of "Ý ] invest " simultaneously.


We noticed that tuning the number of features to keep made little or no difference in the absolute ordering of the different DS models.

we use the optimal number of features averaged over all 6 DS models described in Section 3: the top 13% of MEG sensor/time points, and 3% of fMRI voxels.


Both Levin's and CCA-based verb classes are derived from syntactic behavior.


The decision to use simple features is for optimizing the training speed.

we are able to dedicate more time for tuning the regularization weight.


We use the same parameters ✓ to generate the representations for both ingredients and product categories due to their overall similarity.

v ✓ (x i ) is the vector representation for ingredient i and v ✓ (z p ) is the vector representation for product category p for an RNN model with parameters ✓.


While we use the loss function commonly used for binary classification during training, we ultimately want to frame our question as a ranking problem.

after iterating through the initial observations, we compute the mean average precision (MAP) over the remaining (unseen) ingredients/questions and compare it to the MAP of the baseline RNN model on the same unseen examples.


Sentencelevel statistics would not average the correct counts over all sentences in this same way.

a corpus-level statistic may over-estimate the quality of system output.


However, some well-known regular linguistic shifts include rarer parts of speech like adverbs (included in our case studies below).

we also confirmed that the differences shown in Figure 2 also hold when adverbs and adjectives are included along with the verbs.


In the traditional Encoder-Decoder framework, all utterances in the context are concatenated together before encoding.

information from previous utterances is far outweighed by the most recent utterance.


We define each slot s i as a random variable ranging over an infinite set of letter sequences V , which is represented as follows: where C is a set of characters including the blank character and any other character that potentially appears in the transcription of an utterance.

we regard the set of slots S as also being a random variable that ranges over V M S .


Identifying the set of non-slot phrases is equivalent to identifying the set of slot phrases.

we define a generative model of functional fillers in the same way as the slot values.


We assume that an utterance u is built with phrases provided by S and F .

the conditional distribution p(u|S, F ) basically involves a distribution over the permutation of these slot values with two constraints: f 1 is placed first and f 3 has to be placed last.


However, this is an uncertain hypotheses only, given the ambiguity of language, the noise induced by machine translation and the potential errors of the triple extraction.

our method needs to de-noise these input phrases and quantify to what extent the the relational phrases are indeed synonymous.


the input x consists of two sequences of chunks In our running example, we have s = (Gunmen, abduct, seven foreign workers) and t = (Seven foreign workers, kidnapped).


We will use r to denote the sentence similarity for an input x.

the prediction problem is the following: Given a pair of chunked sentences x = (s, t), predict the alignment y, the alignment similarities z and the sentence similarity r. Note that this problem definition integrates the canonical semantic textual similarity task (only predicting r) and its interpretable variant (predicting both y and z) into a single task.


However, neither our output nor the gold standard similarities are at the granularity of tokens.

to compute the loss, we project the chunk scores z i,j,l for an aligned chunk pair (s i , t j , l) to the tokens that constitute the chunks by equally partitioning the scores among all possible internal alignments.


Although the RNN is able to capture long-span context information to some extent, the local context may very easily dominate the hidden state.

this additional soft headword embedding allows the model to access long-span context information in a different channel.


However, different from the above studies where the alignment information is latent, in dependency parsing, the arc between the modifier and headword is known during training.

we can utilize these labels for attention weights.


We use three basic types: E (Entity), Ev (Event), and Prop (Proposition).

the semantic types of expressions in our system are defined by the rule First-order language can be taken as a fragment of this system; apart from logical connectives and Figure 1: The mapping from syntactic categories to semantic types.


quantifiers, all primitive expressions in first-order logic are confined to constant symbols of type E and predicates of type E ⇒ Prop, E ⇒ E ⇒ Prop, and so on.

adopting higher-order language does not lead to the loss of the expressive power of firstorder language.


When we evaluate the system on previously unseen test data, using BLEU (Papineni et al., 2002), we consistently find the length ratio between MT outputs and human references translations to be very close to 1.0.


Given the alignments of all the training sentence pairs, we add an alignment distance cost to the objective function.

we not only maximize the log translation probabilities, but also minimize the alignment distance cost.


However, the accuracy is still far behind the traditional MaxEnt alignment model in terms of alignment F1 score (Mi et al., 2016b;Tu et al., 2016).

in this section, we explicitly add an alignment distance to the objective function in Eq.


Meanwhile, for the English explicit instance, it is very likely that removing but would not lose any information since its Chinese counterpart ´can be omitted.

it could be used for the English DRR imp , alleviating the meaning shift problem of SynData.


In the current problem, the two tasks are essentially the same, just with different output labels.

as illustrated in Figure 2, M T N shares parameters in all feature layers (L 1 -L 3 ) and uses two separate classifiers in the classifier layer (L 4 ).


Morpheme F 1 , on the other hand, gives us partial credit.

what this shows us is that the WCFG gets a lot more of the morphemes in the held-out set correct, even if it only gets a few complete forms correct.


In this model, the recursive unit uses, instead of latent states from its children, the latent states of the backward LSTM at the start of the span and the latent states of the forward LSTM at the end of the span.

this model encodes the lexical information available in the full model but does not encode the parse structure beyond the local rule production.


Since we assume that the structures of parse trees of s and t are determined by a parser, the values of γ s m,n,i and γ t m,n,i are fixed.

by traversing the parse tree in a bottomup manner, we can identify an LCA (i.e., τ i ) for phrases τ m and τ n while simultaneously computing γ m,n,i .


When TRACE function returns a set of { Ψ k , Φ k }, all ψ ∈ Ψ k are aligned with phrases in the source and their inside probabilities are stored in A.

we can compute the inside probability for each Ψ k , Φ k , which is stored in A to-Source: Relying on team spirit, expedition members defeated difficulties  Although we have discussed using trees for clarity, the alignment is conducted on forests.


Our method is computationally heavy since it handles forests and involves unpacking in the noncompositional alignment process.

we use Viterbi training (Brown et al., 1993) together with a beam search of size µ b ∈ N on the featureenhanced EM.


Due to this conversion, the accuracy on the relation labels is less important.

we reported only the unlabeled attachment score (UAS) 7 .


In addition, 8 cases are due to erroneous original sentences that should be disregarded, e.g., " For two weeks ago,..." and "Accord-ing to the source, will also meet...".

our method corrected 32 errors while introducing 23 errors in reality for 446 errors in 1-best trees, which achieves a 2.5% error reduction.


Higher-order MST models associate higher-order score functions with the construction of individual dependencies.

the deterministic strategy is a prerequisite to incorporate higher-order features.


For sub-problems L, R, LR, and N, each has O(n 3 ) elements, with a unit calculating time O(n).

both algorithms run in time of O(n 4 ) with a space requirement of O(n 3 ).


Supervised training data has been scarce and, while techniques like distant supervision appear to be a promising way to extend knowledge bases at low cost, in practice the training data has often been too noisy for reliable training of relation extraction systems (Angeli et al., 2015).

most systems fail to make correct extractions even in apparently straightforward cases like Figure 1, Carey will succeed Cathleen P. Black, who held the position for 15 years and will take on a new role as chairwoman of Hearst Magazines, the company said.


However, text features embedding may be in a different semantic space with relation types.

we directly learn a mapping g from text feature representations to relation mention representations (Van Gysel et al., 2016a,b) instead of simple heuristic rules like concatenate or average (see Fig.


Because s c,i would not be used in other components of our framework, we integrate out s c,i and write the log likelihood as Note that o * c is a hidden variable but not a model parameter, and J T is the likelihood of ρ c,i = δ(o c,i = o * c ).

we would first infer o * c = argmax o * c J T , then train the true label discovery model by maximizing J T .


Universal Schemas (Riedel et al., 2013;Verga et al., 2015;Toutanova et al., 2015) has been proposed to unify patterns and knowledge base, but it's designed for document-level relation extraction, i.e., not to categorize relation types based on a specific context, but based on the whole corpus.

it allows one relation mention to have multiple true relation types; and does not fit our scenario very well, which is sentence-level relation extraction and assumes one instance has only one relation type.


Although delicately designed, it fails to make full use of the connection between these modules, i.e., not refine the context representation for classifier.

its discriminative module might suffer from the overwhelming size of text features.


Given the weights α, we build a consolidated representation of context event chain e 1 , e 2 , ..., e n−1 as a weighted sum of h 1 , h 2 , ..., h n−1 : The event candidate h c and the new representation of the existing chain h e can be further integrated to deduce a deeper representation of the full event chain hypothesis to the next layer (hop 2), denoted as v. v contains deeper semantic information compared with h c , which encode the temporal order of the event chain [h 1 , h 2 , ..., h n−1 , h c ] without differentiating the weights of each event.

in the next hop, better event weights can potentially be deduced by using v instead of h c in the calculation of attention: In the same way, we stack multiple hops and repeat the steps multiple times, so that more abstract evidences can be extracted according to the chain of existing events.


First, entities in their own pages are usually not annotated.

we annotate these entities with matching between the text and the page title.


Furthermore, only 6.9% of queries in the ERD14 dataset have spelling mistakes, whereas the number in the GERDAQ dataset is 23.0%.

spell-checking is more important in the GERDAQ dataset.


A key difference from many other problems, however, is that the classes to choose from (i.e., the senses of a target word) vary for each word, therefore requiring a separate training process to be performed on a word by word basis.

hundreds of training instances are needed for each ambiguous word in the vocabulary.


(2014) and set the error δ = 0.00001 and the number of iterations to 1 δ = 100, 000.

of this lexical profiling step we have a probability distribution over vocabulary words for each given word sense of interest.


This choice is due to WO's higher performance over cosine similarity for vectors with explicit dimensions (Pilehvar et al., 2013).

each row of the original adjacency matrix M of G will be replaced with the weights calculated in Formula 7 and then normalized in order to be ready for PPR calculation (see Formula 1).


We view this as a significant achievement given the total absence of man-  IMS uses the MFS as a backoff strategy when no sense can be output for a target word in context (Zhong and Ng, 2010).

the performance of the MFS is mixed up with that of the SVM classifier.


Thanks to BabelNet, in fact, Train-O-Matic can be used to generate sense-annotated data for any language supported by the knowledge base.

in order to build new training datasets for the two languages, we ran Train-O-Matic on their corresponding versions of Wikipedia, then we tuned the two parameters K and z on an in-house development dataset 7 .


Of course, this small subset of languages will be different for every linguistic feature.

even for extremely resource-poor languages for which at present no annotated resources exist, SuperPivot will make available richly annotated corpora that should advance linguistic research on these languages.


As a result , it was found that a path which crosses a sphere obliquely existed .

lGP-NMT: その結果、球を斜めに横切る経路が存在することが分かった。 lGP-NMT+: その結果、球を斜めに横切る経路が存在することが分かった。 (, it was found that a path which obliquely crosses a sphere existed .)


(2017) have proposed a model which learns to parse and translate by using automatically-parsed data.

it is also an interesting direction to learn latent structures in a target-side language.


Due to the large number of parameters, it may be difficult for the NMT system to learn the proper sentence representation as the initial state.

the model is very likely to get stuck in local minimums, making the translation process arbitrary and unstable.


As there are hundreds of millions of parameters in the NMT system, it is hard for the model to learn the exact representation of source sentences.

the values of initial state may not be exact during the translation process, leading to poor translation performances.


The two mechanisms could also work together, so that both the encoder and the decoder could be improved: The previously proposed word prediction mechanism could be used only as a extra training objective, which will not be computed during the translation.

the computational complexity of our models for translation stays exactly the same.


The size of this set also varies from scene to scene.

for our evaluation, we restrict the set {y 1 , .


In the original MDN model proposed by Bishop (1994), the parameters of the mixture model are separate functions of input, which is appropriate when the inputs and outputs directly relate to each other, but in the case of geolocation or lexical dialectology, the relationship between inputs and outputs is not so obvious.

it might be a difficult task for the model to learn all the parameters of each sample correctly.


We differentiate the normalization methods to lead the surfer of affinitypreserving random walk to explore more in the neighborhood of the sentences in V rather than end in the absorbing vertex s 0 .

the sentences that satisfy the saliency and diversity constraints will be highlighted even though they are in a small sentence cluster.


We conclude that when the model is provided with syntactic information, it learns to pick S-type candidates, but does not continue to learn deeper features to further distinguish them or needs more data to do so.

the model is not able to point to exactly one antecedent, resulting in a lower s@1 score, but does well in picking a few good candidates, which yields good s@2-4 scores.


For example, given a hypernymhyponym pair animal-bird, the context flap is a distinctive characteristic of bird, because almost every bird can flap, but not every animal can flap.

the context flap is closer to the hyponym bird than to the hypernym animal.


For example, the HyperScore for the pairs duck-animal and animal-duck are 3.02 and 0.30, respectively.

the magnitude proportion of the hypernym-hyponym pair duckanimal is larger than that for the pair animal-duck.


The word "road" is in the definition of "car", but "car" is not in the definition of "road".


However, there is a non-zero probability that w i and w t are related.

the model will move their vectors further instead of moving them closer.


It is common for a Chinese learner to comprehend the meaning of a word from these graphical components.

we propose models that enhance word representations by character glyphs.


The meaning of a Chinese word is often related to the meaning of its compositional characters.

chinese word embedding can be enhanced by its compositional character embeddings (chen et al., 2015;Xu et al., 2016).


This permits us to decouple decisions made in designing the NMT architecture from decisions about which models we will use for learning sentence embeddings.

we can benefit from orthogonal work in designing neural architectures to embed sentences.


However, in practice, since we have so much bitext to draw from, we can easily find data in which they do not match exactly.

our generated data consists of pairs of English references from the bitext along with the NMT-produced English back-translations.


We do the same filtering for the SimpWiki data, though the trend is not nearly as strong.

this is unlikely due to the nature of the evaluation data, and may be due to machine translation quality dropping as sentence length increases.


For a target word w i , the objective function of MGE is almost equivalent to maximizing P (w During the backpropagation, the gradients of h i 1 , h i 2 , h i 3 can be different in our model while they are always same in MGE, so the gradients of the embeddings of words, characters, subcharacter components can be different in our model while they are same in MGE.

the representations of words, characters, and subcharacter components are decoupled and can be better trained in our model.


Recent trends (Tsvetkov et al., 2015) and (Tsvetkov et al., 2016) have suggested that eval-uation methodologies such as word analogy and word similarity tasks may not be holistic.

embeddings generated by our approach should be evaluated by plugging into end-level tasks such as machine translation, POS tagging etc.


For the purpose of completeness, the other non-SVD based embedding models, mainly the more recent neural prediction methods, are also compared in our experiments.

we build the second baseline using the skip-gram model provided by the word2vec software package (Mikolov et al., 2013a), denoted as SGNS.


However, high-quality embeddings can be expensive to train.

rather than training their own embeddings, NLP researchers and practitioners often download pre-trained embeddings from the Web, e.g.


To estimate the relevance of each embedding set for the user's task, these signatures are later compared to corresponding signatures created from the user's corpus.

each signature method has an associated similarity measure, which takes in a pair of signatures (one from a VecShare embedding set, and the other from the user's corpus) and outputs a numeric similarity score for the pair.


Once the target embedding set has been identified, the library downloads the target embeddings for the particular words in the user's vocabulary.

if the user's vocabulary is much smaller than that of the embedding set, this download can be much more compact than the full embedding set.


Finally, as discussed below our experimental results reveal that the different signature methods have distinct strengths.

we also evaluate All, a simple ensemble of the VocabRk, SimCorr, and MaxTkn methods.


4, it does the following sketch update for every word i ∈ [L]: Note that the only difference is in replacing the single vectorc n by the word-specific vector c n i .

this is no longer a rank-one update of the sketch matrix, but a full update.


we obtain: and where We thank the Unbabel research team and the reviewers for their comments.


Also, the non-negativity of the variance gives 0 ≤ V [l].


(2) means that for any ϵ 2 > 0, there exists n ′ such that if n ′ ≤ n then |E[l]| < ϵ 2 .

we have The arbitrary property of ϵ 1 and ϵ 2 allows ϵ 1 + ϵ 2 to be rewritten as ϵ.


To address this difficulty, Z i,w is approximated by the first-order Taylor expansion around The first-order Taylor approximation gives where x=µ .

it can be shown that the first and second order moments of ∆L(θ) have the order of O( log (n) n ) in the smoothed case as well.


Also, their updating method needs to scan not only new but also old training data, and thus is not an incremental algorithm in a strict sense.

it potentially incurs the same time complexity as the retraining.


Thus it is natural to use trained languages models to initialize the encoder and decoder.

the basic procedure of our approach is to pretrain both the seq2seq encoder and decoder networks with language models, which can be trained on large amounts of unlabeled text data.


Our method predicts K attention context vectors while reading the source, and learns to use a weighted average of these vectors at each step of decoding.

we avoid looking back at the source sequence once it has been encoded.


As opposed to scalar or categorical prediction, structured predictions vary in size and complexity.

one must decide not only how to explain the prediction, but also what parts of it to explain.


A good explanation would ideally also decompose into cognitive chunks (Doshi-Velez and Kim, 2017): basic units of explanation which are a priori bounded in size.

we seek a framework that naturally decomposes an explanation into (potentially several) explaining components, each of which justifies, from the perspective of the black-box model, parts of the output relative to the parts of the input.


We can formalize these components as subgraphs x are likely causes for the elements in V k y .

we define an explanation of y as a collection of such components: Our approach formalizes this framework through a pipeline (sketched in Figure 1) consisting of three main components, described in detail in the following section: a perturbation model for exercising F locally, a causal inference model for inferring associations between inputs and predictions, and a selection step for partitioning and selecting the most relevant sets of associations.


The quality of these dependency estimators will depend on the frequency with which each input and output token occurs in the perturbations.

we are interested in obtaining uncertainty estimates for these predictions, which can be naturally done with a Bayesian approach to logistic regression.


Natural language processing methods that derive semantics from large corpora have been shown to incorporate biases present in the data, such as archaic stereotypes of male/female occupations (Caliskan et al., 2017) and sexist adjective associations (Bolukbasi et al., 2016).

there is interest in methods that can detect and address those biases.


More importantly, in the 20-NG and Reuters datasets, increasing the number of pieces from 3 to 5 further reduces perplexity.

we have achieved a new state-of-theart perplexity on 20 News-Groups task and -to the best of our knowledge -better perplexities on the CADE12 and RCV1 tasks compared to using a state-of-the-art model like the G-NVDM.


Attempts to overcome this difficulty are based on a greedy approach which starts with firstorder dependencies between labels and iteratively increases the scope of dependency patterns under the constraint that a high-order dependency is selected only if it extends an existing lower order feature (Müller et al., 2013).

feature selection may only choose only few higherorder features, motivating the need for an effective variable-order CRF (voCRF) training procedure (Ye et al., 2009).


This makes ADAGRAD well-suited for tasks with sparse gradients such as distributed deep networks (Dean et al., 2012), forward-backward splitting (Duchi and Singer, 2009), and regularized dual averaging methods (Xiao, 2010).

it may seem reasonable to apply ADA-GRAD to optimize online topic models.


Since ADAGRAD uses the accumulation of previous gradients as learning rates' denominators, the learning rates shrink very quickly.

the algorithm cannot break the symmetry quickly.


Then, ADAGRAD learning rates decrease too quickly, leaving the tie between top and bottom unbroken.

the algorithm fails to form appropriate topics.


the lower and upper bounds of E λ (t) kv are For a large datasets, Dn v should be large.


This cycle repeats, until some learning rates decrease to zero and learning effectively stops.

the algorithm will never break the tie or infer good topics.


In addition, since ADADELTA computes the moving average of updates, and ADAM needs to compute the bias-corrected gradient estimate, they require more intricate implementations.

these two methods are not as popular as ADAGRAD for beginners.


In addition, the magnitude of gradients is proportional to the dataset's size.

when the dataset is small enough, ADAGRAD will still work.


The weighted memory is designed to up-weight nearer sentiment words, and the recurrent attention module, discussed below, attends long-distance sentiment words.

they work together to expect a better prediction accuracy.


Although multiple-attention mechanism has the potential to synthesize features in complicated sentences, enforcing the model to pay a fix number of attentions to the memory is unnatural and even sort of unreasonable for some cases.

we need a mechanism to stop the attention process automatically if no more useful information can be read from the memory.


But, the key in sentiment analysis lies in its cognitive basis.

we envision that cognition grounded data obtained in text reading should be helpful in building an attention model.


However, the UPA model works only if user and product information is available.

for IMDB2 where user/product information is not available, only CBA and LA models work and the combined use of CBA+LA gives the best performance.


Note that this also includes the restaurants with zero funny votes, by excluding these we found that the ratings were more consistently stable on about 2.1 votes.

regardless of restaurant rating, the funny reviews distribution are stable on average.


This is very useful in the online review setting since there tends to be a cumulative advantage, that is the "rich get richer" effect which limits the exposure that the users get to other helpful reviews.

identifying these types of review early can potentially diversify the types of reviews that the users read.


Also, it is not surprising that some products may always receive low ratings because of their poor quality and vice versa.

it is necessary to leverage individual preferences of users and overall qualities of products in order to achieve better performance.


This may result in words with similar vector representations having an opposite sentiment polarity (e.g., good and bad), thus degrading sentiment analysis performance.

this study proposes a word vector refinement model that can be applied to any pre-trained word vectors (e.g., Word2vec and GloVe).


Sentiment prediction is a core component of an end-to-end stock market forecasting business model.

an efficient sentiment analysis system is required for real-time analysis of financial text originating from the web.


A key difference of GRU with LSTM is that, GRU's recurrent state is completely exposed at each time step in contrast to LSTM's recurrent state which controls its recurrent state.

comparably GRUs have lesser parameters to learn and training is computationally efficient.


In some cases GloVe has the advantage while in others Word2Vec performs better.

we derive a new hybrid word embedding model using a stacked denoising autoencoders (DAWE).


Motivated by this contrasting behavior we choose to combine the predictions of these models.

we construct an ensemble by taking the best performing deep learning (CNN, LSTM and GRU each) and classical feature based (SVR) models using a MLP network.


Note that new words in Chinese are easier to create from individual characters than in other languages.

many of these words are not in the dictionary.


U is essentially regarded as an noisy N class data, i.e., it has a lot of errors (which are hidden P class instances).

this step effectively aims to purify the N class set.


Parameters of Gaussian distribution are obtained using Maximum Likelihood Estimate (MLE) as follows: We set the threshold θ = µ + σ.

those words have probabilities higher than θ are considered very likely to belong to the P class.


For example, the polarity vector (0.9, 0.1) means that 90% of the words in the existing lexicon that contains the character are positive and the other 10% are negative.

for a word with 2 characters, which covers most cases in Chinese, a 4dimensional vector is formed.


We will also see that many new sentiment words that we discovered are not even in an authoritative Chinese language dictionary.

compiling a comprehensive sentiment lexicon is needed.


The syntactic pattern based DP approach performed very poorly because social media posts are brief and the use of patterns to link sentiment words are quite infrequent.


But after that, the results deteriorate for both systems.

the table only shows the first few iterations since the results keep deteriorating after iteration 2.


Despite not being the main goal of a reformulation method, improvements in precision are also expected with a good query reformulation method.


RL-RNN-SEQ performs slightly worse than RL-RNN but produces queries that are three times shorter, on average (15 vs 47 words).

rL-rNN-SEQ is faster in retrieving documents and therefore might be a better candidate for a production implementation.


The proportion of relevant terms selected by the SL-and RL-Oracles over the total number of candidate terms (Table 3) indicates that only a small subset of terms are useful for the reformulation.

we may conclude that the proposed method was able to learn an efficient term selection strategy in an environment where relevant terms are infrequent.


This number further increases to 83% when considering Wikipedia-based datasets (we provide details on these datasets in Section 5).

a generic encoder-decoder model learns to copy all too well at the expense of other rewrite operations, often parroting back the source or making only a few trivial changes.


For most Natural Language Processing (NLP) tasks, obtaining sufficient annotated text for training accurate models is a critical bottleneck.

active learning has been applied to NLP tasks to minimise the expense of annotating data (Thompson et al., 1999;Tong and Koller, 2001; Settles and Craven, 2008).


The model, φ, is trained on one source language, and the policy is learned on a different source language.

policy learning uses Alg 1, with the small change that in step 3 the model is initialised using φ. the learned policy can exploit the knowledge from cross-lingual initialisation, such that it can figure out which aspects that need to be corrected using target annotated data.


However, research in that field is still hampered by the small size of datasets for the task, and the difficulty of generating one (Daume III and Marcu, 2004).

the dataset of Thadani and McKeown (2013) only consists of 1,858 fusion instances of which 873 have two inputs, 569 have three and 416 have four.


Note here, the goal of our model is to obtain responses not only semantically relevant to the corresponding queries, but also of good diversity and novelty.

in this work, embedding-based metrics  are adopted to evaluate semantic the relevance between queries and their corresponding generated responses, while dist-1, dist-2 (Li et al., 2016a) are used as diversity measures.


The additional KL term in Eq (1) prevents this behavior and forces the model to find a solution with, on one hand, low reconstruction error and, on the other, predicted posterior distributions close to the prior.

the decoder part of the VAE is capable of reconstructing a sensible data sample from every point in the latent space that has non-zero probability under the prior.


A recurrent language model instead models the full joint distribution of output sequences without having to make independence assumptions x 1 , z).

adding a recurrent layer on top of our fully feed-forward encoder-decoder architecture relieves it from encoding every aspect of a text fragment into the latent vector and allows it to instead focus on more high-level semantic and stylistic features.


We split  -six or more "funny votes" is funny -three or fewer funny votes is not funny Otherwise the word was discarded.

the final dataset includes 1939 instances of filled-in words: 1449 for training and 490 for validation.


Moreover, a paragraph vector is embedded in the same space as word vectors though it can contain multiple topics and words with multiple senses.

doc2vec (Le and Mikolov, 2014) doesn't perform well on Information Retrieval as described in (Ai et al., 2016a) and (Roy et al., 2016).


As a result, doc2vec (Le and Mikolov, 2014) doesn't perform well on Information Retrieval as described in (Ai et al., 2016a) and (Roy et al., 2016).

we expect a paragraph vector to be embedded in a higher dimensional space.


According to a previous study (Martínez-Gómez et al., 2017), EasyCCG achieves higher accuracy.

when the output of both C&C and EasyCCG can be proved, we use EasyCCG's output for creating features.


In this example, the sub-goals G 2 , G 3 , and G 4 match the premises P 2 , P 3 , and P 4 , respectively.

a ′ ⇒ B ′ can be proved without introducing axioms.


Again, by applying inference rules, we obtain the following sets of premises P and subgoals G: Here, the two sub-goals G 5 and G 6 do not match any of the premises, so the attempted proof of B ′ ⇒ A ′ fails.

we therefore attempt to inject additional axioms, but in this case no predicate in P shares the argument x 2 of the predicates bar(x 2 ) and in(e 1 , x 2 ) in G. no axiom can be generated.


Subject or object words can affect the similarity of sentence pairs.

the number of each case in unproved sub-goals, like subj(e 1 ) in Figures 3 and 5, is used as a feature.


Our logic-based method, however, interpreted the first sentence as a negative existential formula of the form ¬∃xP(x) and the second sentence as an existential formula ∃xP ′ (x).

it could easily handle the semantic difference between the positive and negative sentences.


Morphological analyzers help reduce sparsity by providing several out-of-context morpheme-based analyses for words, but they usually introduce ambiguity by returning multiple analyses for the same surface form.

the model would require a further step of morphological disambiguation to choose the correct analysis in context.


What distinguishes morphologically rich languages (MRL), like Arabic, from other languages is that their words include more morphemes (such as prefixes and suffixes) representing a number of morphological features, e.g., gender, number, person, mood, as well as attachable clitics.

table 1 shows the set of 16 Arabic morphological features we model MRLs tend to have more fully inflected words (types) than their poor counterparts.


For example, a tail consonant ㅆ strongly implies that the word is a past tense verb as in ᄀ ᅡ ᆻ ᄃ ᅡ (went), ᄋ ᅪ ᆻᄃ ᅡ (came), and ᄒ ᅢ ᆻᄃ ᅡ (did).

a jamo-level model can identify unseen verbs more effectively than word-or character-level models.


The evaluation corpora were all annotated with a coarse-grained tagset with 17 tags.

this LSTM tagger seems to be a well-performing, language-independent choice for learning models on coarse-grained tagsets.


Especially for the morphologically fine-grained tagsets this might have been problematic, as it is doubtful if all PoS tags of a morphological tagset do even occur on 50k tokens.

in order to evaluate the taggers using all available data, we will reproduce setups reported in the literature and compare the performance of the taggers to those results.


Taggers for Slavic languages often make use of additional resources such as morphological dictionaries, which we intentionally do not include to avoid human-crafted resources that are not available for all languages.

we do not expect to reach state-of-the-art performance, but we want to quantify the size of the gap.


On German-2 and Swedish-2, the Word-Char+ tagger is able to reach better results than the reported reference values except for Czech which uses a morphologically fine-grained tagset.

language-∆ to reference tagger Corpus Id # Tags Acc (%) HunPos CRF Word-Char+ Czech 1,574 95.9 -4.7 -3.2 -1.5 German-2 54 97.6 -0.1 -0.2 0.9 Swedish-2 153 96.1 0.0 -0.6 0.1 Table 4: Results of reproducing setups in the literature using the full corpus size fitted PoS taggers reach better results than neural networks when training models on corpora with extremely fine-grained PoS tagsets.


This baseline is important since non-neural, feature-based approaches have been found empirically to be more efficient, in the sense that their learning curves tend to be steeper.

in the low-resource setting we would be remiss to not consider a feature-based approach.


In particular, our context window size is five.

each word has four different versions of e c , namely e c −1 , e c −2 , e c +1 , and e c +2 , each taking a distinct embedding matrix.


However, it is hard to guarantee that every turker performs the survey carefully, given the difficult and long passages of high school problems.

to obtain the ceiling human performance on RACE, we manually labeled the proportion of valid questions.


Moreover, we found that both SECT-LSTM and SEDT-LSTM have better performance than their CNN counterparts, which suggests that LSTM can more effectively preserve the syntactic information.

we conducted further analysis of only SECT-LSTM and SEDT-LSTM models in the subsequent subsections and drop the suffix "-LSTM" for abbreviation.


The Children's Book Test focuses on the role of context and memory (Hill et al., 2016a), and the fictional genre makes it difficult to connect the entities in the stories to real-world knowledge about those entities.

language models have proved to be a highly competitive solution to these tasks.


In our problem setting, the model also has access to the lexical resource L = {L e | e ∈ E} where L e = (l e 1 , ..., l e k ) is the lexical definition of entity e extracted from the knowledge base.

the task of the model is to predict the correct missing entities for each empty slot in D. There are several possible ways to specify the candidate set E. For instance, we could define E so that it includes all entities found in the corpus.


For a given context C i , we replace the embedding of the blank token with d e .


We feed the sequence C i into the context encoder and as usual take the last hidden state as the context embedding h e i .

given C i and e ∈ E, the probability of e being the correct missing entity is: where again σ is the sigmoid function, W and b are model parameters.


However, Freebase also contains other types of valuable information, such as relational information between entities.

one potential direction for future work would be to incorporate both relational information and lexical definitions.


However, to train the answer synthesis module, we further augment the human-annotated labels of each paragraph with tags from a simple NER system 6 because labels of answers provided in the train set are underspecified, i.e., many words in the paragraph that could be potential answers are not labeled.

we assume any named entities could also be potential answers of certain questions, in addition to the answers explicitly labeled by annotators.


That is, most equations and structured answers extracted by our method are correct, and many problems are dropped from the dataset.

we get dataset Math23k which contains 23,161 problems labeled with structured equations and answers.


The output of our seq2seq model is a single equation containing one unknown variable.

our approach is only applicable to the problems whose solution involves one linear equation of one unknown variable.


In this example, the wide divergence in answer-space similarity neighborhoods between the two QAs would pull them apart, so they live in different parts of the latent space, reflecting the dissimilarity between their causes.

our contribution in this paper is a neighborhood-preserving method for CQA retrieval, LASER-QA, expanding to LAtent-Space Embedding for Rretrieval in QA archives.


In the datasets that we use, we have labels indicating which QAs are related/relevant to a particular question.

the quality of the scoring function can be evaluated using traditional information retrieval metrics (Robertson and Zaragoza, 2007) such as Precision, MAP, MRR, and NDCG when measured against such labellings.


We intend to learn the co-efficients such that x i may be reconstructed well as a linear combination of the neighbors using the co-efficients.

these co-efficients are computed by minimizing, for every question x i , the regularized reconstruction penalty (RRP) given below: The first term denotes the approximation error in reconstructing x i as a linear combination of its k nearest neighbors using weights w q ij .


Mining question patterns based on question clusters is motivated by the observation that, all questions within a QC tend to ask questions about an identical "question topic" (e.g., nyc in the above example) from different aspects, or the same aspect but using different expressions.

we can leverage the consensus information among questions to detect the boundary of the question topic: the more times an n-gram occurs in different questions within a question cluster, the more likely it is the question topic of the current question cluster.


The program generates Timegraphs to infer relations from the annotated TLINKs.

relations without explicit TLINK labels can still be used if they can be inferred from the annotations.


In particular, we follow the intuition that as CNNs learn the embeddings in a classification setting they should be used in the same way for building the reranking kernel, i.e., we need to use the embeddings in a pointwise reranking fashion.

we propose a hybrid preference-pointwise kernel, which consists in (i) a standard reranking kernel based on CTKs applied to the Q/AP structural representations; and (ii) a classification kernel based on the embeddings learned by neural networks.


Manual feature engineering can provide competitive results (Nicosia et al., 2015), however, it requires significant human expertise in the specific domain and is time-consuming.

machine learning methods for automatic feature engineering are extremely valuable.


3.2 are trained on TRAIN by construction, and therefore cross-validation on TRAIN would exhibit unrealistically high performance.

we employed the following disjoint Cross Validation approach: we train 5 models as in traditional 5-fold crossvalidation on TRAIN.


In contrast, we are interested in how onomatopoeic words relate to regular words (Table 3), which we study by explicit grounding in sound.

while we show competitive performance on this dataset, it might not be best suited for studying the benefits of our approach.


Unlike in COCO, the lack of a particular object label in an image for Visual Genome does not necessarily indicate that the object is not present, both due to annotation noise and the use of multiple synonyms for objects by human labelers.

we restrict the set of candidate irrelevant images to those which contain a matching object to the question premise but a different attribute.


In this work, we focus on stereotypical or prominent characteristics across a range of actions that can be denoted using the same verb.

three investigation points of our work include: (1) crowd-sourcing experiments to estimate the distribution of human judgments on the prominent characteristics of everyday physical action verbs, (2) the feasibility of learning models for inferring the prominent characteristics of the everyday action verbs despite the potential noise in the human annotation, and (3) their predictive power in zeroshot action recognition despite the potential noise from contextual variations of action attributes.


We use logistic regression to predict each attribute, as we found that extra hidden layers did not improve performance.

we let f emb (x v ) = w v , the GloVe embedding of v, and use Equation 1 to get the distribution over labels.


However, we found these annotations to be skewed towards "no emotion", since most verbs do not strongly associate with a specific emotion.


Given a pair of referring expressions , r i and r j are cosituational expressions.

for a word w i ∈ r i , we consider all words w j ∈ r j as its situational context.


(2015) found that the generality or concreteness of a noun's meaning is reflected in the entropy of its embedding, and we adopt that measure for our purposes.

we compute entropies of our word embeddings and report accuracies corresponding to the proportion of noun pairs where the entropy of the more general noun is higher than the more concrete noun.


Inspired by this approach, we use the multimodal corpus of referring expressions to set up a new task for evaluating word embeddings, which consists of capturing approximate inferential relations between referring expressions.

in our case, the hypothesis and the premise are expressions referring to objects, and the task is to determine whether they could (potentially) refer to the same object.


People may type variant length of 4, e.g.,, 4444444 to express their remarks.

alternatively, we model the audience chat with a character-level LSTM-RNN model (Graves, 2013).


VerbNet v3.2 3 can be represented as a 6334 verb × n frame binary matrix, with 1s in cells with attested verbframe pairs (Figure 1a).

each verb is represented as a binary vector of frames.


While generating demonstrations for moving a single block is straightforward, disambiguating action ordering when multiple blocks are moved is challenging.

we focus on instructions where a single block changes its position between the start and goal states, and restrict demonstration generation to move the changed block.


A secondary source of noise is due to discretization of the state space.

the agent often can not reach the exact target position.


However, the temporal relation between two events in a story depends on, and is often dictated by, relations among other events.

effectively identifying temporal relations between events is a challenging problem even for human annotators.


De-spite the existence of denser annotation schemes (e.g., ), the TLINK annotation task is quadratic in the number of nodes, and it is practically infeasible to annotate complete graphs.

the problem of identifying these unknown relations in training and test is a major issue that dramatically hurts existing methods.


Evaluating these systems remains a challenge as it is not economically feasible to exhaustively annotate every possible candidate relation from a sufficiently large corpus.

a poolingbased methodology is used in practice to construct datasets, similar to them methodology used in information retrieval (Jones and Rijsbergen, 1975;Harman, 1993).


This ensures that the convolutional kernel only operates over consecutive terms that existed in the original document.

we end up with l g − 1 matrices C n lq×l d ×n f , and the original similarity matrix is directly employed to handle the signals over unigrams.


In this part, we would like to further examine the performance of the proposed models in re-ranking different sets of search results.

we extend our analysis to re-rank search results from all submitted runs from six years of the TREC Web Track ad-hoc task.


Specifically, a model's retrieval quality can be examined by checking across a range of individual document pairs, namely, how likely a model can assign a higher score for a more relevant document.

it is possible for us to compare different models over the same set of complete judgments, removing the issue of different initial runs.


Specifically, a Nav is more relevant for a user with navigational intent, whereas for other users it may in some cases be less useful than a document that directly includes highly pertinent information content.

we hypothesize that further improvements can be obtained by introducing a classifier for user intents, e.g., navigational pages, before employing neural IR models.


Sentences provide a natural and powerful document decomposition for search that can be easily learnt as a search step: for all the models and configurations considered, the Sentence score was above 88% correct (Table 3) 7 .

sentence selection is the easy part of the problem, and the model can allocate more computation (such as the end-word selection Bi-LSTM) to spans likely to contain the answer.


We update the proposal LSTM by using the sampling-derived P seg as a training target after each batch.

the proposal learns to predict segment boundaries that are likely to result in low reconstruction loss for the main network.


(4) Coverage for the visual information: images that appear in documents and videos often capture event highlights that are usually very important.

the summary should cover as much of the important visual information as possible.


Equations 8,9 and 10 are all monotone submodular functions under the budget constraint.

we apply the greedy algorithm (Lin and Bilmes, 2010) guaranteeing near-optimization to solve the problem.


We argue that this is due to the fact that the output neurons of Tensor Fusion are easy to interpret and semantically very meaningful (i.e., the manifold that they lie on is not complex but just high dimensional).

it is easy for the subsequent layers of the network to decode the meaningful information.


For ease of comparison with related work and within our own study, we associate each user with a single anchor tweet.

both annotators and classifiers are asked to deter-mine the stance of a user using data from one particular time window.


This assumption seems reasonable given that the voting population was strongly polarized during the general (post-primary) election season, and it doubles the amount of data we can use to train the models.

throughout this work the labels we use are taken from the set {"Support Trump / Oppose Clinton" = −1, "Neutral / I don't know" = 0, "Oppose Trump / Support Clinton" = 1}.


By contrast, moderators are baffled when asked to tune t r and t a directly.

we ask them to specify the approximate desired coverage.


According to the distributional hypothesis (Firth, 1957), one can extract a word's meaning from the contexts in which it appears.

if words' meanings change over time, as has been argued at least since Reisig (1839), it follows that the words' contexts should change accordingly, and this change should be detected by our model.


2, it turns out that the SVD procedure distorts data in an uneven manner -frequent words are distorted less than infrequent words.

we demonstrate that this reinstatement of correlation between frequency and change scores is merely an artefactual consequence of the truncated SVD factorization.


In the genuine condition, words appear in different usage contexts between decades, while in the shuffled condition they do not, because the random shuffling creates a homogeneous corpus.

the validity of the control condition is established if: (a) the change scores are diminished as compared to the genuine condition; (b) change scores are uniform across decades (since decades are shuffled); (c) the variance of change scores is smaller that in the genuine condition.


This supports the claim that any measurement is significantly affected by unrelated noise.

we have established that the shuffled condition is a suitable control for meaning change.


Next, frequency and polysemy were set as two fixed effect predictors in a linear mixed effect model, like the one described in Section 2.4.

we were able to replicate the results in the genuine condition as reported in (Hamilton et al., 2016).


We start by observing that, given the definition of the Dirichlet distribution, it follows that Considering the different sets of indices in isolation, let i=j 1 α 2 i , and ψ 2 = km 2 i=k 1 α 2 i .

we rewrite (7) for the two conditions: 1. w has one meaning: 2. w has two meanings: with some algebraic manipulations, it can be shown that M (1) > M (2) if the following holds: when (8) holds, the average cosine distance between two samples of a certain word w gets larger as w acquires more meanings.


Lemma 2 proves that the average cosine distance between two samples of the same word, when using count vector representations, is negatively correlated with the word's frequency.

the role of frequency cannot be evaluated as an independent predictor in any model based on count vector representations.


a compositional function c combines d user factor scores f u,d with original feature values x: a version of each feature exists with and without the factor information integrated.


We then apply our peak detection algorithm (Section 3.2) on the dynamics and use its output as one of the classifier's features (denoted by isPeak).

our feature vector is: We apply our techniques to two corpora: the first is a temporal corpora (Section 3.4.1), which we used for creating word embeddings, and the second is a relational corpora (Section 3.4.2), which we used for training our models and for evaluation.


The dataset is not balanced: it contains more negative examples than positive ones.

for evaluating the methods that involve a classifier we use stratified 10fold cross validation.


The original attention mechanism outputs the alignment weights from an extra full connection layer while we think the inner dot can represent the semantic relatedness adequately.

we consider the i-th row of M as the similarity between the i-th position of S and each position in T and normalize it as follows:  For T counterpart, the alignment representation is .


(2016) has showed the performances on Original TrecQA and Clean version TrecQA are not comparable.

for a fair comparison, we only display the results on Clean version TrecQA which are posted on the website of Wiki of the Association for Computational Linguistics 1 .


We can find the inter-attention layer gives very high weights over 0.25 (while the average weight is about 0.14) to "sleeping" and "eating" which are the only difference between two sentences.

the difference will be attended in following processing.


By exploiting facts derived from domain knowledge bases, a domain taxonomy would be learned via distant supervision and have higher coverage than existing methods (Alfarone and Davis, 2015).

it is an important task to construct a taxonomy based on a text corpus and a knowledge base of a specific domain.


Worse still, even for the same predicate, it is legitimate to use arbitrarily different predicate symbols, e.g., other symbols like hired by or even predicate1 can also be used for the employer predicate, reminiscent of the symbol grounding problem (Harnad, 1990).

directly transferring the mapping from natural language to predicate symbols learned from source domains to the target domain may not be much beneficial.


We ignore the relations here since we focus on aligning concepts.

the linearized concept sequences can be represented as g = {g 1 , g 2 , .


Once we find a match in all the template entries, we would substitute the placeholder in right hand side with the matched substring to get the candidate concept label.

even we haven't seen "teacher", by matching teacher with the regular expression (.


Although these issues would be somewhat alleviated by the creation of larger datasets, collecting human judgments for all possible word pairs is impractical.

the ability to predict association strengths between arbitrary word pairs represents the best solution to these coverage issues (Boyd-Graber et al., 2006).


We analyze the discourse with contrast relationship and find that the contrast information may result from different parts of sentence, e.g., tenses (e.g., previous vs. now), entities (their vs our), or even the whole arguments, etc.

in order to learn the relationship representation between two arguments, we propose an attention mechanism that can select out the most important part from two arguments and perform the information interaction between two arguments.


In this way, the new representations of the two arguments interact with each other.

this attention mechanism enables our model to focus on specific spans in the two arguments, which is crucial to recognize the discourse relations.


However, similar with many models in deep learning, one big issue is the lack of labeled data.

we propose a multi-task attentionbased neural network by integrating the aforementioned model into a multi-task learning framework to address the implicit discourse relation recognition with the aid of large amount of unlabelled data.


A simple and straightforward way is to equally share the knowledge learned from main task and auxiliary task.

the total loss of the multi-task neural network is calculated as: where Loss aux has the same weight as Loss main .


This mechanism acts as a gate to determine how much the information would pass through to the final result.

under the framework of multi-task and gated mechanism, the main and auxiliary tasks are capable of not only sharing the parameters of learning argument pair representation but also interacting the representations learning from each other.


In this paper, we address this issue by building a deep memory network that is capable of encoding zero pronouns into vector representations with information obtained from their contexts and potential antecedents.

our resolver takes advantage of semantic information by using these continuous distributed representations.


As shown in Chen and Ng (2016), traditional hand-crafted features are crucial for the resolver's success since they capture the syntactic, positional and other relationships between an AZP and its candidate antecedents.

to evaluate the importance of each candidate antecedent in a comprehensive manner, following Chen and Ng (2016) who encode hand-crafted features as inputs to their network, we integrate a set of features that are utilized in Chen and Ng (2016), in the form of vector (v (f eature) ) into our attention model.


A comparison between Tables 2 and 3 reveals that the averaging strategy similarly affects both groups of parsers.

the performance level among recent RST discourse parsers is much more homogeneous than the situation depicted in the literature.


Since negative results do not often get reported, there is no clear evidence in the literature regarding the non-utility of particular knowledge sources.

an absence of the explicit modeling of selectional preferences in the recent literature is an indicator that incorporating this knowledge source has not been very successful for coreference resolution.


This result indicates that the incorporation of selectional preferences creates many links for nonanaphoric mentions, which in turn decreases precision.

the overall performance does not improve substantially when system mentions are used.


In addition, GrASP allows to add task-specific attributes.

for context-dependent arguments detection we add boolean attributes indicating whether the term is related to the topic, whether it appears in a lexicon characterizing argumentative texts, or in a lexicon characterizing the topic.


Some terms even include the topic itself, such as "Politics and Government".

we exploited these tags to develop a standard supervised classifier for the topic categorization of editorials.


In total, for each argument component type, we consider 5 types of token-level statistics: the total number of words in the given component type, the length (in terms of word) of the shortest/longest component of the given type, and the mean/variance of the number of words in each component of the given type.

there are in total 7 × 5 = 35 features to represent the token-level statistics.


Similar to the token-level features above, we design 5 types of statistics and their combination ratios.

the dimension for the letter-level features is the same to that of the token-level features.


A given piece of text has a set of ACs, which occur in a specific order in the text: (C 1 , ..., C n ).

at encoding timestep i, the model is fed a representation of C i .


However, recent work has shown that joint models that simultaneously try to complete multiple aspects of the subtask pipeline outperform models that focus on a single subtask (Persing and Ng, 2016;Stab and Gurevych, 2014b;Peldszus and Stede, 2015).

we will modify the single-task architecture so that it would allow us to perform AC classification (Kwon et al., 2007;Rooney et al., 2012) together with link prediction.


This can be thought of as a representation of AC C i .

our joint model will simply pass this representation through a fully-connected layer as follows: where W cls , b cls become elements of the model parameters, Θ.


Lastly, we use softmax to form a distribution over the possible classes.

the probability of predicting the component type at timestep i is defined as: Finally, combining this new prediction task with Equation 2, we arrive at the new training objective: which simply sums the costs of the individual prediction tasks, and the second summation is the cost for the new task of predicting AC type.


In the cases of both WCS and BFG, all hidden units have their own independent peakiness.

s h and s f are vectors, applied elementwise after broadcasting the lattice score.


We check that suggested phrases do not overlap with already translated source words by keeping track of the sum of attention in previous time steps for each source position.

the problem of global reordering is left entirely to the NMT model and we follow the attention when hypothesizing phrases.


In both cases, the translation quality decreases.

both one-word phrases and longer phrases are necessary to obtain the best results.


In contrast, phrases are much better than words as translation units in SMT and have made a significant advance in translation quality.

a natural question arises: Can we translate phrases in NMT?


This is the reason why some sentences do not have corresponding phrasal recommendations (Table 2).

our approach can be further enhanced if we can reduce the error propagations from the segmenter or chunker, for example, by using n-best chunk sequences instead of the single best chunk sequence.


• β = 1.0: In the reverse of the first case, the weight of the leaf nodes is manually set to 0.0.

only the phrase vectors are used to predict the target words.


Furthermore, there have been no large-scale studies of how these hyperparameters affect the performance of NMT systems.

it remains unclear why these models perform as well as they do or how we might improve them.


While some work (Luong et al., 2015b;Luong and Manning, 2016;Wu et al., 2016) has achieved state-of-the-art results using deep networks, others (Jean et al., 2015;Chung et al., 2016;Sennrich et al., 2016b) have produced similar results with far shallower ones.

it is unclear how important depth is, and whether shallow networks are capable of producing results competitive with those of deep networks.


It has also been observed (Tu et al., 2017) that very large beam sizes, even with length penalty, perform worse than smaller ones.

choosing the correct beam width can be crucial to achieving the best results.


Similarly, self-loops will have label self .

we can simply assume that the GCN parameters are label-specific, resulting in the following computation, also illustrated in Figure 2: This model is over-parameterized, 3 especially given that SRL datasets are moderately sized, by deep learning standards.


Statistical models are typically less accurate when they are applied to out-of-domain data.

the predicted syntax for the out-ofdomain test set is of lower quality, which negatively affects the quality of GCN embeddings.


However, many of these logical forms are spurious, in the sense that they do not represent the question's meaning.

the objective must marginalize over the many logical forms generated in this fashion: We optimize this objective function using stochastic gradient descent.


However, this approach does not precompute correct logical forms.

computing its gradient requires running a wide beam search, generating, e.g., 300 logical forms, executing each one to identify which are correct, then backpropagating through a term for each.


The second term favours values of the evaluation variable that conforms with the most likely interpretations of the corresponding statement (s j ) by the semantic parser.

in the E-step, we upweight evaluations of statements that are both discriminative, as well as supported by interpretations from the semantic parser.


While we have chosen a particular instantiation of a semantic parsing formalism, our learning approach is independent of the semantic parsing framework in principle, and only assumes a loglinear parametrization over logical interpretations of sentences.

while we present results for a particular parsing framework and lexicon, the method may conceptually extend to other parsing formalisms such as DCS (Liang et al., 2011).


The architecture in Section 3 also acts as a semantic composition model, extracting the meaning of the phrase by combining the meanings of its component words.

we performed a qualitative experiment to investigate: (1) how well do traditional compositional methods capture metaphors, without any fine-tuning; and (2) whether the supervised representations still retain their domain-specific semantic information.


9 If a human reader were to observe at least one sentence that states a person was killed by police, they would infer that person was killed by police.

we aggregate an entity's mention-level labels with a deterministic disjunction: At test time, z i is latent.


Therefore we aggregate an entity's mention-level labels with a deterministic disjunction: At test time, z i is latent.

the correct inference for an entity is to marginalize out the model's uncertainty over z i : e) ) = 1 − P (y e = 0|x M(e) ) (4) Eq.


The ability to question the actions and intentions of governments is a crucial part of democracy (Pitkin, 1967), particularly in parliamentary systems.

scholars have studied parliamentary questions in detail, in terms of their origins (Chester and Bowring, 1962), their institutionalization (Eggers and Spirling, 2014) and their importance for oversight (Proksch and Slapin, 2011).


If the keywords extracted by our algorithm is located within the boundary of annotation, it is treated as correct.

we can obtain the precision, recall, and F-measure by comparing the proposed keywords with the correct keywords by: Since the reference methods do not focus on the keywords level, we only compare the performance of Memnet and ConvMS-Memnet in Table 6.


The sentiment polarity of a word was determined by a look-up from pre-trained sentiment lexica (Liu et al., 2005;Wilson et al., 2005) 3 .

the L length context can now be viewed as a sequence of its segment's sentiments.


A variety of NLP tasks are limited by scarcity of manually annotated data.

co-occurring emotional expressions have been used for distant supervision in social media sentiment analysis and related tasks to make the models learn useful text representations before modeling these tasks directly.


In many cases, emojis serve as a proxy for the emotional contents of a text.

pretraining on the classification task of predicting which emoji were initially part of a text can improve performance on the target task (see §5.3 for an analysis of why our pretraining helps).


Our hypothesis is that the content obtained from the URL is likely to be important for understanding the emotional content of the text in the tweet.

we expect emojis associated with these tweets to be noiser labels than for tweets without URLs, and the tweets with URLs are thus removed.


We approximate it using three factors, specifying soft membership of products and users (Ding et al., 2006) by finding: where F ∈ R nt×K represents the posterior probability of K topic clusters for each product; S ∈ R K×K encodes the distribution of each topic k; and T ∈ R K×nu indicates the posterior probability of K topic clusters for each user.

of matrix factorization, we directly obtain the probability of each user on each topic from the person-topic matrix T .


However, too many hops leads to over-fitting, thereby harms the performance.

we choose 3 as the number of hops in our final test.


The dual problem is then: Note that, as there is no constraint on z, if i λ i = 0 then L * ({λ i }) = +∞.

we can restrict the domain of {λ i } in the dual problem to the set Λ = {{λ i }| i λ i = 0}.


However, we know that our optimal solution must satisfy tree-shape constraints (5).

every cluster except the root must have exactly one incoming arc and there is at most one arc between two clusters.


We follow a common approach in dependency parsing and assign labels l in a postprocessing step, although our model is able to incorporate label scores directly.

we are left with jointly decoding a dependency structure and assigning a sequence of spines.


Thanks to using a neural network model, which removes the need for feature engineering, we can concentrate on the question of which representations are more relevant for the model at each step of the flow.

we reflect on how alternative representations should impact the task, and we report their relative contribution in an ablation study.


In the case of non-projective parsing, the extraction of the oracle is not trivial because there can be many possible oracles that would derive the same tree.

it is common practice to use some heuristic to extract only one of the possible oracles.


Le and Zuidema (2016) have shown that, for deep and complex hierarchical structures, the models that have a memory akin to the memory in LSTM are much more robust towards the vanishing gradient problem.

in our work we use the Tree-LSTM neural architecture from Tai et al.


Second, our NCRF-AE highlights a unified structure of different loss functions for labeled and unlabeled data with shared parameters.

during training, our model can address both labeled and unlabeled data well by alternating the loss functions.


TAG differs from CCG in having a smaller set of combinatory operations, but a more varied set of elementary objects: the TAG-annotated version of the Penn Treebank that we use (Chen, 2001) includes 4727 distinct supertags (2165 occur once) while the CCG-annotated version (Hockenmaier and Steedman, 2007) includes 1286 distinct supertags (439 occur once).

building a robust, broad-coverage TAG parser has proven difficult.


Each state is represented by c = (s, b, A) where s, b, and A denote the stack, buffer and set of dependency relations derived so far.

our objective is to predict a transition operation given the configuration set c. The initial configuration is defined as s , and A = ∅ where n is the number of tokens in the sentence w 1 w 2 • • • w n .


We expect that the embeddings for common supertags would be better representations than embeddings for rare supertags.

we restricted our experiment to analogies between supertags among the 300 most common ones in the training set.


However, knowing that in the given context, it is the second argument for the relation "live in" helps concluding that it is a location.

we propose a single neural network (NN) for both tasks.


For a sentence classification task, the input sequence for the CRF layer is not inherentely clear.

we propose to model the joint entity and relation classification problem with the following sequence of scores (cf., Figure  2): with r ij being the relation between e i und e j .


Therefore, we propose to model the joint entity and relation classification problem with the following sequence of scores (cf., Figure  2): with r ij being the relation between e i und e j .

we approximate the joint probability of entity types T e 1 , T e 2 and relations R e 1 e 2 as follows: Our intuition is that the dependence between relation and entities is stronger than the dependence between the two entities.


End-to-end learning prevents error propagation in the pipeline approach, and allows cross-task dependencies to be modeled explicitly for entity recognition.

it gives better relation extraction accuracies compared to pipelines.


We shall refer to the task of manually classifying a single belief as true or false as a Belief Evaluation Task (BET).

the crucial problem is: How can we select a subset of beliefs to evaluate which will best estimate the true (but unknown) accuracy of KG and its predicates?


Although distributional models have evolved over time, from count-based (Landauer et al., 1998) and generative (Blei et al., 2003) to prediction-based (Mikolov et al., 2013), the similarity between distributional vectors still indicates only the abstract semantic association and not a precise semantic relation (e.g., vectors of antonyms may be as similar as vectors of synonyms).

a number of approaches have been proposed for specializing distributional spaces for specific lexico-semantic relations, either by (1) modifying the learning objective or regularization of the original embedding model by incorporating linguistic constraints (Yu and Dredze, 2014;Kiela et al., 2015) or (2) retroactively fitting the pre-trained unspecialized embeddings to linguistic constraints (Faruqui et al., 2015;Mrkšić et al., 2016).


The Dual Tensor model can be considered distributional as it requires only distributional vectors of words as input.

in contrast to path-based methods, it is language-independent and more widely applicable.


We assume that we can make a prediction about (h, t), once we know the information of (h, e) and (e, t).

we build the training set by extracting all such relation paths and their sentences from training text, and similar for testing set.


Some relation types may tend to co-occur, such as DI-RECTED BY and PRODUCED BY, whereas others may be restrictive (e. g. one can only have a single PLACE OF BIRTH).

in addition to the target entity pair, we take other entities from the same sentence that were extracted at the data generation step.


However, event pairs can be very far in a sentence and are involved in complex syntactic structures.

we also apply two heuristic rules to deal with complex syntactic structures, e.g., two event mentions are in separate clauses and have a punctuation sign in their context.


Rule 1 (punctuation): Comma directly influences the meaning in text and omitting it may alter the meaning of phrase.

include comma if it precedes or follows e 1 , e 2 or their modifiers.


It should be noted that if a word is too far from entities, it may be not related to the relation.

we choose maximum value e max and minimum value e min for the relative distance.


Among others, it managed to precisely spot the 2001 insurgency of the members of the Kosovo Liberation Army in Macedonia, notwithstanding the fact that the initial set of training pairs did not mention Macedonia at all.

it seems that the models at least partially 'align' new data along the existing semantic axis trained before.


For example, the 2005 model hardly contains much knowledge about the conflict relations between Mexico and the Popular Revolutionary Army (EPR) which stopped its activities after 1996.

we additionally conducted a similar experiment, but this time the projections were learned only on the salient pairs (dubbed previous): that is, the pairs active in the last year up to which the model was trained.


comparable to that of the up-to-now projections on the accuracies @5 and @10, and is even higher on the accuracy @1 (statistically significant with t-test, p < 0.01).

the single-year projections are somewhat more 'focused', while taking much less time to learn, because of less training pairs.


, 1 + max t <t e t }, i.e., the indices of all previously mentioned entities plus an additional value for a new entity.

e t starts as {1} and grows monotonically with t, allowing for an arbitrary number of entities to be mentioned.


This assumption is unrealistic for pure generative models like ours.

we remove this assumption and adapt their prediction results to our formulation by mapping all the predicted entities that have not been mentioned to "new entity".


We defined ρ s 0 as a diagonal N × N matrix containing the classical Maximum Likelihood probability Estimation to have a specific symbol at the first sequence position: where S is again the set of all sequences in the training set and w 1 is the first word in each sequence w. With regard to the ancilla system we do not know anything about it and thus we have to define ρ a 0 as the D × D diagonal matrix .

we can define ρ 0 as Putting all the ingredients together, we can finally write down the formula for the probability P (w|ρ 0 , U) for a sequence w in the QLM specified by ρ 0 and U.


We have to say that in this experiment we were not trying to build the best possible phone recogniser, but simply to compare the relative performances of the analysed LM techniques showing the effectiveness of QLM when used in a real application.

absolute Phone Error Rate is not so important here and it can be certainly possible to devise recognisers with better performances by applying more sophisticated techniques.


In this way, the model can automatically learn when to refer to tokens in the ingredients.

the probability of generating token y v is defined as: If no string match is found for y v , we simply set p copy v (y v ) = 0 in the above objective.


Pointer Switch: We use z i,v ∈ {0, 1} to denote the decision of whether to copy one cell from the table.

we compute this probability as follows: if z i,v = 1, the next token y i,v is generated from the database, whereas if z i,v = 0, then the following token is generated from a softmax.


Actually achieving this maximal value is typically infeasible, since the embedding dimensionality is intentionally limited.

learning word and context embeddings that optimize skip-gram's NEG objective function (4) can be viewed as finding a low-rank approximation of the word-context PMI matrix.


Being aware of which n-grams are important, CNN is able to ex-  The intuition behind our method is simple: Since CNNs essentially capture semantic features of ngrams, we can use important n-grams to initialize the filters.

the filters are able to focus on extracting those important n-gram features at the beginning of the training.


We set the size of the window over the processed text equal to 2.

in our graph-of-words representation of a document, each term is linked with its preceding and its following term with an edge.


Our intuition is that given an initial term, terms that are close to terms that are close to the initial term or beyond, may have also some relation with the initial term, and the strength of this relation decreases as the shortest path length increases.

although the proposed kernel does not incorporate any knowledge of the language being used, it does capture some statistical information and is thus capable of outperforming metrics based on the unigram and even n-gram vector-space model.


In order to determine the kernel value, it is necessary to compute the value of k walk for all pairs of edges between the two transformed graphs.

the number of edges in the transformed graph can be at most n 2 in the case all the shortest paths in the original graph are no longer than d. there are at most n 2 • n 2 = n 4 pairs of edges.


n i has an additional child node which represents v i , to ensure that every word appears at the leaf level at least once ( Figure 2).

3 if the vocabulary size is V , there will be a total of (N + 1)V leaf nodes.


To further complicate matters, some workers may produce spam or malicious responses.

it is not uncommon for workers to correlate poorly with one another.


These methods are best suited for simple, straightforward tasks; with noisier data such as that which may be obtained for more difficult or subjective tasks, these strategies may produce skewed labels that misrepresent the instance.

it is desirable to devise more effective aggregation strategies that consider factors such as label distribution and worker quality, while still avoiding manual adjudication of all instances.


We posit that although annotator quality is an important factor in predicting accurate aggregations, the interplay between it and other factors is more nuanced.

rather than adapting the item-response method, our learning approach incorporates features that address multiple potential flaws or biases in crowdsourced annotations.


The chunking dataset used here is relatively small compared with other low-level tasks, POS tagging and dependency parsing.

these results suggest that the successive regularization is effective when dataset sizes are imbalanced.


Clearly, if the two sets of embeddings align well, the earth mover's distance will be small.

we can try to find the transformation that minimizes the earth mover's distance.


They can generate sharp images if trained well, but they are notoriously difficult to train.

a lot of research efforts have been dedicated to the investigation into stabler training (Radford et al., 2015;Salimans et al., 2016;Nowozin et al., 2016;Metz et al., 2016;Poole et al., 2016;Arjovsky and Bottou, 2017), and the recently proposed Wasserstein GAN ) is a promising technique along this line of research.


As neural networks are universal function approximators (Hornik, 1991), we can attempt to approximate f with a neural network, called the critic D, with weight clipping to ensure the function family is K-Lipschitz.

the objective of the critic is Conceptually, the critic D assigns scores f D to real target embeddings and fake ones generated by the generator G. When the objective (6) is trained until optimality, the difference of the scores will approximate the Wasserstein distance up to a multiplicative constant.


However, the decoding speed is significantly worse since the decoder needs to apply K NMT models rather than only one.

a recent line of research transfers the idea of knowledge distillation (Bucilu et al., 2006;Hinton et al., 2014) to NMT and trains a smaller network (the student) by minimizing the cross-entropy to the output of the ensemble system (the teacher) (Kim and Rush, 2016;Freitag et al., 2017).


The second intuition of the criterion used by Srinivas and Babu (2015) is that neurons with small outgoing weights contribute very little overall.

they search for a pair of neurons i, j ∈ [1, m] according the following term and remove the j-th neuron.


Shrinking the maxout layer remains more of a challenge (rows (g) and (h)), but the number of parameters in this layer is small.

shrinking all layers except the maxout layer leads to almost the same number of parameters (factor 1.05 in row (f)) as the original NMT network (a), and thus to a similar storage size, memory consumption, and decoding speed, but with a 1.8 BLEU gain.


7 Networks with the size factor 1.0 like 3-Unfold-Normal are very likely to yield about the same decoding speed as the Single network regardless of the decoder implementation, machine learning framework, and hardware.

we think that similar results are possible on other platforms as well.


Analyzing user activities (Farajtabar et al., 2017) and interactions (Castillo et al., 2011;Mukherjee and Weikum, 2015) to evaluate the credibility may not be appropriate for satirical news as it cannot prevent the spreading.

we utilize content-based features, including psycholinguistic features, writing stylistic features, structural features, and readability features, to address satirical news detection.


In Wikipedia, citations are implicit (see Table 1) and there are no cue words in text which link to the provided citations.

the proposed methodologies and features from the scientific domain do not perform optimally in our case.