Since its first use for machine translation (Kalchbrenner and Blunsom, 2013;Cho et al., 2014;Sutskever et al., 2014), the encoder-decoder approach has demonstrated great success in many other sequence generation tasks including image caption generation (Vinyals et al., 2015b;Xu et al., 2015), parsing (Vinyals et al., 2015a), dialogue response generation (Li et al., 2016a;Serban et al., 2016) and sentence summarization (Rush et al., 2015;Chopra et al., 2016).

in this paper we focus on sentence summarization, which as its name suggests, consists of generating shorter versions of sentences for applications such as document summarization (Nenkova and McKeown, 2011) or headline generation (Dorr et al., 2003).


Instead, we define the legitimate range of the sequence by setting minimum and maximum lengths.

in addition to the normal beam search procedure, we set two rules: • If the model generates the EOS tag when the output sequence is shorter than the minimum length, we discard the sequence from the beam.


We then replace its last word with the EOS tag and add this sequence to the beam (EOS replacement in Section 4.1).

3 we keep only the sequences that contain the EOS tag and are in the defined length range.


Inspired by previous work that has demonstrated that additional inputs to decoder models can effectively control the characteristics of the output (Wen et al., 2015;Li et al., 2016b), this model provides information about the length in the form of an additional input to the net.

the model uses an embedding e 2 (l t ) ∈ R D for each potential desired length, which is parameterized by a length embedding matrix W le ∈ R D×L where L is the number of length types.


Figure 4 shows the architecture of LenInit.

the model uses the memory cell m t to control the output length by initializing the states of decoder (hidden state s 0 and memory cell m 0 ) as follows: where b c ∈ R H is a trainable parameter and length is the desired length.


While the model of LenEmb is guided towards the appropriate output length by inputting the remaining length at each step, this LenInit attempts to provide the model with the ability to manage the output length on its own using its inner state.

the memory cell of LSTM networks is suitable for this endeavour, as it is possible for LSTMs to learn functions that, for example, subtract a fixed amount from a particular memory cell every time they output a word.


LenEmb (0,L) and LenInit (0,L) in the table).

we use the same beam search as that for f ixRng with minimum length of 0.


• The substitution cipher function applied to one text, results in the other text, and vice versa.


In specific domains (e.g., product reviews) where such public corpora are unavailable, one can utilize distant supervision to automatically label the corpus (Ling and Weld, 2012).

an entity linker will detect mentions m i and map them to one or more entity e i in E Ψ .


Each type label y k ∈ Y is represented by a K-dimensional binary indicator vector y k ∈ {0, 1} K , where y k,k = 1, and 0 otherwise.

we aim to learn a mapping function from the mention's feature space to a lowdimensional vector space, i.e., Φ M (m i ) : and a mapping function from type label space to the same low-dimensional space, i.e., Φ Y (y k ) : In this work, we adopt linear maps, as similar to the mapping functions used in (Weston et al., 2011).


We evaluate personas as features for automatic analysis of framing and tone, as defined in the MFC ( §4).

we build multi-class text classifiers (separately) for the primary frame and the tone of a news article, for which there are 15 and 3 classes, respectively.


We therefore make use of Bayesian optimization (Bayesopt) to make as many modeling decisions as possible (Pelikan, 2005;Snoek et al., 2012;Bergstra et al., 2015;Yogatama et al., 2015).

let F be the set of features that might be used as input to any text classification algorithm.


The Teams corpus presented and used in this paper was designed to add several notable extensions to existing multi-party spoken dialogue resources.

the Teams corpus was experimentally collected to constrain the team processes, tasks, and outcomes in ways that facilitate an investigation of team entrainment.


team size and session length were both significantly positively associated with entrainment on pitch-max.


Specifically, team size and session length were both significantly positively associated with entrainment on pitch-max.

as team size or session length increased, entrainment also increased.


The amount of variance explained for participation dominance was significant above and beyond the variables entered in Model 1, ∆R 2 = 0.08, ∆F (1, 58) = 6.307, p = 0.015.

there was a significant negative association between participation dominance and entrainment on pitch-max, such that greater participation equality was related to greater entrainment.


Task 2) was used to validate whether a receiver prefers the message customized for her or not.

in Task 1, the Turkers were asked to imagine that they work for a marketing firm on a campaign to promote a new car.


In our first study, we employed regression analysis to identify significant correlations between personal traits and aspect ranks.

we trained eight linear regression models, one for each of the eight car aspects.


However, these were developed for sentence-level interpretation and only ensure within sentence consistency of labels (Punyakanok et al., 2004;Toutanova et al., 2005;Lewis et al., 2015), limiting their ability to generate coherent characterizations of the process overall.

the same process participant may appear in text at different syntactic positions, with different wording, and with different verbs, which makes it hard to extract globally consistent descriptions.


We also add features that are specific to the nature of the process sentences.

we encode syntactic relationships of arguments with respect to the process name mention in the sentence.


We also studied the effect of varying the number of arguments that ILP uses to measure the compatibility of role assignments.

we allow inference to use just the top k alignments from the alignment classifier.


We construct three author social networks based on the follower, mention and retweet relations between the 1,317 authors in the NEEL-train dataset, which we refer as FOLLOWER, MENTION and RETWEET.

we use the Twitter API to crawl the friends of the NEEL users (individuals that they follow) and the mention/retweet links are induced from their most recent 3,200 tweets.


User embeddings We obtain low-dimensional Twitter author embeddings E (u) using LINE -the recently proposed model for embedding information networks (Tang et al., 2015).

we train LINE with the second-order proximity, which assumes that Twitter users sharing many neighbors are close to each other in the embedding space.


Compositions of user, mention, and entity The distributed representations of users, mentions, and entities offer additional information that is useful for improving entity disambiguation capability.

we explore the information by making two assumptions: socially connected users are interested in similar entities (entity homophily), and semantically related mentions are likely to be linked to similar entities.


As shown in Table 4, by learning the interactions between mention and entity representations, NTEL with mention-entity bilinear function outperforms the NTEL baseline system by 1.8% F1 on average.

the bilinear function results in considerable performance gains in recalls, with small compromise in precisions on the datasets.


We find that using sparse character category and capitalization features in conjunction with attention and phonological features yields the best 0-shot transfer performance (no training labels in the target language).

attention provides +6 F1 in 0-shot and 5% labeled-target language data scenarios.


This category includes models that approximate (1) based on the Markov dependence assumption of order N −1.

the prediction of the current word depends only on the last N − 1 words in the history.


The results shown in Table 3 generally confirm the conclusions we drew from the PTB experiments above.

we can see that the proposed LSRC model largely outperforms all other models.


In particular, we can see that the proposed LSRC model largely outperforms all other models.

lSRC clearly outperforms lSTM with a negligible increase in the number of parameters (resulting from the additional 200 × 200 = 0.04M local connection weights U c l ) for the single layer results.


To address this need, we have developed an approach on learning procedural tasks jointly from language instruction and visual demonstration.

we use And-Or Graph (AoG), which has been used in many computer vision tasks and robotic applications (Zhao and Zhu, 2013;Li et al., 2016;, to represent a hierarchical task model that not only captures symbolic concepts (extracted from language instructions) but also the corresponding visual state changes from the physical environment (detected by computer vision algorithms).


This step searches for new And-node candidates from Σ∪N , and update N and R with the top-ranked candidates.

we denote an And-node candidate to be searched as A = (s l → s m → s r ).


Our key focus is consistency -correct hypotheses from different modules will be correct in a consistent way, but incorrect hypotheses will be incorrect in incompatible ways.

we develop a MEDIATOR model that scores pairs for consistency and searches over all M 2 pairs to pick the highest scoring one.


This is a competitive single-module baseline.

we use the same parse-based features as our approach, and learn a reranker over the M z parse trees (M z = 10).


The discrepancy gets more severe as the length of the correct sequence increases.

even after the training loss converges to a small value, full inference on the training data can incur errors causing the model to be underfitted for long sequences in spite of low training cost.


For a significant fraction of cases (27% for Reddit, and 21% for Subtitles), the local margin is positive while the global margin is negative.

the ED loss for these sequences is small even though the log-probability of the correct sequence is much smaller than the logprobability of the predicted wrong sequence.


These models represent the current state of the art in NMT, and are also convenient for use in our proposed method.


In our first bias method, we use p l (•) to bias the probability distribution calculated by the vanilla NMT model.

we add a small constant ϵ to p l (•), take the logarithm, and add this adjusted log probability to the input of the softmax as follows: We take the logarithm of p l (•) so that the values will still be in the probability domain after the softmax is calculated, and add the hyper-parameter ϵ to prevent zero probabilities from becoming −∞ after taking the log.


First, compared to the baseline attn, our bias method achieved consistently higher scores on both test sets.

the gains on the more difficult KFTT set are large, up to 2.3 BLEU, 0.44 NIST, and 30% Recall, demonstrating the utility of the proposed method in the face of more diverse content and fewer high-frequency words.


We motivated, by way of examples, as to why similarities along question and answer spaces be better composed using methods other than simple sum or average type aggregation.

we noted that there are potentially different ways to answer questions pertaining to the same root-cause, mitigating the manifestation of the inherent root-cause similarity in the answer-space.


Our novel, character-level encoder-decoder model is compact, requires significantly less data to train than previous work, and is able to generalize well to unseen entities in test time.

without use of ensembles, we achieve 70.9% accuracy in the Freebase2M setting and 70.3% accuracy in the Freebase5M setting on the SimpleQuestions dataset, outperforming the previous state-of-arts of 62.7% and 63.9% (Bordes et al., 2015) by 8.2% and 6.4% respectively.


This indicates that the character-level model learns that a space defines an ending point of a complete linguistic unit.

the hidden state of the LSTM encoder at a space likely summarizes content about the character sequence before that space, and therefore contains important semantic information that the decoder needs to attend to.


Formally, we parametrize the proportions using a softmax link .

the data generator distribution is: where p k (x, y) are data distributions, called base generators, that are provided by domain experts, and ∆ is a K-dimensional centered Gaussian with an identity covariance matrix.


We decompose semantic relations in a story into a set of local, lexical relationships between pairs of words.

we consider semantic relations for noun-noun and verb-verb pairs as provided by WordNet (Miller, 1995).


As shown in Figure 3, a two-layer neural network is used to induce τ jt .

a hidden layer combines h L t and h R t using a nonlinear tanh activation The resulting hidden vector p s j t is then mapped into τ jt using another tanh layer.


We use the attention model  to this end.

the base score features h L t /h R t and the bias score features h L e /h R s are combined in the calculation where and Here σ denotes the sigmoid activation function and ⊕ denotes vector concatenation.


Our method achieves highly competitive accuracies.

compared to sequential LSTMs, our best model gives the top result both on the binary and fine-grained classification task.


The reason is that extraction result is based on ranking and only top ranked event affects the performance.

precision is more important than recall here.


We bring in User Product Attention to capture the crucial components over different semantic levels for sentiment classification.

we employ word-level UPA to generate sentence representations and sentence-level UPA to obtain document representation.


(t) and c (t) are the input gate, forget gate, output gate, and memory cell activation vector at time-step t respectively, all of which have the same size as the hidden vector  LSTM is supposed to capture the long-term and short-term dependencies simultaneously, but when dealing with considerably long texts, LSTM also fails on capturing and understanding significant sentiment message (Le et al., 2015).

the error signal would nevertheless suffer from gradient vanishing in modeling long texts with hundreds of words and thus the network is difficult to train.


This greatly extends the applicability to massive structures in diverse forms, such as structured models and soft logic rules, facilitating practitioners to incorporate rich domain expertise and fuzzy constraints.

we propose a mutual distillation method that iteratively transfers information between DNN and structured knowledge, resulting in effective integration of the representation learning capacity of DNN and the generalization power of structured knowledge.


This section introduces the proposed framework that enables joint learning of knowledge components and their weights with the neural network models.

we generalize the one-sided distillation method of (Hu et al., 2016) (section 3.1), and propose to mutually transfer information between the neural network and the structured constraints for effective knowledge learning (section 3.2), and optimize the weights by considering jointly all components (section 3.3).


(1), and write it in an analogous form to the variational free energy of some model evidence.

let log h φ (X, Y ) Cλf φ (X, Y ), then the objective can be written as Intuitively, we can view the output distribution of the neural network p(Y |X) as a prior distribution over the labels, while considering h φ (X, Y ) as defining a "likelihood" metric w.r.t the observations, making the objective analogous to a (negative) variational lower bound of the respective "model".


We address this by incorporating rich lexicon rules at the clause level.

if a polarity-carrying word (e.g., "good") occurs in the scope of a negator (e.g., "not"), then the sentiment prediction of the clause is encouraged to be the opposite polarity.


We leverage a graph-based algorithm for the computation of the sense biasing words.

we use the Personalized PageRank (Haveliwala, 2002, PPR) algorithm which has been extensively used by several NLP applications (Yeh et al., 2009;Niemann and Gurevych, 2011;.


We achieve this for each sense s i by de-conflating the representation v s i of its corresponding lemma and biasing it towards the representations of the words in B i .

we obtain a representation v * s i for a word sense s i by solving: where v s i and v b ij are the respective word representations (∈ V) of the lemma of s i and the j th biasing word in the list of biasing words for The first term in Formula 1 requires the representation of the word sense s i (i.e., v * s i ) to be similar to that of its corresponding lemma, i.e., v s i , whereas the second term encourages v * s i to be in the proximity of its biasing words in the semantic space.


In this paper, we propose a deep architecture to model the strong interaction of sentence pair with two coupled-LSTMs.

we introduce two coupled ways to model the interdependences of two LSTMs, coupling the local contextualized interactions of two sentences.


The output of coupled-LSTMs at each step depends on both sentences.

we propose two interdependent ways for the coupled-LSTMs: loosely coupled model (LC-LSTMs) and tightly coupled model (TC-LSTMs).


2015, verbs were excluded that occur in certain syntactic environments that interfere with property judgments.

participles and imperatives were excluded, as well as verbs in embedded clauses, in questions, and under negation or auxiliaries.


When an incorrect arc is preferred by the ensemble over the gold arc (hence assigned a cost/margin of 0), the model will still incur a loss if s(h y (m), m, x) < s(h y (m), m, x).

the score of any incorrect arc (including those strongly preferred by the ensemble) cannot be higher than the score of the gold arc.


In this paper, we apply shift-reduce parsing to semantic parsing.

we study transferring a learned Combinatory Categorial Grammar (CCG; Steedman, 1996Steedman, , 2000) from a dynamic-programming CKY model to a shift-reduce neural network architecture.


Thus, toys should be the head of the query, and kids should be its modifier.

when the semantics of a query is understood, we can often recover its syntax.


We may then manually annotate web queries.

given a query, a human annotator forms a sentence that is consistent with the meaning he comes up for the query.


Another approach (Wang et al., 2014) is to use search log to estimate P (e|x, y), where x and y are nouns.

we find queries of pattern x PREP y, where PREP is a preposition {of, in, for, at, on, with, • • • }.


We train head-modifier embeddings for each token, and use such embeddings as pre-trained embeddings.

we use an approach similar to training word2vec embeddings but focusing on head modifier relationships instead of co-occurrence relationships.


According to Lin and Bilmes (2011), there exists a simple greedy algorithm for monotonic submodular function maximization where the solution is guaranteed to be close to the real optimum.

if we denote R i greedy as the approximation optimized by greedy algorithm and R i * as the best possible solution, then F( is the objective function of Equation 1 and e ≈ 2.718 denotes the natural constant.


Therefore, in future work, we plan to expand our work to broader applications where interconnectivity between consecutive instances is crucial, such as educational lectures, news series and book chapters.

tREM can be integrated with video description results to get an end-to-end system that produces video recaps.


if a review example/point x i j is among the K nearest neighbors of the review point x i k , we put an edge linking the two examples, and vice versa.


We then develop a stochastic alternating method to cope with the optimization problem for SMTL-LLR, i.e., alternatively updating one parameter by fixing the other.

we initialize W with the values randomly chosen from [0, 1], and initialize Ω as a diagonal matrix, where Ω ii = 1 m .


In this work, we only consider new comments associated with the threads of the discussion that we are currently following to limit the number of possible sub-actions at each time step and with the assumption that prior context is needed to interpret the comments.

the new recommendation should focus on comments that are in the subtrees of previously recommended comments.


In the context of massive digitization projects, it can identify relationships between text excerpts referring to the same source.

detecting copies of the same historical text that have diverged over time (manuscript studies, a.k.a., Stemma Codicum) is an important task.


For example, Clement's reuse ranges from introducing the overall topic by citing multiple verses, to supporting his argumentation.

mk 10 30 is a fully literal reuse from a passage that discusses the problem of rich men in heaven.


Those nodes may not have many important connections (like their prestigious counterparts), but they are ideally placed at the core of the network.

this switches the objective from capturing the quality and quantity of single node connections, to taking into account the density and cohesiveness of groups of nodes.


More precisely, the Ktruss subgraph of G is its largest subgraph where every edge belongs to at least K − 2 triangles.

every edge in the K-truss joins two vertices that have at least K − 2 common neighbors.


It also appears that document size (i.e., graph-ofwords structure) is responsible for a lot of performance variability.

on longer documents, performance plateaus at higher window sizes.


These prediction results show that relative reading difficulty can be predicted for sentence pairs with high accuracy, even with fairly simple feature sets.

the results for AoA model D, which uses a small number of targeted features, are competitive with the best model B that relies on a much larger feature set.


To interpret this result, note that the parser and reranker likelihood represent the probability the parser and reranker models assign to the syntactic parse produced by the sentence.

they are a measure of how likely it is that the sentence structure occurs, based on the model's training data.


The final Fleiss' Kappa Value is 0.545, which indicates a moderate agreement level (0.41-0.60) based on the Kappa Interpretation Model (Fleiss and Cohen, 1973).

the performance of the annotators reveal that the NLD task is not simple, since there are many cases that are ambiguous and hard to make accurate judgments on, even for humans.


In this paper, we adapt a neural network joint model (NNJM) using L1-specific learner text and integrate it into a statistical machine translation (SMT) based GEC system.

we train an NNJM on general learner text (not L1-specific) and subsequently train on L1-specific data using a Kullback-Leibler divergence regularized objective function in order to preserve generalization of the model.


Dahlmeier and Ng (2011) implemented a system to correct collocation errors, by adding paraphrases derived from L1 into the confusion set.

they use a bilingual L1-L2 corpus, to obtain L2 paraphrases, which are likely to be translated to the same phrase in L1.


They first train an NNJM using in-domain data, and then perform regularized adaptation on the general domain data (concatenation of in-domain and out-of-domain data) which restricts the model from drifting away from the in-domain NNJM.

they add a regularization term J to the objective function in their adaptive training step: where p ID (y|h) id probability distribution estimated by the in-domain NNJM.


These languages have been in contact for a long time, hence there are many lexical and grammatical similarities among them, leading to the subcontinent being considered a linguistic area (Emeneau, 1956).

there is overlap between the vocabulary of these languages to varying degrees due to cognates, language contact and loanwords from Sanskrit (throughout history) and English (in recent times).


Neural Machine Translation Recent advances in neural machine translation (NMT) (Bahdanau et al., 2014;Devlin et al., 2014) using the framework of sequence to sequence learning (Sutskever et al., 2014) have demonstrated the effectiveness of deep learning models at capturing and translating language semantics.

recurrent neural networks augmented with attention mechanisms (Luong et al., 2015) have proved to be successful at handling very long sequences.


This paper investigates how linguistic knowledge mined from large text corpora can aid the generation of natural language descriptions of videos.

we integrate both a neural language model and distributional semantics trained on large text corpora into a recent LSTM-based architecture for video description.


We propose to take advantage of these to aid video description.

we replace the embedding matrix from one-hot vectors and instead use 300-dimensional GloVe vectors (Pennington et al., 2014) pre-trained on 6B tokens from Gigaword and Wikipedia 2014.


A convolutional layer combined with a number of wide convolutional filters is applied to E S .

the k-th convolutional filter operator filter k involves a weight vector w k ∈ R m×t , which works on every t k -sized window of E S , and is accompanied by a bias term b ∈ R. The filter operator is followed by the non-linear function F , a rectified linear unit, ReLU, followed by a maxpooling operation, to generate a hidden activation h k = MaxPooling(F (filter k (E S ; w k , b)).


In order to alleviate these issues, we introduce an RNN-LM that utilizes both character-level and word-level inputs.

our model has a gate that adaptively choose between two distinct ways to represent each word: a word vector derived from the character-level information and a word vector stored in the word lookup table.


(2013), have been based on syntactic structures.

it has been proven that Inversion Transduction Grammar (ITG) (Wu, 1997), which captures structural coherence between parallel sentences, helps in word alignment (Zhang and Gildea, 2004;Zhang and Gildea, 2005).


Calculate a source-to-target posterior probability − → p θ (z|x) and a target-to-source posterior probability ← − p θ (z|x) for each bilingual sentence x = {f , e} under the current model parameters θ, where z denotes an alignment in a sentence pair x.

z i,j =1, if f i is aligned to e j (otherwise z i,j =0).


Here, φ agree is a feature of agreement constraint, which assigns each alignment direction to a sign (i.e., +1 or -1).

φ agree is defined as follows: where − → Z and ← − Z are sets of possible alignments generated by source-to-target and target-to-source alignment models, respectively.


We focus on English to perform a direct comparison with supervised parsers (although unsupervised or weakly supervised approaches are likely to be most beneficial for domains or languages where supervised parsers are not available).

* Equal contribution we evaluate different parsing scenarios with varying amounts of supervision.


Their analysis was based on syntactic evaluation against CCGbank.

they found the most egregious "semantic" errors to be the misuse of verb chains, possessives and PP attachment (bottom of Table 3).


Taking dependency relationships into account (P (1p|r)), we observed shifts in the magnitude of correlations.

we found significant negative correlations between age and using 1st-person singular pronoun in the subject (r = −0.12, p < .001), and the object positions (r = −0.17, p < .001).


This suggests that there is more information to be gained by utilization syntactic and semantic context.

we can achieve a more meaningful, deeper insight into the relationship of subject and object position of the first person in different contexts, revealing a more complex, and more insightful set of relations.


It should also be noted that the results reported in (Klebanov et al., 2014) are not the same, due to the mentioned differences in the implementation of the features and possibly other differences in the experimental setup (e.g., data filtering, pre-processing and meta-parameter optimization).

our implementation of the B features performs better than reported by Klebanov et al.


Even though the evidence is not conclusive, these results suggest that the insights derived from the analysis of PROMETHEUS and captured by the feature set N can also be applied to model word-level metaphor detection across very different genres.

we believe that our initial attempt to encode context and domain information for metaphor detection deserves further investigation.


Data Following past work, our dataset is derived from the Govtrack database.

1 our dataset consists of all votes on the full-text (not amendments) of bills or resolutions from the 106th-111th Congress, six of the most recent Congresses for which bill texts are readily available.


They are designed for different application back-ends, requirements and scenarios at different points in time.

the semantic schema for a domain is designed without considering other domains.


Dealing with mixed case Both approaches require a separate treatment for mixed case words.

we need a gazetteer that maps each word to its mixed case form -either manually created or statistically collected from training data.


, w d ], w j 2 R + that modify the comparison of ingredient and category vectors.

the association is predicted by where diag(w) is a diagonal matrix with the entries specified by w. We assume that, at this stage, the RNN parameters ✓ and therefore the vector representations v ✓ (z p ) and v ✓ (x i ) are nonadjustable.


We focus on metrics that compare it to the ground truth response of the conversation.

we investigate two approaches: word based similarity metrics and word-embedding based similarity metrics.


Our experiments on the dataset show the models instantiated by the static and dynamic modeling outperform a strong baseline.

the model based on the dynamic modeling robustly predicts appropriate addressees and responses even if the number of interlocutors in a conversation increases.


Compared with the baseline, our proposed models achieve better results, which suggests that the models can select the correct addressees that spoke at more previous time steps.

the dynamic model achieves 68% in accuracy, which is 7 point higher than the accuracy of static model.


However, compared with the baseline, our proposed models suppress the performance degradation.

the dynamic model predicts correct addressees most robustly.


However, none of these models are applicable for predicting a variable that ranges over an infinite set, and it is not straightforward to extend them suitably.

a discriminative approach is not applicable for arbitrary slot filling tasks because it requires a fixed finite set of slot values to take statistics.


To reflect this observation, we introduce a filler slot to separately model the functional fillers based on a position feature.

we define three filler slots: beginning filler f 1 , which precedes any slot value, ending filler f 3 , which appears at the end of an utterance, and middle filler f 2 , which is inserted between slot values.


For estimating slot values given u, we adopt a candidate generation approach (Williams, 2014) that leverages another slot filling algorithm to enumerate likely candidates.

2 we assume a candidate generation function g(u) that generates N candidates {S 1 , ..., S N } regarding u.


At the time, it was very successful, but later works noted that it had major problems with lexical memorization (Roller et al., 2014;Weeds et al., 2014;.

when the training and test sets were carefully constructed to ensure they were completely disjoint, it performed extremely poorly.


They propose a new model, Ksim, which addresses their concerns, but lacks any notion of Distributional Inclusion.

they argue for directly including the cosine similarity of w and H as a term in a custom SVM kernel, in order to determine whether w and H are related all.


The vector rejection is orthogonal to the projection, and intuitively, is the original vector after the projection has been removed: Using the vector rejection, we take a learned Hfeature detectorp, and discard these features from each of the word vectors.

for every data point H, w , we replace it by its vector rejection and rescale it to unit magnitude: A new classifier trained on the H i+1 , w i+1 data must now learn a different decision plane thanp, aŝ p is no longer present in any data points.


For each fold F i , we then use all pairs (w, H) where w ∈ F i as the test set pairs.

if "car" is in the test set fold, then "car → vehicle" and "car truck" will appear as test set pairs.


1 Nevertheless, we find that all available datasets of this kind are insufficient for judging verb similarity due to their small size or narrow coverage of verbs.

a number of word pair evaluation sets are prominent in the distributional semantics literature.


We employed the SimLex-999 annotation guidelines.

we instructed annotators to give low ratings to antonyms, and to distinguish between similarity and relatedness.


We start with a large collection of relational triples, obtained by shallow information extraction.

we use the collection of Faruqui and Kumar (2015), obtained by combining the OLLIE tool with Google Translate and projecting multilingual sentences back to English.


To compute clusters of relational phrases, we use modularity-based graph partitioning.

we use the partitioning algorithm of Blondel et al.


The goal of inference is to find the assignment to the inference variables that maximizes total score.

we seek to solve Here 1 represents all the inference variables together and C denotes the set of all valid assignments to the variables, defined by the following set of constraints: 1.


Chunk similarities To predict the chunk similarities, we assume that the label-specific chunk similarities of aligned chunks are the best edge-weights for the corresponding inference variables.

for a pair of chunks (s i , t j ) that are aligned with a label l, the chunk pair similarity z i,j,l is the coefficient associated with the corresponding inference variable.


If the alignment edge indicates an unaligned chunk, then the corresponding score is zero.

but can chunk similarities directly be used to find good alignments?


Thus, to compute the loss, we project the chunk scores z i,j,l for an aligned chunk pair (s i , t j , l) to the tokens that constitute the chunks by equally partitioning the scores among all possible internal alignments.

for a token w i in the chunk s i and token w j in chunk s j , we define token similarity scores as Here, the normalizing function N is the product of the number of tokens in the chunks 3 .


The proposed BiAtt-DP parses a sentence in a linear order via sequentially querying the memory component that stores continuous embeddings for all headwords.

we consider all possible arcs during the parsing.


We use the notion of additive token embedding as in (Botha and Blunsom, 2014) to utilize the available information about the token, e.g., its word form, lemma, part-of-speech (POS) tag, and morphological features.

the token embedding is computed as where e i 's are one-hot encoding vectors for the ith word, and E's are parameters to be learned that store the continuous embeddings for corresponding feature.


For PTB and CTB datasets, we use exactly the same setup as in (Chen and Manning, 2014;.

we convert the English and Chinese data using the Stanford parser v3.3.0 (de Marneffe et al., 2006) and the Penn2Malt tool (Zhang and Clark, 2008), respectively.


Next, in Table 3 we show the parsing accuracy of the proposed BiAtt-DP on 12 languages in the CoNLL 2006 shared task, including comparison with state-of-the-art parsers.

we show UAS of the 3rd-order RBGParser as reported in  since it also uses low-dimensional continuous embeddings.


Unlike the transition-based formulation, the proposed BiAtt-DP directly predicts the headword and the dependency relation at each time step.

there is no explicit representation of actions or headwords in our model.


First, we show that parser bias entails substantial overestimation of parser performance.

we demonstrate that bias towards the output of a specific tagger-parser pair leads to over-estimation of the performance of these tools relative to other tools.


This difference is significant at p < 0.01.

tense agreement provides a strong signal for identifying non-implicative verbs, and thus can help systems avoid false-positive entailment judgements, e.g.


• Fourier: Transformation of HSV vectors into a Fourier basis representation.

the representation f of a color (h, s, v) is given bŷ The Fourier representation is inspired by the use of Fourier feature descriptors in computer vision applications (Zhang and Lu, 2002).


Non-convex denotations The Fourier feature transformation and the nonlinearities in the model allow it to capture a rich set of denotations.

our model addresses the shortcoming identified by McMahan and Stone (2015) that their model cannot capture non-convex denotations.


To address this, we explore using two variants of reinforcement learning to directly optimize a coreference system for coreference evaluation metrics.

we modify the max-margin coreference objective proposed by Wiseman et al.


3.2 From QA-SRL to Open IE SRL and Open IE have been defined with different objectives.

sRL identifies argument role labels, which is not addressed in Open IE.


Considering some of the English connectives (e.g., while) are highly ambiguous, we compare our method with ones that uses only unambiguous connectives.

we first discard as, when, while and since in top 20 connectives, and get 22,999 synthetic instances.


During the first iteration, we start at the root of the tree, and greedily parse until we reach a leaf.

for each node, we use the Stack-LSTM model to calculate scores for each transition (as described in Section 2), and then execute the highest-scoring transition, generating a child node upon which we repeat the procedure.


Recently, recurrent neural networks have reemerged as a powerful tool for learning the latent structure of language.

work on long short-term memory (LSTM) networks for language modeling has provided improvements in perplexity.


Luckily, we can exploit the structure of the problem to efficiently compute the partition function, where T (u) is the set of all trees under the grammar that have yield u.

we make use of the inside algorithm, which is just CKY (Aho and Ullman, 1979) in the (+, ×) semiring (Goodman, 1998), which runs in O(|G| • |u| 3 ) time, where |G| is the size of the grammar.


To investigate this problem, we approach timeline generation as a special single-document summarisation task.

we assume that the information to be summarised is contained in a single history article, and that no further mentions of specific events exist externally.


In this paper, we propose a novel time-aware KB embedding approach taking advantage of the happening time of facts.

we use temporal order constraints to model transformation between time-sensitive relations and enforce the embeddings to be temporally consistent and more accurate.


Instead, the proposed method utilizes external information sources to resolve ambiguities inherent in text interpretation.

our strategy is to find other documents that contain the information sought, expressed in a form that a basic extractor can "understand".


To understand the properties of SQuAD, we analyze the questions and answers in the development set.

we explore the (i) diversity of answer types, (ii) the difficulty of questions in terms of type of reasoning required to answer them, and (iii) the degree of syntactic divergence between the question and answer sentences.


We call a paraphrased pair source sentence s and the other as target t. Superscripts of s and t represent the source and the target, respectively.

τ s , τ t is a pair of source and target phrases.


4.3), it keeps all competing hypotheses of support relations using packed representations.

h i is accompanied by its packed support list as illustrated in Fig.


In addition, we extend W using word embeddings; we use the MVLSA word embeddings (Rastogi et al., 2015) given the superior performance in word similarity tasks.

we compute the cosine similarity of embeddings; words with a higher similarity value than a threshold are determined as similar words.


Alignment Quality Alignment quality was evaluated by measuring the extent that the automatic alignment results agree with those of humans.

we evaluated how goldalignments can be replicated by automatic alignment (called recall) and how automatic alignments overlap with alignments that at least an annotator aligned (called precision) as: where Ha is a set of alignments, while G and G are the ones that two of annotators produce, respectively.


We employ the same model architecture as Kiperwasser and Goldberg (2016a).

we first use a bi-LSTM to encode an n-token sentence, treated as a sequence of per-token concatenations of word-and POS-tag embeddings, into a sequence of vectors is the output of the bi-LSTM at time step i.


(2011) show that all three deduction systems can be directly "tabularized" and dynamic programming (DP) can be applied, such that, ignoring for the moment the issue of incorporating complex features (we return to this later), time and space needs are low-order polynomial.

as the two-index shorthand ri, js suggests, arc-eager and arc-hybrid systems can be implemented to take Opn 2 q space and Opn 3 q time; the arc-standard system requires Opn 3 q space and Opn 4 q time (if one applies the so-called hook trick (Eisner and Satta, 1999)).


Previous work showed that treating semantic dependency parsing as the search for Maximum Subgraphs is not only elegant in theory but also effective in practice (Kuhlmann and Jonsson, 2015;.

our previous work showed that 1-endpoint-crossing, pagenumber-2 (1EC/P2) graphs are an appropriate graph class for modelling semantic dependency structures .


If a secondorder score is negative, our algorithm avoids including it by selecting other paths.


In this section we evaluate the effectiveness of our proposed model and TACRED on improving slot filling systems.

we run two sets of experiments: (1) we evaluate model performance on the relation extraction task using TACRED, and (2) we evaluate model performance on the TAC KBP 2015 cold start slot filling task, by training the models on TACRED.


Thus, the reliability of a labeling function is not consistent among different pieces of instances.

a labeling function could be more reliable for a certain subset (Varma et al., 2016) (also known as its proficient subset) comparing to the rest.


Here, we present REHES-SION, a novel framework capturing context's semantic meaning through representation learning, and conduct both relation extraction and true label discovery in a context-aware manner.


Since different supervision information may have different proficient subsets, we require each labeling function to encode only one elementary supervision information.

in the relation extraction scenario, we require each labeling function to only annotate one relation type based on one elementary piece of information, e.g., four examples are listed in Fig.


1, distributed representation of context bridges relation extraction with true label discovery, and allows them to enhance each other.


2): where z c is the representation of c ∈ C l , W is a n z × n v matrix, n z is the dimension of relation mention embeddings and tanh is the element-wise hyperbolic tangent function.

we represent bag of text features with their average embedding, then apply linear map and hyperbolic tangent to transform the embedding from text feature semantic space to relation mention semantic space.


And our method infers born-in as the true label for the first two relation mentions; after replacing the matched contexts (born) with other words (elected and examined), our method no longer trusts born-in since the modified contexts are no longer matched, then infers None as the true label.

our proposed method infer the true label in a context aware manner.


True label discovery methods have been developed to resolve conflicts among multi-source information under the assumption of source consistency (Li et al., 2016;Zhi et al., 2015).

in the spammer-hammer model (Karger et al., 2011), each source could either be a spammer, which annotates instances randomly; or a hammer, which annotates instances precisely.


NLP algorithms have been investigated for automatically inducing scripts from unstructured texts (Mooney and De-Jong, 1985;Chambers and Jurafsky, 2008).

chambers and Jurafsky (2008) made a first attempt to learn scripts from test inducing event chains by grouping events based on their narrative coherence, calculated based on Pairwise Mutual Information (PMI).


To leverage the advantages of both methods, we propose to integrate chain temporal order information into event relation measuring.

we calculate event pair relations by representing events in a chain using LSTM hidden states, which encode temporal information.


A neural network is used to model chains of extracted events for script learning.

we model the probability of a sub-sequent event given a chain of events.


We keep the entities whose corresponding anchor texts occur in the query as candidates, and treat others as related entities.

we obtain three candidates in this example, namely "Blake Shelton", "Austin, Texas", and "Austin (song)".


We use average F1 designed by ERD Challenge (Carmel et al., 2014) as the evaluation metrics.

given a query q, with labeled entitiesÂ = {Ê 1 , .


To further inspect the ability of T-O-M to enable disambiguation in different domains, we decided to evaluate on specific documents from the various gold standard datasets which could be clearly assigned a domain label.

we tested on 13 SemEval-13 documents from various domains 6 and 2 SemEval-15 documents (namely, maths & computers, and biomedicine) and carried out two separate tests and evaluations of T-O-M on each domain: once using the MFS backoff strategy, and once not using it.


Instead, we rely on a small number of enhancements for our main application-semantic parsing for questionanswering-with the hope that this step can be replaced by an enhanced UD representation in the future.

we define three kinds of enhancements: (1) long-distance dependencies; (2) types of coordination; and (3) refined question word tags.


Since these are crucial to question-answering, we use a small lexicon to refine the tags for determiners (DET), adverbs (ADV) and pronouns (PRON) to DET:WH, ADV:WH and PRON:WH, respectively.

we use a list of 12 (English), 14 (Spanish) and 35 (German) words, respectively.


In addition, we add features encoding the semantic similarity of ungrounded events and Freebase relations.

we used the cosine similarity of the translation-invariant embeddings of Huang et al.


least highly) correlated with the linguistic feature f .

we compute the relative position x of each pivot that occurs in the verse and apply a Gaussian filter (σ = 6 where the unit of length is the character), i.e., we set p(x) ≈ 0.066 (0.066 is the density of a Gaussian with σ = 6 at x = 0) and center a bell curve around x.


To illustrate the potential of our method we select five out of the 100 past tense pivots that give rise to large clusters of distinct combinations.

starting with CRS, we find other pivots that "split" the set of verses that contain the CRS past tense pivot "ti" into two parts that have about the same size.


The encoder then takes the word embedding v enc (w i ) and the hidden state h (2) i as the input to a uni-directional LSMT: is the hidden state corresponding to w i .

the encoder of our model is a three-layer LSTM network, where the first two layers are bi-directional.


DEP is constructed by using pre-trained dependency relations rather than learning them.

p(H w i = w j |w i ) is fixed to 1.0 such that w j is the head of w i .


N for all the words in the same sentence.

the uniform probability distributions are used for equally connecting all the words.


The prediction task could be trained jointly with the translation task in a multi-task learning way (Luong et al., 2015a;Dong et al., 2015;Zhang and Zong, 2016), where both tasks share the same encoder.

word prediction for the initial state could be interpreted as an improvement for the encoder.


With an explicit training, the WP E could achieve a much higher precision in all conditions.


Adapting EG to our decoding problem leads to Algorithm 1.

it can be shown that the EG algorithm is a gradient descent algorithm for minimising the following objective function subject to the simplex constraints: the algorithm looks for the maximum entropy solution which also maximizes the log likelihood under the model.


We outperform previous systems in all metrics.

our single model improves the state-of-the-art average F1 by 1.5, and our 5-model ensemble improves it by 3.1.


The information overlap between summary sentences should be as minimal as possible due to the length limit of summary.

the information coverage of summary is a determinant, which requires that the summary sentences should cover diverse aspects of information.


For simplicity we use the mini-batch for this set, i.e., we choose t 1 for a given s 1 , s 2 as follows: where S b ⊆ S is the current mini-batch.

we want to choose a negative example t i that is similar to s i according to the current model.


(2016) proposed a multi-granularity embedding (MGE) model based on the CWE model, which represents the context as a combination of surrounding words, surrounding characters, and the radicals of the target word.

they developed a dictionary of 20,847 characters and 296 radicals.


showed that relations between words are reflected to a large extent in the offsets between their vector embeddings (queen -king = woman -man), and thus the vector of the hidden word b * will be similar to the vector b − a + a * , suggesting that the analogy question can be solved by optimizing: where V is the vocabulary and sim is a similarity measure.

they used the cosine similarity measure, defined as: resulting in: Equation 4 has been referred to as CosAdd model.


The training dataset is a set of paraphrase pairs in which two sentences in each pair represent the same meanings.

we extract Chinese paraphrases in machine translation evaluation corpora NIST2003 2 and CWMT2015 3 .


Moreover, we select aligned sub-sentence pairs between paraphrases to enlarge the training corpus.

we first segment the sentences into sub-sentences according to punctuations of comma, semicolon, colon, question mark, ellipses, and periods.


Distributional models (DS: Turney and Pantel (2010); Clark (2012); Erk (2012)), and in particular neural network approaches (Bengio et al., 2003;Collobert et al., 2011;Huang et al., 2012;Mikolov et al., 2013), do not fare well in the absence of large corpora.

for a DS model to learn a word vector, it must have seen that word a sufficient number of times.


We want to simulate the process by which a competent speaker encounters a new word in known contexts.


Parameter choice: we experiment with higher learning rates coupled with larger window sizes.

the model should take the risk of a) overshooting a minimum error; b) greedily considering irrelevant contexts in order to increase its chance to learn anything.


Selective training: we only train the nonce.

we only update the weights of the network for the target.


Too find more about the theoretical correctness of FOFE, please refer to (Zhang et al., 2015b).

the FOFE codes can almost uniquely encode any sequences, serving as a fixed-size but theoretically lossless representation for any variable-length sequences.


Figure 3 shows that for a given window, the re-embedding performs better when the dimensionality of the learned manifold is chosen to be closer to the original space dimensionality.

dimensional reduction on the original space will bare a cost in performance.


Unlike the prior work, this paper proposes MUSE 2 -a novel modularization framework incorporating sense selection and representation learning models, which implements flexible modules to optimize distinct mechanisms.

mUSE enables linear time sense identity decoding with a sense selection module and purely senselevel representation learning with a sense representation module.


The architecture is similar to continuous bag-of-words (CBOW) (Mikolov et al., 2013a).

given a word embedding matrix P , the local context can be modeled as the summation of word embeddings from its contextC t .


We note that modularized model enables efficient sense selection by leveraging word-level tokens, while remaining purely sense-level tokens in the representation module.

if n denotes max k |Z k |, decoding L words takes O(nL) senses to be searched due to independent sense selection.


Based on different modeling methods ((1) or (2)) in the sense selection module, we connect the model to respective reinforcement learning algorithms to solve the MDP.

we refer (1) to policy distribution and refer (2) to Q-value estimation in the reinforcement learning literature.


In addition, as the skip-gram formulation in (4) is unidirectional (L(z ik | z jl ) =L(z jl | z ik )), we perform one-side optimization for the target sense z ik to stabilize model training 3 .

for the target word w i and the collocated word w j given respective contextsC t andC t (0 < |t − t | ≤ m), we first draw a sense z jl for w j from the policy π(• |C t ) and optimize the expected collocation likelihood for the target sense z ik as follows, Note that (4) can be merged into (5) as a single objective.


We further follow the convention of recent neural reinforcement learning by reducing the reward range to aid model training (Mnih et al., 2013).

we replace the log likelihood logL(•) ∈ (− inf, 0] with the likelihoodL(•) ∈ [0, 1] as the reward function.


To evaluate the quality of the learned sense embeddings, we compute the similarity score between each word pair given their respective local contexts and compare with the human-judged score using Stanford's Contextual Word Similarities (SCWS) dataset (Huang et al., 2012).

given a list of word pairs with corresponding contexts, S = {(w i ,C t , w j ,C t )}, we calculate the Spearman's rank correlation ρ between human-judged similarity and model similarity estimations 4 .


When F is not an integer, the copies are generated so that their expected number becomes F .

⌈F ⌉ copies are added to T with probability F − ⌊F ⌋, and ⌊F ⌋ copies are added otherwise.


Since it is difficult to assess the main result in Section 4.2.2 directly, the theorems in Sections 4.2.1, from which the main result is readily derived, were investigated.

the first and second order moments of ∆L(θ) were computed on datasets of increasing sizes to empirically investigate the convergence property.


The results in Table 4 indicate that cross-model transfer is indeed possible, with most transferred feature sets achieving similar results or even outperforming features learned with the Bi-LSTM.

transferred diversity significantly outperforms its in-model equivalent.


In the above formulation, the predictions of attention contexts are symmetric.

c i is not forced to be different from c j =i .


tor representations work well by revealing a hidden structure of the original word vectors.

it is meaningful to transform the hard-to-interpret dimensions of the pre-built word vectors, which are widely used, to more interpretable vectors.


(* p < .05, ** p < .01, *** p < .001) Overall Interpretability of both directions The first two columns (A) and (B) in table 6 show the overall distance ratio computed over the top words extracted by descending order and ascending order, respectively.

(A) refers to the top words having the highest positive values in each dimension, while (B) uses the lowest negative values.


This differs from our approach in several respects.

we focus on explaining the behavior of the "black box" mapping F only locally, around the current input context, not globally.


At a high level, this is a robust optimization formulation which seeks to minimize worst case cut values, and can be cast as a Mixed Integer Programming (MIP) problem.

for a bipartite graph G = (U, V, E) Algorithm 1 Structured-output causal rationalizer 1: procedure SOCRAT(x, y, F ) 2: (µ, σ) ← ENCODE(x) 3: end for 8: Ex →y ← BIPARTITION(G) 10: Ex →y ← SORT(Ex →y ) By cut capacity 11: return Ex →y 12: end procedure with edge weights given as uncertainty intervals θ ij ±θ ij , the partitioning problem is given by where x u ik , x v jk are binary variables indicating subset belonging for elements of U and V respectively, y ij are binary auxiliary variables indicating whether i and j are in different partitions, and Y is a set of constraints that ensure the K-partition is valid.


For Gaussian latent variables, the prior distribution mean and variances are encoded using linear transformations of a hidden state.

the prior distribution covariance is encoded as a diagonal covariance matrix using a softplus function: where Enc(c) is an embedding of the conditioning information c (e.g.


These two methods avoid ADAGRAD's problem.

aDaM performs much better for prediction.


In this paper, we propose a novel framework to solve the above problems in target sentiment analysis.

our framework first adopts a bidirectional LSTM (BLSTM) to produce the memory (i.e.


To ease this problem, we adopt an intuitive method to edit the memory to produce a tailor-made input memory for each target.

the closer to the target a word is, the higher its memory slide is weighted.


One might want to use the length of the path from the specific word to the target in the dependency tree as the distance, which is a worthwhile option to try in the future work, given the condition that dependency parsing on the input text is effective enough.

the weight for the word at position t is calculated as: where t max is truncation length of the input.


For example, "except" and "don't play well" in "Except Patrick, all other actors don't play well" are attended by different attentions, and combined to produce a positive sentiment on "Patrick".

we employ a GRU to update the episode e after each attention.


Figures 3a and 3b present a case that there are more than one opinion targets in a comment, which cannot be analyzed with sentence-level sentiment analysis methods properly.

it's a comparative sentence in which the reviewer has a positive sentiment on the first commented person, but a negative sentiment on the second person, and our model predicts both of them correctly.


Furthermore, the retrieved results of SURF are categorized according to their aspects making them easy to interpret.

if a target sentence has multiple aspects, then SURF will retrieve results for each aspect.


The theory of incongruity states that we laugh because there is something incongruous (Attardo, 1994), in other words, there is a change from our expectation.

we propose a general generative language model to model the generation of humorous text.


In this section we describe some observations in our data that have motivated our approach to solving the problem.

we show that humorous reviews tend to reference aspects which deviate from what is expected.


In particular, we show that humorous reviews tend to reference aspects which deviate from what is expected.

in funny reviews, the authors tend to use referential humor, in which specific concepts or entities are referenced to produce comedic effects, which we call aspects.


We use recent advancements in Wikification, which aims to connect important entities and concepts in text to Wikipedia, it is also known as disambiguation to Wikipedia.


Most existing sentiment classifiers rely on labeled training data and the data are usually language-dependent.

a sentiment classifier is learned from a labeled dataset in a specific language and this sentiment classifier can be used for sentiment classification in this language.


Among these languages, the English language is the source language with labeled training data, and we do no use any labeled data in the other languages.

we use the multilingual multidomain Amazon review dataset 4 provided by (Prettenhofer and Stein, 2010) and the NLPC-C2013 dataset 5 .


In the latter case, the sentiment classifier learned in the source language can be used for sentiment classification in both languages.

wan (2009) used machine translation to translate the source language to the target language to bridge the gap and applied the co-training approach.


Bilingual word embeddings have been induced for cross-lingual NLP tasks (Vulić and Moens, 2015;Guo et al., 2014;Zou et al., 2013;.

proposed the BiSkip model to induce bilingual word embeddings, which is extended from the monolingual skip-gram model in word2vec to a bilingual model.


In this paper, we proposed an approach to build a universal sentiment classifier in multiple languages.

we proposed a pivot-based model to transfer the sentiment information from the source language to any resource-poor language via pivot languages.


In order to measure the performance of our model, here we use three metrics.

we use accuracy to measure the overall sentiment classification performance, M AE and RM SE to measure the divergences between prediction py and ground truth gy.


We use ω to control the influence from the venue and the author.

ω is the parameter of a Bernoulli distribution, from which a binary variable c is generated to determine whether the document is influenced by venue or user.


As described in Section 3.3, we use a small set of seed words as our topic prior knowledge.

the seed words list contains 5 to 10 seed words for each of the five topics of interest 3 about Hurricane impact, public utility, food, shelter, medical, respectively.


Our model outperforms the baseline models.

the perplexity of our model is 1122, which is 40 and 116 lower than that of JST and TS.


The ground truth of sentiment classification labels of tweets are set by using human annotation.

we randomly select 1000 documents from the dataset, and label each document as positive, negative or neutral manually.


Instead of building a new word embedding model from labeled data, we propose a word vector refinement model to refine existing semantically oriented word vectors using sentiment lexicons.

the proposed model can be applied to the pre-trained vectors obtained by any word representation learning models (e.g., Word2vec and GloVe) as a post-processing step to adapt the pre-trained vectors to sentiment applications.


This weight is used to control the movement direction of the target word towards to its nearest neighbors.

the target word will be moved closer to the higher-ranked sentimentally similar neighbors and further away from lowerranked dissimilar neighbors, as shown in Fig.


Tang et al., (2014) proposed a method to learn sentiment specific word embeddings from tweets with emoticons as distantsupervised corpora without any manual annotation.

they developed three neural networks to effectively incorporate the supervision from sentiment polarity of text in their loss functions.


To encourage the agent to find predictive paths, our reward functions include the following scoring criteria: Global accuracy: For our environment settings, the number of actions that can be taken by the agent can be very large.

there are much more incorrect sequential decisions than the correct ones.


For a typical KG, the RL agent is often faced with hundreds (thousands) of possible actions.

the output layer of the policy network often has a large dimension.


We use the same dimension as TransE, R to embed the entities.

the state vector we use has a dimension of 200, which is also the input size of the policy network.


In this paper, we propose a reinforcement learning framework to improve the performance of relation reasoning in KGs.

we train a RL agent to find reasoning paths in the knowledge base.


The relevance reward r R is simply the cosine similarity between these two vectors: We use a sequence auto-encoder (SAE; Dai and Le 2015) to train the LSTM sentence encoder on both the complex and simple sentences.


We also constructed WikiLarge, a larger Wikipedia corpus by combining previously created simplification corpora.

we aggregated the aligned sentence pairs in Kauchak (2013), the aligned and revision sentence pairs in Woodsend and Lapata (2011), and Zhu's (2010) WikiSmall dataset described above.


Evaluation Following previous work (Woodsend and Lapata, 2011; Xu et al., 2016) we evaluated system output automatically adopting metrics widely used in the simplification literature.

we used BLEU 5 (Papineni et al., 2002) to assess the degree to which generated simplifications differed from gold standard references and the Flesch-Kincaid Grade Level index (FKGL;Kincaid et al.


We also evaluated system output by eliciting human judgments via Amazon's Mechanical Turk.

(selfreported) native English speakers were asked to rate simplifications on three dimensions: Fluency (is the output grammatical and well formed?


Since Newsela contains high quality simplifications created by professional editors, we performed the bulk of our experiments on this dataset.

we set out to answer two questions: (a) which neural model performs best and (b) how do neural models which are resource lean and do not have access to linguistic annotations fare against more traditional systems.


Narayan and Gardent (2014) describes a sentence simplification approach which combines a probabilistic model for splitting and deletion with a phrase-based statistical machine translation (SMT) and a language model for rephrasing (reordering and substituting words).

the splitting and deletion components exploit the deep meaning representation (a Discourse Representation Structure, DRS) of a complex sentence produced by Boxer (Curran et al., 2007).


In this model, we learn a multi-source model which takes into account not only the input complex sentence but also the associated set of RDF triples available in the WEBSPLIT dataset.


For the convenience of further discussions, we introducer to denote the underlying (distributional) fake response produced by the decoder.

,r stands for a sequence of word distributions projected from the hidden layers of the decoder RNN, based on which one would sample the output response utterance in a traditional Seq2Seq generator.


One can view a VAE as a traditional Autoencoder with some restrictions imposed on the internal representation space.

using a sample from the q(z|x) to reconstruct the input instead of a deterministic z, forces the model to map an input to a region of the space rather than to a single point.


Finally, we have extensively evaluated the trade-off between the KLterm and the reconstruction loss.

we investigated the effect of the receptive field size on the ability of the model to respect the latent vector which is crucial for being able to generate realistic and diverse samples.


in The player hit the ball, humans would certainly exclude from the meaning of ball semantic dimensions that are strictly related to its dancing sense).

not all the features of the semantic representations are active, and the composition process makes some features more 'prominent', while moving others to the background.


We utilize this fact to make the document vector dv Dn sparse by zeroing attribute values whose absolute value is close to a threshold (specified as a parameter), which results in the Sparse Composite Document Vector SCDV Dn .

let p be percentage sparsity threshold parameter, a i the value of the i th attribute of the non-Sparse Composite Document Vector and n represent the n th document in the training set:    Flowcharts depicting the formation of wordtopics vector and Sparse Composite Document Vectors are shown in figure 1 and figure 2 respectively.


While there seems to be a consensus concerning the usefulness of word embeddings and how to learn them, this is not yet clear with regard to representations that carry the meaning of a full sentence.

how to capture the relationships among multiple words and phrases in a single vector remains an question to be solved.


We subsequently describe the architectures that we investigated for the sentence encoder, which, in our opinion, covers a suitable range of sentence encoders currently in use.

we examine standard recurrent models such as LSTMs and GRUs, for which we investigate mean and maxpooling over the hidden representations; a selfattentive network that incorporates different views of the sentence; and a hierarchical convolutional network that can be seen as a tree-based method that blends different levels of abstraction.


We hypothesize that the semantic nature of NLI makes it a good candidate for learning universal sentence embeddings in a supervised way.

we aim to demonstrate that sentence encoders trained on natural language inference are able to learn sentence representations that capture universally useful features.


The key idea is that not only the entailment results but also the theorem proving process can be considered as features for learning textual sim-ilarity.

by taking into account not only whether a theorem is proved but also how it is proved, we can capture the semantic relationships between sentence pairs in more depth.


However, the information needed to correctly choose among these patterns for a given input is not necessarily available to the network.

the network is only aware of important disambiguating historical (e.g., is the input of Latin or Greek origin) and lexical-semantic (e.g., is the input verb transitive or intransitive) factors to the extent that they are implicitly encoded in the input character sequence.


In this paper, we bridge this gap by introducing a novel compositional neural architecture that explicitly leverages the sub-character information.

we perform Unicode decomposition on each Korean character to recover its underlying jamo letters and construct character-and word-level representations from these letters.


Finally, we derive each word embedding vector v i from a characterlevel bidirectional LSTM embedder.

namely, we define each word embedding as the concatenation we run a bidirectional LSTM over the character stream.


To allow the architecture to encode morphological features specific to one language, e.g., the third person present plural ending in Spanish is -an, but -ão in Portuguese, we modify the creation of the character-level embeddings.

we augment the character alphabet Σ with a distinguished symbol that indicates the language: id .


However, in the multi-task scenario, we find that the neural tagger's learning curve is even steeper.

if we have to train a tagger on very little data, we are better off using a neural multi-task approach than a feature-based approach; preliminary attempts to develop a multitask version of MARMOT failed (see Figure 3).


To capture order information (Ling et al., 2015), we use different embedding matrices for context embedding in different context positions, training different embeddings for the same word when they reside on different locations as the context word.


Using the general idea of segmentation-free word embedding, we can extend existing word embedding models.

in this paper, we propose a segmentationfree version of the widely used skip-gram model with negative sampling (SGNS) (Mikolov et al., 2013), which we refer to as SGNS-sembei.


Collected from the English exams for middle and high school Chinese students in the age range between 12 to 18, RACE consists of near 28,000 passages and near 100,000 questions generated by human experts (English instructors), and covers a variety of topics which are carefully designed for evaluating the students' ability in understanding and reasoning.

the proportion of questions that requires reasoning is much larger in RACE than that in other benchmark datasets for reading comprehension, and there is a significant gap between the performance of the state-of-the-art models (43%) and the ceiling human performance (95%).


Moreover, while MCTest is designed for 7 years old children, RACE is constructed for middle and high school students at 12-18 years old hence is more complicated and requires stronger reasoning skills.

rACE can be viewed as a larger and more difficult version of the MCTest dataset.


In this section, we study the nature of questions covered in RACE at a detailed level.

we present the dataset statistics in Section 3.1, and then analyze different reasoning/question types in RACE in the remaining subsections.


The higher difficulty level of RACE is justified by its higher ratio of reasoning questions in comparison to CNN, SQUAD and NEWSQA.

59.2% questions of RACE are either in the category of single-sentence reasoning or in the category of multi-sentence reasoning, while the ratio is 21%, 20.5% and 33.9% for CNN, SQUAD and NEWSQA respectively.


(2016), we adopt a bilinear attention form.

similarly, we use bidirectional GRUs to encode option o i into a vector h o i .


We pass the scores through softmax to get a probability distribution.

the probability of option i being the right answer is calculated as Gated-Attention Reader Gated AR (Dhingra et al., 2016) is the state-of-the-art model on multiple datasets.


With a multi-hop architecture, GA also enables a model to scan the document and the question iteratively for multiple passes.

the multi-hop structure makes it possible for the reader to refine token representations iteratively and the attention mechanism find the most relevant part of the document.


Note that our implementation slightly differs from the original GA reader.

the Attention Sum layer is not applied at the final layer and no character-level embeddings are used.


To take a closer look at how syntactic sequences affect the performance, we removed the character/word embedding from our model seen in Figure 3 and conducted experiments based on the syntactic input alone.

we are interested in two aspects related to syntactic sequences: First, the ability to predict answers of questions of syntactic sequences compared to complete random sequences.


Experimental results on SQuAD data set showed that our proposed approaches outperform the state-of-the-art BiDAF model, proving that the proposed embeddings play a significant part in correctly identifying answers for the machine comprehension task.

we found that our model can perform especially well on exact match metrics, which requires syntactic information to accurately locate the boundaries of the answers.


In this paper, we introduce a task and several models to drive progress towards this goal.

we propose the task of rare entity prediction: given a web document with several entities removed, models are tasked with predicting the correct missing entities conditioned on the document context and the lexical resources.


We use a significantly pre-processed and augmented version of the Wikilinks dataset for the purpose of entity prediction, which we call the Wikilinks Rare Entity Prediction dataset.

we parse the HTML texts of the web pages and ex- 238,025 (97.1%) Table 2: Statistics for the augmented version of the Wikilinks dataset, where n represents the entity frequency in the corpus.


In this paper, we examined the use of external knowledge in the form of lexical resources to solve reading comprehension problems.

we propose the problem of rare entity prediction.


Transfer learning (Pan and Yang, 2010) has been successfully applied to numerous domains in ma-chine learning, such as machine translation (Zoph et al., 2016), computer vision, (Sharif Razavian et al., 2014), and speech recognition (Doulaty et al., 2015).

object recognition models trained on the large-scale ImageNet challenge (Russakovsky et al., 2015) have proven to be excellent feature extractors for diverse tasks such as image captioning (i.e., Lu et al.


We use a bi-directional Long-Short Term Memory Network (Bi-LSTM) (Hochreiter and Schmidhuber, 1997) for tagging.

we project each word p i → p * i into a continuous vector space via pretrained GloVe embeddings (Pennington et al., 2014).


We use an architecture motivated by latent predictor networks (Ling et al., 2016) to force the model to learn when to copy vs. directly predict the word, without direct supervision of what action to choose.

at every time step i, two latent predictors generate the probability of generating word w i , a pointer network C p (Vinyals et al., 2015) which can copy a word from the context paragraph, and a vocabulary predictor V p which directly generates a probability distribution of choosing a word w i from a predefined vocabulary.


We implement a rule-based extraction method for this purpose, which achieves very high precision and medium recall.

most equations and structured answers extracted by our method are correct, and many problems are dropped from the dataset.


In our work, we employ bidirectionally entailing rules from PPDB.

we focus on lexical (single word) and phrasal (multiword) rules which we use to paraphrase questions by replacing words and phrases in them.


We estimate p θ (q |q) using neural networks given their successful application to paraphrase identification tasks (Socher et al., 2011;Yin and Schütze, 2015;.

the input question and its paraphrases are encoded as vectors.


Every word is initially mapped to a d-dimensional vector.

vector q t is computed via q t = W q e (q t ), where W q ∈ R d×|V| is a word embedding matrix, |V| is the vocabulary size, and e (q t ) is a one-hot vector.


The third baseline (DATAAUGMENT) employs paraphrases for data augmentation during training.

we use the question, its paraphrases, and the correct answer to automatically generate new training samples.


Our goal was to see if a relatively simple architecture with minimal capacity for retaining information was able to incorporate the information required to identify temporal relations in text.

we use several simple LSTMbased components to recover ordering relations between temporally relevant entities (events and temporal expressions).


Recent work on learning to rank (L2R) has shown that deep learning and kernel methods are two very effective approaches, given their ability of engineering features.

in question answering (QA), Convolutional Neural Networks (CNN), e.g., (Severyn and Moschitti, 2015;Miao et al., 2016;Yin et al., 2016) can automatically learn the representation of question and answer passage (Q/AP) in terms of word embeddings and their non-linear transformations.


In this work, we further compare and investigate different ways of combining CTKs and CNNs in reranking settings.

we follow the intuition that as CNNs learn the embeddings in a classification setting they should be used in the same way for building the reranking kernel, i.e., we need to use the embeddings in a pointwise reranking fashion.


The intuition about the hybrid models is to add CNN layer vectors, not their difference, to the preference CTK.

cNN layers are still used as they were used in a classification setting whereas cTKs follow the standard SVM Rank approach.


Additionally, we encode the relational information between Q and AP, by injecting relation features into the network.

we associate each word w of the input sentences with a word overlap binary feature indicating if w is shared by both Q and AP.


We have shown that the combination of CTKs and CNNs with a hybrid kernel in the reranking setting outperforms the state of the art on AST and is comparable to the state of the art in cQA.

in cQA, a combination of CTKs and CNNs performs comparably to the systems using domain specific features that were manually engineered.


In an orthogonal and interesting direction, other recent works (Chung et al., 2016;He et al., 2016;Settle and Livescu, 2016) learn word representations based on similarity in their pronunciation and not the sounds associated with them.

phonetically similar words that have near identical pronunciations are brought closer in the embedding space (e.g., flower and flour).


To quantify this effect and compare to QRPE, we pair each irrelevant image-question pair (I − , Q) from VTFQ with a relevant image from the VQA dataset.

we find the nearest neighbor question Q nn in the VQA dataset to Q based on an average of the word2vec (Mikolov et al., 2013) embedding of each word, and select the image on which Q nn was asked as I + to form (I + , Q, P, I − ) tuples like in our proposed dataset.


This model (which we denote QC-Sim) uses a pretrained captioning model to automatically provide natural language image descriptions and reasons about relevance based on a learned similarity between the question and image caption.

the approach uses NeuralTalk2 (Karpathy and Li, 2015) trained on the MS COCO dataset (Lin et al., 2014) to generate a caption for each image.


We additionally see that both joint dictionaryembedding models outperform the dictionary-only models overall.

the BGRU+GloVe model performs especially well at determining the aspect and motion attributes of verbs, particularly relative to the baseline.


Although our results appear promising, we argue that our model still fails to represent much of the semantic information about each image class.

our model is prone to hubness: the overprediction of a limited set of labels at test time: those that closely match signatures of examples in the training set.


Second, while our experiments use attributes as a pivot between language and vision domains, the effects of this could be explored more in future work.

since our experiments show that unsupervised word embeddings significantly help performance, it might be desirable to learn data-driven attributes in an end-to-end fashion directly from a large corpus or from dictionary definitions.


This further corroborates the finding that different dimensions of grounding lead to complementary notions of similarity.

correlation between visual and situational embeddings is relatively low, as compared to more fuzzy textual embeddings which correlate well with denotational embeddings.


left-right, old-young) which occur frequently in our data and which are frequently used to distinguish between objects occurring in the same situation.

as the scenes in our data sets contain many objects of the same category (e.g.


given a source sequence  Similarly, a backward RNN ← − Φ enc reads X from right to left, word by word, and generates a sequence of backward annotation vectors (1) The final annotation vector for a given time step i is the concatenation of forward and backward vec- each source sequence X is encoded into a sequence of annotation vectors , which are in turn used by the decoder: essentially a neural language model (LM) (Bengio et al., 2003) conditioned on the previously emitted words and the source sentence via an attention mechanism.


We use the projected image as the first and/or last word of the source sentence and let the attention model learn when to attend to the image representation.

given the global image feature vector q ∈ R 4096 , we compute (6): where W 1 I ∈ R 4096×4096 and W 2 I ∈ R 4096×dx are image transformation matrices, b 1 I ∈ R 4096 and b 2 I ∈ R dx are bias vectors, and d x is the source words vector space dimensionality, all trained with the model.


Total first-pass first fixations The total number of first fixations occurring in the first pass of a trial.

a first-pass first fixation occurs when it is the first fixation on the given word, and there have been no fixations on any words occurring later in the text.


Total later-pass first fixations The total number of first fixations occurring outside the firstpass of a trial.

a later-pass first fixation occurs when it is the first fixation on the given word, but there have already been fixations on words occurring later in the text.


Total multi-fixations The total number of fixations on a word in the first-pass, excluding the first fixation.

a multi-fixation occurs when a word is fixated on multiple times in the run which starts with a first-pass first fixation.


We then define an augmented feature set, hereafter Biondi+word, which takes into account these word-level annotations.

we create new features corresponding to each of the fixationbased baseline features.


In every case, we observe that the classification accuracies are the same or worse on Trial 2 compared to Trial 1.

we are able to extract less diagnostically-useful information when the participant is reading aloud than when they are reading silently.


It may be that the participants were too early in their decline (and the texts too linguistically simple) for any effect to be seen, or it could be that these variables are not capturing the most relevant linguistic information.

the features were very coarse, making only a binary distinction between high/low frequency words and function/content words.


The most recent evaluation metric in TE3, i.e., the temporal awareness (UzZaman and Allen, 2011), is adopted in this work.

let G sys and G true be two temporal graphs from the system prediction and the ground truth, respectively.


(1) cannot be decoupled for eachŷ i and is usually formulated as an ILP problem (Roth and Yih, 2004;Chambers and Jurafsky, 2008;.

let I r (ij) ∈ {0, 1} be the indicator function of relation r for event i and event j and f r (ij) ∈ [0, 1] be the corresponding soft-max score.


With the inference solver defined above, we propose to use the structured perceptron (Collins, 2002) as a representative for the inference based training (IBT) algorithm to learn those weight vectors w r .

let L = {x k , y k } K k=1 be the labeled training set of K instances (usually documents).


In this work, however, we propose to mimic this mental process by a post-filtering method 2 .

we take each TLINK produced by ILP and determine whether it is vague using its relative entropy (the Kullback-Leibler divergence) to the uniform distribution.


Harnessing the wealth of information present in unstructured text online has been a long standing goal for the natural language processing community.

knowledge base population seeks to automatically construct a knowledge base consisting of relations between entities from a document corpus.


We can measure how well the current evaluation set covers the predictions X m by using a conservative estimate of the variance ofπ (joint) m .

7 the variance ofπ (joint) m is a monotonically decreasing function in n m , the number of samples drawn from X m .


We conjecture that a suitable combination of convolutional kernels and recurrent layers can lead to a model that better accounts for these factors.

we present a novel re-ranking model called PACRR (Position-Aware Convolutional-Recurrent Relevance Matching).


We first encode the query-document relevance matching via query-document similarity matrices sim |q|×|d| that encodes the similarity between terms from a query q and a document d, where sim ij corresponds to the similarity between the i-th term from q and the j-th term from d. When using cosine similarity, we have  (Hui et al., 2017), query-document similarity matrices preserve a rich signal that can be used to perform relevance matching beyond unigram matches.

n-gram matching corresponds to consecutive document terms that are highly similar to at least one of the query terms.


Relevance matching can be extracted in terms of pieces of text that include relevant information.

one can segment documents according to relevance relative to the given query and retain only the text that is highly relevant to the given query.


This sequence of vectors for each query term q i is passed into a Long Short-Term Memory (LSTM) recurrent layer (Hochreiter and Schmidhuber, 1997) with an output dimensionality of one.

the LSTM's input is a sequence of query term vectors where each vector is composed of the query term's normalized IDF and the aforementioned salient signals for the query term along different kernel sizes.


Due to limited computational resources, we determine the range of hyper-parameters to consider based on pilot experiments and domain insights.


DRMM (DRMM LCH×IDF ), DUET, Match-Pyramid and K-NRM are trained under the same settings using the hyperparameters described in their respective papers.

as our focus is on the deep relevance matching model as mentioned in Section 1, we only compare against DUET's local model, denoted as DUETL.


All models are implemented with Keras (Chollet et al., 2015) using Tensorflow as backend, and are trained on servers with multiple CPU cores.

the training of PACRR takes 35 seconds per iteration on average, and in total at most 150 iterations are trained for each model variant.


Thus, we extend our analysis to re-rank search results from all submitted runs from six years of the TREC Web Track ad-hoc task.

we only consider the judged documents from TREC, which loosely correspond to top-20 documents in each run.


The ranking of documents can be decomposed into rankings of document pairs as suggested in (Radinsky and Ailon, 2011).

a model's retrieval quality can be examined by checking across a range of individual document pairs, namely, how likely a model can assign a higher score for a more relevant document.


Note that the manual judgments that indicate a document as relevant or non-relevant relative to a given query contain disagreements (Carterette et al., 2008;Voorhees, 2000) and errors (Alonso and Mizzaro, 2012).


In addition, as for the distinction between Nav documents and Rel or HRel documents, we argue that since Nav actually indicates that a document mainly satisfies a navigational intent, this makes such documents qualitatively different from Rel and HRel documents.

a Nav is more relevant for a user with navigational intent, whereas for other users it may in some cases be less useful than a document that directly includes highly pertinent information content.


In this work, we have demonstrated the importance of preserving positional information for neural IR models by incorporating domain insights into the proposed PACRR model.

pACRR captures term dependencies and proximity through multiple convolutional layers with different sizes.


Conditioned on the question vector, we compute a representation of each document word that is sensitive to both the surrounding context and the question.

each word in the document is represented as the concatenation of its word vector d i,j , the question vector q, boolean features indicating if a word appears in the question or is repeated, and a question-aligned embedding from Lee et al.


1 Our implementation of the GNR is 24.7 times faster at inference time than the official Bi-Directional Attention Flow implementation 2 .

on a machine running Ubuntu 14 with 40 Intel Xeon 2.6Ghz processors, 386GB of RAM, and a 12GB TitanX-Maxwell GPU, the GNR with beam size 32 and batch size 32 requires 51.58 ± 0.266 seconds (mean ± std) 3 to process the SQUAD validation set.


The final hidden state of the question-encoding LSTMs naturally cluster based on question type (Table 1).

the task induces a question encoding that superficially respects type information.


with documents where we can safely assume that the majority of the entities will be contained in a large knowledge base (KB) such as Wikidata Vrandečić and Krötzsch (2014) we find that simple string matching techniques are sufficiently accurate.

we use a part of speech tagger (Honnibal, 2017) to extract nominal groups in the training data and string-match them with entities in Wikidata.


In this work, we provide a methodology that overcomes several limitations of existing approaches to extractive question answering.

our proposed model, the Globally Normalized Reader, reduces the computational complexity of previous models by casting the question answering as search and allocating more computation to promising answer spans.


For instance, compare the top left subplot, with 0 dropout and good performance with H p = 20, H u = 100, to subplot 3,3, with optimal performance at H p = 80, H u = 200.

limiting the system's memory resources is indeed the key to its performance.


In contrast, for MMS, considering the unsatisfactory quality of speech recognition, symmetric affinity matrices are inappropriate.

to improve the readability, for a speech transcription, if there is a sentence in document that is related to this transcription, we would prefer to assign the text sentence a higher salience score than that assigned to the transcribed one.


The abrupt transition of RGB histogram features often indicates shot boundaries (Zhuang et al., 1998).

when the transition of the RGB histogram feature for adjacent frames is greater than a certain ratio 3 of the average transition for the whole video, we segment the shot.


In comparison, our TFN approach seems to capture more complex interaction through bimodal and trimodal dynamics and thus performs better.

in the first example, the utterance is weakly negative where the speaker is referring to lack of funny jokes in the movie.


However, weaknesses in the annotation process itself are easy to overlook.

scholars often choose what information to give to annotators without examining these decisions empirically.


(2010), who find that even lowquality annotators are useful, we find that lowquality contexts can be useful.

we find that the classifier produced from our model performs better than any classifier trained by majority vote from the same labels.


When the item is viewed with context c, the item's true label Y i is transformed by noise into a "context-specific" label S c i ∈ V .

the true label may appear differently when seen through the lens of each context.


On the other extreme, in the Political Party context, annotators selected "Trump" or "Clinton" too often when the model settled on the "Neutral" option.

even when a user's stance was not clear to annotators in other conditions, annotators who saw political party still inferred stance from the text.


In our empirical investigation, we highlight the critical role of control conditions in the validation of experimental findings.

we argue that every observation about a change of meaning over time should be subjected to a control test.


As we show in Section 4, some recently reported laws of historical meaning change do not survive this proposed test.

similar results are obtained in the genuine and control conditions.


This embedding was shown to improve results on downstream tasks (Baroni et al., 2014;Bullinaria and Levy, 2012;Turney and Pantel, 2010).

the top 300 elements of the diagonal matrix of singular values Σ, denoted Σ d , were retained to represent a new, dense embedding of the word vectors, using the truncated left hand orthonormal matrix U d : These representations were subsequently aligned with the orthogonal Procrustes method following (Hamilton et al., 2016).


While identifying such laws is probably within the reach of NLP analyses of massive digital corpora, we argued that a more stringent standard of proof is necessary in order to put them on a firm footing.

it is necessary to demonstrate that any proposed law of change has to be observable in the genuine condition, but to be diminished or absent in a control condition.


As such, it would be surprising if word frequency, polysemy, and prototypicality were to capture too high a degree of variance.

since semantic change may result from the interaction of many factors, small effects may be a priori more credible than large ones.


Sequence learning, especially using LSTM (Hochreiter and Schmidhuber, 1997;Graves and Schmidhuber, 2005;Graves, 2013), has become a well-established standard in numerous NLP tasks (Zhou and Xu, 2015;Ma and Hovy, 2016;Wang and Chang, 2016).

sequence-to-sequence models (Sutskever et al., 2014) have grown increasingly popular and are used extensively in, e.g., Machine Translation (Cho et al., 2014;Bahdanau et al., 2015), Sentence Representation (Kiros et al., 2015), Syntactic Parsing , Conversation Modeling (Vinyals and Le, 2015), Morphological Inflection (Faruqui et al., 2016) and Text Summarization (Gu et al., 2016).


In the context of WSD framed as a sequence learning problem, a sequence-to-sequence model takes as input a training set of labeled sequences (cf.

section 3.1) and learns to replicate an input sequence x while replacing each content word with its most suitable word sense from s. sequence-to-sequence WsD can be viewed as the combination of two sub-tasks: • A memorization task, where the model learns to replicate the input sequence token by token at decoding time; • The actual disambiguation task where the model learns to replace content words across the input sequence with their most suitable senses from the sense inventory s. In the latter stage, multi-word expressions (such as nominal entity mentions or phrasal verbs) are replaced by their sense identifiers, hence yielding an output sequence that might have a different length than x.


The encoder-decoder architecture generalizes over both the models in Sections 3.1 and 3.2.

we include one or more bidirectional LSTM layers at the core of both the encoder and the decoder modules.


In line with previous approaches, and guided by the intuition that WSD is strongly linked to other NLP tasks at various levels, we also design and study experimentally a multitask augmentation of the models described in Section 3.

we consider two auxiliary tasks: • Part-of-speech (POS) tagging, a standard auxiliary task extensively studied in previous work .


In this paper, we focus on the task of measuring word relatedness over time.

we infer whether two words (tokens) relate to each other during a certain time range.


Our technique allows us to produce temporally appropriate expansions.

given a query of n entities Q = {e 1 , e 2 , .


Using our technique, the temporal focus of a query can be identified and appropriate expansions offered to the end-user.

we can analyze the relationship between the query entities to identify the "most relevant" time period -when those entities were strongly connected.


The task addressed in this paper has not been extensively studied for under-resourced languages.

pattern-based methods, although effective for English, are language-dependent to a large extent.


We propose an end-to-end neural model for idiomaware distributed semantic representation.

in terms of invisibility, we introduce a neural idiom detector to adaptively distinguish literal and idiomatic meaning of each phrase when learning sentence representations.