To that end, for each of the three languages, we selected a morphological class of interest and evaluated for all words in the full UD test set that were assigned to that class what the most important character set of length one, two and three was.

we evaluated for each word for which the class was cor- rectly predicted, which character set had the highest positive contribution towards predicting that class.


In this paper, we take a different task: focusing instead on the similarity of the surface forms, phonology, or morphology of the two transfer languages.


To overcome this issue, we build HYPO, the first dataset focused on exaggeration, and introduce the task of automatic hyperbole detection.

the goal of this work is to find an automatic solution to establish if the topic of a sentence is aggrandized or, on the contrary, is presented as it is in reality.


It makes things sound bigger than they actually are, but with the goal of emphasizing them, of clarifying ideas, of adding interest and humor to the conversation (Roberts and Kreuz, 1994).

speakers exaggerate to accentuate the one element, account or aspect of reality that is important to them, in order to make others see truth from their perspective (Ritter, 2012).


Each of the 854 sentences was expected to be judged by 5 annotators provided with minimal instructions on the subject at hand.

the annotators decided if a sentence contained a hyperbole (Question 1), and if so, they highlighted the words that caused its hyperbolic bent (Question 2), paraphrased it without exaggerating (Question 3), classified the hyperbole as quantitative or qualitative (Question 4), rated the degree of the exaggeration (Question 5) and established whether it was creative or conventional (Question 6).


Hence, its main contribution to the field of NLP is the proposal of a new task together with the construction of a corpus of hyperboles, and its main achievement is the devising of a procedure to learn such figurative mechanism.

the described experiment tested the hypothesis that quantity and quality, which emerge from the body of literature as two core features of exaggerations, are also useful for the automatic processing of the figure.


Observing that the situation entity type of a clause depends on discourse functions the clause plays in a paragraph and the interpretation of discourse functions depends heavily on paragraph-wide contexts, we propose to build context-aware clause representations for predicting situation entity types of clauses.

we propose a hierarchical recurrent neural network model to read a whole paragraph at a time and jointly learn representations for all the clauses in the paragraph by extensively modeling context influences and inter-dependencies of clauses.


Therefore, we aim to build context-aware clause representations dynamically which are informed by their paragraph-wide contexts.

we propose a hierarchical recurrent neural network model to read a whole paragraph at a time and jointly learn representations for all the clauses in the paragraph.


However, a major limitation in these previous research is that they are assuming that for each post, the main thread where it belongs is given.

the problem they solve is finding the post for which a post is immediately replying, rather than treating the corpus as a single set of posts with no known information about the threads, the initial post of each thread, and the posts that belong to each thread.


2) We assume hierarchy structure of a global cluster and a local cluster.

one global cluster can consist of multiple local clusters.


The fact that we observe no gains from the additional sentence-level Bi-LSTM encoder for the DRC and SAC tasks suggests that the content of the sentence informs its discourse role and subjective aspect much more strongly than neighboring sentences.

the DRC and SAC seem to be more localized classification tasks than SRC.


However, the performance for the argumentation component identification does not improve in MTL.

the extraction of fine-grained argumentative components seems to inform higher-level rhetorical analysis tasks, but not vice-versa.


Through a careful examination of the data, we notice that time expressions or events are selective as to what types of time expression or events can be their parent.

the category of the child time expression or event has a strong indication on which candidate can be its parent.


Researchers who, instead, tried to build the end-to-end parsing pipelines considered a wider range of approaches including sequence models and RNNs (Biran and McKeown, 2015;Feng and Hirst, 2014;Ji and Eisenstein, 2014;Li et al., 2014).

when they tried to utilize the discourse structures for out-domain applications, they used RNNbased models and found that those models are advantageous for their downstream tasks (Bhatia et al., 2015;Ji and Smith, 2017).


Then, we conducted univariate feature selection to restrict all remaining features to those showing at least a small relationship with the outcome.

we keep all features passing a family-wise error rate of α = 60 with the given outcome.


This might have limited performances of more complex models, but considering the processing time and the computation load, the combination of the linear model and the RNN-based model of our pipeline obtained both the high performance and efficiency for the practical applications to downstream tasks.

it's possible the linear model will not perform as well if the training size is increased substantially.


In this section we describe a user study we ran for testing the hypotheses about which entities constitute useful recommendations.

we test whether what a user writes next in a document is actually considered to be a good recommendation by the raters and whether IDF score correlates with how useful an entity is considered.


Notice how predictions from XML-CNN base are more generic than from XML-CNN IDF.

for the first input related to Antarctic Ice Sheet gaining ice entities Earth and research are recommended.


Good entity recommendations need to be both relevant and interesting, which we motivated with a user study.

we showed how entities which users write in the document are considered as good recommendations and how IDF score correlates with how useful an entity is considered to be.


The problem of recommending content to document writers has recently been addressed by industry with tools like Google Explore in Docs and Microsoft Researcher, however the systems are proprietary and the methods have not been published.

this work is the first to formalize the problem and provide insights about it to the research community.


The sentiment of an aspect in a sentence can influence the succeeding aspects due to the presence of conjunctions.

for sentences containing conjunctions like and, not only, also, but, however, though, etc., aspects tend to share their sentiments.


We proposed a novel interpretable neural network approach for relevant emotion ranking.

motivated by transfer learning, the neural network is initialized to make the hidden layer approximate the behavior of topic models.


In this paper, we have proposed a novel interpretable neural network for relevant emotion ranking.

motivated by transfer learning, the neural network is initialized to make its hidden layer approximate the behavior of a topic model.


In this paper, we propose a multi-grained attention network to address the above two issues in aspect level sentiment classification.

we propose a fine-grained attention mechanism (i.e.


We employ the pretrained word vector, GloVe (Pennington et al., 2014), to obtain the fixed word embedding of each word.

we denote the embedding lookup matrix as L ∈ R dv×|V | , where d v is the word vector dimension and |V | is the vocabulary size.


We employ a bidirectional Long Short-Term Memory Network (BiLSTM) on top of the embedding layer to capture the temporal interactions among words.

at time step t, given the input word embedding x, the update process of forward LSTM network can be formalized as follows: Where σ is the sigmoid activation function, i t , f t , o t are the input gate, forget gate and output gate, respectively.


In addition, considering that the context words with closer distance to an aspect may have higher influence to the aspect, we utilize the position encoding mechanism to simulate the observation.

formally, the weight for a context word w j , which has l word-level distance from the aspect (here we treat the aspect phrase as a single unit), is defined as follows: we treat the weights of words within the aspect as 0 in order to focus on the context words in the sentence.


In terms of the previous example, the aspect "speaker quality" should pay more attention on "lacking" and less attention on "like", compared with aspect "Mac OS" due to their different sentiment polarities.

for each aspect pair a i and a j in aspect list A, we compute the square loss on the coarse-grained attention vector a ca i and a ca j , and also estimate the distance d ij ∈ [0, 1] between a i and a j as the loss weight.


Compared with MGAN-CF, the complete MGAN model gains further improvement by bringing the aspect alignment loss, which is designed to capture the aspect level interactions.

we collect the statistics of sentencelevel with different aspect amounts, which is shown in Table 5.


In this paper, we propose a multi-grained attention network (MGAN) for aspect-level sentiment classification.

we propose a finegrained attention mechanism, which is responsible for linking and fusing the words from the aspect and context, to capture the word-level interaction.


Ablation Study In this section, we study the impacts and contribution of the different components of our model.


Removing either or both can cause performance to degrade.

we also observe that CA seems to be less important than CC, i.e., performance drops more as compared to removing CA.


The objective is to learn the joint-association between the multiple modalities & utterances, and to emphasize on the contributing features by putting more attention to these.

we employ bi-modal attention framework, where an attention function is applied to the representations of pairwise modalities i.e.


Assuming a particular video has 'u' utterances, the raw utterance level multi-modal features are rep- Three separate Bi-GRU layers with forward & backward state concatenation are first applied on the raw data followed by the fully-connected dense layers, resulting in T ∈ R u×d (text), V ∈ R u×d (visual) and A ∈ R u×d (acoustic), where 'd' is the number of neurons in the dense layer.

finally, pairwise-attentions are computed on various combinations of three modalities- the attention between V and T is computed as follows: • Bi-modal Attention: Modality representations of V & T are obtained from the Bi-GRU network, and hence contain the contextual information of the utterances for each modality.


For the three modalities, three separate attention blocks are required, where each block takes multi-utterance information of a single modality and computes the self attention matrix.

the MU-SA attention (A v ) on V (visual) will be computed as follows, Mv(i,j) u k=1 e Mv(i,k) for i, j = 1, .., u The attention matrix A p ∈ R 3×d is computed for p = 1 st , 2 nd , ...u th utterances.


Thus, adaptation performance will decline with an increase in distribution difference.

in sentiment analysis, reviews of different products have different vocabulary.


Our main intuition is to treat the problem as a semi-supervised learning task by considering target instances as unla-beled data, assuming the domain distance can be effectively reduced through domain-invariant representation learning.

the proposed approach jointly performs feature adaptation and semi-supervised learning in a multi-task learning setting.


Existing work on this topic has not explicitly addressed the problem of local feature redundancy.

when two features are equally predictive of an outcome, which of them should be included in the saliency mask for that decision?


We also show that manipulating the model bias term to set a semantically appropriate "default behavior" or "null hypothesis" for the model significantly improves performance.

by explicitly choosing what output a zero-information, empty explanation should correspond to, the model is able to learn explanations that correspond more closely with human-generated data.


Term 3.4, which is new, ensures recall by encouraging the adversarial predictor's prediction on the antirationale to be similar to the prediction it would make with no information at all (aka the default value).

the antirationale should contain no predictive signal.


What this means is while 65% of distinct phrases discovered by the generator are unnecessary for accurate prediction, the 35% of them that are necessary lead to a major decrease in predictive accuracy.

the generator "hedges its bets" with respect to predictive accuracy by including more information in the rationales than it has to, and experiences a better global optimum as a result.


We construct the first corpus of human-annotated vague words and sentences and present empirical studies on automatic vagueness detection.

we investigate context-aware and context-agnostic models for predicting vague words, and explore auxiliary-classifier generative adversarial networks for characterizing sentence vagueness.


Our methods for automatically detecting vague words and sentences are based on deep neural networks, which have demonstrated impressive recent success.

we investigate context-aware and context-agnostic models for predicting word vagueness, where feature representations of words are built with and without considering their surrounding words.


Privacy policies are often verbose, difficult to read, and perceived as ineffective (McDonald and Cranor, 2008).

vague language in these documents hurts understanding.


It uses intrinsic feature representations of words without considering the context.

we represent a word using its word2vec embedding, then feed it to a feedforward layer with sigmoid activation to obtain the prediction (Figure 2).


We further investigate the feasibility of predicting vague words and sentences using deep neural networks.

we investigate context-agnostic and context-aware models for detecting vague words, and AC-GAN for detecting vague sentences.


Building on this insight, we propose a new model MVDAM: Multi-view document attention model to detect the ideological bias of news articles by leveraging cues from multiple views: the title, the link structure, and the article content.


Our work draws inspiration from both of these works but differentiates itself from these in the following aspects: We leverage the structure of a news article by noting that an article is just not free-form text, but has a rich structure to it.

we model cues from the title, the inferred network, and the content in a joint generic neural variational inference framework to yield improved models for this task.


We control for time by obtaining article content over a fixed time-period for all sources.

we spider several news sources and perform data cleaning.


Specifically, we spider several news sources and perform data cleaning.

the spidering component collates the raw HTML of news sources into a storage engine (MongoDB).


4 Data Pre-processing and Cleaning Since the labels were derived from the source, we are careful to remove any systematic features in each article which are trivially reflective of the source, since that would result in over-fitting.

we perform the following operations: (a) Remove source link mentions When modeling the link structure of an article, we explicitly remove any link to the source itself.


Second, we also explicitly remove any systematic link structures in articles that are source specific.

some sources may always have links to other domains (like their own franchisees or social media sites).


Bayesian approaches with stochastic attention have been noted to be quite effective at modeling ambiguity as well as avoiding over-fitting scenarios especially in the case of small training data sets (Miao et al., 2016).

we assume a latent representation h learned from the multiple modalities in X which is then mapped to the label space Y .


In the most general setting, instead of learning a deterministic encoding h given X, we posit a latent distribution over the hidden representation h, Pr(h|X) to model the overall document where Pr(h|X) is parameterized by a diagonal Gaussian distribution N (h|µ(X), σ 2 (X)).

consider the distribution Pr(Y |X) which can be written as follows: As noted by Miao et al.


The inference network takes as input the features X and learns a corresponding hidden representation h. More specifically, it outputs two components: (µ, ς) corresponding to the mean and log-variance of the gaussian parametrizing the hidden representation h. We model this using a "multi-view" network which incorporates hidden representations learned from multiple modalities into a joint representation.

given ddimensional hidden representations corresponding to multiple modalities z title , z network , and z content the model first concatenates these representations into a single 3d-dimensional representation z concat which is then input through a 2-layer feed-forward network to output a d-dimensional mean vector µ and a d-dimensional log-variance vector ς that parameterizes the latent distribution governing h. We now discuss the models used for capturing each view.


Convolutional networks have been shown to be very effec-tive for modeling short sentences like titles of news articles.

we use the same architecture proposed by (Kim, 2014).


We use a state-of-the-art network representation learning algorithm to learn representations of nodes in a social network.

we use Node2Vec (Grover and Leskovec, 2016), which learns a d-dimensional representation of each source given the hyperlink structure graph G. Node2Vec seeks to maximize the log likelihood of observing the neighborhood of a node N (u), given the node u.


Having learned the embedding matrix F for each source node, we now model the link structure of any given article A simply by the average of the network embedding representations for each link l referenced in A.

we compute z network as: To model the content of an article, we use a hierarchical approach with attention.


In particular, we compute z network as: To model the content of an article, we use a hierarchical approach with attention.

we compute attention at both levels: (a) words and (b) sentences.


Since not all words contribute equally to the representation of the sentence, we introduce a word level attention mechanism which attempts to extract relevant words that contribute to the meaning of the sentence.

we learn a word level attention matrix W w as follows is the latent representation of the sentence i.


Once again, since not all sentences contribute equally to the representation of the article, we introduce a sentence level attention mechanism which attempts to extract relevant sentences that contribute to the meaning of the article.

we learn the weights of a sentence level attention matrix s α s h s , where z content is the latent representation of the article.


The KL Divergence term in Equation 2 can thus be analytically computed.

the KL Divergence between two K dimensional Gaussian distributions A, B with means µ A , µ B and diagonal co-variances κ A , κ B is: Parameter Estimation Having described precisely, the models for each of the components in Equation 2, we can reformulate the maximization of the variational lower bound to the following loss function on the set of all learn-able model parameters θ: J (θ) as follows: where NLL is the negative log likelihood loss computed between the predicted label and the true label, and λ is a hyper-parameter that controls the amount of regularization offered by the KL Divergence term.


Finally, all flavors of our model outperform the baselines.

observe that incorporating the network cues outperforms all uni-modal models that only model either the title, the network, or the content.


Table 2, we highlight some of the challenges of our model.

our model finds it quite challenging to identify the political ideology of the source for articles that are non-political and related to global events, or entertainment.


(2018) focused on the process of proliferation of true and false news online.

they analyzed 126K stories tweeted by 3M people more than 4.5M times, and confirmed that "fake news" spread much wider than true news.


In order to predict the factuality of reporting and the bias for a given news medium, we collect information from multiple relevant sources, which we use to train a classifier.

we collect a rich set of features derived from (i) a sample of articles from the target news medium, (ii) its Wikipedia page if it exists, (iii) its Twitter account if it exists, (iv) the structure of its URL, and (v) information about the Web traffic it has attracted.


The information that can be extracted from a news medium's Twitter profile can be valuable for our tasks.

we use the following features: • Has Account: Whether the medium has a Twitter account.


For this work, we use the length of the URL, the number of sections and the excessive use of special characters such as digits, hyphens and dashes.

we identify whether the URL contains digits, dashes or underscores as individual symbols, which were found to be useful as features for detecting phishing URLs (Basnet et al., 2014).


Despite these efforts in designing efficient features and employing advanced NLP techniques, LJP is still confronted with two major challenges: Multiple Subtasks in Legal Judgment: Practically, legal judgment usually consists of detailed and complicated subclauses, such as charges, the term of penalty, and fines.

for those countries with the civil law system (e.g., China, France, and Germany), the prediction of relevant articles is also considered to be one of the fundamental subtasks, which will guide the prediction for other subtasks.


Specifically, for those countries with the civil law system (e.g., China, France, and Germany), the prediction of relevant articles is also considered to be one of the fundamental subtasks, which will guide the prediction for other subtasks.

all these subtasks compose the complete form of judgment pre-diction.


Accordingly, we introduce topological learning for LJP and propose a unified framework, named as TOPJUDGE.

given the encoded representation of the fact description, TOPJUDGE predicts the outputs of all the subtasks following the topological order, and the output of a specific subtask will be affected by all the subtasks it depends on.


This helps RFA to benefit from the advantages of both residualized control and factor adaptation.

rFA combines linguistic and extra-linguistic features on both the feature/data level and the model level.


The framing lexicons well-capture the numerous themes in this article.

the frames identified and the related framing lexicon words are: Political: electorate, election, former, preelection, political scientists, congress, president, post, bush Public Sentiment: elections, campaign, preelection, democrats, republicans External Regulation and Reputation: west, war, former, washington, politics, summit, exacerbation, west, decision, bush, president Morality: peace, sins, ideals, love, values Fairness and Equality: politics, love, values This article uses several strategies to promote unity in Russia and actively separate Russia from Western culture, including criticizing American politics and emphasizing a difference of values.


This investigation aims to use intra-textual sentiment analysis to examine the continuous sentiment styles used in YouTube vlogs.

we primarily aim to identify distinct trajectories of continuous sentiment.


The rationale is inspired by the algorithm behind sentimentr (Rinker, 2018b) but extends that approach to data that are nonpunctuated or very brief.

our algorithm identifies each sentiment as matched with the 'Jockers & Rinker Polarity Lookup Table' from the lexicon R package (Rinker, 2018a) and then constructs a "naïve context" cluster (i.e., without relying on punctuation or other structure) around that sentiment (here: two words before and after the sentiment).


We address the task of native language identification in the context of user generated content (UGC) in online communities.

we use a large corpus of English Reddit posts in which the L1 of authors had been accurately annotated (Rabinovich et al., 2018).


Furthermore, we define a plethora of features, some which have been used in earlier works but others that are novel.

we define a set of features that rely on the characteristics of the social media outlet, thereby extending the task somewhat, from linguistic analysis to user profiling.


Subreddits This feature set works so well because many of the most popular subreddits in which users post are culturally revealing.

there is a significant presence in this list to (subreddits focusing on) specific countries.


Instead of solving the problem, in this paper, we aim to deeply understand the causes of the problem.

we want to answer the following two questions: • Is error propagation the main cause of accuracy drop?


As aforementioned, reinforcement learning (RL) is leveraged to bridge the gap between training and inference of NMT, by directly optimizing the evaluation measure (e.g., BLEU) at training time.

nMT model can be viewed as an agent, which interacts with the environment (the previous words y <t and the context vector z available at each step t).


The former one sequentially leverages the source-side and target-side monolingual data for RL training.

we first train an MLE model using the bilingual data and source-side (or target-side) monolingual data; based on this MLE model, we then use REINFORCE for training with target-side (or source-side) monolingual data.


we meta-learn the initialization from auxiliary tasks and continue to learn the target task.


In addition to that, the results for the constrained variants of this SMT system justify some of the simplifications required by our approach.

removing lexical reordering and constraining the phrase table to the most frequent n-grams, as we do for our initial system, has a relatively small effect, with a drop of less than 1 BLEU point in all cases, and as little as 0.28 in some.


Our method is also connected to some previous approaches to improve machine translation using monolingual corpora.

the generation of a synthetic parallel corpus through backtranslation (Sennrich et al., 2016), which is a key component of our unsupervised tuning and iterative refinement procedures, has been previously used to improve NMT.


1 has the more semantic connection with the image.

we produce a set of weights β = {β 1 , β 2 , .


In this study, we propose a novel task/method to address QA sentiment analysis.

we create a high-quality annotated corpus with speciallydesigned annotation guidelines for QA-style sentiment classification.


Second, we propose a hierarchical matching network model to address the challenges of QA-style sentiment classification.

we first segment both the question and answer text into sentences and construct the [Q-sentence, A-sentence] units for each QA text pair.


Figure 3 depicts the detail architecture of QA bidirectional matching mechanism.

we first calculate the bidirectional pair-wise matching matrix by using the fol-lowing formula: where is the score that measures how well the word in S Q i semantically matches the word in S A j and vice versa.


This may be due in part to the limitations of the underlying models and training resources, particularly as they relate to heterogeneous sources.

most current approaches to AM are designed for use with particular text types, faring poorly when applied to new data .


We are releasing the gold-standard annotations for this dataset, and code for retrieving   We model the identification of arguments as a sentence-level classification task.

given a sentence ς ς ς with words u 1 , .


To let the model learn which parts of the sentence are relevant (or irrelevant) to the given topic, we use an attentionbased neural network (Bahdanau et al., 2014) that learns an importance weighting of the input words depending on the given topic.

we adopt an outer-attention mechanism similar to the one proposed by Hermann et al.


When comparing between aspect-based alternatives, we observe that the extraction accuracy correlates with the quality of aspect prediction.

ranking using MILNET+MATE+MT gives best results, with a 2.6% increase in MAP against MILNET+MATE and 4.6% against MIL-NET+ABAE init .


9 We use a zero-padding strategy as in (Kim, 2014  We construct a set of hashtags to collect a separate dataset of English tweets from the Twitter API.

we use the eight basic emotions: anger, anticipation, disgust, fear, joy, sadness, surprise, and trust.


Conditional models are frequently encountered in practice; however there has not been a rigorous theoretical analysis of NCE in this setting, and we will argue there are subtle but important questions when generalizing NCE to the conditional case.

we analyze two variants of NCE for conditional models: one based on a classification objective, the other based on a ranking objective.


1, and we will argue there are subtle but important questions when generalizing NCE to the conditional case.

the joint model in Eq 2 has a single partition function Z(✓) which is estimated as a param-eter of the model (Gutmann and Hyvärinen, 2012) whereas the conditional model in Eq 1 has a separate partition function Z(x; ✓) for each value of x.


Levy and Goldberg (2014) make a connection between the NCE-based method of (Mikolov et al., 2013), and factorization of a matrix of pointwise mutual information (PMI) values of (x, y) pairs.

consistency of the NcE-based method under assumption 2.2 implies a similar result, specifically: if we define p N (y) = p Y (y), and define converge to values such that following (Levy and Goldberg, 2014), the inner product x is an estimate of the PMI up to a constant offset H(✓).


Recently, deep neural networks have achieved state-of-the-art results in various tasks including computer vision, natural language processing, and speech processing.

neural text generation models, central focus of this work, have led to great progress in central downstream NLP tasks like text summarization, machine translation, and image captioning.


Interestingly, the reduced examples transfer to other architectures.

when we feed fifty reduced SNLI inputs from each class-generated with the BIMPM model (Wang et al., 2017)-through the Decomposable Attention Model (Parikh et al., 2016), 2 the same prediction is triggered 81.3% of the time.


We provide empirical evidence for this conjecture by comparing BANDITSUM to the sequential RL model proposed by Wu and Hu (2018) (Figure 2) on two subsets of the data: one with good summary sentences appearing early in the article, while the other contains articles where good summary sentences appear late.

we construct two evaluation datasets by selecting the first 50 documents (D early , i.e., best summary occurs early) and the last 50 documents (D late , i.e., best summary occurs late) from a sample of 1000 documents that is ordered by the average extractive label index idx.


We make the first attempt at utilizing publicly available datasets originally prepared for data-to-text language generation to produce such annotations for words and phrases in natural language without additional supervision.

we conduct our study on a recently released dataset of descriptions for NBA basketball games with structured tables of game records.


In this work, we use the most straightforward heuristics to perform template induction, utilizing the established correspondences and inferred parameters.

we first blank out the correspondences of numerics-to-numerics and string-to-string to be empty slots and replace with the tag names.


Neural architectures have often yielded performance gains when trained for multiple tasks together (Collobert et al., 2011;Luong et al., 2015;Chen et al., 2017;Hashimoto et al., 2017).

performance of semantic role labeling tasks improves when done jointly with other semantic tasks (FitzGerald et al., 2015;Peng et al., 2017Peng et al., , 2018.


This inspires us to design a novel agreement-based model to explore such semantic coherency information.

we define a metric for comparing predicate-argument structures and searching for relatively good automatic syntactic and semantic annotations to extend the training data for SRL systems.


To evaluate the robustness of state-of-the-art SRL algorithms, we evaluate two representative SRL frameworks.

one is a traditional syntaxbased SRL system that leverages a syntactic parser and manually crafted features to obtain explicit information to find semantic roles (Gildea and Jurafsky, 2000;Xue, 2008) we employ the system introduced in Feng et al.


The second SRL framework leverages an endto-end neural model to implicitly capture local and non-local information (Zhou and Xu, 2015;He et al., 2017).

this framework treats SRL as a BIO tagging problem and uses a stacked BiLSTM to find informative embeddings.


To train the three SRL systems as well as the supporting parsers, we use the CTB and CPB data 4 .

the sentences selected for the CoNLL 2009 shared task are used here for parameter estimation.


The constituent "常常" (often) are adjuncts attached to VP structure governed by the verb "练习"(practice), which will not be labeled as AM in terms of the verb "做饭"(cook-meal).

the hierarchical structure can help in argument identification and assignment by exploiting local information.


We explore the valuable information about the semantic coherency encoded in the L2-L1 parallel data to improve SRL for learner Chinese.

we introduce an agreement-based model to search for high-quality automatic syntactic and semantic role annotations, and then use these annotations to retrain the two parser-based SRL systems.


To address this issue, we propose a practical teacherstudent framework to extend RL-based dialog systems without retraining from scratch.

the "student" is an extended dialog manager based on a new ontology, and the "teacher" is existing resources used for guiding the learning process of the "student".


The idea is to transfer the knowledge of existing resources to a new dialog manager.

after the system is deployed, if developers find some intents and slots missing before, they can define a few simple dialog rules to handle such situations.


The difference between A u and A u is reflected on the domain specific intents 7 .

in addition to the intents of Sim1, A u includes the "confirm" intent.


Moreover, we narrow down the action space into a small subspace instead of that used in full-domain setting, and gradually introduce more complex user goals and expand the action space as the training proceeds.

we start from a set of necessary slots and actions to accomplish most of the user goals, and then extend the action space and complexity of user goals once every 20 epoch (after epoch 50).


In this paper, we introduce an approach that leverages context-aware features at each time step.

we make use of self-attention to produce context-aware representations of the embeddings.


Lastly, the attention output is a weighted sum of M c .

we consider different K parts in detail for k = 1, .., K: where M k,a ∈ R (dm/K)×T is the k-th part of M a which is transformed from M by W a .


The intent label is provided by correct label during training phase, and by the output from intent classification layer in the test phase.

for t = 1, ...T : We use element-wise multiplication to model the interaction between BiLSTM outputs and the gate vector.


In this paper, we generalize the third type of post-editing into a more general scenario, named Quantifiable Sequence Editing (QuaSE).

in the training stage, each input sentence is associated with a numeric outcome.


To further enhance the capability of generating expected output, we introduce a dual reconstruction structure which exploits the couplings of latent factors of pseudo-parallel sentences.

it attempts to reconstruct one sentence in a pair from the combination of its outcome factor and the other sentence's content factor, based on the intuition that the wording difference in a pair is outcome-related.


The left hand side models individual sentences.


Different from reconstructing an original sentence in Section 2.2, in the dual reconstruction, given a sentence x, we reconstruct its dual sentence x .

we first encode x and x into their outcome factors y/y and content factors z/z .


Our dataset contains sentences extracted from Yelp reviews 2 , where each review is associated with a rating in {1, 2, 3, 4, 5}.

we employ the sentences with sentiment polarity (i.e.


We employ the sentiment analyzer in Stanford CoreNLP (Manning et al., 2014) to do so.

we first invoke CoreNLP to output the probability of each rating in {1, 2, 3, 4, 5} for a sentence, then we take the sum of the probabilitymultiplied ratings as the sentence rating.


For preparing the pseudo-parallel sentences, we first follow the ideas in (Guu et al., 2018) to generate some initial pairs.

we first calculate the Jaccard Index (JI) for each pair of sentences, and keep those with JI values no less then 0.5 as the initial pairs.


In this paper, we present a deep reinforcement learning approach to paraphrase generation.

we propose a new framework for the task, which consists of a generator and an evaluator, both of which are learned from data.


We propose a framework, which contains a generator and an evaluator, called RbM (Reinforced by Matching).

for the generator we adopt the Seq2Seq architecture with attention and copy mechanism (Bahdanau et al., 2015;See et al., 2017), and for the evaluator we adopt the decomposable attention-based deep matching model (Parikh et al., 2016).


The encoder RNN transforms the input sequence X into a sequence of hidden states The decoder RNN generates an output sentence Y on the basis of the hidden states.

it predicts the next word at each position by sam- where s t is the decoder state, c t is the context vector, y t−1 is the previous word, and g is a feed-forward neural network.


The pointer-generator model allows either generating words from a vocabulary or copying words from the input sequence.

the probability of generating the next word is given by a mixture model: where q(s t , c t , y t−1 ) is a binary neural classifier deciding the probability of switching between the generation mode and the copying mode.


The reward R is given by the evaluator M φ .

when a pair of input sentence X and generated paraphraseŶ = G θ (X) is given, the reward is calculated by the evaluator R = M φ (X,Ŷ ).


In our case, we use the edit distance E(X, Y ) between X and Y as the measure of difficulty for paraphrasing.

w k is determined by w k ∼ Binomial(p k , 1), and , where K denotes the batch size for training the evaluator.


In both settings, we find that the proposed RbM-SL and RbM-IRL models outperform the baseline models in terms of all the evaluation measures.

in Quora-II, RbM-SL and RbM-IRL make significant improvements over the baselines, which demonstrates their higher ability in learning for paraphrase generation.


Please judge the paraphrases from the following two criteria: (1) Grammar and Fluency: the paraphrase is acceptable as natural language text, and the grammar is correct; (2) Coherent and Consistent: please view from the perspective of the original poster, to what extent the answer of paraphrase is helpful for you with respect to the original question.

you can consider following aspects: -Relatedness: it should be topically relevant to the original question.


We use the pointer-generator network (See et al., 2017) for the copy mechanism.

the binary variable z t is calculated from the context vector c t , the decoder state d t and the decoder input y t−1 : where vectors w c , w d , w y and the scalar b ptr are learnable parameters, and σ is the sigmoid func-tion.


A quantization layer is designed to map the scalar values into several bins, namely quantization units.

we feed each scalar value op scl i to a softmax layer, and its representation h res i is computed as the weighted sum of all quantization embeddings: where W q and b q are trainable parameters, e scl is the quantization embedding and L is the size of quantization units.


We randomly select some games from testing set, and entrust a professional crowdsourcing company to annotate the generated texts 5 .

three native English workers who are familiar with NBA games are hired.


Then, we utilize a convolutional neural network (CNN) to learn the matching score between L i g and T , where the score is interpreted as the degree of thematic consistency.

in the discriminator, we treat L i g and L i as the negative and positive instance, referring to thematically inconsistent and consistent case, respectively.


Key-AS2S and Mem-AS2S are AS2S with keywords planning (Wang et al., 2016c) and a memory module (Zhang et al., 2017a), respectively.

they are dedicated models designed for Chinese poem generation.


In this paper, we proposed an effective approach that integrates CVAE and adversarial training for classical Chinese poem generation.

we used CVAE to generate each line of a poem with novel and diverse terms.


In this paper, we implement this approach and carefully study its generation results.

we randomly sample 130 questions generated by the approach, and manually judge their quality by comparing them with the references.


propose to combine templates and sequence-tosequence model.

they mine question patterns from a question answering community and apply sequence-to-sequence to generate question patterns for a given text.


To control the balance among different modes, we introduce a 3dimensional switch probability: The final probability distribution is calculated via a weighted sum of the three mode probability distributions: Another main issue of NQG is that the generated question copies the context words that are distant from and irrelevant to the answer, instead of the words that are close and relevant to the answer.

attention is distracted by irrelevant words far away from the answer.


We use the output of the language model, cross-entropy, as the reward.

given a sentence y t , the cross-entropy based reward for the k th word is calculated as We maximize the reward of real-world text and minimize the reward of generated text to train the discriminator.


Both the generator and the discriminator are pre-trained for 10 epochs before adversarial learning.

for PG-BLEU and SeqGAN, before reinforcement learning or adversarial learning, we pre-train the sequence-to-sequence model for 10 epochs like DP-GAN.


Given the rising interest in such automatic question generation (AQG) systems, it is important to objectively examine whether these metrics are suitable for this task.

it is important to verify whether such metrics used for evaluating AQG systems focus on answerability of the generated question by preferring questions which contain all relevant information such as question type (Wh-types), entities, relations, etc.


Before this field grows further, it is important that the community critically examines the current evaluation metrics being used for this task.

there is a need to closely examine the utility of existing n-gram based similarity metrics such as BLEU (Papineni et al., 2002), METEOR (Lavie and Denkowski, 2009), NIST (Doddington, 2002), etc.


To concretize the intuitions developed with the help of the above examples, we first collect human judgments.

we take questions from existing datasets for document QA, knowledge base QA and visual QA and add systematic noise to these questions.


NIST: NIST is a variant of the standard BLEU metric that takes into account the relative importance of each n-grams in the sentence.

the metric gives a high weightage to ngrams which have a lower frequency in the corpus and hence are more informative as compared to very frequent n-grams which are less informative.


However, when we asked human evaluators to analyze 200 questions generated by an existing AQG system, they reported that the quality was poor.

after having discussions with annotators, we found that using this output, it would be very difficult to conduct such a systematic study to assess the importance of different words in the question.


We did an initial pilot using the instructions mentioned in Table 2, but due to the subjective nature of the task, it was difficult for the annotators to agree on the notion of important information.

we found that the annotators disagreed between most important information and all important information (i.e., they were confused between rating 1 v/s 2 and 4 v/s 5).


For each source, we used 300 of these annotations and used bagging to find the optimal weights.

we drew 200 samples randomly from the given set of 300 samples and did a grid search to find w i 's and δ such that the Q-METRIC computed using Equation 1 had maximum correlation with human scores.


The correlation scores obtained for different Q-METRICs are indeed encouraging.

we observe that while the correlation of existing metrics with noisy questions generated was very low (Table 4), the correlation of the modified metrics is much higher.


the Q-BLEU4 metric than the BLEU4 metric (i.e., in general, higher Q-BLEU4 indicates better performance and lower Q-BLEU4 indicates poor performance).

notice that BLEU4 gives much importance to stop words, but these words hardly have any influence on the final performance.


The main aim of this work was to objectively examine the utility of existing metrics for AQG.

we wanted to see if existing metrics account for the answerability of the generated questions.


We denote the character vocabulary as V .

we use bidirectional LSTM (bi-LSTM) (Hochreiter and Schmidhuber, 1997;Schuster and Paliwal, 1997) model as the encoder to project the input sentence X into the vector space.


However, there is no theoretical guarantee that the output sentence generated by the decoder is strongly correlated to the style id input one hot(k).

when the input style id is changed, the output sentence would probably be the same because no supervised loss is given to force the output sentences to follow the onehot style representation.


In this paper, we employ neural network to parametrize the posterior distribution estimation function Q.

we first compute the average character embeddings of sequence Y and then use a linear projection with softmax normalizer to get the style distribution.


Note that our model is fully unsupervised: the one-hot style id conveys no meaningful message unless the mutual information regularization is considered.

the onehot style id and mutual information regularization cannot be torn apart.


In this work, given a dialogue history, we regard as a coherent response an utterance that is thematically correlated and naturally continuing from the previous turns, as well as lexically diverse.

for example, in figure 1 the response "the stove."


We define a learnable prior over alignment variable z i .

we let the multinomial distribution on z i be parameterized in a log-linear fashion using feature function g(z i ).


In particular, we let the multinomial distribution on z i be parameterized in a log-linear fashion using feature function g(z i ).

we consider the following four features: the length, width, and area of the smallest rectangular region enclosing cluster z i , and the number of active elements in mask C z i .


Specifically, we consider the following four features: the length, width, and area of the smallest rectangular region enclosing cluster z i , and the number of active elements in mask C z i .

we let P (z i |X; w) ∝ exp(w T g(z i )).


However, instead of considering attention over the entire image, we restrict attention over image features to the cluster mask determined by the alignment variable, C z i .

we project binary mask C z i from the input image dimensionality (224*224) to the dimensionality of the visual features (14*14).


To measure object hallucination, we propose a new metric, CHAIR (Caption Hallucination Assessment with Image Relevance), which captures image relevance of the generated captions.

we consider both ground truth object annotations (MSCOCO Object segmentation (Lin et al., 2014)) and ground truth sentence annotations (MSCOCO Captions (Chen et al., 2015)).


We also consider the Robust split, introduced in (Lu et al., 2018), which provides a compositional split for MSCOCO.

it is ensured that the object pairs present in the training, validation and test captions do not overlap.


In general, we see that models which perform better on standard evaluation metrics, perform better on CHAIR, though this is not always true.

models which optimize for CIDEr frequently hallucinate more.


As we explore later, this is likely because they have  a better understanding of the image.

methods that incorporate bounding box attention (as opposed to relying on coarse feature maps), consistently have lower hallucination as measured by our CHAIR metric.


The multi-modal hierarchical beam search algorithm comprises K-best word-level beam search and Nbest sentence-caption-level beam search.

we use the corresponding captions instead of images in beam search algorithm for the attTI mechanism which attends images.


Notably, correlation constraints in this paper are defined as keyphrases that should cover all topics in the source document and differ from each other.

we employ a coverage mechanism (Tu et al., 2016) to memorize which parts in the source document have been covered by previous phrases.


The second phase is to rank the candidates with machine learning methods.

some researchers Hulth, 2003;Medelyan et al., 2009;Gollapalli and Caragea, 2014) formulate the keyphrase extraction as a supervised classification problem, while others apply unsupervised approaches (Mihalcea and Tarau, 2004;Grineva et al., 2009;Liu et al., 2009Liu et al., , 2010Zhang et al., 2013;Bougouin et al., 2013Bougouin et al., , 2016 on this task.


In this paper, we propose to leverage the structural information of both documents and multi-sentence summaries to improve the document summarization performance.

we import both structural-compression and structuralcoverage regularization into the summarization process in order to capture the information compression and information coverage properties, which are the two most important structural properties of document summarization.


It outperforms Opinosis and PG-Original by a large margin in terms of R-2 F-scores (5.03/6.03/8.73 for DUC-04 and 5.12/6.40/10.92 for TAC-11).

pG-Original is the original pointer-generator networks with multidocument inputs at test time.


Early work mostly focuses on the keyphrase extraction task, and a two-step strategy is typically designed.

a large pool of candidate phrases are first extracted according to predefined syntactic templates (Mihalcea and Tarau, 2004;Wan and Xiao, 2008;Liu et al., 2009bLiu et al., , 2011 or their estimated importance scores (Hulth, 2003).


We adopt a simple alternating training strategy to switch training between the main task and the auxiliary task.

we first estimate parameters on auxiliary task with D u for one epoch, then train model on the main task with D p (labeled dataset) for T epochs.


There are 6.56 images on average in each article and not every image is closely related to the event of the article.


Moreover, the idea of constructing our MMAE can be easily extended to other modalities.

we both consider the intramodality salience and the intermodality relevance.


In this case, we can construct a function F : P → O that maps the parameter space P onto the output space O.


Its key feature is constructing a pyramid that consists of Elementary Discourse Units (EDUs), clause-like text units introduced in Rhetorical Structure Theory (Mann, William Charles and Thompson, Sandra Annear, 1988), in the source documents.


By following PEAK, we find the optimal alignment by solving the Linear Assignment Problem.

we compute all similarity scores between EDUs in the summary and pyramid and then find the maximal score so that each EDU in the system summary is matched to at most one EDU in the pyramid.


Both pyramid method and responsiveness were used in manual evaluations.

tAC-2008 to 2011 have large numbers of participating systems, from 44 to 48.


The scores achieved were around 0.9 and better than those on DUC-2003 to 2007.

prop(BE) achieved the best scores in some cases.


The reconstructor R reconstructs the original text x from y s .

we seek an output of reconstructor x that is as close to the original text x as possible; hence the loss for training the reconstructor, R loss , is defined as where the reconstruction loss l s (x,x) is the crossentropy loss computed between the reconstructor output sequencex and the source text x, or the negative conditional log-likelihood of source text x given word sequence y s sampled from G(x).


Thus, we model the dependencies between the output variables by learning (globally normalized) node and edge factor functions that jointly optimize a global performance criterion.

we represent the cQA setting as a large undirected graph G=( As shown in Figure 2b, the graph contains six subgraphs: are associated with the three subtasks, while the bipartite subgraphs We associate each node u ∈ V t with an input vector x u , representing the embedding for subtask t, and an output variable y u , representing the class label for subtask t. Similarly, each edge (u, v) ∈ E st is associated with an input feature vector µ(x u , x v ), derived from the node-level features, and an output variable y uv ∈ {1, 2, • • • , L}, representing the state transitions for the pair of nodes.


The effect on accuracy-P-R-F 1 is the same as observed with the pipeline approach but with larger differences.

4 accuracy improves by more than two points absolute, and recall increases, which boosts F 1 to almost 60.


Although the question, starting with when, requires an answer that is expressed as a moment in time, there is only one such expression (i.e., November 2014) in the given text (we refer to the text as the context).

the question has only a single candidate answer.


By contrast, in QAngaroo and the multiple-choice datasets, the performance did not degrade so strongly.

the difference between the scores on the full and k = 1 questions in QAngaroo was 1.8.


Our motivation can be summarized as follows: (i) How many questions are valid in each dataset?

the hard questions may not in fact be hard, but just unsolvable, as indicated in Chen et al.


It first creates step-specific context and query encodings via cell-specific bidirectional LSTMs: Then, we use bidirectional attention (Seo et al., 2017) to emulate a hop of reasoning by focusing on relevant aspects of the context.

we first compute context-to-query attention: 3 are trainable parameters, and is elementwise multiplication.


Commonsense Selection: We conducted manual analysis on a 50 sample subset of the NarrativeQA test set to check the effectiveness of our commonsense selection algorithm.

given a context-query pair, as well as the commonsense selected by our algorithm, we conduct two independent evaluations: (1) was any external commonsense knowledge necessary for answering the question?


The edge-weights around S q are distributed equally among all edges of the same type, and they are weighted such that edges relevant to the question receive a higher weight than those which are not.

we average word vectors to compute a relation vector v(r) from the surface form of the relation, and a question vector v(q) from the words in the question, and use cosine similarity between these as the edge weights.


Inspired by these findings, we propose to exploit deep representations for NMT models.

we investigate two types of strategies to better fuse information across layers, ranging from layer aggregation to multi-layer attention.


However, we note that the proposed approach is generally applicable to any other type of NMT architectures.

the encoder is composed of a stack of L identical layers, each of which has two sublayers.


Residual connection and layer normalization are also employed.

both x and y have residual connections to the output.


We use two-layer attention for illustration, while the approach is applicable to any layers lower than l. transforming information across layers.

for constructing each hidden state in any layer-l, we allow the self-attention model to attend any layers lower than l, instead of just layer l-1: where C l −i is sequential vectors queried from layer l-i using a separate attention model, and AGG(•) is similar to the pre-defined aggregation function to transform k vectors {C l −1 , .


Therefore, we explicitly add a regularization term to encourage the diversities between layers: where λ is a hyper-parameter and is set to 1.0 in this paper.

the regularization term measures the average of the distance between ev-ery two adjacent layers: Here D(H l , H l+1 ) is the averaged cosine-squared distance between the states in layers The cosine-squared distance between two vectors is maximized when two vectors are linearly independent and minimized when two vectors are linearly dependent, which satisfies our goal.


In this work, we propose to better exploit deep representations that are learned by multiple layers for neural machine translation.

the hierarchical aggregation with diversity regularization achieves the best performance by incorporating more depth and sharing across layers and by encouraging layers to capture different information.


We argue other abilities of non-recurrent networks could be responsible for their strong performance.

we hypothesize that the improvements in BLEU are due to CNNs and Transformers being strong semantic feature extractors.


• We empirically show that the number of attention heads in Transformers impacts their ability to capture long-distance dependencies.

many-headed multi-head attention is essential for modeling longdistance phenomena with only self-attention.


A clear understanding of those strengths and weaknesses is important to guide further work.

given the idiosyncratic limitations of recurrent and selfattentional models, combining them is an exciting line of research.


ATR also uses gates to bypass the vanishing gradient problem so as to capture long-range dependencies.

we use the addition and subtraction operations between the weighted history and input to estimate an input and forget gate respectively.


In practice, 4-layer QRNN encoder and decoder are used to gain translation quality that is comparable to that of singlelayer LSTM/GRU NMT.

our onelayer model achieves significantly higher performance than a 10-layer SRU system.


It is clear that both gates are highly non-linearly correlated, and that there are regions where σ(x + y) is equal to, greater or smaller than σ(x − y).

by adapting the distribution of input  and forget gates, the twin-gated mechanism has the potential to automatically seek suitable regions in Figure 2 to control its preference between the new and past information.


We also provide results of several existing systems that are trained with comparable experimental settings to ours.

our single model yields a detokenized BLEU score of 21.99.


In comparison to several advanced deep NMT models, such as the Google NMT (8 layers, 24.61 tokenized BLEU)  and the LAU-connected NMT (4 layers, 23.80 tokenized BLEU) (Wang et al., 2017a), the performance of our shallow model (23.31) is competitive.

when replacing LSTM in the Google NMT with our ATR model, the GNMT+ATR system achieves a BLEU score of 24.16, merely 0.45 BLEU points lower.


In addition, our model can also detect non-consecutive longdistance dependencies.

the prediction of "Parteien" relies heavily on the token "unglücklichen", which actually entails an amod linguistic dependency relationship.


To apply this idea to NMT, however, is much more involved.

in the process of beam search, we cluster similar target hidden states to construct equivalence classes, the three dimensions of which are target words in the target vocabulary, part translations retained in the beam search and different combinations of similar target hidden states, respectively.


The system is an improved version of attentionbased NMT system named RNNsearch (Bahdanau et al., 2015) where the decoder employs a conditional GRU layer with attention, consisting of two GRUs and an attention module for each step 5 .

equation (6) is replaced with the following two equations: Besides, for the calculation of relevance in equation (4), s j−1 is replaced withs j−1 .


Regarding lexical choice, the two systems differ not only in the number of errors, but in the nature of those errors.

the BPE model had more trouble handling German compound nouns.


It is expected that the generalized templates can constrain the search space in generation and the model achieves higher fluency by loss of expressive power.

we find that only 0.48%, 1.01%, and 1.20% of the unigrams, bigrams, and trigrams are unique in the stories generated by the GE-Seq2Seq model, while the percentages are 3.16%, 15.33%, and 29.67% in the stories generated by our proposed model.


To overcome the above limitation, in this work, we develop a novel deep attentive sentence ordering network (referred as ATTOrderNet) by inte- grating self-attention mechanism (Vaswani et al., 2017) with LSTMs to learn a relatively reliable paragraph representation for subsequent sentence ordering.

the bidirectional LSTM is first adopted as a sentence encoder to map the input sentences to the corresponding distributed vectors, and then a self-attention based paragraph encoder is introduced to capture structural relationships across sentences and to obtain a hierarchical context representation of the entire set of sentences.


The sentence ordering task aims to order a set of sentences as a coherent text.

a set of n sentences with the order The goal is to find the correct order o * for them, , with which the whole sentences have the highest coherence probability: where o indicates any order of these sentences and ψ denotes the set of all possible orders.


In this paper, we use self-attention mechanism for paragraph encoder instead.

we employ this mechanism without encoding any positional information of the sentences.


For a sentence, we first apply word embedding matrix to translate the raw words in the sentence into distributional representations, and then adopt bidirectional LSTMs to learn a sentence-level representation for summarizing its high level semantic concepts.

assume that a sentence , these words are transformed to dense vectors through a word embedding matrix W e : is then fed into bidirectional LSTMs sequentially to compute a semantic representation of the sentence.


Here, we adopt bidirectional LSTMs (Bi-LSTMs) to take full advantages of additional backward information and enhance the memory capability.

the Bi-LSTMs contain the forward LSTMs which process the sentence s o i from w 1 to w n w and backward LSTMs which read s o i in the reversed direction: where h t denotes the representation of position t by concatenating the forward hidden state − → h t and backward hidden state ← − h t together.


The improvement is regardless of data sizes.

for smaller datasets such as Accident and Earthquake datasets, ATTOrderNet outperforms the previous best baseline methods by 6% and 7% tau score respectively.


Sentences in darker shades show more attention weight which reflects stronger links they have with the example sentence.

figure 2 shows the attention distribution for the first sentence in the original document.


Update states For each parameter k, we blocksample a whole given tree.

we implement a Bayesian version of Felsenstein's tree-pruning algorithm, which is akin to the forward filtering-backward sampling algorithm for Bayesian hidden Markov models.


Step 3 We analyze the TRMs by simulating language evolution.

given the latent representation of language l, we stochastically generate its descendant l after some time span t. we draw z l ,k according to the transition probability of Eq.


The approach we adopt in the present study is to let a simpler model explain the model's complex behavior.

we used linear regression with L1 regularization (i.e., lasso).


For each word order i, the target variable was θ l ,f (n,i) of each sample of simulated language l whereas explanatory variables were the items of the probability vector θ l , * .

we aimed at finding out features that were characteristic of the specified word order.


of 36), although some pairs do have high correlations.

active interactions tend to be both concurrent (0.81) and spatially near (0.71), and spatially near interactions tend to be concurrent (0.89).


We use a combination of features extracted directly from the conversation turn, sentiment lexica and context.

we extract (a) the first word in the conversation turn, (b) bag-of-words features (binary flags and tf-idf scores), and (c) the root verb, and flags indicating the presence of exclamation, question marks and negation cues from (Morante and Daelemans, 2012) (other).


Model We experiment with a multi-layer perceptron (MLP) applied to sentence-level classification tasks.

input to the MLP is a sentence or short text, represented as a fixed-size vector embedding.


This means that if hyperparameter search is done carefully, the choice of activation function is less important for these sentence classification tasks.

the (binary) tasks MR and SUBJ appear robust against the choice of activation function, with the difference between the best and worst function being always less than 1pp, in all settings.


The model then applies 1Dconvolution on top of these embeddings.

a filter w of size h takes h successive embeddings x i:i+h−1 , performs a scalar product and obtains a feature c i : Here, f is the activation function and b is a bias term.


, |x|} |y| , which has an interpretation as the set of all (potentially non-monotonic) alignments from x to y with the restriction that each output symbol y i aligns to exactly one symbol in x ∈ Σ * x .

a is the set of all manyto-one alignments between x and y where many may be as few as zero.


Second, IBM Model 1 does not condition on previously generated words on the output side.


What is crucial about this RNN, like the α j (i), is that it only summarizes the characters decoded so far independently of the previous attention weights.

the attention weights at time step i will have no influence from the attention weights at previous time steps, shown in Fig.


To solve these tasks, the proposed models read all the text available.

models read every token or word of the text from beginning to end.


Inspired by these studies, we propose more flexible strategies for modeling localness for self-attention networks in this work.

we cast the localness modeling as a learnable Gaussian bias, in which a central position (i.e.


The mechanism allows to dynamically select related representations as needed.

it is very useful for generation models such as machine translation (Bahdanau et al., 2015;Luong et al., 2015; and image captioning (Xu et al., 2015).


, x I }, each hidden state in the l-th layer is constructed by attending to the states in the (l − 1)-th layer.

1 the (l − 1)-th layer H l−1 ∈ R I×d is first transformed into the queries Q ∈ R I×d , the keys K ∈ R I×d , and the values V ∈ R I×d with three separate weight matrices.


The distribution of attention is then regularized with the learned Gaussian bias to produce the final distribution, which pays more attention to the local context around the word "talk".

a Gaussian bias G is placed to mask the logit similarity energy in Equation 2, namely: The first term is the original dot product selfattention model.


We solve such corner cases by assigning the character-pixel to the box that has the closest box center.

the chargrid representation is constructed as follows: for each character c k at location b k , the area covered by that character is filled with some constant index value E(c k ).


Considerable efforts were spent to ensure that the labels are correct.

each invoice was analyzed by three annotators plus a reviewer.


The computation resembles other recurrent networks such as LSTM, GRU and RAN .

a forget gate f t controls the information flow (Equation 1) and the state vector c t is determined by adaptively averaging the previous state c t 1 and the current observation Wx t according to f t (Equation 2).


The second step computes the remaining pointwise operations.

we compile all point-wise operations into a single fused CUDA kernel and parallelize the computation across each dimension of the hidden state.


The implementation of Word2Vec 1 from (Mikolov et al., 2013) is employed.

we employ CBOW with the dimension set to 300, window size to 10, minimum count to 5, and a subsampling threshold of 10 −3 .


pACRR-firstk is employed where the first 1, 000 terms are used to compute the similarity matrices, and the original configuration from  is used.


Akin to Xiong et al., 2017;, the NIRM baselines and the proposed NPRF are employed to re-rank the search results from BM25.

the top-10 documents from the unsupervised baseline are used as the pseudo relevance feedback documents D q as input for NPRF, where each d q ∈ D q is represented by its top-20 terms with the highest tfidf weights.


Figure 2 illustrates the sensitivity of NPRF relative to two parameters: the number of feedback documents m within D q and the number of terms k that are used to summarize each d q ∈ D q .

figure 2 shows the performance of NPRf ds as the number of feedback documents m varies (top), and as the number of top terms k varies (bottom).


Next, word representations are passed into a stacked recurrent encoder layer.

we use Bidirectional LSTM encoders at this layer.


We also investigate how vMF VAE performs in a different setting, one less plagued by the KL collapse issue.

the Neural Variational Document Model (NVDM), proposed by Miao et al.


To verify this, we investigate whether the BoW representation and the learned latent code can be reconstructed from each other.

given a sentence x we can compute BoW as defined above and µ = enc(x), the latent encoding of x as represented by the mean vector output by the encoder.


We propose a mean-max attention autoencoder (mean-max AAE) to model sentence representations.

an encoder performs the MultiHead self-attention on an input sentence, and then the combined mean-max pooling operation is employed to produce the latent representation of the sentence.


Compared the mean RAE with LSTM-SDAE, both of which use the RNN-RNN encoderdecoder network to reconstruct the input sequence, our mean RAE consistently obtains better performance than LSTM-SDAE across all considered tasks.

it yields a performance gain of 10.4 on the TREC dataset and 29 on the STS14 dataset.


In this paper, we build on this recent innovation D2KE (Wu et al., 2018a), and present the Word Mover's Embedding (WME), an unsupervised generic framework that learns continuous vector representations for text of variable lengths such as a sentence, paragraph, or document.

we propose a new approach to first construct a positive-definite Word Mover's Kernel via an infinite-dimensional feature map given by the Word Mover's distance (WMD) to random documents from a given distribution.


It is desirable to set these to a small number, in part because this length is indicative of the number of hidden global topics, and we expect the number of such global topics to be small.

these global topics will allow short random documents to align with the documents to obtain "topic-based" discriminatory features.


We defer the proof to the appendix A. R (x, y) be the difference between the exact kernel (2) and the random approximation (3) with R samples, we have uniform convergence where d is the dimension of word embedding and L is a bound on the document length.

to guarantee | R (x, y)|  ✏ with probability at least 1 , it suffices to have We conduct an extensive set of experiments to demonstrate the effectiveness and efficiency of the proposed method.


Our clustering approach is part of a larger media monitoring project to solve the problem of monitoring massive text and TV/Radio streams (speechto-text).

media monitors write intelligence reports about the most relevant events, and being able to search, visualize and explore news clusters assists in gathering more insight about a particular story.


In this section, we give more details about the way we construct the document representations in the monolingual and crosslingual spaces.

we introduce the definition of the similarity functions Γ 0 and Γ 1 that were referred in §3.


Additionally, we also experimented with computing the monolingual clusters with the same embeddings as used in the crosslingual clustering phase, which yielded poor results.

this system achieved F 1 score of 74.8 for English, which is below the bag-of-words baseline presented in Table 2.


This proves our capsule networks are effective for text.

our capsule network outperforms conventional CNNs like DCNN, CNN-MC and VD-CNN with a large margin (by average 1.1%, 0.7% and 1.0% respectively), which shows the advantages of capsule network over conventional CNNs for clustering features and leveraging the position information.


In the framework of Pearl (1995), we consider a counterfactual variable of interest: what would have been the cancer incidence among smokers if smoking had been randomized?

we consider a causal effect as the counterfactual outcome of a hypothetical intervention on some treatment variable.


F (A) contains matrices that shifted each row of A by an arbitrary real number.

if we take a matrix from F (A) and apply the softmax function to each of its rows, we obtain a matrix that consists of true probability distributions.


In addition, since we can imagine many kinds of contexts, it is difficult to assume a basis that represents a conditional probability distribution for any contexts.


DOC computes probability distributions from the middle layers in addition to the final layer.

dOC directly connects the middle layers to the output.


Therefore, an RNN language model with DOC can output a distribution matrix whose rank is identical to one of the true distributions.

,Ã is a better approximation of A than the output of a standard RNN language model.


This table demonstrates that the coefficient of variation decreases with growth in λ β .

the model trained with a large λ β assigns balanced weights to each probability distribution.


Table 5 shows the ranks of matrices containing log probability distributions from each method.

table 5 describesÃ in Equation 9 for each method.


We trained AWD-LSTM and AWD-LSTM-MoS by setting the non-monotone interval to 60, as with AWD-LSTM-DOC.

we used hyperparameters identical to the original ones to train AWD-LSTM and AWD-LSTM-MoS, except for the non-monotone interval.


The ensemble technique provided further improvement, as described in pre- 8 We exclude models that use the statistics of the test data (Grave et al., 2017;Krause et al., 2017) from these tables because we regard neural language models as the basis of NLP applications and consider it unreasonable to know correct outputs during applications, e.g., machine translation.

we focus on neural language models as the foundation of applications although we can combine the method using the statistics of test data with our AWD-LSTM-DOC.


Moreover, EncDec outperformed the state-of-the-art method (Zhou et al., 2017) on the ROUGE-2 and ROUGE-L F1 scores.


(1) In disfluent structures, the repair (e.g., to Denver) frequently seems to be a "rough copy" of the reparandum (e.g., to Boston).

they incorporate the same or very similar words in roughly the same word order.


Our baseline CNN outperforms both the LSTM and the BLSTM, while the ACNN model clearly outperforms the baseline CNN, with a further 5% increase in f-score.

the ACNN noticably improves recall without degrading precision.


This architecture gives strong results on wordlevel language modeling while reducing the number of parameters significantly.

pRU improves the perplexity of a recent state-of-the-art language model Merity et al.


Standard dropout: PRUs achieve either the same or better performance than LSTMs.

the performance of PRUs improves with the increasing value of g. At g = 4, PRUs out-perform LSTMs by about 4 points on the PTB dataset and by about 3 points on the WT-2 dataset.


This process, however, has not been explicitly taken into consideration by conventional language models.

although in most cases words are atomic language units, words are not necessarily atomic semantic units for language modeling.


The above limitation motivates us to develop a supervised topic model which can jointly model documents and labels.

we propose a Siamese Labeled Topic Model (SLTM) to exploit the information of documents and labels based on the Siamese network (Bromley et al., 1993;Hu et al., 2014;Wang and Zhang, 2017), where weight matrices in SLTM represent conditional distributions.


For instance, the word embedding of v j is generated from the topic-word distribution W 2 as: The generated word embeddings can be used for specific applications, such as label prediction.

we firstly represent a new document d n by its document embeddings e(d n ), where e(d n ) is the sum of word embeddings of all words in d n .


Finally, we adopt two hybrid methods by combining LDA and supervised learning algorithms as baselines.

a softmax classifier and a liner regression (LR) are used to predict categorical and real-valued labels for documents, respectively.


Although L-LDA and Dep-LDA can identify label-specific topics on ISEAR, these models' one-to-one mapping of labels and topics makes them unsuitable in this evaluation.

l-lDA and Dep-lDA constraint each topic to words in certain documents with the same label, which renders their coherence scores being estimated by a subset of the corpus only.


Both Gibbs Sampling and VEM are not directly applicable to new variations of the topic model.

the inference algorithm requires re-deriving for any minor changes to the model.


To address these issues, we propose a design that models representations of comments jointly along observed reply links.

we attach a vector of latent binary variables to each comment in a collection of social data, which in turn connect to each other according to the observed reply-link structure of the dataset.


We now present an overview of our model.

it will take the probabilistic form of an undirected graphical model whose architecture mirrors the tree structure of the threads in our data.


Inference in this model class in intractable, so as has been done in previous work on topic modeling (Ng and Jordan, 2003) we rely on variational methods to approximate the gradients needed during training as well as the posteriors over the topic bit vectors.

we will need the gradients of the normalizer and the sum of the energy function over the hidden variables which we refer to as the marginal energy.


In contrast, our proposed model and mean-field procedure can be scaled efficiently to large data because they are amenable to GPU implementation.

the described inference procedure can be viewed as the output of a neural network.


To assess the relative importance of the specific encoder model architecture used, we conduct experiments in which we fine-tune standard document representation models via triplet-based training.

we consider a single-layer MLP with BoW inputs, and a Neural Variational Document Model (NVDM) (Miao et al., 2016).


We first evaluate embeddings quantitatively with respect to retrieval performance.

we assess whether the induced representations afford improved retrieval of abstracts relevant to a particular systematic review (Cohen et al., 2006;Wallace et al., 2010).


We now turn from the specialized domain of biomedical abstracts to more general applications.

we consider learning disentangled representations of beer, hotel and restaurant re-   views.


5 In Table 7, we evaluate cross AUC performance for beer by first 'decorrelating' the aspects.

for each cell (k, k ) in the table, we first retrieve the subset of reviews in which the sentiment w.r.t.


We hypothesize that different source domains are aligned to different sub-spaces of the target domain.

in this paper, we model the domain relationship with a mixtureof-experts (MoE) approach (Jacobs et al., 1991b).


All supervised objectives are optimized jointly with the encoder being shared, resulting in the following multi-task learning objective: Adversary-augmented MoE We use MMD (Gretton et al., 2012) as the adversary to minimize the divergence between the marginal distribution of target domain and source domains.

at each training epoch, given the K batches {x S 1 , x S 2 , ..., x S K } from all the source domains, we sample a batch (unlabeled) x T from our target domain, and minimize the MMD: where measures the discrepancy between D S and D T based on Reproducing Kernel Hilbert Space (RKHS).


In addition, as the models are trained with more data, the odds of answering easy examples correctly increases at a faster rate than the odds of answering a difficult example correctly.

performance starts to look more human, in the sense that humans learn easy items faster than they learn hard items.


2 To gain further insights into our results, we examine the attention weights computed by FANs during verb-number prediction (supervised objective).

for each attention head at each layer of the FAN, we compute the percentage of times the subject is the most attended word among all words in the history.


To this end, we propose a label-wise attention mechanism.

we apply the same type of attention, but repeating it |L| (number of labels) times, where each attention module is reserved for a specific label l: This section describes the main experiment w.r.t the performance of our proposed attention mechanism, in comparison with existing emoji prediction systems.


Finally, as a final (and perhaps thoughtprovoking) finding, the highest attention weights associated to the emoji were given to the words game, boys and football, in that order.

the model relies more on the word boys than on the actual description of the emoji.


The generation process is guided by a CFG over target trees, which is constructed by taking all production rules extracted from the trees of all sentences in the training corpus.

a rule RNN first generates the top of the tree structure, and continues until a preterminal is reached.


In addition to constituency parse trees, TrDec can also utilize dependency parse trees via a simple procedure that converts a dependency tree into a constituency tree.

this procedure creates a parent node with null tag for each word, and then attaches each word to the parent node of its head word while preserving the word order.


and the predict precision of n-grams changes intõ Finally, the probabilistic BLEU (P-BLEU) is defined as Probabilistic GLEU (P-GLEU) can be defined in a similar way.

we denote the probabilistic precision of n-grams as P-Pn.


Finally, it introduces the notion of pronoun equivalence, assigning partial credit to pronoun translations that differ from the reference translation in specific ways deemed to be acceptable.

it considers six possible cases when comparing the translation of a pronoun in MT output and the reference.


The distribution of the disagreements over APT cases is very skewed and ranges from 8% for case 1 to 32% for case 2 and 49% for case 3.

aPT identifies correct pronoun translations with good precision, but relatively low recall.


In this work, we propose a novel method based on representation matching to transfer the knowledge learned from the WSD data to the DL models for ED.

two separate deep learning models are employed to model the context for WSD and ED.


We use the same data split with the prior work (Chen et al., 2015;Nguyen and Grishman, 2015b).

40 newswire documents are used for testing, 30 other documents are reserved for validation, and the 529 remaining documents form the training data.


Our method differs from them in that it is proposed for specific target refinement.

the refined vectors by general purpose refinement method can be further refined to extract a specific knowledge by our method.


We evaluate our models on the Ontonotes 5.0 (Weischedel et al., 2012) benchmark dataset and the WinoBias dataset (Zhao et al., 2018).

6 the WinoBias dataset is composed of pro-stereotype (PRO) and antistereotype (ANTI) subsets.


To perform fuzzy clustering, we follow AI-KU and associate each instance with k representatives, but deviate in the way the representatives are generated.

each representative is a set of size 2 , containing samples from the forward distribution and samples from the backward distribution.


For example, when presented with the query The human heart not only makes heart sounds and , the forward LM predicted in its top twenty suggestions the word muscle, followed by a next-word prediction of movements.

the symmetry extends beyond "sounds" to the phrase "heart sounds" which could be substitutes by "muscle movements".


In order to better understand how different modifications of the soft label loss affected model performance on transfer tasks, we conducted some additional experiments.

we investigated the effects of (a) varying the number N of word neighbors used to compute the loss function, and (b) removing the normalization factor by getting rid of the denominator in Eq.


a layer h X which is shared between the main task and the auxiliary task is split into two subspaces: a shared subspace h X S and a private subspace h X P .


We notice interesting patterns, visible in Figure 2.

pSN appears markedly better at sentences with named entities (ART, pER, GEO, ORG) and temporal entities (DOM) than both ST and the FSN.


The CI indicate the difference in % NomAnaph ClauseAnaph Pleon  395  2  56  NomAnaph  53  11  25  ClauseAnaph  93  3 176 Pleon example, measures such as first fixation duration were not included in any of the models, while revisits as late as the third run (the third time the eyes pass over the region of interest) occurred in these regions and provided a strong signal.

useful features were the durations of the second and last fixations, as well as the information about the run (pass) during which they occur.


We show that properly training and tuning a relatively simple architecture with a minimal feature set and greedy search achieves state-of-the-art accuracies and beats more complex neural-network architectures.

the model itself is a straightforward stacked bidirectional LSTM (Figure 1) with just two input features at each position (character and bigram).


Therefore, K denotes a trade-off between the accuracy of short words and long words.

we investigate two major segmentation problems that might affect the accuracy of word segmentation performance, namely, insertion errors and deletion errors.


In this paper, we present an investigation into these questions.

we consider two simple but competitive neural network topologies in order to investigate the effect of training example order from these datasets on performance.


(2017) explored the effect of minibatching on the learning of Neural Machine Translation models, carrying out their experiments on two datasets (two language-pairs).

they studied the strategies of (1) sorting by length of the source sentence, (2) target sentence, or (3) both, among other things.


In this paper, we take a step toward explaining such deep learning based models through a case study on a popular neural model for NLI.

we propose to interpret the intermediate layers of NLI models by visualizing the saliency of attention and LSTM gating signals.


In this work, we are primarily interested in the internal workings of the NLI model.

we focus on the attention and the gating signals of LSTM readers, and how they contribute to the decisions of the model.


We propose to examine the saliency of attention.

given a premise-hypothesis pair and the model's decision y, we consider the similarity between a pair of premise and hypothesis hidden states e ij as a variable.


Interestingly, the saliencies are clearly different across the examples, each highlighting different parts of the alignment.

for h1, we see the alignment between "is playing" and "taking a nap" and the alignment of "in a garden" to have the most prominent saliency toward the decision of Contradiction.


LSTM gating signals determine the flow of information.

they indicate how LSTM reads the word sequences and how the information from different parts is captured and combined.


Comparing the behavior of the input LSTM and the inference LSTM, we observe interesting shifts of focus.

we see that the inference LSTM tends to see much more concentrated saliency over key parts of the sentence, whereas the input LSTM sees more spread of saliency.


We hypothesize that by leveraging syntactic structure while training neural SRL models, we may achieve robust performance, especially for low resource scenarios.

we propose to leverage syntactic parse trees as hard constraints for the SRL task i.e., we explicitly enforce that the predicted argument spans of the SRL network must agree with the spans implied by the syntactic parse of the sentence via scoring function in the training objective.


In addition to the log-likelihood objective, we also include syntactic inconsistency loss (defined in Section 2.3) which quantifies the hard constraint (spans implied by syntactic parse) violations in our training objective.

while training our model, we enforce the outputs of our system to agree with the spans implied by the syntactic parse of the sentence as much as possible.


(2) where f (x, y) incorporates those constraints and report that SRL constraints do not show significant improvements over the ensemble model.

by using syntactic constraints, (He et al., 2017) achieves up to +2 F1 score on CoNLL-2005 dataset via A* decoding.


Due to their diversity in content and design, web pages provide a rich environment for natural language grounding tasks.

we consider the task of mapping natural language commands to web page elements (e.g., links, buttons, and form inputs), as illustrated in Figure 1.


As such, we first consider a simple retrieval-based model that uses the command as a search query to retrieve the most relevant element based on its TF-IDF score.

each element is represented as a bag-of-tokens computed by (1) tokenizing and stemming its text content, and (2) tokenizing the attributes (id, class, placeholder, label, tooltip, aria-text, name, src, href) at punctuation marks and camel-case boundaries.


Although these methods have reported state-of-the-art results for languages with rich resources, no significant improvement has been announced for low-resource languages.

feature-engineered statistical models still perform better than these neural models for low-resource languages.


In this paper, we argue that amid this rapid progress on empirical benchmarks, crucial steps are sometimes skipped.

we demonstrate that the level of difficulty for several of these tasks is poorly characterized.


This self-attention is performed for each token for each head, and the results of the H self-attentions are concatenated to form the final self-attended representation for each token.

consider the matrix S (j 1) of T token representations at layer j 1.


Instead, we propose training the self-attention to attend to specific tokens corresponding to the syntactic structure of the sentence as a mechanism for passing linguistic knowledge to later layers.

we replace one attention head with the deep bi-affine model of Dozat and Manning (2017), trained to predict syntactic dependencies.


Since POS is a strong predictor of predicates 4 and the complexity of training a multi-task model increases with the number of tasks, we combine POS tagging and predicate detection into a joint label space: For each POS tag TAG which is observed co-occurring with a predicate, we add a label of the form TAG:PREDICATE.

we feed the representation s (r) t from a layer r preceding the syntacticallyinformed layer p to a linear classifier to produce per-class scores r t for token t. We compute locally-normalized probabilities using the softmax function: is a label in the joint space.


This is confirmed on low-resource languages.

on ro → en, our unsupervised PB-SMT model obtains a BLEU score of 23.9, outperforming Gu et al.


assumption upon which supervised machine learning typically relies.

the training and test data no longer reflect the same underlying data distribution.


Once we have acquired datasets from each acquisition model D A , we evaluate the performance of each possible successor model when trained on D A .

we train each successor model S on the acquired data D t A for all t in the range [0, T ], evaluating its performance on a held-out test set (distinct from U).


To compare the learning curves, we select incremental points along the x-axis and report the performance at these points.

we report results with training sets containing 10% and 20% of the training pool.


random sampling, but our results further suggest that this comes at an additional cost: the acquired dataset may not generalize well to new learners.

models trained on foreign actively acquired datasets tend to underperform those trained on i.i.d.


To investigate this, we measure pairwise distances between instances in the learned feature space after training.

for each test instance we calculate its cosine similarity to all other test instances, inducing a ranking.


As reported in Table 4, despite the aforementioned differences in predictive performance, the learned representations seem to be similar.

sampling under foreign acquisition models does not lead to notably different representations.


Data annotation is a key bottleneck in many data driven algorithms.

deep learning models, which became a prominent tool in many data driven tasks in recent years, require large datasets to work well.


Furthermore, due to the stochastic nature of the estimation, the task A labels need not be fully accurate, allowing us to make use of noisy predictions which are assigned by an automatic classifier (Sections 3.1 and 4).

given a medium-sized sentiment corpus with sentencelevel labels, and a large collection of un-annotated text from the same distribution, we can train an accurate aspect-level sentiment classifier.


Consider two classification tasks over a shared input space, a source task s from X to Y s and a target task t from X to Y t , which are related through a conditional distribution P (y t = i|y s = j).

a labeling decision for task s induces an expected label distribution over the task t. For a set of datapoints x 1 , ..., x n that share a source label y s , we expect to see a target label distribution of P (y t |y s ) =p y s .


At each training step we compute the loss and update the gradient with respect to a different random subset.

in each training step we sample a random pair (U j ,p j ), sample a random subset U of U j of size k, and compute the local XR loss of set U : wherep u is computed by summing over the elements of U rather than of U j in equations (1-2).


The self similarity of w in layer is (1) where cos denotes the cosine similarity.

the self-similarity of a word w in layer is the average cosine similarity between its contextualized representations across its n unique contexts.


In all three models, the higher the layer, the lower the self-similarity is on average.

the higher the layer, the more contextspecific the contextualized representations.


We arrive at a similar coefficient (but with different centering) as a special case of CKA, which in the general case makes no parametric assumptions about disbtributions whatsoever.

our version is suitable for comparing sets containing just one word vector, whereas the method of Nikolentzos et al.


Suppose we have a word embeddings matrix W ∈ R N ×D , where N is the number of words in the vocabulary and D is the embedding dimension (usually 300).

each row w (i) of W is a D-dimensional word vector.


Finally, very similar conclusions were shown to hold for sentence representations obtained by word vector averaging, also referred to as meanpooling.

averaged fastText vectors compared with rank correlation coefficients already show impressive results on standard STS tasks, rivaling much more sophisticated systems.


The sense vectors that maintain high performances are SensEmbed, AutoExtended and Chen2014.

chen2014 has high performances with all the word embedding combinations.


This is not surprising since they are able to produce contextualised word representations, performing, in fact, a preliminary disambiguation of the words.


(2016) conducted a series of experiments to study the effect that the variation of sense distributions in the training set has on the performances of It makes sense (Zhong and Ng, 2010).

they increased the volume of training examples (V) by enriching SemCor with senses inferred from BabelNet; increased the number of least frequent senses (LFS) (V+LFS); and overfitted the model constructing a training set proportional to the correct sense distribution of the test set (GOLD,GOLD+LFS).


This is particularly useful since it can serve as a test bed for new word and sense embeddings.

it will be interesting to test new sense embedding models based on contextual embeddings.


The framework consists of three modules: a multi-domain Dialog State Tracker (DST) at the dialog act level, a dialog policy module for deciding the next dialog act, and a reward estimator for policy evaluation.

given a set of collected human dialog sessions D = {τ 1 , τ 2 , .


In this paper, we will analyze the side effect of using too many context utterances and propose a multi-hop selector network (MSN) to alleviate the problem.

mSN firstly utilizes a multi-hop selector to select the relevant utterances as context.


In order to make full use of word and utterance selectors, we design a combined strategy to fuse two scores.

we use the weighted sum of two scores for selection: where α is a hyper-parameter and s (1) is the final score that hop1 selector produces.


Then we compute matching score g(U i , r i ) based on the matching features.

we use the final state of GRU output h L as features and apply a single-layer perceptron to obtain score: where W and b are learnt parameters, σ(•) is sigmoid activation function.