Activation, in its most common usage, refers to the way nodes in semantic networks become easier to retrieve after adjacent nodes have been activated, typically through a presentation (Collins and Loftus, 1975).

activation makes a direct claim that more highly activated words require less time to retrieve, and vice versa (Anderson, 1983).


More relevantly, Dell and O'Seaghdha (1992) examine the time course of lexical access in language production.

they use series of three words and EEG data to estimate lexical retrieval time.


Further, their model of the effects of word-properties relied on primarily qualitative attributes, such as semantic or phonetic relatedness.

they find additional evidence for lemma and phonological retrieval taking place in separate stages, based on inhibition and facilitation effects.


After we simulate the memory retrievals for each word, we can compare this information to the actual empirical timing data in the corpus.

we rely on Anderson's (1983) original account of memory.


However, we discovered a robust effect in the opposite direction: higher delays imply shorter expected retrieval times, and shorter delays imply longer expected retrieval times.

when people are expected to need the longest to retrieve words, they actually speak more quickly, and vice versa.


This allows us to observe how certain processes involved in fluent language production overlap.

the data support a model according to which lexical retrievals can happen quite early.


Motivated by the reasoning network for document comprehension (Shen et al., 2016), we propose multi-turn reasoning neural networks that generate the proper response (or, "answer") by attention-based reasoning from current conversation session ("document") and current query (identical to "question" in document comprehension) from the user.

our networks utilize conversation context and explicitly separate speakers' interventions into sentence-level and conversation-level memories.


Note that, traditional single-turn conversation models or NMT models are a special case of this model by simply setting T d to be 2.

the generation of the next sentence is sessioninsensitive and is only determined by the former single sentence.


However, we can add constraints to determine whose responses to be generated, either one speaker or both of them.

when i takes odd integers of 1, 3, 5 and so on, we are modeling the first speaker.


In this situation, only the previously predicted words are known and the probability of current sequence is approximated by dropping the posterior context.

we construct a sequence generation LM using sequence-to-sequence neural network f .


We refer to the set of annotations from the same speaker as one memory.

the sentences for which the probabilities are being predicted as M r (response memory, specially corresponds to the chatbot's side) and the question set as M q (query memory, specially corresponds to the user's side).


One configuration in MULTI and REASON is that, we respectively use the reference contexts (instead of former automatically generated sentences) to generate current sentence.

when generating s i , we use the golden contextual sentences of from s 1 to s i−1 .


Such user teaching precisely addresses Run π θ (a|s) with user to collect new dialogue samples D π Ask user to correct the mistakes in the tracked user's goal for each dialogue turn in D π Add the newly labeled dialogue samples to the existing corpora: Train model end-to-end on D and obtain an updated policy π θ (a|s) eq 4 7: end for 8: for learning iteration k = 1 : N do 9: Run π θ (a|s) with user for a new dialogue 10: Collect user feedback as reward r 11: Update model end-to-end and obtain an updated policy π θ (a|s) eq 5 12: end for the limitations of the currently learned dialogue model, as these newly collected dialogue samples are driven by the agent's own policy.

in this study we let an expert user to correct the mistake made by the agent in tracking the user's goal at the end of each dialogue turn.


On this basis, the MaxBLEU metric is defined as: where r k• denotes all the references in the kth group.

we begin by calculating all the multi-BLEU scores between each hypothesis and grouped references, and pick the score for the sentence with the highest BLEU as the score for this set of hypothesis, so that we make an alignment between generated hypothesis h to the group-aware references r. For simplicity, one response can only be aligned to one group reference, and multi-group references are not considered in this work.


In this paper, we introduce a new variant of matching networks (Vinyals et al., 2016;Snell et al., 2017) to address the EMR coding problem.

we combine the non-parametric idea of k-NN and matching networks with traditional neural network text classification methods to handle both frequent and infrequent labels encountered in EMR coding.


Before we present our results, we discuss an essential distinction between the MIMIC II and MIMIC III datasets.

we are interested in the differences concerning label imbalance.


Various aspects of the embedding spaces produced by these algorithms have been previously studied.

the effect of parameter choices has a large impact on how all three of these algorithms behave (Levy et al., 2015).


This regularization mitigates the effects of multicollinearity (when two features are highly correlated).

given N ground-truth data points with M extracted features per data point, let x n ∈ R 1×M be the features for sample n and let y ∈ R 1×N be the set of labels.


Given this model, we create training instances by observing the stability of a large number of words across various combinations of two embedding spaces.

given a word W and two embedding spaces A and B, we encode properties of the word W , as well as properties of the datasets and the algorithms used to train the embedding spaces A and B.


We repeat this process for more than 2,500 words, several datasets, and three embedding algorithms.

we consider all the words present in all seven of the data domains we are using (see Section 4.3), 2,521 words in total.


Deep learning is leveraged to mine evidences from large scale textual data, which contain valuable market information.

distributed word similarities trained over large scale raw texts are taken as a basis of relevance measuring, and deep reinforcement learning is leveraged to learn a strategy of topic expansion, given a small amount of manually labeled data from financial analysts.


We leverage a small amount of manually labeled data, downloaded from financial analysis websites, for guiding evidence mining.

we take a reinforcement learning method, which regards the evidence mining process as a decision process.


we discover PetroChina as a concept stock of 'petroleum' from the document 'PetroChina acquires Keppel's entire stake in Singapore Petroleum') and the task is to automatically discover these relations and select a subset of stocks as c's concept stocks based on the data    Motivated by the success of embedding-based models Pennington et al., 2014) in capturing semantic regularities, we use embeddings to represent concepts, stocks and documents.

we adopt Chinese word segmentation (Yang et al., 2017) to obtain words from documents.


Doc2Vec (Le and Mikolov, 2014) is then used on the documents of each data source S i to obtain a local word embedding matrix E i and a local document embedding matrix F i , where each column of E i (F i ) corresponds to a word (document) vector representation of S i .

we use embeddings, E i c and E i o as the local concept representation of c and the local stock representation of o in data source S i , respectively.


It improves speakerbased emotion modeling by using memory networks which are efficient in capturing long-term dependencies and summarizing task-specific details using attention models (Weston et al., 2014;Graves et al., 2014;Young et al., 2017).

the memory cells of CMN are continuous vectors that store the context information found in the utterance histories.


To get our sentence representation, we use a simple CNN with one convolutional layer followed by max-pooling (Kim, 2014;Poria et al., 2016).

the convolution layer consists filters of sizes 3, 4 and 5 with 50 feature maps each.


In a simple 2D Cartesian coordinate system (x, y), the slopes of a linear function remain consistent when x changes.

the importance of preceding utterances linearly declines as the distance between the previous utterance and the target utterance becomes larger.


To further investigate what the universal time-decay attention learns, we inspect the learned weights w i and find that the convex attention function almost dominates the whole function.

our model automatically learns that the convex timedecay attention is more suitable for modeling contexts from the dialogue data than the other two types.


Therefore, we focus on whether our time-aware attention mechanisms can compensate the poor attention learned from the content-aware model.

we are not going to verify whether our time-aware attention mechanisms could collaborate with the contentaware attention mechanism, instead, we focus on examining how much our proposed time-aware attention could mitigate the detriment of the contentaware attention.


The probable reason may be that it is difficult to model attention for individual sentences given the unsuitable time-decay functions.

if designs of attention functions are unsuitable for dialogue contexts, the encoded sentence embeddings would be weighted by improper attention values.


Since the intention is to measure the student's reading ability, any difference in performance that is not due to reading ability confounds the measurement.

since comprehension complexity of a passage is known to impact reading comprehension, it seems reasonable to assume that it would also impact other aspects of reading skill, including oral reading fluency.


The average complexity score for passages in the training set was 613.1, with a large variation across passages: min=240, max=1,019, SD=154.75.

the selected book offers its readers much variety in the configurations of textual features across its different passages.


Our models substantially outperformed the baseline with r increasing from 0.4 to 0.7-0.8.

the final models explain much of the variability in reading rates.


Variation across passages in the use of everyday words appears non-random.

the first third of the book averages 41.4 matches per passages; the rest of the book averages 37.3.


We use this to address the tricky problem of speaker evaluation.

we use the speaker model and an application of Bayes' rule to infer the most likely target color given a human utterance, and we report the accuracy of that process at identifying the target color.


As mentioned in Section 1, we evaluate the two models on a measure of pragmatic informative-ness: how well does the model represent a human speaker, such that a generative model of a listener can be built from it to interpret utterances?

formally, for a speaker S(u | , c t , C) and an example consisting of an utterance, language identifier, and color context (u, , C), we identify the t * that maximizes the probability of u according to S: L uses a noisy-channel model with a uniform prior over target colors and S as a generation model to infer the most likely target color given the input utterance.


3 However, for comparing monolingual and bilingual models, we found perplexity to be unhelpful, owing largely to its vocabulary-dependent definition.

if we fix the vocabulary in advance to include tokens from both languages, then the monolingual model performs unreasonably poorly, and bilingual training helps immensely.


We identified the 10 most frequent color-related words in each language to translate.

(we did not use this process to find translations of function words like "the" or the Chinese nominalization/genitive particle 的 de, but we show proposed translations that were not color-related, such as 灰 huī being translated as the English comparative ending "-er".)


We do not specifically claim these should be considered basic color terms, since Berlin and Kay give a theoretical definition of "basic color term" that is not rigorously captured by our model.

they explicitly exclude 灰色 huīsè from the set of basic color terms, despite its frequency, because it has a mean-ing that refers to an object ('ashes').


We note that this result does not rule out several competing hypotheses.

we do not exclude improvements in compositional semantics or syntax, nor do we distinguish improvements in  specific linguistic areas from broader regularization effects of having additional data in general.


In this work, we are interested in developing a learning method that enjoys the benefits of both approaches.

we seek an intermediate language of task representations that, like in program synthesis, is both expressive and compact, but like in multitask approaches is learnable directly from training data without domain engineering.


Inspired by related techniques aimed at making synthesis more efficient (Devlin et al., 2017), we use learning to help us develop an effective optimization procedure for natural language parameters.

we simply use the languagelearning datasets, consisting of pairs (x where q provides a (suitably normalized) approximation to the distribution of descriptions given task data.


Data augmentation Due to their comparatively small size, a data augmentation scheme (Jia and Liang, 2016) is employed for the regular expression and navigation datasets.

wherever a description contains a recognizable entity name (i.e.


We apply our method to four large multimodal datasets, ranging from image captions to image/text data scraped from Wikipedia, 2 to examine the relationship between concreteness scores and the performance of machine learning algorithms.

we consider the cross-modal retrieval problem, and examine a number of NLP, vision, and retrieval algorithms.


Our goal is to measure how "clustered" a word is in image feature space.

we ask: for each image v ∈ V w , how often are v's nearest neighbors also associated with w?


To demonstrate the effectiveness of our speaker naming approach, we design a model based on an end-to-end memory network (Sukhbaatar et al., 2015), namely Speaker-based Convolutional Memory Network (SC-MemN2N), which relies on the MovieQA dataset, and integrates the speaker naming approach as a component in the network.

we use our speaker naming framework to infer the name of the speaker for each segment of the subtitles, and prepend the predicted speaker name to each turn in the subtitles.


We also used "deep visual features" of the image as additional auxiliary features.

we use the 4, 096 features from VGGNet's (Simonyan and Zisserman, 2015) f c7 layer .


This leads us to conclude that the performance gain is actually obtained from using the auxiliary features.

using explanations generated by various deep learning models as auxiliary features improved performance.


Second, explanations from the component systems could be used to build an explanation for the overall ensemble.

by combining multiple component explanations, SWAF could also produce more comprehensible results.


A notable difference between longstanding work in natural language generation and recent "neural" models is in the treatment of entities and the words used to refer to them.

in the generation of narrative text, character-centered generation has been shown important in character dialogue generation (Walker et al., 2011;Cavazza and Charles, 2005) and story planning (Cavazza et al., 2002).


We use a max-pooling function to form a context vector c t with the same dimensionality as h t−1 (and, of course, p t and e current ).

at time step t, each element of the combined context vector c t is calculated as follows.


For an input we encode this input as $s$ and thus assume that s 0 = $ and s n+1 = $.

our RNNs use ReLUs (Rectified Linear Units), so for every v ∈ Q N we let σ v (the ReLU activation) be the vector the ReLUs act like identities on nonnegative inputs, but clip negative inputs to 0.


be an RNN, s an input string of length n and 0 ≤ t ≤ n a time step.

we define • the hidden state vector h s,t ∈ Q N given by where h s,−1 = h −1 and we use standard matrix product and point-wise vector addition, Finally, the RNN R computes the weighted language R : Σ * → R, which is given for every in- each component h s,t (n) of the hidden state vector is the ReLU activation applied to a linear combination of all the components of the previous hidden state vector h s,t−1 together with a summand w st that depends on the t-th input letter s t .


There exists an RNN with saturated linear activation, input alphabet Σ = {a}, and 1 designated neuron n ∈ N such that for all s ∈ Σ * and 0 ≤ t ≤ |s| • h s,t (n) = 0 if M does not halt on ε, and • if M does halt on empty input after T steps, then h s,t (n) = 1 if t = T 0 otherwise.

such RNNs with saturated linear activation can semi-decide halting of an arbitrary Turing machine in the sense that a particular neuron achieves value 1 at some point during the evolution if and only if the Turing machine halts on empty input.


On the other hand, the non-existence of such efficient or exact algorithms gives evidence for the necessity of approximation, greedy or heuristic algorithms to solve those problems in practice.

since finding the highest-weighted string in RNN is the same as finding the most-likely translation in a sequence-to-sequence RNN decoder, our NPcompleteness result provides some justification for employing greedy and beam search algorithms in practice.


Unlike the previous section, which only uses the vector of the NP head to calculate relatedness, here we include all common nouns occurring before the NP head as well because they also represent the core semantic of an NP (e.g., "earthquake victims" and "the state senate").

given an NP, we first construct a list N which consists of the head and all common nouns appearing before the head, we then represent the NP as a vector v using the following formula, where the suffix " PP" is added to each n if the NP is a bridging anaphor: v = n∈N embeddings P P n |N | (1) Table 3 shows that adding the constraint based on embeddings PP improves the result of MLN model II by 4.5%.


Hence, in this work we use the annotations produced by such systems to adapt a standard RNN layer by introducing a bias towards coreferent recency.

given an input sequence and coreference clusters extracted from an external system, we introduce a term in the update equations for Gated Recurrent Units (GRU) (Cho et al., 2014) which depends on the hidden state of the coreferent antecedent of the current token (if it exists).


To train the dialogue models, we use the Open-Subtitles dataset (lis, 2016).

we use the pre-processed data by (Li et al., 2016a) and further removed duplicates.


It is pre-trained on parallel French-English WMT14 corpus.

2 we use 2-layer LSTM cells with 1000 hidden cells for each layer.


(Bojar et al., 2016(Bojar et al., , 2017.

we investigate the following challenges: Distinct criteria: Traditionally, NLG outputs are evaluated according to different criteria, such as naturalness and informativeness (Gatt and Krahmer, 2017).


The encoder reads through X and computes a sequence of hidden states h 1:Tx : where F enc is a non-linear activation function (e.g., LSTM), h t is the hidden state at time t. Each time the model generates a target word y t , the decoder looks at a set of positions in the source sentence where the most relevant information is located.

another non-linear activation function F dec is used for the decoder where the hidden state s t at time t is computed by: Here, the context vector c t is computed as a weighted sum of the hidden vectors h 1:Tx : where is the dot product of two vectors.


In addition, we also evaluated system output by eliciting human judgments.

we randomly selected 40 sentences from each test set, and included human reference simplifications and corresponding simplifications from the systems above 2 .


Optimized  We note that the performance of the SoA feature sets on the new data is somewhat below the published results for the VUA data.

the v-16 system posted an F-score of 0.60 when trained and tested on verbs-only VUA data .


In this section, we consider the relationship between holistic scores of essay quality and use of metaphor, in argumentative essays.

we ask the following research questions: (1) Is there a relationship between essay score and metaphor use?


We show that, in a significant portion of such data, this protocol leaves clues that make it possible to identify the label by looking only at the hypothesis, without observing the premise.

we show that a simple text categorization model can correctly classify the hypothesis alone in about 67% of SNLI (Bowman et al., 2015) and 53% of MultiNLI (Williams et al., 2018).


To determine the degree to which such artifacts exist, we train a model to predict the label of a given hypothesis without seeing the premise.

we use fastText (Joulin et al., 2017), an off-the-shelf text classifier that models text as a bag of words and bigrams, to predict the entailment label of the hypothesis.


We conduct experiments to confirm USIM's validity.

we show that (1) UCCA can be consistently and automatically applied to learner language (LL) ( §4.2), (2) USIM is not prone to unduly penalize valid corrections ( §4.2), and (3) USIM assigns a lower score to corrections of poor quality ( §4.5).


We obtain a USIM score of 0.7 between the parses of the reference correction and the source, which is comparable to the parser's reported performance (0.73 in-domain, 0.68 out-of-domain), despite not performing any domain adaptation to LL.

the UCCA parses of the source and the correction are roughly as similar to each other as they are to their gold standard parse.


Future work will conduct user studies to assess the relative importance of different evaluation criteria.

we will explore to what extent users are tolerant to invalid changes to the sentence's structure, i.e., violation of conservatism, relative to their tolerance to invalid changes to the sentence's meaning, i.e., violation of faithfulness.


In this paper, we classify events based on their expected duration.

we differentiate between events whose duration is less than a day, and events whose duration is a day or more.


The function g shares an updating paradigm similar to negative sampling: where σ denotes the sigmoid function and γ the discounting learning rate.

d is the target label specifying the relative direction of w t+i given w t , defined as according on the relative position of w t+i respect to w t .


Nevertheless, DSG yields robust results on different scale of training data, which suggests that our model provides an effective solution to learn embeddings with exploiting the structure in context, while not severely suffered from the data sparsity problem.

among all SG models, DSG produces the best performance when trained on either the large or the small corpus.


STM assumes that different word vector specializations emphasize different subsets of semantic properties of words that are more informative for predicting some lexico-semantic relations than others.

we assume that a particular specialization function f (i) S can be trained to transform the input vectors x 1 and x 2 into vectors that encode properties suitable for predicting a particular relation, e.g., hypernymy.


Using the Google dataset GL (Mikolov et al., 2013b)   the CosAdd method (Mikolov et al., 2013c) shown in (5).

we determine a fourth word d such that the similarity between (b − a + c) and d is maximised.


Thus, we opt to construct a balanced corpus of positive and negative examples for a specific kind of redundancy in a specific configuration.

we extract all sentences that contained a pair of adjacent words that are likely to be semantically similar.


Semantic Verbal Fluency tests have been used in the detection of certain clinical conditions, like Dementia.

given a sequence of semantically related words, a large number of switches from one semantic class to another has been linked to clinical conditions.


Our model significantly outperforms the only previous work on sluice resolution on available newswire corpora, but also has a number of advantages over this work.

our model (a) does not require full syntactic parsing as a pre-processing step, (b) does not require manual feature engineering, and (c) is more robust when evaluated on speech corpora, because it is not dependent on full syntactic parsers (a).


All systems perform significantly worse on outof-domain data than on newswire.

we see all models struggle with root sluices.


There are some important problems in the evaluation of word embeddings using standard word analogy tests.

in virtue of the assumptions made by systems generating the embeddings, these remain tests over randomness.


5 Let π : V → V be a permutation of word vectors such that similar word remain close in the space.

let's suppose that π shuffles the vectors of all nationalities, like Dane and Frenchman, but maintains the same language vectors like Danish and French.


Firstly, even if word analogies as described by Equation (4) existed in the data, it would only be a matter of chance that applying PCA to the entire dataset would recover even slightly these parallel (analogous) word relations visually.

there is no reason to believe that the line through the words in a pair is not almost perpendicular to the surface they are mapped to.


b) Sentiment influence in conjunctions, in which, the sentiment of an aspect in a sentence influences the succeeding aspects due to the presence of conjunctions.

for sentences containing conjunctions like and, not only, also, but, however, though, etc., aspects tend to share/contrast their sentiments.


In this paper, we facilitate such phenomena by proposing a neural network where the information is shared among the aspects by means of a Long Short-Term Memory (LSTM) network (Hochreiter and Schmidhuber, 1997).

we model the sequential relationship between the aspects as per their occurrence in the sentence.


In other words, we model the sequential relationship between the aspects as per their occurrence in the sentence.

our model first takes a sentence along with all of its aspect-terms and then generates the sentential representations relative to each aspect to get better aspect-oriented features (Tang et al., 2016a).


To this end, we design a neural network architecture, capable of tracking and updating the states of entities at the right time with external memory, making it a natural fit for the task.

our model maintains a number of "memory chains" h j , one for each entity with the key k j and dynamically updates the states (h j ) of them as it progresses through the sentence with the help of the delay recurrence d j , taking previous activations into account.


With this delayed gate activation mechanism, we essentially decouple the duty of capturing transitions of activations between steps from the task of entity state tracking.

h j t is now dedicated to keeping track of the state of the j-th entity only and released from the burden of monitoring the activation of the update gate.


Observe that, when n < 5, the model suffers from insufficient capacity (not enough memory chains) to capture the various aspects required by the task, with aspect detection F 1 remaining below 78.

when n = 2 (no unconstrained chains), model performance drops substantially to a F 1 of 76.6 ± 0.4.


We apply z-score normalization by gender to our acoustic-prosodic features.

for each feature value we subtract the gender mean and then divide by gender standard deviation.


It is worth mentioning that the greater significance for SBC is attributable to the size of the corpus alone, as the average differences in similarities are comparable in both corpora.

even though conversations in SBC are less restricted than in CGC, the partner vs. nonpartner comparison is still "fair".


Most correlations, however, are very weak, with only a few reaching |r| > 0.1, all between pairs of measurements on the same feature.

we find correlations between local and global convergence for each prosodic feature (+0.14 ≤ r ≤ +0.47) and local and global similarity on pitch (r = +0.16) and intensity (r = +0.26).


This input matrix is fed to the convolution layer where different filters of the same 1 × 3 shape are used to extract the global relationships among same dimensional entries of the embedding triple.

these filters are repeatedly operated over every row of the input matrix to produce different The score function f (h, r, t)  feature maps.


Experimental results show that ConvKB obtains better link prediction performance than previous SOTA embedding models.

convKB obtains the best mean rank and the highest Hits@10 on WN18RR, and produces the highest mean reciprocal rank and highest Hits@10 on FB15k-237.


ConvKB obtains better scores than ConvE on both datasets (except MRR on WN18RR and MR on FB15k-237), thus showing the usefulness of taking transitional characteristics into accounts.

on FB15k-237, ConvKB achieves improvements of 0.394 − 0.316 = 0.078 in MRR (which is about 25% relative improvement) and 51.7 − 49.1 = 2.6% in Hits@10, while both ConvKB and ConvE produce similar MR scores.


The entity pairs in both sets are identical, except the roles are changed.

in E BW the Wikipedia entity is the target while the Baidu entity is the context, and in E WB the roles are reversed.


We propose a new task of detecting the most dominant event in a news article, which is an event assumed to govern and connect other foreground events and background events.

removal of the central event can break the entirety of a document and decompose the document into sections describing disjoint sets of situations.


In addition, we identify an event set E R which includes events whose contextual modality or realis status is "Actual" and use the set for constraining central event identification.

we define three rule based classifiers which begin with strict rules followed by relaxed rules in subsequent passes.


The classifier was trained on the corpus used in the Event Nugget Detection and Coreference task in the TAC KBP 2016 (Ellis et al., 2015).

the classifier uses a common neural layer shared between two event mentions that embed event lemma and parts-of-speech tags and then calculates cosine similarity, absolute and euclidean distances between two event embeddings.


We found that both types of event coreference resolution errors, missed coreference links as well as wrong links, cause problems, especially in calculating the Size Rank and the Stretch Rank for an event.

the first type of errors can demote both ranks of the correct central event while the second type of errors can wrongly promote one of the two ranks for non-central events.


based on a consistency score, which measures the similarity between target and source data distribution.

the consistency score is derived from the KL divergence between φ T (x) and φ S (x) for every word in the sentence in the source training data.


Here we describe a globally constrained decoding (Graves et al., 2012) method used in our model.

we use external knowledge information to guide the decoding process at the document level.


We propose a novel approach to semisupervised learning for information extraction that uses ladder networks (Rasmus et al., 2015).

we focus on the task of named entity classification, defined as identifying the correct label (e.g., person or organization name) of an entity mention in a given context.


To the best of our knowledge, our work is one of the first applications of LN to any NLP task.

our contributions are as follows: (1) We provide a novel application of LNs to an IE task, in particular semi-supervised named entity classification (NEC).


4) are critical for semisupervised learning.

it is important for unlabeled data to be used for regularization to be able to learn good abstractions in the different layers.


Further, we observe that LN predicts more PERSON and ORG entities as these as the most frequently appearing types in this dataset.

lN follows closely the underlying distribution of the data when making predictions, unlike EPB.


We automatically generate labeled training data for event trigger identification leveraging this wealth of event descriptions 1 .

we first group together paraphrases of event mentions.


Our experiments show that this data can be used with limited amounts of gold data to achieve significant improvement over both standard and neural event extraction systems.

it achieves 1.1 and 1.3 point F1 improvements over a state-of-the-art system in trigger identification on TAC-KBP and ACE data respectively.


These include lexical features, such as tokens, part-of-speech tags, and lemmas, syntactic features, such as dependency types and arcs associated with each token, and entity features, including unigrams/bigrams normalized by entity types, and the nearest entity in the sentence.

we use the Evento system from Ferguson et al.


We conjecture that semi-supervision will be more beneficial when gold data is very limited, but the conclusion isn't obvious, since semi-supervision is more likely to add noisy examples in this case.

we limit the number of positive gold examples for each event by randomly sampling the overall set.


The convolution layer applies a filter w j ∈ R (h+1)k to each ngram window of h+1 tokens.

writing x i:i+h for the concatenation of the word vectors x i , x i+1 , ..., x i+h , we have: where b j ∈ R is a bias term and f is the ReLu activation function (Nair and Hinton, 2010).


This approach draws from previous work on modeling phrases by composing their parts and the relations that link them (Socher et al., 2011(Socher et al., , 2013(Socher et al., , 2014.

let S d be the list of head-modifier relations obtained by parsing 4 sentence S. Each relation r in S d is a head-modifier tuple h, m, l .


This is the idea of our dynamic oracle, which therefore is a correct dynamic oracle only with respect to a preset criterion for plane assignment, and not for all the possible plane assignments that would produce the gold dependency structure.

given a 2-planar gold dependency tree whose set of arcs is t G , we need to divide it into two gold arc sets t 1 G and t 2 G , associated with each plane.


However, to make the oracle correct for the practical version, where consecutive Switch transitions are disallowed, we need to modify the cost calculation for the Switch transition.

applying a Switch transition does not affect the loss, so (Switch(c)) − (c) is always 0.


's (2011) system can be modified to generate a new family of deduction systems with corresponding transition systems.

we present three novel variants of the degree-2 Attardi parser, summarized in Fig.


Since each such deduction rule corresponds to a reduce transition, each revision to the deduction system yields a variant of Attardi's (2006) parser.

generalization of the deduction system gives rise to a family of nonprojective transition-based dependency parsers.


Among these, we have highlighted novel algorithms that (1) increase non-projective coverage without affecting computational complexity for exact inference, and (2) improve the time and space complexity for exact inference, even while providing better coverage than the original parser.

our ALLs 0 s 1 runs in Opn 6 q time and Opn 4 q space (improving from Opn 7 q and Opn 5 q, respectively) while providing coverage of 93.16% of the non-projective sentences in UD 2.1.


While tractable, this approach has the shortcoming of only considering a small part of the true prior, which potentially decreases the extent to which pragmatic reasoning will be able to apply.

if a useful caption never appears in the sampled prior, it cannot appear in the posterior.


To evaluate the success of S 1 as compared to S 0 , we define a listener L eval (image|caption) / P S 0 (caption|image), where P S 0 (caption|image) is the total probability of S 0 incrementally generating caption given image.

l eval uses Bayes' rule to obtain from S 0 the posterior probability of each image w given a full caption u.


Besides the NLVR corpus with a focus on complex and natural compositional language (Suhr et al., 2017), other useful visual reasoning datasets have been proposed for navigation and assembly tasks (MacMahon et al., 2006;Bisk et al., 2016), as well as for visual Q&A tasks which focus more on complex real-world images (Antol et al., 2015;Johnson et al., 2017).

for the NLVR dataset, previous models have incorporated property-and count-based features of the objects and the language (Suhr et al., 2017), or extra semantic parsing (logical form) annotations (Goldman et al., 2017) -we focus on end-toend models for this visual reasoning task.


The calculated probability is p(•|S\{w i }).

we use a bi-directional LSTM-RNN (Hochreiter and Schmidhuber, 1997) LM.


We alter a LM to a label-conditional LM, i.e., for position i in sentence S with label y, we aim to calculate p τ (•|y, S\{w i }) instead of the default p τ (•|S\{w i }) within the model.

we concatenate each embedded label y with a hidden layer of the feed-forward network in the bidirectional LM, so that the output is calculated from a mixture of information from both the label and context.


This data construction process is similar to (Schamoni et al., 2014) who made an English-German CLIR dataset, but ours is at a larger scale.

we use Wikipedia dumps released on August 23, 2017.


Inspired by the above methods, in this paper we propose a novel technique for multitask learning of domain-general representations.

1 we propose deep learning architectures for multi-domain learning, featuring a shared representation, and domain private representation.


Overall, the raw COND and GEN perform better than the baseline.

for COND, we observed performance gains on EuroPARL, T-BE and T-SC.


While GCNs were introduced  Baseline (Bastings et al., 2017) 14.9 12.6 +Sem 15.6 13.4 +Syn (Bastings et al., 2017) 16.1 13.7 +Syn + Sem 15.8 14.3 for modeling undirected unlabeled graphs (Kipf and Welling, 2016), in this paper we use a formulation of GCNs for labeled directed graphs, where the direction and the label of an edge are incorporated.

we follow the formulation of  and Bastings et al.


In our experiments, we used GCNs on top of a standard BiRNN encoder and a CNN encoder (Figure 2).

the initial representations of words fed into GCN were either RNN states or CNN representations.


The end-to-end nature of the Neural MT architecture (Sutskever et al., 2014;Bahdanau et al., 2015) provides a natural mechanism 1 to integrate stream decoding.

the recurrent property of the encoder and decoder components provide an easy way to maintain historic context in a fixed size vector.


(2017) can be inefficient for large numbers of constraints.

all unsatisfied constraints are expanded at each time step t which increases decoding complexity from O(tk) for normal beam search to O(tk2 c ).


Here we compare the results of InferSent (Conneau et al., 2017), a more involved representation that was found to provide a good sentence representation based on NLI data.

we concatenate the forward and backward encodings for each sentence, and maxpool over the length of the sentence, resulting in v and u (in R 2d ) for the context and hypothesis.


( §7) In order to perform experiments in a controlled, multilingual setting, we created a parallel corpus from TED talks transcripts.

3 we prepare data between English (EN) and three pairs of languages, where the two languages in the pair are similar, with one being relatively lowresourced compared to the other: Galician (GL) and Portuguese (PT), Azerbaijani (AZ) and Turkish (TR), and Belarusian (BE) and Russian (RU).


Finally, it is of interest to consider pre-training in multilingual translation systems that share an encoder or decoder between multiple languages (Johnson et al., 2016;Firat et al., 2016), which is another promising way to use additional data (this time from another language) as a way to improve NMT.

we train a model using our pairs of similar low-resource and higher-resource languages, and test on only the low-resource language.


We argue for a fairer metric based on the bits per utterance using utterance-aligned multi-text.

we train and test on "the same" set of utterances in each language, modulo translation.


We find that under standard approaches, text tends to be harder to predict in languages with finegrained inflectional morphology.

language models perform worse on these languages, in our controlled comparison.


We say that a surface form f is syncretic if two slots s 1 = s 2 exist such that some paradigm π(t, ) maps both s 1 and s 2 to f .

a single form fills multiple slots in a paradigm: syncretism may be thought of as intra-paradigmatic ambiguity.


The goal of this paper is the disambiguation of raw surface form counts, taken from an unannotated text corpus.

given such counts, we seek to impute the fractional counts for individual lexical entries (4-tuples), which are unannotated in raw text.


To that end, we employ a re-embedding component in which a contextual and a non-contextual representation are explicitly combined per token.

for a sequence of word-embeddings w 1 , .


We investigate explicit interaction between the target answer and the passage, so that contextual information can be better considered by the encoder.

matching is used between the target answer and the passage for collecting relevant contextual information.


It employs a pointer network (Vinyals et al., 2015) to compute the distribution of the next token in the decoder.

the probability of selecting the i-th token w i in the natural language question as the next token in the substring is computed as: where h is the hidden state of the previously generated token.


We focus on a simple weighted bag-of-words neural model for direct comparison to other popular methods, motivated by the fact that many virtual communities form around shared interests in particular topics.

let c p,i denote the number of times person p uses word v i ∈ V , where V is the vocabulary, and w p,i = log(c p,i + 1) be the log-scaled word count.


(2015) used to train a face recognition system.

: xwhere is an embedding made from a subset of another person's Tweets.


Then we use the discourse-related information to modify the word-level attention function.

the context vector representing the source document is: where h (e) (j,i) shows the encoder state of word i in discourse section j and α (t) (j,i) shows the corresponding attention weight to that encoder state.


The probability is learnt during training according to the following equation: ) Then the next word y t is generated according to: The joint probability is decomposed as: p g is the probability of generating a word from the vocabulary and is defined according to Equation 7. p c is the probability of copying a word from the source vector x and is defined as the sum of the word's attention weights.

the probability of copying a word x is defined as: Decoder coverage In long sequences, the neural generation models tend to repeat phrases where the softmax layer predicts the same phrase multiple times over multiple timesteps.


We attempt to solve the problem of standard table summarization by leveraging the hierarchical nature of fixed-schema tables.

rows consist of a fixed set of attributes and a table is defined by a set of rows.


In this paper, we present an unsupervised technique (Key2Vec) that leverages phrase embeddings for ranking keyphrases extracted from scientific articles.

we propose an effective way of processing text documents for training multi-word phrase embeddings that are used for thematic representation of scientific articles and ranking of keyphrases extracted from them using theme-weighted PageRank.


We take steps in cleaning the individual single word and multi-word tokens that we obtain.


(θ, Gen) is consistently and often significantly better than baselines across datasets and metrics.

humans tend to prefer the summaries extracted by  (θ, Gen).


It implicitly promotes the number of topics covered in the selected keyphrases by dampening intra-topic recommendation, and captures the mutually reinforcing relationship between topics and keyphrase candidates.

removing edges between candidates belonging to a single topic ensures that the overall recommendation of each topic is distributed throughout the entire graph.


Other research has focused on characterizing narratives in terms of their structure.

seminal formalisms such as plot units (Lehnert, 1981) and Story Grammars (Rumelhart, 1980) have been used to analyze story plots.


To this end, we present a dataset of movie plots extracted from Wikipedia.

we scraped lists of movies from the 'Lists of film remakes' page on Wikipedia, which consist of entries of movies considered remakes of previous movies.


These clusters were then manually pruned to remove errors, and the statistics of the curated dataset are shown in Table 1.

we observe that the average summary is quite long (564 words), which would make human annotations of similarity for such narratives difficult.


Character Alignment Kernel: This component compares two narratives by aligning characters of one with similar characters in the other.

we align each character, c i , of a story, s i , to a character, c j , of the other story, s j .


text, images, audio, video) (Morency et al., 2011;Poria et al., 2015;Tran and Cambria, 2018).

when we deal with Social Media posts, the presence of both textual and visual content has promoted a number of investigations on sentiment or emotions (Baecchi et al., 2016;You et al., 2016b,a;Yu et al., 2016;Chen et al., 2015) or emojis (Cappallo et al., 2015(Cappallo et al., , 2018.


Previous research such as (Fernández-González and Gómez-Rodríguez, 2012) and (Qi and Manning, 2017) proves that the widely-used projective arc-eager transition-based parser of Nivre (2003) benefits from shortening the length of transition sequences by creating non-local attachments.

they augmented the original transition system with new actions whose behavior entails more than one arc-eager transition and involves a context beyond the traditional two focus words.


Moreover, we adapt the featurization process with biaffine combination described in Qi and Manning (2017) for the arc-swift system to be used on the original Covington and NL-Covington parsers.

arc transitions are featurized by the concatenation of the representation of the head and dependent words of the arc to be created, the No-Arc transition is featurized by the rightmost word in λ 1 and the leftmost word in the buffer B and, finally, for the Shift transition only the leftmost word in B is used.


From the table, we can see that the models employing multi-task learning improve the performance on most tasks beyond the single-task models, in which our model achieves the highest accuracies.

our model boosts the performance by 3.3% over the best single-task model BiLSTM, outstripping other multi-task models by at least 0.9%.


The specific semantic parsing problem we study in this work is to map a natural language question to a SQL query, which can be executed against a given table to find the answer to the original question.

we use the currently largest natural language questions to SQL dataset, WikiSQL (Zhong et al., 2017), to develop our model and to conduct the experiments.


We follow the data preprocessing in (Wang et al., 2017).

we first preprocess the dataset by running both tables and question-query pairs through Stanford Stanza (Manning et al., 2014) using the script included with the WikiSQL dataset, which normalizes punctuations and cases of the dataset.


One is based on quantization (Botha et al., 2017;Han et al., 2016), which changes the way parameters are stored.

it seeks to find shared weights among embedding vectors and only keeps scale factors for each word.


This poses difficulty in practice due to the discrete nature of the clustering.

maximizing over the latent variables involves nondifferentiable argmax operations: To tackle this problem, we use the recently proposed Gumbel-Softmax to approximate the clustering decision during training.


While this approach achieves good verb prediction and sense disambiguation accuracy, it is not clear to what extend the model captures human intuitions about visual verbs.

it is interesting to ask whether the image regions that the model identifies as salient for a given verb correspond to the regions a human observer relies on when determining which verb is depicted.


Language is a weaker source of supervision for colorization than user clicks.

language lacks ground-truth information about the colored image (e.g., the exact color of a pixel or region).


We integrate h into every convolutional block of the FCNN, allowing language to influence the computation of all intermediate feature maps.

say Z n is the feature map of the nth convolutional block.


The research results show that spatial entities can be extracted with a good accuracy, however, spatial relation extraction is still challenging (Kordjamshidi et al., 2017a;Pustejovsky et al., 2015).

when the sentences convey more than one relationship, finding the right links between the spatial objects and spatial prepositions becomes difficult.


Our HACA model is an encoder-decoder framework comprising multiple hierarchical recurrent neural networks (see Figure 2).

in the encoding stage, the model has one hierarchical attentive encoder for each input modality, which learns and outputs both the local and global representations of the modality.


Cross-modal Attentive Decoder In the decoding stage, the representations of different modalities at the same granularity are aligned separately with individual attentive decoders.

one decoder is employed to align those highlevel features and learn a high-level (global) crossmodal embedding.


For the experiments, the originally proposed values for all parameters were maintained.

the bidirectional LSTMs for characterbased embeddings are trained on the corpus at hand with dimensions set to 25.


The singular unit can thus use these gates to reliably store number information across long-range dependencies.

the unit can (enumeration follows the same order as the panels in Figure 1c): (1) encode subject number viaC t subject with different values for singular and plural; (2) open the input gate only when a singular subject is presented (i t subject = 1 in light-blue curves only) and protect it from interfering nouns (i t = 0, t subject < t < t verb ); (3) at the same time, clear the cell from previously stored information (f t subject = 0) and then store subject number across the entire dependency (f t = 1, t subject < t < t verb ); (4) this will result in stable encoding of subject number in the cell C t throughout the dependency; (5) finally, output subject number at the right moment, when predicting the verb form (o t verb −1 = 1) (Eq.


We analyze different queues to measure the extent of diversity that each queue introduces to training data during our normal self-training process  where, at every iteration, only instances of the designated queue are added to training data.

we compute the extent of diversity that each given queue introduces as follows: 1 − cosine(T e , concat(T e , Q e )) where, as before, T e and Q e indicate the representations of training and queue instances at episode e respectively, and concat(.,.)


We create the diverse queue by sampling instances with high prediction confidence from all queues.

at every episode, we rank instances of each queue based on their prediction confidence and create a diverse queue by combining top r% instances of each queue, where r indicates the rate of adding new instances and set to r = 10%.


Effective semi-supervised learning algorithms based on pretraining techniques (Hinton and Salakhutdinov, 2006;Bengio et al., 2007;Erhan et al., 2010) have been developed for text classification, deep belief networks (Hinton and Salakhutdinov, 2006), and stacked autoencoders Bengio et al., 2007).

dai and Le (2015) developed an autoencoder for the later supervised learning process.


In this work, we extend previous work in self-training by developing a new and effective data sampling policy based on spaced repetition (Dempster, 1989;Cepeda et al., 2006;Averell and Heathcote, 2011) which addresses some of the above challenges.

our model's data sampling policy is not predetermined, it explores the entire data space and dynamically selects unlabeled instances with respect to the strength of a learner on a target task and easiness of unlabeled instances, and it relaxes the same model constraint of pretraining-based approaches by decoupling pretraining and fine-tuning steps.


In this work, we use a targeted evaluation approach (Marvin and Linzen, 2018) to elicit evidence for syntactic state representations from language models.

we examine language model behavior on artificially constructed sentences designed to expose behavior that is crucially dependent on syntactic state representations.


That is, we examine language model behavior on artificially constructed sentences designed to expose behavior that is crucially dependent on syntactic state representations.

we study complex subordinate clauses and garden path effects (based on mainverb/reduced-relative ambiguities and NP/Z ambiguities).


However, we only find a significant no-matrix penalty for GRNN and the RNNG (p < .001 in both): the other models do not significantly penalize an ungrammatical continuation (p = .9 for JRNN; p = .5 for TinyLSTM).

jRNN and TinyLSTM give no indication that (2-b) is less probable than (2-c).


convolution which is left padded) ensures that the pair-embeddings are aligned so that the prediction targets correspond to the later word in the pair.

the pair can be thought of as representing the 'current' and 'previous' words together.


Many of the measures are highly correlated, but the pattern of correlations is different from the pattern of benefits that we find during joint-training.

we note that the N400 is correlated with the other ERP signals, and yet we do not see benefit in prediction of the N400 when jointly training a model to predict it and other signals.


We also see that with transfer learning, a model trained on only 5 hours of ST data achieves a BLEU of 9.1, nearly as good as the 10.8 of a model trained on 20 hours of ST data without transfer learning.

fine-tuning an English ASR modelwhich is relatively easy to obtain-produces similar results to training an ST model on four times as N = 0 2.5 5 10 20 50 base 0 2.1 1.8 2.1 10.8 22.7 +asr 0.5 5.7 9.1 14.5 20.2 28.2  much data, which may be difficult to obtain.


Formally, we define untranslated terminology with respect to a source sentence S, sentence created by a translator R, and sentence created by an interpreter I.

we define any consecutive sequence of words s i:j , where 0 ≤ i ≤ N − 1 (inclusive) and i < j ≤ N (exclusive), in source sentence S 0:N that satisfies the following criteria to be an untranslated term: • Termhood: It consists of only numbers or nouns.


Since our definition of terminology only allows for nouns and numbers, we restrict prediction to words of the corresponding POS tags Q = {CD, NN, NNS, NNP, NNPS} using the Stanford POS tagger (Toutanova et al., 2003).

we assign a POS tag p i to each word from s i and only extract features/predict using the classifier if p i ∈ Q; otherwise we always assign the Outside tag.


terms all noun/# B-rank 1,256 10.8 45.4 A-rank 1,206 10.4 43.6 S-rank 812 7.0 29.6 Table 2: Final untranslated term count and number of Itags after filtering based off of the relevancy constraint ( §2).

only the raw untranslated source terms that appear in the translator's transcript are truly considered untranslated.


Subsequently, we apply temporal attention over it; the attention weight is calculated as in .

we use the global method for each t in {h f low t } T t=1 : where h dec m ∈ R D o is the state of the decoder LSTM, c m ∈ R 4D 2 is the context vector, α m ∈ R T is the attention mask, and Next, we obtain the output word probability: where Finally, we select the output word as y m+1 = argmax s∈V (s m ).


We perform several quantitative evaluations to provide more insights about our AudioCaps dataset.

our experiments are designed to answer the following questions: 1.


The semantic attention is also favorable for captioning performance, as SemTempAtt(1NN)-VGGish-LSTM in Table 3 slightly outperforms TempAtt-VGGish(FC2)-LSTM in Table 2.

the additional use of semantic attention enhances the temporal attention model.


It shows the proportion of time the editors substituted a word at each relative word position bin compared to if they randomly chose a word to substitute.

the red curve in the plot shows the proportion of replacement word locations if they were chosen randomly from those available in the editing task.


In order to strengthen the reward signal, we reinterpret the UTILITY calculator (coupled with the answer generator) as a discriminator in an adversarial learning setting.

instead of taking the UTILITY calculator to be a fixed model that outputs the expected quality of a (question, answer) pair, we additionally optimize it to distinguish between true (question, answer) pairs and model-generated ones.


6 and 8) Since the v j i s and ρ were selected arbitrarily, we can obtain any element of L(G, A ) in that manner.


The question now is to find when to pass down a model to its children.

at which stage has a model learned the most it could from its data and should start to diverge to improve again?


Discontinuous constituency trees can be seen as derivations of Linear Context-Free Rewriting Systems (LCFRS, Vijay-Shanker et al., 1987), a class of formal grammars more expressive than context-free grammars, which makes them much harder to parse.

exact CKY-style LCFRS parsing has an O(n 3f ) time complexity where f is the fan-out of the grammar (Kallmeyer, 2010  A natural alternative to grammar-based chart parsing is transition-based parsing, that usually relies on fast approximate decoding methods such as greedy search or beam search.


2012, we eliminate spurious ambiguity by selecting a canonical derivation for a gold tree.

we design the static oracle (i) to apply COMBINE as soon as possible in order to minimize the size of the memory (ii) to combine preferably with the most recent set in the memory when several combinations are possible.


We then compute the score of each structural action: where FF s is a feedforward network with two hidden layers, a tanh activation and a single output unit.

it outputs a single scalar for each column vector of matrix M .


When = 0, and q (z|x) is a delta distribution, we recover the AE.

the AE does not regularize the variational distribution toward a prior distribution, and there is only a point-estimate to represent the text sequence's latent feature.


The KL term L R pushes q (z|x) close to an uninformative prior p(z): the posterior becomes more like an isotropic Gaussian noise, and less representative of their corresponding observations.

l R blocks Path A, and thus z remains uninformative during the entire training process: it starts with random initialization and then is regularized towards a random noise.


Formally, has the form: where t is the iteration number, T is the total number of training iterations, f is a monotonically increasing function, and we introduce two new hyper-parameters associated with the cyclical annealing schedule: • M : number of cycles (default M = 4); • R: proportion used to increase within a cycle (default R = 0.5).

we split the training process into M cycles, each starting with = 0 and ending with = 1.


We present the results of our study into the ability of state-of-the-art first-order neural graphbased parsers, with seemingly simple architectures, to explicitly forego structured learning and prediction.

1 this is not due to a significantly faster, simpler, algorithm for projective maximum spanning tree (MST) decoding than Eisner (1996)'s algorithm, which we formally prove to be impossible, given the Exponential Time Hypothesis.


B More analysis on the effectiveness of NFSTs B.1 Does feeding alignments into the decoder help?

we attribute our models' outperforming Neuralized IBM Model 1 to the fact that a complete history of past alignments is remembered in the RNN state.


An example triple in Freebase is the subject Clarissa, predicate book.written work.author and object Samuel Richardson, explaining that the book Clarissa is written by author Samuel Richardson.

we take advantage of these relations between entities, which describe facts, to help with factoid question answering.


FreebaseQA contains over 54K matches from about 28K unique questions that can be used to train machine learning (ML) models and help the development of factoid QA systems for more realistic applications.

these matches may be used to train ML models to align natural language questions with Freebase predicates to search for the correct answers in Freebase.


As a result, these questions show much richer linguistic variation and complexity than almost all existing KB-QA data sets.

we use the TriviaQA (Joshi et al., 2017) data set as the primary source of our QA pairs, while also including questions scraped from the trivia websites, KnowQuiz (http://www.


We denote the set of n-grams of m q as W n (m q ) and use W n (m q ) to retrieve the subgraph as Next, to select the correct fact (the answer) in the subgraph, we match the mention m q with candidate subjects in S q , and match the pattern p q with candidate relations in R q .

we want to maximize the log-likelihood ( 1)The probabilities in (1) are modeled by s ∈Sq e h(f (mq ),f (s )) (2) r ∈Rq e h(g(pq ),g(r )) , where f : M S → R d maps the mention and the subject onto a d-dimensional differentiable manifold embedded in the Hilbert space and similarly, g : P R → R d .


Hence, we propose a ranking score with literal closeness and semantic closeness.

the literal closeness is measured by the length of the longest common subsequence |σ|(s, m) between a subject s and a mention m. The semantic closeness is measured by the co-occurrence probability of the subject s and the mention m P(s, m) = P(s|m)P(m) = P(w1, .


First, instead of considering the match of mention-subjects and pattern-relations separately, (13) jointly considers both input pairs and their dependency.

(13) incorporates such dependency as the weight factors |I| (for subjects) and |J| (for relations).


To address these difficulties, we propose an essential-term-aware Retriever-Reader (ET-RR) model that learns to attend on essential terms during retrieval and reading.

we develop a two-stage method with an essential term selector followed by an attention-enhanced reader.


We decompose the task of relation extraction in the knowledge graph into two subtasks: knowing where to go, and knowing when to stop (or to halt).

single-hop relation extraction and termination decision.


In the UHop framework, the model is trained to favor the correct relation over incorrect relations.

to select the correct outbound single-hop relations from current entity e, the model prefers the correctr over the other relations R −r of e; to terminate at the current entity e, the model favors the correct relationr linked to the current entity e over the outbound R relations from e. To continue the iteration, it proceeds likewise.


Therefore, we propose a progressive method which treats the termination decision as a comparison.

the model stops when it cannot extract any relation better than that from its previous hop.


The model must follow the instructions to arrive at the destination node.

the task is to navigate to an answer cell (answer entity) starting from a random cell (topic entity) given a sequence of instructions (questions).


In our case, unlike the previous approaches, that simply apply attention over the contextual utterance for classification, we take a different approach.

we attend over the contextual utterances by computing correlations among the modalities of the target utterance and the context utterances.


We also conduct a comparative experiment to verify that the classification based on a sentence-pair is better than the single-sentence classification with fine-tuned BERT, which means that the improvement is not only from BERT but also from our method.


Compared with the polarity of each aspect, it is relatively easy to obtain overall polarities.

they minimize the square loss between the overall polarity and the weighted sum of all aspect polarities.


We consider our approach as weakly supervised learning because there is no direct supervision from polarity of each aspect.

our model includes two classifiers: a sentiment polarity classifier and an opinion word classifier.


Experiments on three dialogue emotion datasets, IEMOCAP, Friends, and Emo-tionPush demonstrate that our proposed Hi-GRU models attain at least 8.7%, 7.5%, 6.0% improvement over the state-of-the-art methods on each dataset, respectively.

by utilizing only the textual feature in IEMO-CAP, our HiGRU models gain at least 3.8% improvement over the state-of-the-art conversational memory network (CMN) with the trimodal features of text, video, and audio.


The results demonstrate that our proposed HiGRU models achieve at least 8.7%, 7.5%, 6.0% improvement over state-of-the-art methods on each dataset, respectively.

by utilizing only the textual feature in IEMO-CAP, our proposed HiGRU models gain at least 3.8% improvement over the existing best model, conversational memory network (CMN) with not only the text feature, but also the visual, and audio features.


Hence, this variant is termed HiGRU-sf, representing HiGRU with selfattention and features fusion.

we apply self-attention upon the forward and backward hidden states separately to produce the left context embedding, h l k (H l j ), and the right context embedding, h r k (H r j ), respectively.


Early stopping with a patience of 10 is adopted to terminate training based on the accuracy of the validation set.

following the best models on each dataset, the parameters are tuned to optimize WA on the validation set of IEMOCAP and to optimize UWA on the validation set of Friends and Emo-tionPush, respectively.


Even when brought to scale with the NLL loss, it shows much larger fluctuations as the training progresses.

it shows major increases at the re-starts of the optimizer for the simulated annealing that are not compensated for by the rest of the training.


We propose instead to combine benefits from differentiable sampling and bi-directional NMT to obtain a compact model that can be trained endto-end with back-propagation.

• Translations are sampled using the Straight-Through Gumbel Softmax (STGS) estimator (Jang et al., 2017;Bengio et al., 2013), which allows back-propagating reconstruction errors.


In future work, we will investigate the use of differentiable reconstruction from sampled sequences in unsupervised and semi-supervised sequence generation tasks.

we will exploit monolingual corpora in addition to parallel corpora for NMT.


these methods regard one-to-many translation lexicons as a suggestion.


(b) We also sample bilingual sentences that match two source-target phrase pairs.

given a combination of two phrase pairs, we index bilingual sentences that match both simultaneously.


For time step i in layer j, the hidden state h i,j is calculated by employing self-attention over the hidden states in layer j − 1, which are {h 1,j−1 , h 2,j−1 , ..., h m,j−1 }, where m is the number of source-side words.

h i,j is calculated as follows: First, a self-attention sub-layer is employed to encode the context.


For time step t in layer j, a selfattention sub-layer of hidden state s t,j is calculated by employing self-attention mechanism over hidden states in previous target layer, which are Then, a second target-tosource sub-layer of hidden state s t,j is inserted above the target self-attention sub-layer.

the queries(Q) are projected from s ′ t,j , and the keys(K) and values(V ) are projected from the source hidden states in the last layer of encoder, which are {h 1,n , h 2,n , ..., h m,n }.


Definition of the relative tasks Suppose Y k denotes any partition (or clustering) regarding the output label space (target vocabulary) Y.

y k is a set of subsets We define the following relative task: given a context x, y <t , predict the subset or the cluster to which the t th token y t belongs in y k , denoted as y k (y t ).


3 We use multi-class logistic regression to fit the layer-wise hidden representation learned by a well-trained 6-layer Transformer (Vaswani et al., 2017) over each training instance x, y <t .

given a context x, y <t , for each task Y k and a hidden representation h l (x, y <t ) of this context, which is fixed as constant, we predict the cluster Y k (y t ) according to the following probability: where θ l Y k is the parameter of the logistic regression model for task Y k at l th layer.


Motivated by the observation that representations from higher layers have better expressibility than lower layers, as claimed in §2.2, we instead employ a coarse-to-fine strategy to select one task for each layer: finer-grained tasks for higher layers while coarser-grained task for lower layers.

suppose 1 ≤ s(l) ≤ K is the selected index regarding task Y s(l) for the l th layer, then it subjects to s(l) < s(l + 1) for each l. In addition, to encourage the diversity among the selected L tasks, we require s(l + 1) − s(l) to be large enough for all l. Formally, the loss of the hierarchical regularization (HR) method is: where is similar to Eq.


There are two types of feature template: a) the source-side bag-of-word (BOW) features; and b) the target-side order-aware BOW features.

given a context x, y <t , we extract features according to the above templates:  Figure 8: (a) The original hierarchical clustering tree, with left branching to have all leaves at the same tree depth; (b) The hierarchical clustering tree with depth 2, and a virtual task is constructed at depth 3.


For selecting the depths in a coarse-to-fine manner, we follow the heuristics mentioned in Section 3.1, that is, we select depths which are diverse enough so as to have better coverage of all the relative tasks.

we follow a quotient between two adjacent selected tasks of 5, and select from the task which has the cardinality of 5000, then we select tasks of cardinalities around 1000, 200, 40 respectively.


This is an issue for text with words or phrases that are ambiguous in terms of stylistic weight, e.g.

"dog" in "a man with a dog" vs. "That man is a dog."


Previous work has compared models with respect to a single aspect of interest at a time, but has only, to a limited degree, considered how relationships between multiple aspects influence these comparisons.

concurrent work by (Li et al., 2018) examines tradeoff plots, but focuses primarily on variants of its own model, while including only a single point on the plots of style transfer models from other papers.


In Section 6, we present benchmark experiments on using BiRD as a testbed to evaluate various common semantic composition methods using various pre-trained word representations.

we conduct experiments to gain insights into research questions such as: Which common mathematical operations for vector composition (e.g., vector addition, vector multiplication, etc.)


We used an English-French phrase table from the Portage Machine Translation Toolkit (Larkin et al., 2010) to determine additional pairs AB-X.

8 for each AB-F entry in the phrase table (where F is a French term) we keep the five most frequent English unigrams and the five most frequent English bigrams (other than AB) that are also aligned to F. Among the 4,095 ABs, 454 occurred in the phrase table.


These results differ substantially from the observations by Mitchell and Lapata (2010).

in their work the multiplication model showed the best results, markedly outperforming the addition model.


We build on prior work employing online crowd workers to create data by paraphrasing.

we refine the idea of iteratively asking for paraphrases, where each round prompts workers with sentences from the previous round, leading to more diverse data (Negri et al., 2012;Jiang et al., 2017;Kang et al., 2018).


However, the relative performance of the neural methods differs substantially between the two settings.

(1) SIF performs better than an unweighted average on artificial data, but on real data we see the opposite trend, (2) combining rankings with Borda appears to help on the artificial data, but not on the real data, (3) ranking by length is surprisingly effective on the real data, and (4) results tend to be lower on the real data than the artificial (even at lower values of p).


"), but to infer the reason why the narrator is sharing this story (i.e., "To get advice on how to stay alert in the next few hours.").

we are not concerned with the intradiegetic aspects of the narrative, but with the extradiegetic intention of the narrator in sharing the story.


We propose another dimension to this line of work by taking an outside-the-story (extradiegetic) perspective 3 and aiming to understand why the story is shared by the narrator, potentially inferred from how the narrator decides to construct it.

the task introduced here is to infer the advice-seeking intention of the narrator.


16 We are also interested in understanding the types of skills needed to solve the task.

we want to estimate the proportion of the task instances that can not be solved by mere factual reasoning.


Inherently, this is a natural language understanding task, and we propose to address it as such.

we propose the task of substantiated perspective discovery where, given a claim, a system is expected to discover a diverse set of well-corroborated perspectives that take a stance with respect to the claim.


Today, we make use of search engines or recommendation systems to retrieve information relevant to a claim, but this process carries multiple forms of bias.

they are optimized relative to the claim (query) presented, and the popularity of the relevant documents returned, rather than with respect to the diversity of the perspectives presented in them or whether they are supported by evidence.


In order to ensure that our dataset contains more realistic sentences, we use web search to augment our pool of perspectives with additional sentences that are topically related to what we already have.

we use Bing search to extract sentences that are similar to our current pool of perspectives, by querying "claim+perspective".


Instead, we make use of a retrieval system to annotate only the relevant pairs.

we create an index of all the perspectives retained from step 2a.


We provide brief statistics on the sources of different content in our dataset in Table 4.

this table shows: 1. the size of the data collected from online debate websites (step 1).


Cognitive scientists have argued that such images are  much like text contributions in the way their interpretation connects to the broader discourse.

inferences analogous to those used to interpret text seem to be necessary with such images to recognize their spatio-temporal perspective (Cumming et al., 2017), the objects they depict (Abusch, 2013), and their place in the arc of narrative progression (McCloud, 1993;Cohn, 2013).


Note, by contrast, that subjects characterized 12.7% of images as introducing a new tool: this includes many cases where the same subjects say the image is not required.

subjects' intuitions suggest that coherent imagery typically does not contribute instruction content, but rather serves as a visual signal that facilitates inferences that have to be made to carry out the instruction regardless.


Our annotation reveals cases where a single image does include more information than could be packaged into a single textual discourse unit (the proverbial thousand words).

such imagery participates in more complex coherence relationships than we find between text segments.


We, therefore, propose a novel framework for DST that identifies relevant historical context by referring to the past utterances where a particular slot-value changes and uses that together with weighted system utterance to identify the relevant context.

we use the current user utterance and the most recent system utterance to determine the relevance of a system utterance.


Each input is represented as a sequence of words which is encoded to a vector representation via global-local self-attentive encoder (GLE) module (Zhong et al., 2018).

gLE employs local slot-specific bidirectional LSTMs and a global bidirectional LSTM (Hochreiter and Schmidhuber, 1997) that is shared across all slots for encoding the input sequence into a sequence of hidden states (H), followed by a self-attention layer (Lin et al., 2016) to obtain a fixed dimension vector representation (c).


The GLE modules are used to encode the antecedent user utterance (H u p , c u p ), the current user utterance (H u , c u ), the previous system utterance (H s , c s ), each of the system acts (H a i , c a i ), as well as the previous slot-value (H v p , c v p ) and the candidate ( Referential Context Scorer: uses the candidate slot value (c v ), the antecedent user utterance as well as the previous slot-value to determine if the candidate slot value was referenced in the antecedent utterance.

the scorer uses the representation of the candidate slot value c v to attend over hidden states of the antecedent user utterance and the previous slot-value, H u p and H v p , and then computes attention weights for each of the hidden states.


System Act Scorer: is the same as the action scorer proposed by (Zhong et al., 2018).

the scorer uses attention from c u over C a to calculate action summary followed by a linear layer with sigmoid activation to calculate the score y a that determines the relevance of the candidate slot-value based on the previous system actions (Equation 5).


We develop CLEVR-Dialog, a large diagnostic dataset for studying multi-round reasoning in visual dialog.

we construct a dialog grammar that is grounded in the scene graphs of the images from the CLEVR dataset.


We construct a dialog grammar that is grounded in these scene graphs.


To mimic this setup, we condition our question generation at round t only on the partial scene graph S t q that accumulates information received so far from the dialog history (and not on S a ).

we use a set of caption {T C i } and question {T Q i } templates, which serve as the structural units of our dialog grammar.


we use a set of caption {T C i } and question {T Q i } templates (enumerated in Tab.


For example, all Filter primitives filter out objects from an input set of objects according to certain constraints.

filter[color](blue) filters out blue objects from a given set of objects, while filter[shape](sphere) filters out all spheres.


The role of the caption is to seed the dialog and initialize S 0 q .

caption gives Q-er partial information about the image so that asking follow-up questions is possible.


We study the behavior of CorefNMN on CLEVR-Dialog both qualitatively and quantitatively.

we visualize qualitative examples and develop metrics to quantitatively evaluate the textual and visual grounding.


• Interestingly, the second best model HRE-QIH has the best accuracy on 'All' questions, even beating CorefNMN by a margin of 20%.

hRE-QIh (and even MN-QIh) is able to answer 'All' questions significantly better than CorefNMN perhaps due to the ability of its dialog-level RNN to summarize information as the dialog progresses.


Please note that these experiments do not capture the full impact of the bias in this dataset.

there will be more biased words beyond the 25 words we identified on the list of top 100 words ranked according to PMI since the cut-off value of 100 was arbitrarily chosen.


(2016b), there is no gender bias if each nonexplicitly gendered word in the vocabulary is in equal distance to both elements of all explicitly gendered pairs.

if one cannot determine the gender association of a word by looking at its projection on any gendered pair.


We evaluate biased and debiased word embeddings on several downstream tasks.

the CoNLL 2003 shared task (Tjong Kim Sang and De Meulder, 2003) which provides evaluation data for NER, POS tagging, and POS chunking.


Table 4 shows effect size and significance at 0.01 before and after applying the Holm-Bonferroni multiple testing correction (Holm, 1979) for a subset of tests and models; complete results are provided in the supplement.

8 Context  we select Caliskan Test 1 associating flowers/insects with pleasant/unpleasant, Test 3 associating European/African American names with pleasant/unpleasant, and Test 6 associating male/female names with career/family, as well as the angry black woman stereotype and the competent and likable double bind tests.


Moreover, the counterintuitive sensitivity of SEAT on some models and biases suggests that biases revealed by SEAT may not generalize beyond the specific words and sentences in our test data.

our results invalidate the assumption that each set of words or sentences in our tests represents a coherent concept/attribute (like African American or pleasant) to the sentence encoders; hence, we do not assume the encoders will exhibit similar behavior on other potential elements of those concepts/attributes (other words or sentences representing, for example, African American or pleasant).


In addition, we examine how gender bias in ELMo propagates to the downstream applications.

we evaluate a state-of-the-art coreference resolution system  that makes use of ELMo's contextual embeddings on WinoBias (Zhao et al., 2018a), a coreference diagnostic dataset that evaluates whether systems behave differently on decisions involving male and female entities of stereotyped or anti-stereotyped occupations.


When assigning quantitative labels to a dataset, different methodologies may rely on different scales.

when assigning polarities to words in a sentiment lexicon, annotators may use binary, categorical, or continuous labels.


The generative process starts by drawing each latent polarity value z w from a three-dimensional Dirichlet prior, parameterized by α w = (1, 1, 1): If the word is uncontroversial, 3 we spur this prior somewhat using the number of lexica in which the word appears c(w).

we add c(w) to the parameter for the sentiment associated with that word in the lexica, e.g., α SUPERB = (1 + c(SUPERB), 1, 1).


Because computing the normalizing constant P (X ) is intractable, we instead approximate the posterior with a family of distributions Q λ (Z), indexed by variational parameters λ.

we use To construct β w , we first define a neural network g(•; φ d ), with a single 32-dimensional hidden layer, which "encodes" x w d into a threedimensional vector.


(Unlike SentiVAE, this concatenation does not yield a single sentiment lexicon that retains scale coherence, while achieving maximal coverage over words.)

we replace each token with its corresponding polarity value, and then average the these values (Go et al., 2009;Ozdemir and Bergler, 2015;Kiritchenko et al., 2014).


It is vital to find a good middle ground for both these parameters or else the curves will either altogether drop too quickly with all significant features eliminated in a single round, or runtime is wasted without benefit.

removals and vector size need to be balanced so as to optimally capture enough of the highly significant features and not only the tail of the distribution.


We alleviate these problems with the Straight-Through estimator (ST) (Bengio et al., 2013).

in the forward pass of training we discretizeẽ c t by using the arg max (Eq.


The Pyramid method is mainly a measurement of recall, which thus also applies to our lightweight Pyramid; but other measurements for summary quality, such as precision, nonredundancy and grammaticality, may also be considered.

it may be possible to extend our design of crowdsourcing tasks to supply indications for these complementary measurements as well.


(2018) and introduce a simpler left-to-right transition system that requires neither a stack nor a buffer to process the input sentence and where, instead of selecting a child of the word on top of the stack, the network points to the parent of the current focus word.

in our proposed approach, the parsing configuration just corresponds to a focus word pointer i, that is used to point to the word currently being processed.


We intuit that entities are normally only used in underspecified contexts if they are expected to be known to the reader.

they are either more clearly introduced in an earlier sentence, or part of general in-domain knowl-  edge a reader is expected to have.


(2018) and follow most hyperparameter suggestions as given by the in-depth study presented in Reimers and Gurevych (2017).

we use an LSTM with 256 hidden states and one layer (Hochreiter and Schmidhuber, 1997), a locked dropout value of 0.5, a word dropout of 0.05, and train using SGD with an annealing rate of 0.5 and a patience of 3.


Next, we perform k-fold cross-validation on the training set.

each time we train a model with (k-1) folds of the data and based on the learned model we define our q distribution.


(2016) and apply the bidirectional long short-term memory (Hochreiter and Schmidhuber, 1997) (BiLSTM) networks as the neural architecture for all baselines and our approaches.

we implement the following baselines: a Simple model which is a linearchain LSTM-CRF model and we treat all missing labels as O; the missing label CRF (Bellare and McCallum, 2007;Greenberg et al., 2018) (LSTM-M-CRF) model; the partial perceptron (Carlson et al., 2009) model, which is a structured perceptron (Collins, 2002) but only considers the scores on the words with available labels; the transductive perceptron (Fernandes and Brefeld, 2011) model where they introduce a Hamming loss function during the perceptron training process; lastly, we train an LSTM-CRF (Lample et al., 2016) with complete annotations as the upper bound (Com-plete).


To solve the first challenge, we transform multilabel classification to multiple binary classification problems.

a given sentence s attached each pre-defined event type t forms an instance, which is expected to be labeled with 0 or 1 according to whether s contains an event of type t. For example, suppose there totally are 3 predefined types of events(denoted by t 1 , t 2 and t 3 ), and sentence s contains two events of type t 1 and t 3 , then it could be transformed to the following three instances: In this paradigm, sentences that convey multiple events will yield multiple positive pairs, thus the multi-label problem could be well solved.


The output o is expected to be 1 if the given sentence conveys an event of the target type, otherwise 0 (the output should be 1 for the example given in Figure 1).

given a sentence, the proposed model first transforms the input tokens into embeddings, and applies an LSTM layer to calculate a context-dependent representation for each token.


As illustrated in Figure 1, the attention vector α is calculated based on the target event type embedding t 1 and the hidden states h yielded by LSTM.

the attention score for the k-th token in a given sentence is calculated by the following equation: In this model, trigger words of the target event type are expected to obtain higher scores than other words.


the value of this term is 1 for negative samples (y (i) is 0) and 1 + β for positive samples (y (i) is 1), where β ≥ 0.


Therefore, we remove trigger annotations from the corpus.

we employ Stanford CoreNLP Toolkit to split each document into sentences, and assign each sentence with a set of labels according to the original annotations in ACE 2005 corpus.


We address this shortcoming by framing the sub-event detection problem in social media streams as a sequence labeling task and adopt a neural sequence architecture that explicitly accounts for the chronological order of posts.

we (i) establish a neural baseline that outperforms a graph-based state-of-the-art method for binary sub-event detection (2.7% micro-F 1 improvement), as well as (ii) demonstrate superiority of a recurrent neural network model on the posts sequence level for labeled sub-events (2.4% bin-level F 1 improvement over non-sequential models).


In contrast, our work does take into account that chronological order and we predict the presence and the type of a sub-event exploiting information from previous tweets.

we (i) propose a new neural baseline model that outperforms the state-of-the-art performance on the binary classification problem of detecting the presence/absence of sub-events in a sports stream, (ii) establish a new reasonable baseline for predicting also the sub-event types, (iii) explicitly take into account chronological information, i.e., the relation among consecutive tweets, by framing sub-event detection as a sequence labeling problem on top of our baseline model, and (iv) perform an experimental study, indicating the benefit of sequence labeling for sub-event detection in sports Twitter streams.


Twitter streams have been extensively studied in various contexts, such as sentiment analysis (Kouloumpis et al., 2011), stock market prediction (Nguyen and Shirai, 2015) and traffic detection (D'Andrea et al., 2015).

for sub-event detection in Twitter, several approaches Figure 1: Our sub-event detection model comprises: (a) a bin layer, (b) a unit layer, (c) a word embeddings layer, (d) a representation layer and (e) a chronological LSTM layer to model the natural flow of the sub-events within the event.


The goal is, given a main event (i.e., soccer match), to identify its core sub-events (e.g., goals, kick-off, yellow cards) from Twitter streams.

we consider a supervised setting, relying on annotated data (Meladianos et al., 2018).


In this work, we frame the problem of sub-event detection in Twitter streams as a sequence labeling task.

we (i) propose a binary classification baseline model that outperforms state-of-the-art approaches for sub-event detection (presence/absence), (ii) establish a strong baseline that additionally predicts sub-event types, and then (iii) extend this baseline model with the idea of exchanging chronological information between sequential posts, and (iv) prove it to be beneficial in almost all examined architectures.


a bidirectional long-short term memory (BiLSTM) followed by a conditional random fields (CRF) module).

in the textual IE task, we obtain an improvement of 0.5% over SeqIE on the CONLL03 dataset, and an improvement of 1.4% on the chemical entity extraction (Krallinger et al., 2015).


g (0) i = Enc(s i ), and conducts graph convolution on every node, propagating information between its neighbors, and integrating such information into a new hidden representation.


In this way, the graph contextual information is propagated to each word through the LSTM.

we have where h 1:k are the output hidden states of the encoder, GCN(s i ) represents the initial state, and Θ dec is the decoder parameters.


Continuing with our example E1 -if we know from existing knowledge that "Leonardo" is a famous actor and has rarely directed or written a movie, we can decide with a high confidence that this predicate maps to @film.actor in this triple, even if our knowledge graph knows nothing about the new movie "Robin Hood".

we make three contributions in this paper.


Given (i) an existing knowledge base KB of triples (s, p, o) -where s, o ∈ E KB (set of KB entities) and p ∈ R KB (set of KB relations), and (ii) a set of instances (s , p , o ) from OpenIE extraction (s and o may not belong to E KB , and p are text predicates) For example, given E1 and E2 as OpenIE extractions and background knowledge bases (KB) like IMDB, we want to predict "@film.actor" relation given E1 and "@film.directed by" relation given E2 as the target KB relations between the participating entities.

we want to perform this relation inference at instance-level, which can be different for different entities sharing the same predicate.


On a large dataset with over 10,000 free-form types, the graph-enhanced model equipped with an attention-based matching module is able to achieve a much higher recall score while maintaining a high-level precision.

it achieves a 15.3% relative F1 improvement and also less inconsistency in the outputs.


To effectively capture the underlying label correlations without access to known type structures, we propose a novel label-relational inductive bias, represented by a graph propagation layer that operates in the latent label space.

this layer learns to incorporate a label affinity matrix derived from global type co-occurrence statistics and word-level type similarities.


Instead of using an explicit graphical model, we enforce a relational bias on model parameters, which does not introduce extra burden on label decoding.

the task we consider takes a raw sentence C as well as an entity mention span M inside C as inputs, and aims to predict the correct type labels T m of M from a candidate type set T , which includes more than 10,000 free-form types.


In order to create meaningful edges using training data as the only resource, we utilize the type co-occurrence matrix: if two type t 1 and t 2 both appear to be the true types of a particular entity mention, we will add an edge between them.

we are using the co-occurrence statistics to approximate the pair-wise dependencies and the co-occurrence matrix now serves as the adjacent matrix.


To encode the neighbor information into each node's representation, we follow the propagation rule defined in Graph Convolution Network (GCN) (Kipf and Welling, 2016).

with the adjacent or co-occurrence matrix A, we define the following propagation rule on W o : Here T ∈ R d f ×d f is the transformation matrix and I N is an identity matrix used to add selfconnected edges.D is a diagonal degree matrix withD ii = jÃ ij , which is used to normalize the feature vectors such that the number of neighbors does not affect the scale of transformed feature vectors.


In this paper, we present an effective method to impose label-relational inductive bias on finegrained entity typing models.

we utilize a graph convolution layer to incorporate type co-occurrence statistics and word-level type similarities.


To better understand the behavior of our system, we perform a case study on manually-generated event pairs.

for a given pair of event mentions, we first alter only one of the arguments and keep the rest of the content fixed.


We further propose to improve this approach to alleviate the forgetting problem by anchoring the sentence embedding space.

we utilize an explicit alignment model to mitigate the sentence embedding distortion of the learned model when training on new data and new relations.


To achieve this, we propose an alignment model that explicitly anchors the sentence embeddings derived by the neural model.

the alignment model treats the saved data from previous tasks as anchor points and minimizes the distortion of the anchor points in the embedding space in the lifelong relation extraction.


At the same time, we require the model f learned after step k could still perform well on the previous k − 1 tasks.

we evaluate the model by using the average accuracy of k tasks at each step as 1 k k j=1 acc f,j .


After training on each task k, EMR selects several training examples to store in the memory M, denoted as M T To handle the scalability, EMR stochastically replays the memory.

when training on task k with each mini-batch D replay .


The key idea of GEM (Lopez-Paz and Ranzato, 2017) is to constrain the new task learning with previous task data stored in memory.

it constrains the gradients during training with the following operation.


Also, deep networks are highly non-linear models, and the parameter dimensions have complex interactions, making the Euclidean space of parameters not a proper delegate of model behavior (French, 1999).

a slight change in parameter space could affect the model prediction unexpectedly.


So we propose to add a transformation a on the top of the original embedding and learn the basic model f and the transformation a automatically.

at the k-th task, we start with the model f (k−1) , and the transformation a (k−1) , that trained on the previous k − 1 tasks.


Zero-Shot Learning has been studied for a number of NLP tasks including event extraction (Huang et al., 2018;Lee and Jha, 2018;Srivastava et al., 2018), relation extraction (Liu et al., 2014), Conversational Language Understanding (Lee and Jha, 2018).

zero shot entity typing has also been explored, where most of the prior methods adopt the idea of learning a shared semantic space for representing the entities as well as the types, but differ in how they construct the type embeddings.


Figure 1 analyzes the relationship between the length of the Wikipedia description as one criterion of the description quality and the performance of Multi-rep method.

we group the types based on the length of their Wikipedia descriptions and provide the five-number summary box plot of the F-scores for each group.


were embedded into the center of the star, whereas the most specific sentences from a given author were placed into the rays of the star.

some sentences included character names, thus further research is required to mitigate this problem.


One issue that consistently appears is temporal inconsistency between the target and generated post-modifiers.

the model may make an error since it is unaware of the time period that the article is written in (and also may not be aware of the periods of time for which a claim are true).


F (•) refers to the overall coreference scoring function for each n regarding p. Following (Mitkov, 1998), all non-pronominal noun phrases in the recent three sentences of the pronoun p are selected to form N .

in our setting, we want to leverage both the local contextual information and external knowledge in this task, thus for each n and p, F (.)


The architecture of our model is shown in Figure 2, where we use two layers to incorporate contextual information and external knowledge.

the first layer takes the representations of different n and the p as input and predict the relationship between each pair of n and p, so as to compute F c .


We employ the Stanford parser 8 , which generates plurality, animacy, and gender markups for all the noun phrases, to annotate our data.

the plurality feature denotes each n and p to be singular or plural.