The question constitutes the direct context for the answer and dictates its information structure and information requirements to fulfill.

the type of wh-phrase (if present) of a question is a useful indicator of the type of required information: a who-question, such as 'Who rang the doorbell?


To explicitly tie answer properties to question properties, we explored different combinations of the features described above.

we encoded the current word's POS depending on the question form, and the current word's POS depending on the wh-word's POS.


Motivated by these observations, we aim to improve implicit discourse relation detection by making use of easily identifiable explicit discourse relations and taking into account global patterns of discourse relation distributions.

we add an extra CRF layer at the top of the softmax prediction layer (shown in figure 3) to fine-tune predicted discourse relations by considering their inter-dependencies.


In this paper, we investigate the use of discourse-aware rewards with reinforcement learning to guide a model to generate long, coherent text.

we propose to learn neural rewards to model cross-sentence ordering as a means to approximate desired discourse structure.


Here, the definition of coverage is similar in spirit with the conventional recall as both measure how many gold words are included in the output.

c part is essentially the same as recall with respect to its own cases.


In order to detect biased language, we developed subjectivity lexicons for the Portuguese language.

a linguist built a list of Portuguese lexicons based on the analysis of expressions that seem to express some subjectivity of the human evaluator.


Then, we discuss our evaluation procedure and report the results obtained with our debiasing approach.

our experiments aim to answer the following research questions: RQ1: How scores are distributed across the essays?


We implemented the different AES models using scikit-learn (Pedregosa et al., 2011).

we learn AES models using Support Vector Regression (SVR), Random Forests (RF), Logistic Regression (LR), Gradient Boosting (GB), and Multi-Layer Perceptron (MLP).


Each cluster contains ≈ 80% of the vectors associated with one of the misalignment levels inside the cluster.

20% of the essays will be removed from the training set (i.e., α = 0.2).


Overall, discussion about the quality of the ratings in automated essay scoring is lacking, and this was a central interest in this paper.

we create a subjectivity space from which potentially biased scores/ratings can be identified.


• Use Siamese Embeddings -For the majority of our experiments, we use a Siamese model (Bromley et al., 1993).

the textual embeddings for the query text and abstract share the same weights.


When obtaining nearest neighbors for negative examples during training, we use a heuristic to find a subset of the fetched nearest neighbors that are sufficiently wrong.

these are non-citation samples that look dissimilar in the original text but similar in the embedding space.


Machine Comprehension of natural language text is a fundamental challenge in AI and it has received significant attention throughout the history of AI (Greene, 1959;McCarthy, 1976;Reiter, 1976;Winograd, 1980).

in natural language processing (NLP) it has been studied under various settings, such as multiplechoice Question-Answering (QA) (Green Jr. et al., 1961), Reading Comprehension (RC) (Hirschman et al., 1999), Recognizing Textual Entailment (RTE) (Dagan et al., 2013) etc.


In this work, we propose a multi-sentence QA challenge in which questions can be answered only using information from multiple sentences.

we present MultiRC (Multi-Sentence Reading Comprehension) 1 -a dataset of short paragraphs and multi-sentence questions that can be answered from the content of the paragraph.


This removes the over-reliance of current approaches on answer-options and forces them to decide on the correctness of each candidate answer independently of others.

unlike previous work, the task here is not 1 http://cogcomp.org/multirc/ S3: Hearing noises in the garage, Mary Murdock finds a bleeding man, mangled and impaled on her jeep's bumper.


questions from paragraphs, or the reverse).

paragraphs' source could influence the complexity and diversity of the language of the paragraphs and questions, and hence the required level of reasoning capabilities.


In this step, we collect answer-options that will be shown with each question.

for each verified question from the previous steps, we ask 3 turkers to write as many correct and incorrect answer options as they can think of.


To investigate these further, we conducted a follow-up experiment in which manually annotated all questions that required coreference resolution into finer categories.

each question was shown to two annotators who were asked to select one or more of the following categories: entity coreference (between two entities), event coreference (between two events), set inclusion coreference (one item is part of or included in the other) and 'other'.


This baseline selects answer-options that best match sentences in a text corpus .

for each question q and answer option a i , the IR solver sends q + a i as a query to a search engine (we use Lucene) on a corpus, and returns the search engine's score for the top retrieved sentence s, where s must have at least one non-stopword overlap with q, and at least one with a i .


Since the solver is designed for multiple-choice with single-correct answer, we adapt it to our setting by running it for each answer-option.

for each answeroption, we create a single-candidate question, and retrieve a real-valued score from the solver.


As a neural baseline, we apply this solver by , which was originally proposed for SQuAD but has been shown to generalize well to another domain .

since BiDAF was designed for cloze style questions, we apply it to our multiple-choice setting following the procedure by : we score each answer-option by computing the similarity value of it's output span with each of the candidate answers, computed by phrasal similarity tool of Wieting et al.


Instead of training the LSTM T&N and LC models separately and then concatenating their output representations, we propose a framework where both models are trained jointly, and where the final network has then the capacity to predict AES scores and flag adversarial input (Figure 3).

the LSTM T&N and LC networks predict an essay and coherence score respectively (as described earlier), but now they both share the word embedding layer.


To detect whether an essay is adversarial, we further augment the system with an adversarial text detection component that simply captures adversarial input based on the difference between the predicted essay and coherence scores.

we use our development set to learn a threshold for this difference, and flag an essay as adversarial if the difference is larger than the threshold.


In the joint learning setup, we augment the original ASAP dataset with a subset of the synthetic essays.

we randomly select 4 permutations per essay to include in the training set, 11 but include all 10 permutations in the test set.


Computers can help humans edit text more efficiently.

statistical models are used for that purpose, for instance to help correct spelling mistakes (Brill and Moore, 2000) or suggest likely completions of a sentence (Bickel et al., 2005).


In this work, we rely on statistical learning to enable a computer to rephrase a sentence by only pointing at words that should be avoided.

we consider the task of reformulating either a sentence, i.e.


In addition to marking only a subset of the incorrect tokens at inference time, we also train new models for which the training data also only had a subset of incorrect tokens marked.

we train three models QE25, QE50, QE100 for which either 25%, 50% or 100% of incorrect guess tokens were marked.


Formally, the set of words similar to the word w is represented by the set Φ(w) shown in Equation 2, where S denotes the semantic relation between a pair of words.

the set Φ(w) depends on the definition of the semantic relation S between two words, which we will describe in Section 3.3.


To define the semantic relation S, we take into account a temporal context specified by a time window of 26 minutes as reported in (Lucchese et al., 2013).

if two queries belong to the same search session, as defined by a fixed length time window, then each constituent word pair within them is considered to be members of the set S. In real-life settings, even within a session of a specified time length, users often multi-task their activities (possibly by using multiple browser tabs or windows) (Lucchese et al., 2013;Wang et al., 2013;Mehrotra and Yilmaz, 2017).


In contrast, since we investigate cross-session task extraction, the time length threshold is not applicable to our annotation scheme, as a result of which, we re-annotated the task labelled dataset of (Lucchese et al., 2013).

our annotation scheme was solely based on the underlying search intent of the query.


when the semantic context used to learn the transformation matrix for the word vectors is restricted to sim- ilar queries within search sessions, the effectiveness improves further.

table 2 shows that both tempo and tempo-lexical transformations are able to improve recall significantly suggesting that the transformation helps to group more truly task-related queries into the same cluster.


We conduct experiments aimed at addressing three primary research questions: (1) How does each taxonomic organization algorithm perform?

how do DAG algorithms compare to tree-constrained ones, and how do transitive algorithms compare to their non-transitive counterparts?


The mapping q projects the multilingual words into the universal space based on their semantic similarity.

q(u|x) is a distribution based on the distance D s (u, x) between u and x as: where τ is a temperature and D(u i , x) is a scalar score which represents the similarity between source word x and universal token u i : where E K (u) is the "key" embedding of word u, E Q (x) is the "query" embedding of source word x.


In our case, we create one expert per auxiliary language.

we train to only use expert f i when training on a parallel sentence from auxiliary language i.


Assume the language 1...K − 1 are the auxiliary languages.

we have a multi-task objective as: We do not update the MoLE module for training on a sentence from the low-resource language.


This paper revisits several objective functions that have been commonly used for structured prediction tasks in NLP (Gimpel and Smith, 2010) and apply them to a neural sequence to sequence model (Gehring et al., 2017b) ( §2).

we consider likelihood training at the sequencelevel, a margin loss as well as expected risk training.


Label smoothing addresses this by acting as a regularizer that makes the model less confident in its predictions.

we smooth the target distribution with a prior distribution f that is independent of the current input x (Szegedy et al., 2015;Pereyra et al., 2017;Vaswani et al., 2017).


The motivation is to maintain good token-level accuracy while optimizing on the sequence-level.

a sample is processed with the sequence loss if the token loss under the current model is at least as good as the token loss of a baseline model L b TokLS .


Inspired by prior models (Koren et al., 2009;McAuley and Leskovec, 2013), we propose the following latent factor model to describe p u,c : γ U u and γ C c are K-dimensional latent vectors that encode topic-specific information (where K is the number of latent topics) for users and conversations.

γ U u reflects the topical interests of u, with higher value γ U u,k indicating greater interest by u in topic k. γ C c captures the extents that topics are discussed in conversation c. Similarly, D-dimensional vectors δ U u and δ C c capture discourse structures in shaping reply behaviors (where D is the number of discourse clusters).


For instance, a statement message may contain more content words than a rhetorical question.

x c,m,n ∼ M ulti(τ dc,m ), where τ d is a 3-dimension stochastic vector that expresses the appearing probabilities of three kinds of words (DISC, TOPIC, BACK), when the discourse assignment is d. Stop words and punctuations are forced to be labeled as discourse or background.


the parameters that give the lowest value for our objective are selected.

the number of discourse modes (D) and topics (K) are tuned to be 10.


To test the model performance based different levels of user engagement history, we further experiment with varying the length of conversations for training.

in addition to using 75% of conversation history, we also extract the first 25% and 50% of history as training.


While 41% arguments were categorized as abusive, other categories (tu quoque, circumstantial, and guilt by association) were found to be rather ambiguous with very subtle differences.

we observed a very low percentage agreement on these categories and a label distribution spiked around two or more categories.


The motivation for a topicincorporating model was based on our earlier observations presented in section 3.

we trained an LDA topic model (k = 50) (Blei et al., 2003) on the heldout OPs and during training/testing, we merged the estimated topic distribution vector with the output layer after convolution and pooling.


As our task at hand deals with multi-party discourse that presumably involves temporal relations important for the learned representation, we opted for a state-of-theart self-attentive LSTM model.

we re-implemented the Structured Self-Attentive Embedding Neural Network (SSAE-NN) (Lin et al., 2017) which learns an embedding matrix representation of the input using attention weights.


In this paper, we study the problem of parsing structured knowledge graphs from textual descriptions.

we consider the scene graph representation (Johnson et al., 2015) that considers objects together with their attributes and relations: this representation has been proved useful across a variety of vision and language applications.


We show that our carefully customized dependency parser is able to generate high quality scene graphs by learning from data.

we use the Visual Genome dataset (Krishna et al., 2017), which provides rich amounts of region description -region graph pairs.


parsed scene graphs are used in Schuster et al.


Instead of running two separate stages, our work proposed to use a single dependency parser that is end-to-end.

our customized dependency parser generates the scene graph in an online fashion as it reads the textual description once from left to right.


In recent years, dependency parsing with neural network features (Chen and Manning, 2014;Dyer et al., 2015;Cross and Huang, 2016;Kiperwasser and Goldberg, 2016;Dozat and Manning, 2016;Shi et al., 2017) has shown impressive performance.

kiperwasser and Goldberg (2016) used bidirectional LSTMs to generate features for individual words, which are then used to predict parsing actions.


Second, not every word in the sentence will be (part of) a node in the scene graph (see Figure 2 for an example).

some words in the sentence may not have a parent word, which violates the dependency parsing setting.


We use a slightly modified version of SPICE score (Anderson et al., 2016) to evaluate the quality of  scene graph parsing.

for every region, we parse its description using a parser (e.g.


Our method takes a slightly different approach: instead of predicting actual images, we train a deep recurrent neural network to predict the latent feature representation of images.

we are specifically interested in the semantic content of visual representations and how useful that information is for learning sentence representations.


Recent work has ranged from trying to learn to compose word embeddings (Le and Mikolov, 2014;Pham et al., 2015;Wieting et al., 2016;Arora et al., 2017), to neural architectures for predicting the previous and next sentences  or learning representations via largescale supervised tasks (Conneau et al., 2017).

skipThought  led to an increased interest in learning sentence representations.


Multi-modal semantics Language grounding in semantics has been motivated by evidence that human meaning representations are grounded in perceptual experience (Jones et al., 1991;Perfetti, 1998;Andrews et al., 2009;Riordan and Jones, 2011).

despite ample evidence of humans representing meaning with respect to an external environment and sensorimotor experience (Barsalou, 2008;Louwerse, 2008), standard semantic models rely solely on textual data.


In the first method, we try to predict the image features (Cap2Img).

we learn to map the caption to the same space as the image features that represent the correct image.


To learn sentence representations, we employ a bidirectional LSTM architecture.


To exploit contextual information in both input directions, we process input sentences using a bidirectional LSTM, that reads an input sequence in both normal and reverse order.

for an input sequence x of length T, we compute the hidden state at time t, h t ∈ R 2m via Here, the two LSTMs process x in a forward and a backward order, respectively.


Since GloVe representations are not tuned to represent grounded information, we learn a global transformation of GloVe space to grounded word space.

let x ∈ R n be the original GloVe embeddings.


To ground captions in the images that they describe, we map h T into the latent space of image representations such that their similarity is maximized.

we aim to predict the latent features of an image from its caption.


By jointly training the BiLSTM with these latent projections, we can then ground the language model in its visual counterpart.

let =1 be the parameters of the BiLSTM as well as the projection layers.


(2016), leading to multimodal sentence representations as described in Section 3.5.

we concatenate Cap2Cap, Cap2Img or Cap2Both and Skip-Thought with Layer Normalization (ST-LN) representations, yielding GroundSent-Cap, GroundSent-Img and GroundSent-Both representations, respectively.


Inspired by this converging evidence, the present work proposes a computational framework to explore various quantification tasks in the visual domain (see Figure 1).

we investigate whether ratio-based quantification tasks can be modeled by a single, multi-task learning neural network.


(Collobert et al., 2011;Søgaard and Goldberg, 2016;Bingel and Søgaard, 2017).

søgaard and Goldberg (2016) showed the benefits of keeping low-level tasks at the lower layers of the network, a setting which enables higher-level tasks to make a better use of the shared representations.


Tasks (a) and (c) are operationalized as classification problems and evaluated through accuracy.

only one answer out of 3 and 17, respectively, is considered as correct.


The intuition is that solving the lower-level tasks would be beneficial for tackling the higher-level ones.

providing a proportional estimation ('80%') after performing vagueQ ('most') and setComp ('more') should lead to a higher accuracy in the highest-level task, which represents a further step in complexity compared to the previous ones.


These probability distributions had been collected by asking participants to select, from a list of 9 quantifiers (reported in § 3.1), the most suitable one to describe the target objects in a visual scene presented for 1 second.

they were computed against the proportion of targets in the scene, which in that study was shown to be the overall best predictor for quantifiers.


The frozen representation of the scene had been previously extracted using the state-of-art Inception v3 CNN (Szegedy et al., 2016) pretrained on ImageNet (Deng et al., 2009).

the network is fed with the average of the features computed by the last Convolutional layer, which has size 25*2048.


The model has a core structure, represented by layers 1-5 in the figure, which is shared across tasks and trained with multiple outputs.

(a) layers 1, 2, and 3 are trained using information regarding the output of all 3 tasks.


In particular, (a) layers 1, 2, and 3 are trained using information regarding the output of all 3 tasks.

these layers are updated three times by as many backpropagation passes: One on the top of setComp output, the second on the top of vagueQ output, the third on the top of propTarg output; (b) layers 4 and 5 are affected by information regarding the output of vagueQ and propTarg, and thus updated twice; (c) layers 6 and 7 are updated once, on the top of the output of propTarg.


In setComp, all the models are neatly above chance/majority level (0.47).

the one-task-end2end model achieves a remarkable 0.90 acc., which is more than 10% better compared to the simple one-task-frozen model (0.78).


To shed light on this issue, we tested the multi-task-prop model after reversing its architecture.

propTarg is now the first task, followed by vagueQ, and setComp.


In contrast with the pattern of results obtained by the original pipeline, no benefits are observed for this version of MTL model compared to one-task networks.

both vagueQ (0.32 r) and propTarg (0.08 acc.)


The multi-task-prop model, in contrast, shows a fairly high generalization power.


Through careful analysis of the results attained by state-of-the-art learning models and human annotators on existing datasets, we show that the design of the decoy answers has a significant impact on how and what the learning models learn from the datasets.

the resulting learner can ignore the visual information, the question, or both while still doing well on the task.


We conduct extensive empirical and human studies to demonstrate the effectiveness of our procedures in creating high-quality datasets for the Visual QA task.

we show that machines need to use all three information (image, questions and answers) to perform well -any missing information induces a large drop in performance.


We propose automatic procedures to revise VQA and Visual7W (and to create new datasets based on COCOQA (Ren et al., 2015), VQA2 (Goyal et al., 2017), and Visual Genome) such that the decoy generation is carefully orchestrated to prevent learning algorithms from exploiting the shortcuts in the datasets by overfitting on incidental statistics.

our design goal is that a learning machine needs to understand all the 3 components of an image-question-candidate answers triplet in order to make the right choiceignoring either one or two components will result in drastic degradation in performance.


We have also discovered that the targets (i.e., correct answers) are infrequently used as decoys.

among the 69,817 training samples, there are 19,503 unique correct answers and each one of them is used about 3.6 times as correct answers to a question.


The decoys need to be plausible to the image.

they should appear in the image, or there exist questions so that the decoys can be treated as targets to the image.


Effectiveness of new decoys A better set of decoys will force learning models to integrate all 3 pieces of information -images, questions and answers -to make the correct selection from multiple-choices.

they should prevent learning algorithms from exploiting shortcuts such that partial information is sufficient for performing well on the Visual QA task.


The main idea is to parse each sentence in the input corpus and to use the dependency tree to derive the co-occurrence count.

for a target word w with modifiers m 1 , ..., m o and a head h, we considered the dependency labels , where lbl is the type of the dependency relation between the head and the modifier and lbl −1 is used to mark the inverse relation.


Before introducing the attr2vec model we briefly describe the GloVe factorization approach (Pennington et al., 2014).

gloVe employs a matrix factorization model using as input the word-word co-occurrence count.


The model we propose, attr2vec, employs a matrix factorization model based on factorization machines.

we associate with each variable v ∈ V a bias term b v ∈ R and a latent factor vector f v ∈ R d , where the dimensionality d of the latent factor space is a hyperparameter of the model.


Our evaluation closely follows the one in .

we used the WordSim353 dataset (Finkelstein et al., 2001;Agirre et al., 2009) containing pairs of similar words that reflect either relatedness (topical similarity) or similarity (functional similarity) relations.


We trained 200-dimensional attr2vec embeddings using news topics as additional contextual information and a linear bag-of-words approach -i.e., each row in the feature matrix consists of a pair of words and the topic of the news article where such pair has been observed (see the first and the last group of columns for the example in Figure 1).

we used the same collection of ∼8M news articles presented in Section 4.1 and we considered the following two article topics: general news stories (G) and sport news (SP O).


Figure 4: Two-dimensional projection of the 200dimensional vector space, that contains representations of both words and topics, using t-SNE.

we considered two topics: general news stories (G) and sport news (SP O).


Moreover, attr2vec can mimic existing word embedding algorithms when no additional contextual information is considered.


We train our model using a modified version of the SGNS loss function.

our positive samples are now triples (w, c, t) where w is a target word, c is a context word, and t is a time, instead of pairs (w, c) which are typically used in SGNS.


For every tuple where r 2 is the target word, we replace the target word with r 1 •r 2 and we multiply the frequency by 1 − shift(t; r 1 •r 2 ).

in the modified corpus, r 1 •r 2 has shift(t; r 1 •r 2 ) percent of r 1 's contexts at time t and 1 − shift(t; r 1 •r 2 ) percent of r 2 's contexts at time t. We train a model mod on this modified train-ing data.


log( P (w,c) P (w)•P (c) ) < log(k f )).

we set #(w, c) = 0 in the objective function.


This paper presents a corpus and experiments to mine possession relations from text.

we target alienable and control possessions, and assign temporal anchors indicating when the possession holds between possessor and possessee.


Alienable and control possessions, however, have opposite temporal anchors for before and after.

most alienable possessions are true before and after verb x (69.8% and 92.6% respectively), and most control possessions are not true before and after verb x (71.2% and 66.7%).


we extract alienable and control possessions, and specify temporal anchors with respect to the verb of which the possessor is the subject.


The sentence discourse vector can hence be obtained by subtracting c 0 estimated by the first principal component ofc s 's on a set of sentences.

the sentence embeddings are obtained by a weighted average of the word vectors stripping away the syntax by subtracting the common discourse vector and down-weighting frequent tokens.


We investigate this problem by learning domain-specific representations of input sentences using neural network.

a descriptor vector is learned for representing each domain, which is used to map adversarially trained domaingeneral Bi-LSTM input representations into domain-specific representations.


We address this issue by investigating a model that learns domain-specific input representations for multi-domain sentiment analysis.

given an input sentence, our model first uses a bidirectional LSTM to learn a general sentence-level representation.


Since DSR-at has m outputs (one for each training domain), we adopt an ensemble approach to obtain a single output for unknown test domains.

since the domain classifier outputs probabilities on how likely the test data come from each training domain, we use these probabilities as weights to average the m outputs.


The final objective function can be reformulated as: As can be seen, Eq.9 consists of two parts balanced by the trade-off parameter λ.

the first part corresponds to the maximum margin of the learning system and it can also represent the complexity of the learning system, while the second part corresponds to the emotion loss function of the learning system implemented in hinge form.


We set the number of epochs and patience value as 100 & 20 respectively.

we run the experiments for maximum 100 epochs and if validation loss does not reduce for consecutive 20 epochs training stops and reports the best epoch attained so far.


6 Larger improvements are visible for proportional F1 score than for binary F1 score.

more data and SRL knowledge helps the model to better annotate the scope of opinion roles.


Subsampling Monolingual Data To simulate low-resource behavior along another axis, we subsample the monolingual corpora used by BISPARSE-DEP to induce monolingual vectors, X e , X f .

we learn X e and X f using progressively smaller corpora.


Further analysis revealed that the good performance of the delexicalized model is due to the relative robustness of the delexicalized parser on frequent contexts in the co-occurrence matrix.

we found that in French and Russian, the most frequent contexts were derived from amod, nmod, nsubj and dobj edges.


Many times we fail to capture the temporal orientation of a text using just the tense information or the existing temporal keywords.

the tweet "Today I have a plan for a meeting at night."


Results in Table 3 demonstrate that any user's past-orientation is significantly correlated (0.4677) with their age.

it suggests that when people age they think more about the past than the present and future.


These variations were tested to ensure the observed patterns reported in the experiments, but we report numerical results only for best performing models.

higher dimensional vectors with dim=300 produced consistently better alignment with human scoring data.


Variable n indicates the total number of guesses a model is allowed to make when responding to a given cue word.

n is the size of the subspace explored around the cue word within each distributional semantic space.


We examined our proposal using a recently published dataset of production norms (Jouravlev and McRae, 2016) and confirmed when people were explicitly asked to recall thematically related words, their responses were more likely located within the context embedding space in the vicinity of the cues word embedding.

wC, where w is the word embedding of the cue and C is the context embedding of the response, best measures forward thematic relatedness.


In this paper, we propose to use entities found in the original text to infer the summary topic, miti-gating the aforementioned problem.

we leverage on linked entities extracted by employing a readily available entity linking system.


For this purpose, we employ an RNNbased model to globally disambiguate the entities.

we use BiGRU and concatenate the forward and backward hidden state vectors as the new entity vector: Locally disambiguating encoder Another way to disambiguate an entity is by using only the direct neighbors of the entity, putting no importance value to entities that are far.


To do this, we employ a CNN-based model to locally disambiguate the entities.

we do the convolution operation using filter matrices W f ∈ R h×d with filter size h to a window of h words.


Extracted from search and social media metadata between 1998 and 2017, these high-quality summaries demonstrate high diversity of summarization styles.

the summaries combine abstractive and extractive strategies, borrowing words and phrases from articles at varying rates.


In this paper, we explore the idea of polyglot semantic translation, or learning semantic parsing models that are trained on multiple datasets and natural languages.

we focus on translating text to code signature representations using the software component datasets of Richardson and Kuhn (2017a,b).


Table 8 shows that we can achieve similar separation between implicatives and nonimplicatives as the feature mining strategy presented in §5.

those features may be redundant with information already learnable from factuality datasets (UDS-IH2).


To appropriately handle the semantic variety of entities/relations in distinct triples, we propose an accurate text-enhanced knowledge graph representation learning method, which can represent a relation/entity with different representations in different triples by exploiting additional textual information.

our method enhances representations by exploiting the entity descriptions and triplespecific relation mention.


Concretely, we employ BiLSTM model (Schuster and Paliwal, 1997;Graves and Schmidhuber, 2005) with mutual attention mechanism  to learn representations for relation mentions and entity descriptions.

in order to generate triple-specific textual representation of entities and relation, a mutual attention mechanism is proposed to model relation between entity descriptions and relation mention of one triple.


To gain more insight, we present a failure analysis to explore possible limitations and weaknesses of our model.

several illustrative triples from the test set of FB15K are listed in Table 4.


There is one critical reason that the word-toword axiom injection described in Section 3.2 fails to detect phrase-to-phrase correspondences.

the natural deduction mechanism decomposes the goal G into atomic sub-goals that are then proved one-by-one (word-by-word), independently of each other except for the variable unification effect.


The corresponding subgraph in P to each of these partitions is given by the vertices and edges connected with a path of length one to the unified variables that appear in Reach(v).

g denotes the vertices in the subgraph of g induced by the partition Reach(v).


paraphrases we extracted from the natural deduction proof in the training set.

the examples of verb phrases show our method has the potential to capture long paraphrases.


Experiments on standard benchmark datasets show that SGTB significantly improves upon published results.

sGTB outperforms the previous state-of-the-art neural system by near 1% absolute accuracy on the popular AIDA-CoNLL dataset.


To capture the non-linear relationships between the low-dimensional dense features like statistical features, sophisticated machine learning models such as neural networks and gradient tree boosting are preferred over linear models.

gradient tree boosting has been shown to be highly competitive for ED in recent work (Yang and Chang, 2015;Yamada et al., 2016).


We adopt beam search to address this problem, in which we keep track of multiple hypotheses and sum over the paths in the beam.

we propose Bidirectional Beam Search with Gold path (BiBSG) technique that is specifically designed for SGTB model training.


Therefore, we choose the following decomposition: where F (x, y t , y 1:t−1 ) is a factor scoring function.

a local prediction y t depends on all the previous decisions, y 1:t−1 in our model, which resembles recurrent neural network (RNN) models (Elman, 1990;Hochreiter and Schmidhuber, 1997).


The key idea of SGTB is that, instead of defining a parametric model and optimizing its parameters, we can directly optimize the factor scoring function F (•) iteratively by performing gradient descent in function space.

suppose F (•) = F m−1 (•) in the m-th iteration, we will update F (•) as follows: where is the functional gradient, η m is the learning rate, and 1 [•] represents an indicator function, which returns 1 if the predicted sequence matches the gold one, and 0 otherwise.


Following previous work (Guo and Barbosa, 2016; Ganea and Hofmann, 2017), we evaluate our models on both in-domain and cross-domain testing settings.

we train our models on AIDA-train set, tune hyperparameters on AIDAdev set, and test on AIDA-test set (in-domain testing) and all other datasets (cross-domain testing).


As shown, SGTB achieves much better performance than all previously published results.

sGTB-BiBsG outperforms the previous sOTA system (Ganea and Hofmann, 2017) by 0.8% accuracy, and improves upon the best published results when employing the PPRforNED candidate selection system by 1.9% accuracy.


Nevertheless, our SGTB models achieve better performance than best published results on three of the five popular ED datasets.

sGTB-Bs outperforms the prior sOTA system by absolute 4% F1 on the CWEB dataset, and sGTB-BiBsG performs consistently well across different datasets.


we did a coarse grid search over a parameter space for κ 1 , κ 2 , κ 3 , r, and thres.


The overall performance (micro-precision, micro-recall, micro-F1) of the systems is tested upon 21 different test cases.

we summed up the individual true positives, false positives and false negatives based on the system results for the different ontology matching tasks and, in the next step, we computed the performance metrics.


The group recurrent layer we proposed is equivalent to the standard recurrent layer with a blockdiagonal sparse weight matrix.

our model employs a uniform sparse structure which can be computed very efficiently.


layer which adopts a group strategy to approximate the standard recurrent layer.

we consider to split both the input x t and hidden state h t into K disjoint groups as i t represent the input and hidden state for i-th group at time step t. Based on this split, we then perform recurrent computation in every group independently.


For En-De translation, we use a widely adopted dataset (Jean et al., 2015;Wu et al., 2016).

part of data in WMT'14 is used as the training data, which consists of 4.5M sentences pairs.


From Table 2, we can observe that on De-En task, Group RNNSearch models achieve comparable or better BLEU score compared with the RNNSearch but with much less number of parameters.

with group number 2 and 4, we achieve about 28% and 43% parameter reduction of recurrent layers respectively.


We use global statistics in the relation graph to train the embedding model.

we model the semantics of a textual relation as its cooccurrence distribution of KB relations, and learn textual relation embedding to reconstruct the corresponding co-occurrence distributions.


The number of contextual sentences positively supporting a relational fact is also an important signal, but is lost in mean pooling.

instead, we use summation with a trainable cap: we additively aggregate the signals from all the contextual sentences, but only to a bounded degree.


Under manual evaluation, PCNN+ATT+GloRE achieves the best performance in the full range of N .

for the top 1,000 predictions, GloRE improves the precision of the previous best model PCNN+ATT from 83.9% to 89.3%.


We can flexibly adapt our method for training data generation to this case.

we create extra negative training events, in which an argument of the positive event has been moved to another argument position in the same event, as shown in Figure 6.


When generating TEMPROB, we need to process a large number of articles, so we adopt the greedy inference strategy described earlier due to its computational efficiency Mirza and Tonelli, 2016).

we apply the same-sentence relation classifier before the neighboring-sentence relation classifier; whenever a new relation is added in this article, a transitive graph closure is performed immediately.


We first show extreme cases that some events are almost always labeled as T-Before or T-After in the corpus.

for each pair of verbs v i , v j ∈ V , we define the following ratios: where where I {•} is the indicator function.


Here we quantify the "correctness" of η b and η a based on TBDense.

we collected all the gold T-Before and T-After pairs.


3.2.1 as the baseline, and added the prior distribution obtained from TEMPROB on top of it.


4.2.1, the prior distributions from TEMPROB can also be used to regularize the conventional ILP formulation.

in each document, let I r (ij) ∈ {0, 1} be the indicator function of relation r for event i and event j; let x r (ij) ∈ [0, 1] be the corresponding softmax score obtained from the local classifiers (depending on the sentence distance between i and j).


Word embeddings are obtained from an unsupervised learning model that learns co-occurrence statistics of words from a large external corpus, yielding word embeddings as distributional semantics (Mikolov et al., 2013).

we use pre-trained embeddings from GloVE (Pennington et al., 2014).


We express our structured prediction problem such that it corresponds to building a hypergraph that encodes the token-level gold labels for all entities in the input sentence.

3 we represent the problem as a directed hypergraph.


(2) The word and preposition representations via tensor decomposition are simple features leading to a standard classifier.

we do not use dependency parsing (which many prior methods have relied on) or handcrafted features (Prokofyev et al., 2014) or train task-specific representations on the annotated training dataset (Belinkov et al., 2015).


Tensor decomposition: We combine standard ideas from word embedding algorithms and tensor decomposition algorithms to arrive at the lowrank approximation to the tensor log(1 + X).


Weighted Decomposition (WD): Based on ideas from the literature on word embedding algorithms, we also consider weighting different elements of the tensors differently in order to reduce the effect of the large dynamic range of the tensor values.

we employ the GloVe objective function to our tensor model and minimize the objective function (2): where b U i is the scalar bias for the word i in the matrix U .


We note that the proposed tensor embeddings achieve the best performance among all approaches.

the tensor with weighted decomposition has the highest F1 score on the CoNLL dataset-a 6% improvement over the stateof-the-art.


The precision-recall analysis shows that the predicted marginal probabilities are meaningful in a ranking sense, but we can also ask whether they are meaningful in a sense of calibration: predictions are calibrated if, among all structures with predicted probability q ± , they exist in the gold parses with probability q.

predictions with confidence q have precision q.


20 (If they are not directly connected, the model instead conditions on a 'no edge' feature.)

probabilities are maximum likelihood estimates from the training data's (predicate, argument label, path) counts, from either greedy 19 for each unique (lemma, framesetID) pair, such as (view, view-02).


The belief state is the posterior probability distribution over the underlying state u t of the system.

s t deterministically keeps track of all possible states that the OOHMM might be in-just as the state of a determinized FSA keeps track of all possible states that the original nondeterministic FSA might be in.


In this paper, we simply try to tune φ to yield good proposals.

we try to ensure that q φ (y | x) in equation 27is close to p(y | x) from equation (1).


For example, we might try to ensure thatp closely matches the model's distribution p (equation (28))-the "na-guarantee it (assuming that the model p is correct).

we seek to minimize (1 − λ)KL(p||q φ ) + λKL(q φ ||p) (with λ ∈ [0, 1]) (32) averaged over examples x drawn from a training set.


Nonetheless, we try to use a reasonable estimate so that the reported KL divergence is interpretable in an absolute sense.

we take z(x) = log y∈Yp (y | x) ≤ log Z, where Y is the full set of distinct particles y that we ever drew for input x, including samples from the beam search models, while constructing the experimental results graph.


18 We would get the same gradient for any rescaled version of the unnormalized distributionp, but the formula for obtaining that gradient would be different.

if we rewrite the above derivation but add a constant b to both logp(y) and log Z throughout (equivalent to adding b to G T ), we will get the slightly generalized expectation formulas in place of equations (43) and (44).


We observed that the character-level BiLSTM outperformed the CNN by 0.1% on the PTB-WSJ development set, and hence in all of our experiments we use the character-level BiLSTM.

we generate a character-level representation for each word by feeding its character embeddings into the BiLSTM and obtaining the concatenated final states.


With this motivation, we apply a conditional random field (CRF) (Lafferty et al., 2001) on top of the word-level BiLSTM to perform POS tag inference with global normalization, addressing the "label bias" problem.

given an input sentence, we pass the output sequence of the word-level BiLSTM to a firstorder chain CRF to compute the conditional probability of the target label sequence: where θ represents all of the model parameters (in the BiLSTMs and CRF), s and y denote the input embeddings and the target POS tag sequence, respectively, for the given sentence.


Given a classifier, we first generate input examples that are very close to original inputs (so should yield the same labels) yet are likely to be misclassified by the current model.

these adversarial examples are generated by adding small perturbations to the inputs in the direction that significantly increases the loss function of the classifier (worstcase perturbations).


As can be seen from the table, for both languages, adversarial training (AT) results in cleaner word embedding distributions than the baseline, with a higher cosine similarity within each POS cluster, and with a clear advantage in the average tightness across all the clusters.

the learned word vectors show stronger correlations with their POS tags.


With this change in place, let s best (i, j) denote the score of the best subtree spanning (i, j).

for spans of length one, we need only consider the choice of label: for general spans (i, j), we have the following recursion: we can independently select the best label for the current span and the best split point, where the score of a split is the sum of the best scores for the corresponding subtrees.


Since ∆ decomposes over spans, the inner lossaugmented decode max T [s(T ) + ∆(T, T * )] can be performed efficiently using a slight modification of the dynamic program used for inference.

we replace s , where * ij is the label of span (i, j) in the gold tree T * .


One possible explanation is that the neural machinery, in particular the LSTM, is handling much of the reconciliation between labels that was previously handled by an inference procedure.

instead of using local information to suggest several brackets and letting the grammar handle interactions between them, the LSTM may be making decisions about brackets already in its latent state, allowing it to use the result of these decisions to inform other bracketings.


This work exploits translation data as a source of semantically relevant learning signal for models of word representation.

we exploit equivalence through translation as a form of distributional context and jointly learn how to embed and align with a deep generative model.


Training data We train our model on bilingual parallel data.

we use parliament proceedings (Europarl-v7) (Koehn, 2005) from two language pairs: English-French and English-German.


First, we apply a context distribution smoothing mechanism to enlarge the probability of sampling a rare context.

all context counts are scaled to the power of α.


With the dataset in hand, we then propose the novel task of studying persuasion (1) at the level of individual users, and (2) in a setting that can control for selected user-level factors, in our case, the prior beliefs associated with the political or religious ideology of the debaters and voters.

previous studies focus on predicting the winner of a debate based on the cumulative change in pre-debate vs. post-debate votes for the opposing sides (Zhang et al., 2016;Potash and Rumshisky, 2017).


Similar to the setting described above, Task 2 controls for political ideology.

we only use debates where the two debaters are from different political ideologies (CONSERVATIVE vs. LIBERAL).


(2017) is the most relevant work to ours since they consider features of the audience on persuasion.

they studied the effect of an individual's personality features (open, agreeable, extrovert, neurotic, etc.)


In this paper, we employ advancements in extrapropositional semantics extraction within NLP to study how author commitment reflects the social context of an interactions.

we investigate whether the level of commitment expressed by individuals in an organizational interaction reflects the hierarchical power structures they are part of.


We use the committed belief analysis framework introduced by (Diab et al., 2009;Prabhakaran et al., 2015) to model different levels of beliefs expressed in text.

in this paper, we use the 4-way belief distinction -COMMITTED-BELIEF, NONCOMMITTEDBELIEF, REPORTED-BELIEF, and NONAPPLICABLE-introduced in (Prabhakaran et al., 2015).


We found evidence that subordinates use significantly more propositional hedges than superiors, and that superiors and subordinates use significantly different proportions of different types of beliefs in their messages.

subordinates use significantly more non-committed beliefs than superiors.


Prior works (Iwata et al., 2010;Pruteanu-Malinici et al., 2010;Saha and Sindhwani, 2012;Schein et al., 2016) have investigated Bayesian modeling of topics in time-stamped documents.

blei and Lafferty (2006) developed a LDA based dynamic topic model (DTM) to capture the evolution of topics in a time sequence collection of documents; however they do not capture explicitly the topic popularity and usage of specific terms over time.


For each model, we apply a document classifier trained on the model parameters to test whether CNPMI is consistent with classification accuracy.

we want our classifier to transfer information from training on one language to testing on another (Smet et al., 2011;Heyman et al., 2016).


Succeeding at NLI does not require a system to solve any difficult machine learning problems except, crucially, that of extracting effective and thorough representations for the meanings of sentences (i.e., their lexical and compositional semantics).

a model must handle phenomena like lexical entailment, quantification, coreference, tense, belief, modality, and lexical and syntactic ambiguity.


Most mistakes involve concept identification.

relevant words are often erroneously ignored by the parser.


For parsing, since the derivation tree in a lexicalized TAG is a type of dependency tree (Rambow and Joshi, 1994), we can directly apply dependency parsing models.

we use the biaffine graph-based parser proposed by  together with our novel techniques for supertagging.


A highway connection is a gating mechanism that combines the current and previous layer outputs, which can prevent the problem of vanishing/exploding gradients (Srivastava et al., 2015).

in networks with highway connections, we replace Eq.


We first perform unlabeled arc-factored scoring using the final output vectors from the BiLSTMs, and then label the resulting arcs.

suppose that we score edges coming into the ith word in a sentence i.e.


Then, the probability distribution i over the possible dependency relations of the arc pointing from the p i th word to the ith word is calculated by: where We generally follow the hyperparameters chosen in .

we use BiLSTMs layers with 400 units each.


At the other end, NOUN [PP] VERB (participial) and NOUN [PP] ADVERB ADJ are difficult.

in the nonce condition, where semantics is unhelpful or even misleading, the target could easily be interpreted as a modifier of the noun embedded in the preceding prepositional phrase.


(2) meta-testing: the trained metrics in M is applied to meta-testing tasks denoted as where each T 0 i is a fewshot learning task consisting of both training and testing data as is a small labeled set for generating the prediction model M 0 i for each T 0 i .

m 0 i s are kNN-based predictors built upon the metrics in m. We will detail the construction of m 0 i in Section 3, Eq.


To address the aforementioned challenges, we propose a novel task clustering algorithm based on the theory of matrix completion (Candès and Tao, 2010).

we deal with the huge number of entries by randomly sample task pairs to evaluate the S ij and S ji scores.


Score Filtering First, we use only reliable task pairs to generate a partially-observed similarity matrix Y.

if S ij and S ji are high enough, then it is likely that tasks {i, j} belong to a same cluster and share significant information.


Then, based on these cluster centers and using l 1 -norm, we found a one-to-one association of each word to the closest cluster center in the word embedding space.

for each word v i , we can find an associated cluster center or meta-word c u with u ∈ {1, 2, .


We denote this mapping by c u = closest(v i , C), where closest returns the centroid in C with the minimum distance to v i .

finally, the BoC k representation for each document where c corresponds to each of the k centroids and n = |{v BoC k (d j ) corresponds to a histogram of centroid frequencies, where each pair (c , n ) represents a centroid (meta-word) and its corresponding frequency in the document.


We introduce the MAN architecture in §2 and prove in §3 that it directly minimizes the (generalized) f-divergence among multiple distributions so that they are indistinguishable upon successful training.

for MDTC, MAN is used to overcome the aforementioned limitation in prior art where domain-specific features may sneak into the shared model.


In this paper, we strive to tackle the text classification problem in the real-world setting in which texts come from a variety of domains, each with a varying amount of labeled data.

assume we have a total of N domains, N 1 labeled domains (denoted as L ) for which there is some labeled data, and N 2 unlabeled domains ( U ) for which no annotated training instances are available.


In this section, we present a theoretical analysis showing the validity of MAN.

we show that MAN's objective is equivalent to minimizing the total f-divergence between each of the shared feature distributions of the N domains, and the centroid of the N distributions.


Nonetheless, we adopt the same dataset for fair comparison and employ a MLP as our feature extractor.

we take the 5000 most frequent features and represent each review as a 5000d feature vector, where feature values are raw counts of the fea-tures.


In this paper we present the Pivot Based Language Model (PBLM), a representation learning model that marries together pivot-based and NN modeling in a structure aware manner.

our model processes the information in the text with a sequential NN (LSTM) and its output consists of a context-dependent representation vector for every input word.


We present (Section 3) the Pivot Based Language Model (PBLM) -a domain adaptation model that (a) is aware of the structure of its input text; and (b) outputs a representation vector for every input word.

the model is a sequential NN (LSTM) that operates very similarly to LSTM language models (LSTM-LMs).


1 PBLM learns the connection between easy -an adjective that is often used to describe kitchen appliances, but not books -and great.

pBLM is able to learn the connection between witty and easy which will facilitate adaptation between the books and kitchen appliances domains.


Interestingly, in the product domains unlabeled reviews tend to be much more positive than in the airline domain.

in the B domain there are 6.43 positive reviews on every negative review; in D the ratio is 7.39 to 1; in E it is 3.65 to 1; and in K it is 4.61 to 1.


From the decomposition in Equation 4, it seems that the outputs of an RNN with the nCARNN unit can be efficiently computed in parallel.

we can compute the weight for each input in parallel, and take their weighted sum to produce any desired hidden vector output.


(2017); Bordes and Weston (2017); Liu and Perez (2017)).

the systems have to produce complete responses and learn the dialog behaviour solely from the ground truth responses without help from manual features, rules or templates.


In this paper, we improve the model by incorporating multiple levels of granularity.

we propose (1) an encoder with character attention which augments the (sub)word-level representation with character-level information; (2) a decoder with multiple attentions that enable the representations from different levels of granularity to control the translation cooperatively.


In this paper, we propose a simple, fast, and effective method for recalling previously seen translation examples and incorporating them into the NMT decoding process.

for an input sentence, we use a search engine to retrieve sentence pairs whose source sides are similar with the input sentence, and then collect n-grams that are both in the retrieved target sentences and aligned with words that match in the source sentences, which we call "translation pieces".


A number of methods have been proposed to ameliorate these problems, including methods that incorporate symbolic knowledge such as discrete translation lexicons (Arthur et al., 2016;He et al., 2016;Chatterjee et al., 2017) and phrase tables (Zhang et al., 2017;Tang et al., 2016;Dahlmann et al., 2017), adjust model structures to be more conducive to generalization (Nguyen and Chiang, 2017), or incorporate additional information about domain (Wang et al., 2017a) or topic  in translation models.

one paradigm of interest is recent work that augments NMT using retrieval-based models, retrieving sentence pairs from the training corpus that are most similar to the sentence that we want to translate, and then using these to bias the NMT model.


In this paper, we propose a simple and efficient model for using retrieved sentence pairs to guide an existing NMT model at test time.

the model collects n-grams occurring in the retrieved target sentences that also match words that overlap between the input and retrieved source sentences, which we will refer to as "translation pieces" (e.g., in Figure 1, the blue part of the retrieved target sentence is collected as translation pieces for the input sentence).


The probability of generating next word y t is, 2 where z t is a decoding state for time step t, computed by, c t is a source representation for time t, calculated as, where α t,i scores how well the inputs around position i and the output at position t match, computed as, The standard decoding algorithm for NMT is beam search.

at each time step t, we keep n-best hypotheses.


(2017) which rewards n-grams from syntactic translation lattices during NMT decoding, we add an additional reward for n-grams that occur in the collected translation pieces.

as shown in Figure 2, at each time step t, we update the probabilities over the output vocabulary and increase the probabilities of those that result in matched n-grams according to where λ can be tuned on the development set and δ (•) is computed as Equation 8 if y t t−n+1 ∈ G X , otherwise δ (•) = 0.


In addition, their method runs the NMT model for each retrieved sentence pair to obtain the NMT encoding and decoding information of the retrieved sentences as key-value memory to guide the NMT model for translating the real input sentence, which changes the NMT model structure and increases both the training-time and test-time computational cost.

at test time, running the NMT model for one retrieved sentence pair costs the same time as translating the retrieved source sentence with beam size 1.


With this result in hand, we propose a method for more directly capturing contextual information that may help disambiguate difficult-to-translate homographs.

we learn from neural models for word sense disambiguation (Kalchbrenner et al., 2014;Iyyer et al., 2015;Kågebäck and Salomonsson, 2016;Yuan et al., 2016;Šuster et al., 2016), examining three methods inspired by this literature ( §4).


2 we show the translation accuracy of an NMT system with respect to words of varying levels of ambiguity.

we use the best baseline NMT system to translate three different language pairs from WMT test set (detailed in §6) and plot the F1-score of word translations by the number of senses that they have.


Bi-directional LSTM (BiLSTM) Kågebäck and Salomonsson (2016) leveraged a bidirectional LSTM that learns a context vector for the target word in the input sequence and predicts the word sense with a multi-layer perceptron.

we can compute the context vector c t for tth word similarly to bi-directional encoder as follows, are forward and backward LSTMs repectively, and f c (x t ) = M c 1(x t ) is a function that maps a word to continous embedding space.


Given the context vector from this language model, they predict the word sense with a WSD classifier.

we can compute the context vector c t for tth word by first replacing tth word with a special symbol (e.g.


5 Adding Context to NMT Now that we have several methods to incorporate global context regarding a single word, it is necessary to incorporate this context with NMT.

we propose two methods to either Gate or Concatenate a context vector c t with the Lookup embedding M e • 1(x t ) to form a context-aware word embedding before feeding it into the encoder as shown in Fig.


Such an objective poses a hidden danger to NMT models.

the model may generate the best candidate word for the current time step yet a bad component of the whole sentence in the long run.


Actually, the BR-CSGAN is a weighted sum of the naive GAN (λ=1) and MRT (λ=0), and it incorporates the advantages of the two approaches.

compared to naive GAN which is trained without specific objective guidance, BR-CSGAN utilizes the BLEU objective to guide the generator to generate sentences with higher BLEU points.


The deep encoder consists of multiple layers, where the hidden states in layer − 1 are the inputs to the hidden states at the next layer .

where is the hidden state of the 'th layer RNN encoder for the i'th source sentence word.


We share the parameters of the deep-level encoders and decoders of the auxiliary tasks with those of the main task.

where L m enc and L m dec specify the deep-layer RNNs need to be shared parameters.


The second term ensures that the parameters of the shared layers are trained so that they confuse the discriminator by maximising the entropy of its predicted distribution over the task identities.

we add the term L adv2 (Θ mtl ) to the training objective defined as: where H[.]


The parameters Θ are learned by training a sequence-to-sequence neural model on a corpus of parallel sentences.

the learning objective is to maximize the following conditional log-likelihood: The models typically use gated recurrent units (GRUs) (Cho et al., 2014) or LSTMs (Hochreiter and Schmidhuber, 1997).


On the contrary, the other two models distribute attention more evenly among all previous words.

the self-attentive RNN uses a longer context than the self-attentive residual connections but, as the performance on BLEU score shows, this fact does not necessarily mean better translation.


Unfortunately, even though the entire translation y is available in training, during the inference it is unknown in advance but to be generated sequentially.

when calculating α i , one can make use of the information only from x and y <i but nothing from y i .


The beam size for decoding is 12, and the cost function is optimized by Adadelta with hyper-parameters suggested by Zeiler (2012).

for TFA-NMT, the foresight embedding is also 620, and the hyper-parameter λ is 1.


Since our task is closer to the problem of morphological reinflection described above, we changed some of the default model parameters to follow those used in systems that performed well in the 2016 and 2017 SIGMORPHON Shared Tasks (Kann and Schütze, 2016;Bergmanis et al., 2017).

we reduced the number of hidden units to 100 and the encoder and decoder embedding size to 300.


We propose that what is traditionally categorized as noise (i.e., misspellings, inconsistent orthography, emerging abbreviations, and slang) should be modeled as is since it is an inherent characteristic of SM text.

the proposed models attempt to address i) misspellings using subword level representations, ii) grammatical mistakes with SM-oriented Part-of-Speech tags (Owoputi et al., 2013), iii) sounddriven text with phonetic and phonological features , and iv) the intrinsic skewness of NER datasets by applying class weights.


We take advantage of this resource as it easily adapts to other SM environments besides Twitter (Aguilar et al., 2017  the grammatical role that the words play on a sentence.

a word that can be a verb or a noun in different scenarios may conflict with the interpretations of the models; however, by providing syntactical information the models can improve their decisions.


Even though the spellings of both phrases are significantly different, by using the phonological (articulatory) aspects of those phrases it is possible to map them to the same phonetic representation.

our assumption is that social media writers heavily rely on the way that words sound while they write.


This task focuses on the Begin-Inside-Outside (BIO scheme) level of the tokens.

for a given NE, the model has to predict whether a word is B, I, or O regardless of the entity type.


Most notable are the works of (Mattiello, 2005(Mattiello, , 2008(Mattiello, , 2013, who argues that slang exhibits extragrammatical properties that distinguish it from the standard form.

linguistic phenomena like alphabetisms, blending, clippings, and reduplicatives abound in slang (see Table 1) 2 .


The simplicity of our models not only implies better generalization, more robust estimation of parameters in the wake of small dataset sizes, and better interpret-ability but also yields state of the art performance.

we show that by exploiting structural constraints, blend formation can be modeled as a simple sequence labeling problem as opposed to prior work which models it as a general sequence to sequence problem.


Second, the length of the blends in these cases is at-most the length of the components.

the blend B can thus be encoded as a string of the exact same length as C by noting that B only contains characters copied from C or deleted from C. we represent B by E(B) denoting the sequence of copy (C) and delete (D) operations needed to transform C to B. E(B) can easily be computed by the edit distance function between C and B.


Since the size of the training set is relatively small, we reduce the number of learn-able parameters by using pre-trained character embeddings frozen during training.

we use 50 dimensional character embeddings from (Gangal et al., 2017) obtained by training an LSTM language model on an English Dictionary.


We capture this notion through a probabilistic model.

we model Pr(Blend len |Component len ) by fitting a Bayesian Ridge Regression model to the training data and score each candidate on this model as well.


The second insight guides our model in determining how much to retain (or clip).

we can capture this in two ways: (a) Working in the phoneme space and (b) Learn a function to predict the length of the clipping (which encodes the number of syllables implicitly).


CLIPPHONE operates by mapping the word w to a sequence of phonemes, explicitly clipping in the phoneme space and mapping the phoneme sequence back to the grapheme space.

the model can be described as follows: (1) Let θ and π represent multinomial distributions over clipping types and the number of syllables respectively.


Most relevant is the work of (Mattiello, 2005(Mattiello, , 2008(Mattiello, , 2013 who argues that slang is pervasive on the Internet, suggests its extra-grammatical nature and outlines some phonological and morphological properties.

these phenomena are discussed followed by a qualitative analysis on a manually compiled dataset of 1580 words from various sources.


from a finite-state morphological analyzer/generator) into a neural language model, combining it with existing word-and characterlevel modelling techniques, in order to create a model capable of successfully modelling morphologically complex languages.

our model achieves three desirable properties.


Additionally, 17% of the errors belong to type error.


Figure 3 demonstrates that we achieve the best performance with 450-dimensional BiLSTMs.

using BiLSTMs with lower dimensionality causes the model to suffer from the lack of space for capturing proper information and dependencies.


It is noteworthy that DR-BiLSTM (Single) performs better than ESIM in more frequent categories.

the performance of our model in "Neutral", "Negation", and "Quantifier" categories (improvements of 1.4%, 3.5%, and 1.9%, respectively) indicates the superiority of our model in understanding and disambiguating complex samples.


Our investigations indicate that ESIM generates somewhat uniform attention for most of the word pairs while our model could effectively attend to specific parts of the given sentences and provide more meaningful attention.

the dependent reading strategy enables our model to achieve meaningful representations, which leads to better attention to obtain further gains on such categories like Negation and Quantifier sentences (see Section 3 in the supplementary material for additional details).


It would be interesting to evaluate the multimodal architecture on other predicate-argument frameworks, e.g., script knowledge or VerbNet style Semantic Role Labeling.

the exploration our findings on visual contributions to FrameId in the context of further event prediction tasks forms an interesting next step.


Our joint inference formulation handles missing annotations by treating the structures that are not present in a given training example as latent variables ( §3).

1 semantic dependencies are treated as a collection of latent variables when training on FrameNet examples.


Synthetic negative examples are generated following the protocol introduced by Munteanu and Marcu (2005) to distinguish parallel from non-parallel segments.

candidate negative examples are generated starting from the positive examples {(e i , f i ) ∀i} and taking the Cartesian product of the two sides of the positive examples{(e i , f j )∀i, j s.t.


Although the multi-task approach aims to smooth the target distribution, the training process is still driven by the imperfect target text.

at each time step t the algorithm feeds the previous word w t−1 of the target text and evaluates the prediction against the target w t .


We optimized content alignment on the development set against manual alignments.

two annotators aligned 132 sentences to their infoboxes.


First, note that while generating such descriptions from structured data, a human keeps track of information at two levels.

at a macro level, she would first decide which field to mention next and then at a micro level decide which of the values in the field needs to be mentioned next.


Further, unlike the feed-forward neural network based model proposed by (Lebret et al., 2016) we use a sequence to sequence model and introduce components to address the peculiar characteristics of the task.

we introduce neural components to address the need for attention at two levels and to address the stay on and never look back behaviour required by the decoder.


To ensure this, we reweigh the micro level attention weights based on the corresponding macro level attention weights.

we fuse the attention weights at the two levels as: where F (j) is the field corresponding to the j-th value, c w t is the macro level context vector.


For example, if we are focusing on the occupation field at this timestep then we are likely to focus on it for the next few timesteps till all relevant values in this field have been included in the generated description.

we want to remember the field context vector c g t for a few timesteps.


For example, once we have rendered all the occupations in the generated description there is no need to return back to the occupation field.

once we have moved on (f t → 0), we want the successive field context vectors c g t to be very different from the previous field vectors c t−1 .


Further, to deal with the large vocabulary (∼400K words) we use a copying mechanism as a postprocessing step.

we identify the time steps at which the decoder produces unknown words (denoted by the special symbol UNK).


First, the amount of training data in these two languages is smaller than that in English.

the amount of training data available in French (German) is only 24.2 (7.5)% of that available for English.


Note that if we train a model from scratch with only limited data from the target domain instead of finetuning a model trained on a different source domain then the performance is very poor.

training a model from scratch with 10K training instances we get a BLEU score of 16.2 and 28.4 for arts and sports respectively.


With our proposed model, a text is encoded to a vector representation from an wordlevel to a chunk-level to effectively capture the entire meaning.

by adapting the hierarchical structure, our model shows very small performance degradations in longer text comprehension while other state-of-the-art recurrent neural network models suffer from it.


In addition, we crawl hierarchical product category information related to QA pairs.

mobile, office, photo, tv/video, accessories, and home appliance as top-level categories, and specific categories like galaxy s7, tablet, led tv, and others are used.


Figure 3 demonstrates that the HRDE models, in darker blue and red colors, shows better performance than the RDE models, in lighter colors, for every "number of chunks" evaluations.

the HRDE models are consistent when the "number-of-chunks" increased, while the RDE models degrade as the "numberof-chunks" increased.


The performance of both models on a TOEFL listening comprehension test (Tseng et al., 2016) and MCTest (Richardson et al., 2013) is significantly improved via a simple transfer learning technique from MovieQA (Tapaswi et al., 2016).

one of the models achieves the state-of-the-art on all target datasets; for the TOEFL listening comprehension test, it outperforms the previous best model by 7%.


Both models are pre-trained on MovieQA and then fine-tuned on each target dataset, so that their performance on the two target datasets are significantly improved.


We also consider unsupervised transfer learning where the correct answer to each question in the target dataset is not available.

the entire process is supervised during pre-training, but unsupervised during fine-tuning.


We use the released implementations 10 (with default hyper-parameter values), and retrained them on our dataset, adapted to the standard bAbI QA format.

we create three separate variations of data by adding three bAbI-style questions after each step in a paragraph, respectively: Q1.


We can observe that all models gain better recommendation accuracy with the increment of user rating number N .

user and item latent features can be better extracted with more useful information.


Our methods for lexicon induction have connections to recent advances in the causal inference literature.

and  propose an algorithm for counterfactual inference which bear similarities to our Adversarial Selector (Section 3.2), Imai et al.


With this assumption in hand, then the causal effects of T , E Var Y (T ) C, {Y (t)} t∈T , matches I(L) as described in equation (1) (Imbens and Rubin, 2015).

given the same assumptions often made in observational studies, the informativeness coefficient of the full, uncompressed text in fact corresponds to the amount of variation in Y due to the causal effects of T .


Here, we are interested in scenarios where we must generate our labels via an annotation process.

2 assume that we have some annotation function, A, which produces a distribution over the K mutually exclusive labels, conditional on text.


Given that our objective is to mimic the annotation process, we seek to estimate the proportion of documents in the target corpus expected to be categorized into each of the K categories, if we had an unlimited budget and full access to the target corpus at the time of annotation.

we wish to estimate q (T ) , which we define as: (2) Given a set of documents sampled from the source corpus and L applications of the annotation function, we can obtain, at some cost, a labeled training corpus of L documents, i.e., D (train)  .


In both cases, there is also the problem of classifier bias or miscalibration.

in the case of unbalanced labels, a standard classifier is likely to be biased, overestimating the probability of the more common labels, and vice versa (Zhao et al., 2017).


To be clear, calibration refers to the long-run accuracy of predicted probabilities.

a probabilistic classifier, h θ (x), is well calibrated at the level µ if, among all instances for which the classifier predicts class k with a probability of µ, the proportion that are truly assigned to class k is also equal to µ.


We refer to this method as calibrated PCC (PCC cal ).

we select regularization strength via grid search, choosing the value that leads to the lowest average calibration error across training / held-out splits.


As with all datasets, we create a source and target corpus by dividing the datasets by year.

for this dataset, it seems reasonable to posit that the annotation function was relatively constant between source and target, as the annotators worked without explicit knowledge of the article's date (Card et al., 2015).


Yelp reviews: As a primary example of a large dataset with intrinsic labels, we make use of the Yelp10 dataset, treating the source location of the review as the label of interest.

we create binary classification tasks by choosing pairs of cities with approximately the same number of reviews, and again use year of publication to divide the data into source and target corpora, creating multiple subtasks per pair of cities.


Another line of work has approached the problem of aggregating labels from multiple annotators (Raykar et al., 2009;Hovy et al., 2013;Yan et al., 2013).

if we believe that some annotators are more reliable than others, it might make sense to try to determine this in an unsupervised manner, and give more weight to the annotations from the reliable annotators.


One interesting feature that the multi-agent model showcases is its simplification property, which accounts for its strength in abstraction.

it simplified the bold long sentence in the document starting with "couric will... and only generated the salient words.


Keyphrase extraction is one of the techniques that can meet this demand, because it is defined to identify salient phrases, generally formed by one or multiple words, for representing key focus and main topics for a given collection (Turney, 2000;Zhao et al., 2011).

for microblogs, keyphrase extraction has been proven useful to downstream applications such as information retrieval (Choi Target post for keyphrase extraction: "I will curse you in that forum" is the lowest of low.


We use four context encoders in our model, namely, averaged embedding, RNN (Pearlmutter, 1989), attention (Bahdanau et al., 2014), and memory networks (Weston et al., 2015), which are proven useful in text representation Weston et al., 2015;Nie et al., 2017).

in this task, to the best of our knowledge, we are the first to encode conversations for detecting keyphrases in microblog posts.


The encoder representation is thus given by the hidden units at the last state: In this paper, RNN-based encoders have four variants, namely, RNN, GRU, LSTM, and BiL-STM.

as BiLSTM has two opposite directions, its context representation takes the concatenation of the last states from both directions, which come from two ends of a given context.


Similar to attention encoder, the MemNN encoder aims to generate a representation, which addresses the important part in the conversation context that helps tagging keyphrases in target post x i .

the sum of C i and matching matrix P i serves as the encoded representation for conversation context: both attention and MemNN explores salient words in conversations that describe main focus of the conversation, which helps indicate keyphrases of a target post.


For Weibo dataset, we used NLPIR tool 10 (Zhang et al., 2003) for Chinese word segmentation.

weibo conversations have an relatively wide range (from 3 to 8,846 words), e.g., one conversation could contain up to 447 messages.


As a result, N ranges from 2 to 7 for various methods.

since TF-IDF and TextRank extract keywords instead of keyphrases, we aggregate the selected keywords according to Bellaachia and Al-Dhelaan (2012).


Formally, we write the step-wise reward s(y t |y 1:t 1 , Y ⇤ ) as follows: where N (Y,Ỹ ) represents the occurrence of sub-sequenceỸ in whole sequence Y .

if a certain sub-sequence y t n+1:t from Y appears less times than in the reference Y ⇤ , y t receives reward.


As shown in Figure 1, the performance of Seq2Seq is degraded when compressing long sentences.

the performance significantly falls if sentence length exceeds 26 words.


Evaluation results on the Google sentence compression dataset (Filippova and Altun, 2013) show that HiSAN achieved the best F 1 , ROUGE-1,2 and L scores 83.2, 82.9, 75.8 and 82.7, respectively.

hiSAN attained remarkable compression performance with long sentences.


Following their results, we also introduce syntactic features into the embedding layer.

we combine the surface token embedding w i , POS embedding p i , and dependency relation label embedding r i into a single vector as follows: where [] represents vector concatenation, and e i is an embedding of token x i .


To make compressed sentences, the dependency-tree-based approach (Filippova and Strube, 2008) is often used, which is advantageous in that each compressed sentence preserves its original dependency relations.

given a set of dependency trees constructed for sentences in the original documents, a summary is obtained by extracting some rooted subtrees; each subtree corresponds to a compressed sentence.


Our method is faster than existing compressive summarization methods since it employs the efficient greedy algorithm.

given a set, V , of all textual units contained in the document data and a summary length limitation value, L, our method requires at most O(L|V |) objective function evaluations.


Furthermore, ROUGE is known to be monotone and submodular (Lin and Bilmes, 2011).


With regard to the running times, our method substantially outperformed the ILP-based method.

our method was about 170, 380, and 120 times faster than the ILP-based one for the coverage function, the one with rewards, and the ROUGE 1 objective function, respectively.


The agent is then given a "reward" r commensurate with how well the extract resembles the gold-standard summary.

as reward function we use mean F 1 of ROUGE-1, ROUGE-2, and ROUGE-L. Unigram and bigram overlap (ROUGE-1 and ROUGE-2) are meant to assess informativeness, whereas the longest common subsequence (ROUGE-L) is meant to assess fluency.


Let indicate the probability that a span of form (t 1 ) r (t 2 ) extracted from sentence s is coherent.

we model p(c = 1|s, (t 1 ) r (t 2 )) using logistic regression, with features based on the position of partof-speech tags and dependency edges in s. each sentence in the filtered dataset contains a span of the form (t 1 ) r (t 2 ).


This work takes a step toward automatic script summarization by jointly modeling the tasks of movie attribute identification and overview generation.

we propose a novel neural network architecture which draws insights from encoder-decoder models recently proposed for machine translation (Bahdanau et al., 2015) and related sentence generation tasks (Wen et al., 2015;Mei et al., 2016;Lebret et al., 2016).


Graph-based Features Our graph-based features are similar to those described in Gorinski and Lapata (2015).

we view screenplays as weighted, undirected graphs, where vertices correspond to movie characters and edges denote character-to-character interactions (essentially the number of times two characters talk to each other or are involved in a common action).


In order to assess encoder's ability to induce structure over screenplays, we focus solely on the top part of the architecture in Figure 2.

we trained stand-alone models for the six attributes shown in Table 1 on the gold data provided in the Jinni dataset.


(2017) use an auxiliary bag-of-words loss on the latent variable to force the model to use z.

they train an auxiliary network that predicts bag-of-words representation of the target utterance based on z.


The variational posterior is a factorized Gaussian distribution where the mean and the diagonal variance are predicted from the target utterance and the context as follows: where A known problem of a VAE that incorporates an autoregressive RNN decoder is the degeneracy that ignores the latent variable z.


Therefore, our second idea is to apply an utterance drop regularization to effectively regularize the hierarchical RNNs, in order to facilitate the use of latent variables.

at each time step, the utterance encoder vector h enc t is randomly replaced with a generic unknown vector h unk with a probability p. This regularization weakens the autoregressive power of hierarchical RNNs and as well alleviates the data sparsity problem, since it induces noise into the context vector h cxt t which conditions the decoder RNN.


On the other hand, the global variable z conv allows the VHCR to predict a reasonable prior for local variable z utt t even in the presence of the utterance drop regularization.

z conv can act as a guide for z utt by encoding the information for local variables.


The VHRED with the bag-of-words loss (bow) achieves the highest KL divergence, however, at the cost of high NLL values.

the variational lower-bound minimizes the minimum description length, to which the bow loss works in an opposite direction by forcing latent variables to encode bag-of-words representation of utterances.


Our goal in this paper is to study conversational features that lead to egregious conversations.

we consider customer inputs throughout a whole conversation, and detect cues such as rephrasing, the presence of heightened emotions, and queries about whether the chatbot is a human or requests to speak to an actual human.


Examples include (Kiseleva et al., 2016b,a;Jiang et al., 2015).

these works evaluated chat functionality by asking users to make conversations with an intelligent agent and measured the user satisfaction along with other features such as the automatic speech recognition (ASR) quality and intent classification quality.


The works of (Sarikaya, 2017;Sano et al., 2017) studied reasons why users reformulated utterances in such systems.

in (Sarikaya, 2017) they reported on how the different reasons affect the users' satisfaction.


The EGR model significantly outperformed both baselines 10 .

for the egregious class, the precision obtained by the text-based and EGR models were similar.


First, we detected customer rephrasing as described in Section 3.2.1, and then assigned to each its motivation.

in our setting, the relevant motivations are 12 : (1) Natural language understanding (NLU) error -the agent's intent detection is wrong, and thus the agent's response is semantically far from the customer's turn; (2) Language generation (LG) limitation -the intent is detected correctly, but the customer is not satisfied by the response (for example, the response was too generic); (3) Unsupported intent error -the customer's intent is not supported by the agent.


These representations can be derived not only for words (Mikolov et al., 2013) but also sentences and documents (Le and Mikolov, 2014).

convolutional neural networks (CNNs) have been shown to efficiently and effectively preserve important semantic and syntactic information from embedded text sequences (Blunsom et al., 2014).


Hence, we adjusted the datasets to be more similar to real conversations.

we removed some conversations so that every dataset has at most ten conversations at any point in time.


In recent years, there are surging interests in designing machine learning algorithms for complex reasoning tasks, especially in large knowledge graphs (KGs) where the countless entities and links have posed great challenges to traditional logic-based algorithms.

we situate our study in this large KG multi-hop reasoning scenario, where the goal is to design an automated inference model to complete the missing links between existing entities in large KGs.


Arrows with solid border represent the prior and likelihood distributions.

the ELBO (Kingma and Welling, 2013) is composed of three different terms -likelihood p θ r|L), prior p β L|(e s , e t )), and posterior q ϕ L|(e s , e d ), r).


Our TORDERs show a more balanced distribution of labels, which suggests that this method possibly encodes more informative temporal orders compared to the traditional TLINKs.

tORDERs show extremely rare VAGUE labels in Event-DCt pairs.


freeze the assignment by solving Equation 1 after setting β = 0.

ψ(e, E D ) is not used while assigning entities to unambiguous mentions.


Please note that PMI, and there by M , are symmetric.

the expanded set of edges, F + , is now defined as F + = F ∪{(e, e ), (e , e) | e ∈ E + , e ∈ E, M e,e > 0} we augment the set of initial edges F with additional edges connecting entities in E + with entities in E such that PMI between the entities is positive.


", one can infer that a tasterelated word like "bland" fits, but a word like "rude" does not, even though both have negative sentiment.

while the deleted sentiment words do contain non-sentiment information too, this information can often be recovered using the other content words.


During training, we apply some noise to a(x, v src ) by randomly altering each attribute marker in it independently with probability 0.1.

we replace an attribute marker with another randomly selected attribute marker of the same attribute and word-level edit distance 1 if such a noising marker exists, e.g., "was very rude" to "very rude", which produces a (x, v src ).


For each dataset, we train an attribute classifier on the same training data.

we encode the sentence into a vector by a bidirectional LSTM with an average pooling layer over the outputs, and train the classifier by minimizing the logistic loss.


Inducing paraphrases from bilingual data has long been an effective method to overcome data limitations.

bilingual pivoting (Bannard and Callison-Burch, 2005) finds quality para-phrases by pivoting through a different language.


Our key insight is that target transformations can be detected (with some noise) simply by parsing these pairs.

3 we parse the backtranslated paraphrases using the Stanford parser , 4 which yields a pair of constituency parses p 1 , p 2 for each sentence pair s 1 , s 2 , where s 1 is the reference English sentence in the CzEng corpus and s 2 is its backtranslated counterpart.


We incorporate the target syntax p 2 into the generation process by modifying the inputs to the decoder.

a standard decoder LSTM receives two inputs at every time step: (1) the embedding w t−1 of the ground-truth previous word in s 2 , and (2) an attention-weighted average a t of the encoder's hidden states.


5 Since we cannot generate a valid paraphrase for every template, we postprocess to remove nonsensical outputs.

we filter generated paraphrases using n-gram overlap and paraphrastic similarity, the latter of which is computed using the pretrained WORD,TRIAVG sentence embedding model from Wieting and Gimpel (2017).


This network takes the auxiliary task outputs as input.

we define the output label embedding o i of task T i as the sum of the task's label embeddings l j weighted with their probability p T i j : The label embeddings l encode general relationship between labels, while the model's probability distribution p T i over its predictions encodes finegrained information useful for learning (Hinton et al., 2015).


To mitigate this discrepancy, we augment the LTN's input with features that have been found useful for transfer learning (Ruder and Plank, 2017).

we use the number of word types, type-token ratio, entropy, Simpson's index, and Rényi entropy as diversity features.


For our experiments, we evaluate on a wide range of text classification tasks.

we choose pairwise classification tasks-i.e.


This change resulted in a proliferation of heterogeneous formats and, in parallel, often smallsized, non-interoperable resources (lexicons and corpus annotations).

the limitations in size hampered the application of deep learning methods in this area because they typically require large amounts of input data.


Yet, they all have in common that they treat the prediction of the different emotional dimensions as separate tasks.

they fit one individual model per VAD dimension without sharing parameters between them.


Wilson and Sperber (2004) suggest that, when we comprehend arguments, we reconstruct their warrants driven by the cognitive principle of relevance.

we go straight for the interpretation that seems most relevant and logical within the given context (Hobbs et al., 1993).


The intuition of alternative warrants is key to the systematic methodology that we develop in this paper for reconstructing a warrant for the original claim of an argument.

we first 'twist' the stance of a given argument, trying to plausibly explain its reasoning towards the opposite claim.


As the task we propose involves natural language comprehension, we also review relevant work outside argumentation here.

the goal of the semantic inference task textual entailment is to classify whether a proposition entails or contradicts a hypothesis (Dagan et al., 2009).


Number of turns was increased in dialogue segments where the interviewer did not ultimately believe the interviewee response.

more follow up questions were asked when an interviewer did not believe their interlocutor's response, which is an intuitive behavior.


The RILE score is calculated directly from the sentence labels, based on mapping each label according to its positioning on policy themes, as LEFT, RIGHT and NEUTRAL (Volkens et al., 2013).

13 out of 57 classes are categorized as LEFT, another 13 as RIGHT, and the rest as NEUTRAL.


(2015)), and temporal dependency provides further improvement against CHES.

combining content, network and temporal features provides the best results.


Our results demonstrate a number of strengths and weaknesses in the models' ability to generalise to new in-domain instances.

while strong performance is possible on unseen hypernyms, unseen antonyms are more challenging for all the models.


We also consider an ex situ condition in which we remove the word pair from the original premise and hypothesis and analyze the effect of the transformation T within a simplified sentential context which is the same in premise and hypothesis.

we randomly select either the premise or hypothesis context from the original instance and copy it into both positions.


2-way tests of homogeneity show that the difference in accuracy of predictions in instances containing seen or unseen word pairs is nonetheless significant for all models (CE: χ 2 (1) = 19.46, DAM: χ 2 (1) = 74.16, ESIM: χ 2 (1) = 39.33).

the models struggle to recognize the reversed antonym pairs, even though they were all seen in their original order at training time.


Our framework builds on previous research in psycholinguistics demonstrating that the eyetracking record reflects how readers interact with the text and how language processing unfolds over time (Frazier and Rayner, 1982;Rayner, 1998;Rayner et al., 2012).

it has been shown that key aspects of the reader's characteristics and cognitive state, such as mind wandering during reading (Reichle et al., 2010), dyslexia (Rello and Ballesteros, 2015) and native language (Berzak et al., 2017) can be inferred from their gaze record.


We hypothesize that language proficiency influences the way that learners process a second language, which in turn will be reflected in eye movement patterns in reading.

we propose to examine whether the more proficient is an ESL learner, the more similar are their reading patterns to those of native English speakers.


EyeScore Reading Speed Normalization To reduce bias towards fast readers, the feature representations used for Eyescore are normalized to be invariant to the reading speed of the participant.

for the S-Clusters and WFC feature sets we follow the normalization procedure of Berzak et al.


In future work, we plan to extend the analysis of the validity and consistency of our approach, and further explore its applications for language proficiency evaluation.

we will examine the impact of factors that can undermine the validity of language proficiency tests, such as test specific training, familiarity with the evaluation system's features (Powers et al., 2002), and cheating via unauthorized prior access to test materials.


This is because prediction probability/loss of spurious instances tend to be lower than that of genuine instances (He and Garcia, 2009).

the Bayesian Uncertainty model (Feinman et al., 2017) defines spurious instances as those that have greater uncertainty (variance) in their stochastic predictions, and the Variational Inference model (Rehbein and Ruppenhofer, 2017;Hovy et al., 2013) expects greater posterior entropy in predictions made for spurious instances.


function in Figure 2 scores instances based on their frequency of occurrence in the hard batch.

for each instance h i : where S e (h i ) is the score of h i at iteration e, 1 Y (x) is an indicator function which is 1 when x ∈ Y and otherwise 0, hard batch e indicates the set of hard instances at iteration e, and loss e (h i ) is the loss of the network for h i at iteration e. The above function assigns higher scores to instances that are frequently considered as hard instances by the curriculum learning framework (such instances are ranked higher in the final ranked list of spurious instances).


function in Figure 3 scores instances based on their occurrence in q 0 .

for each instance h i : where |q e 0 | indicates the number of instance in q 0 at iteration e. The above function assigns higher scores to instances that frequently occur in q 0 .


Furthermore, Figure 4(e) shows that Lit generates a more accurate ranked list of spurious instances and consistently outperforms other models at almost all ranks.

it maintains a MAP of around 60% at rank 20, while other models have consistently lower MAP than 50% at all ranks.


In this paper, we propose a cognitive model of fluency, in which lexical memory retrievals may explain some of the variability in speech rates.

frequency, context and recent uses together have the potential to quantify retrieval delays through activation (Anderson, 1991).