Different from previous methods that treat sentence scoring and sentence selection as two tasks, our method integrates the two steps into one endto-end trainable model.

nEUSUM is a neural network model without any handcrafted features that learns to identify the relative importance of sentences.


Therefore, we employ another GRU as the recurrent unit to remember the partial output summary, and use a Multi-Layer Perceptron (MLP) to score the document sentences.

the GRU takes the document level representation s t−1 of the last extracted sentenceŜ t−1 as input to produce its current hidden state h t .


Contributions: while we retain the use of WordNet and of a language model, we show that, without deteriorating the quality of the results, steps (1) and (2) above (Community Detection) can be performed in a much more simple, completely unsupervised way, and that step (3) can be removed.

the MSCG is powerful enough to remove redundancy and ensure informativeness, should proper edge weights and path re-ranking function be used.


Our systems outperform all baselines on AMI (including Oracle) and all baselines on ICSI (except Oracle).

our System is best on ICSI, while our System (KeyRank) is superior on AMI.


Multi-layer neural networks have also been used towards this purpose (Lee and Choeh, 2014).

there seems to be progress toward more sophisticated models.


They showed that the existence and strength of connections between reviewers and users in a social network, along with product rating history similarity, moderated the general user opinion of a particular reviewer's reviews.

they analyzed social network connections in Ciao's circle of trust, a social network where a user connects to a reviewer if they consistently find their reviews helpful, along with users' and reviewers' product rating histories, and made the following observations: 1.


Furthermore, a variety of insightful observations have been made on moderating factors.

product opinions, user idiosyncrasy, product and review nature, along with review voting context have been shown to influence the way users vote.


"名 古 屋" and each Chinese pseudo-word, and compose the social word vector S 名古屋 .

for each culture/language, the new word vectors like S W are constructed based on the monolingual similarities of each word to the vectors of a set of task-related words ("social words" in our case).


Task definition: This task is to discover and quantify cross-cultural differences of concerns towards named entities.

the input in this task is a list of 700 named entities of interest and two monolingual social media corpora; the output is the scores for the 700 entities indicating the crosscultural differences of the concerns towards them between two corpora.


As a result, it is necessary to correctly model the relevant contextualizing information.

we are interested in exploring how political ideology, language, and framing interact to represent morality on Twitter.


Then, a second decoder fills in missing details by conditioning on the natural language input and the sketch itself.

the sketch constrains the generation process and is encoded into vectors to guide decoding.


We compare our model (COARSE2FINE) against several previously published systems as well as various baselines.

we report results with a model which decodes meaning representations in one stage (ONESTAGE) without leveraging sketches.


To model such inputs with rich internal structures, we follow , and model the distribution over a sequential surface representation of z, z s instead.

we have p(z) p(z s ) and p θ (x|z) p θ (x|z s ) 2 .


As we will explain in § 4.3, we find these two components perform well with linearization.

the prior is parameterized by a Long Short-Term Memory (LSTM) language model over z s .


The learning signal weights the gradient for each latent sample z.

in REiN-FORCE, to cope with the high variance of the learning signal, it is common to use a baseline b(x) to stabilize learning, and re-define the learning signal as in STRUCTVAE, we define where log p(x) is a pre-trained LSTM language model.


In this paper, we propose a new neural semantic parsing framework -Sequence-to-Action, which can simultaneously leverage the advantages of semantic graph representation and the strong prediction ability of Seq2Seq models.

we model semantic parsing as an end-to-end semantic graph generation process.


Based on the above action encoding mechanism, this section describes our encoder-decoder model for mapping sentence to action sequence.

similar to the RNN model in Jia and Liang (2016), this paper employs the attentionbased sequence-to-sequence RNN model.


Concretely, we transform logical forms into semantic graphs using a depth-first-search algorithm from root, and then generate the action sequence using the same order.

entities, variables and types are nodes; relations are edges.


It is obviously that the next action has a strong correlation with the partial semantic graph generated to current, and illegal actions can be filtered using structure and semantic constraints.

we incorporate constraints in decoding using a controller.


The semantic constraints ensure the constructed graph must follow the schema of knowledge bases.

we model two types of semantic constraints.


(Artetxe et al., 2017;Smith et al., 2017).

we use identically spelled words that occur in the vocabularies of both languages as bilingual seeds, without employing any additional transliteration or lemmatization/normalization methods.


As discussed before, previous evaluation has focused on favorable conditions.

existing unsupervised methods have almost exclusively been tested on Wikipedia corpora, which is comparable rather than monolingual, exposing a strong cross-lingual signal that is not available in strictly unsupervised settings.


Because BWEs represent words from different languages in a shared space, BLI can be performed via cosine similarity in this space.

given a BWE representing two languages V s and V t , the translation of each word s ∈ V s can be induced by taking the word t ∈ V t whose representation x t in the BWE is closest to the representation x s .


We reimplement BiDAF with simplified parameterizations and change the prediction layer so that it can predict multiple answer spans.

we encode the input story {x 1 , ..., x T } and a given question {q 1 , ..., q J } at the character level and the word level, where the character level uses CNNs and the word level uses pre-trained word vectors.


We can see that BiDAF-M and DrQA-M perform surprisingly well in the within-world evaluation even though they do not use any entity span information.

drQA-M outperforms BidAF-M which suggests that modeling question-context interactions using simple bilinear terms have advantages over using more complex bidirectional attention flows.


As a final baseline, we consider training models with the sigmoid loss objective function.

we compute a start/end probability for each token by applying the sigmoid function to the start/end scores of each token.


With increasing complexity of models for tasks like classification (Joulin et al., 2016), machine comprehension (Rajpurkar et al., 2016;Seo et al., 2017), and visual question answering (Zhu et al., 2016), models are becoming increasingly challenging to debug, and to determine whether they are ready for deployment.

these complex models are prone to brittleness: different ways of phrasing the same sentence can often cause the model to In the United States especially, several high-profile cases such as Debra LaFave, Pamela Rogers, and Mary Kay Letourneau have caused increased scrutiny on teacher misconduct.


Given two datasets 2 } which represent two different styles s 1 and s 2 , respectively, our task is to generate sentences of the desired style while preserving the meaning of the input sentence.

we generate samples of dataset X 1 such that they belong to style s 2 and samples of X 2 such that they belong to style s 1 .


Then z is given by: where, x f is the sentence x in language f .

x f is the output of e → f translation system when x e is given as input.


In the future work, we will also explore whether an enhanced back-translation by pivoting through several languages will learn better grounded latent meaning representations.

it would be interesting to back-translate through multiple target languages with a single source language (Johnson et al., 2016).


We draw our inspiration from recent progress in inverse reinforcement learning (Ho and Ermon, 2016;Finn et al., 2016;Fu et al., 2017) and propose the AREL algorithm to learn a more intelligent reward function.

we first incorporate a Boltzmann distribution to associate reward learning with distribution approximation, then design the adversarial process with two models -a policy model and a reward model.


For evaluation, we conduct both automatic metrics and human evaluation but observe a poor correlation between them.

our method gains slight performance boost over the baseline systems on automatic metrics; human evaluation, however, indicates significant performance boost.


Relying on bilingual image tags and descriptions, our approach conditions text embedding induction on the shared visual information for both languages, producing highly correlated bilingual embeddings.

we propose a novel model based on Partial Canonical Correlation Analysis (PCCA).


1 Research in multi-modal semantics deals with the grounding problem (Harnad, 1990), motivated by evidence that many semantic concepts, irrespective of the actual language, are grounded in the perceptual system (Barsalou and Wiemer-Hastings, 2005).

recent studies have shown that performance on NLP tasks can be improved by joint modeling of text and vision, with multimodal and perceptually enhanced representation learning outperforming purely textual representa-tions (Feng and Lapata, 2010;Kiela and Bottou, 2014;Lazaridou et al., 2015).


• We highlight the importance of the convolutional network used to extract embeddings.

networks trained with semantic labels result in better embeddings than those trained with visual labels, even when evaluating similarity on concrete words.


For computing a score between 2 word pairs with Picturebook, we set s(w (1) , w (2) ) = min i,j d(e i , e (2) j ).

2 the score is minus the smallest cosine distance between all pairs of images of the two words.


For each action, three students searched the web and collected a set of images depicting potential effects.

given a verb-noun pair, each of the three students was asked to collect at least 5 positive images and 5 negative images.


Given an image, the model will output a probability distribution q over the candidate actions (i.e., verb-noun pairs) that can potentially cause the effect depicted in the image.

for model training, we are given a set of human annotated seeding image data {x, t} and a set of web-search image data {x , t }.


In the experiments, the negative sampling ratio is set to 25.

for each positive action-effect pair, 25 negative pairs are created through random sampling.


To address this issue, this paper introduces a new task on action-effect prediction.

we focus on modeling the connection between an action (a verb-noun pair) and its effect as illustrated in an image and treat natural language effect descriptions as side knowledge to help acquiring web image data and bootstrap training.


2 Moreover, it is hard for vanilla CNN to differentiate opinion words of multiple targets.

multiple active local features holding different sentiments (e.g., "great food" and "service was dreadful") may be captured for a single target, thus it will hinder the prediction.


This strategy preserves context information by directly feeding the features before the transformation to the next layer.

the input h (l+1) i of the (l + 1)-th CPT layer is formulated as: i is the input of the l-th layer andh is the output of TST in this layer.


The idea is a closer opinion word is more likely to be the actual modifier of the target.

we first calculate the position relevance v i between the i-th word and the target 4 : where k is the index of the first target word, C is a pre-specified constant, and m is the length of the target w τ .


The hyper-parameters of TNet-LF and TNet-AS are listed in Table 2.

all hyperparameters are tuned on 20% randomly held-out training data and the hyper-parameter collection producing the highest accuracy score is used for testing.


LSTM-FC-CNN-LF and LSTM-FC-CNN-AS are built by applying FC layer to replace TST and keeping the context-preserving mechanism (i.e., LF and AS).

the concatenation of word representation and the averaged target vector is fed to the FC layer to obtain targetspecific features.


The TNet variants can still outperform LSTM-FC-CNN-LF/AS with significant gaps, e.g., on LAPTOP and REST, the accuracy gaps between TNet-LF and LSTM-FC-CNN-LF are 0.42% (p < 0.03) and 0.38% (p < 0.04) respectively.

as our TNet involves multiple CPT layers, we investigate the effect of the layer number L. we conduct experiments on the held-out training data of LaPTOP and vary L from 2 to 10, increased by 2.


Our TNet model is carefully designed to solve these issues.

we propose target specific transformation component to better integrate target information into the word representation.


However, we found that using existing MNs to deal with ASC has an important problem and simply relying on attention modeling cannot solve it.

their performance degrades when the sentiment of a context word is sensitive to the given target.


Attention: Attention can be obtained based on the above input representation.

an attention weight α i for the context word x i is computed based on the alignment function: where M ∈ R d×d is the general learning matrix suggested by Luong et al.


This contradiction also generalizes to realworld sentences.

although real-world review sentences are usually longer and contain more words, since the attention mechanism makes MNs focus on the most important sentiment context (the context with high α i scores), the problem is essentially the same.


This representation is for capturing the global correlation between context and different sentiment classes.

e i is another output representation for x i , which is coupled with the sentiment interaction factor d i , d t .


Based on this association we assign a polarity orientation to a word in the domain.

if a word is found significant by χ 2 test, then the exact class of the word is determined by comparing c w p and c w n .


Essentially, system-3 does not give much improvement with iterations, unlike system-2.

addition of the iterative process with the shared representation given by SCL overcomes the errors introduced by SCL.


The input sentiment signal determines which decoder to use.

we use the seq2seq model (Sutskever et al., 2014) for implementation.


TextCNN achieves the accuracy of 89% and 88% on two datasets.

we generate sentences given sentiment s, and use the pre-trained sentiment classifier to assign sentiment labels to the generated sentences.


Inspired by (Seo et al., 2016) and (Chen et al., 2017b), we use a modeling layer to capture the interaction between the premise and the hypothesis.

we use bi-directional LSTMs as building blocks: Here, p M i and u M j are the modeling vectors which contain the crucial information and relationship among the sentences.


In Figure 2, we show the performance on the three relation labels when the model is pre-trained on different discourse markers sets.

we removed discourse marker from the original set each time and use the rest 7 discourse markers to pre-train the sentence encoder in the DMP task and then train the DMAN.


To overcome the weaknesses of standard sequential models such as recurrent neural networks, our work is based on the intuition that modeling intra-sentence relationships can not only improve classification performance but also pave the way for more explainable neural sarcasm detection methods.

our key intuition manifests itself in the form of an attention-based neural network.


We propose a multi-dimensional intra-attention recurrent network that models intricate similarities between each word pair in the sentence.

our novel deep learning model aims to capture 'contrast' (Riloff et al., 2013) and 'incongruity' (Joshi et al., 2015) within end-to-end neural networks.


Both CNN-LSTM-DNN (Ghosh and Veale, 2016) and GRNN (Zhang et al., 2016) are state-ofthe-art models for document-level sarcasm detection and have outperformed numerous neural and non-neural baselines.

both works have well surpassed feature-based models (Support Vector Machines, etc.


More advanced gradient estimators exist that also reduce variance (Grathwohl et al., 2017;Tucker et al., 2017;Liu et al., 2018), but for simplicity we use the self-critical baseline method (Rennie et al., 2016), where the baseline is b(x) = l ω (y + , y , x), or b(x) = −s ω (y , x) in the separable loss case, and y = argmax i g θ (y i |x).

the baseline is the reward of the most likely sample according to the generator.


In this paper, we propose Adversarial Contrastive Estimation as a general technique for improving supervised learning problems that learn by contrasting observed and fictitious samples.

we use a generator network in a conditional GAN like setting to propose hard negative examples for our discriminator model.


For detection problems evaluated using F-measure, our framework proves that the utility of correctly predicting one more positive instance (marginal positive utility) and that of correctly predicting one more negative instance (marginal negative utility) are different and dynamically changed during model training.

the importance of instances of each class is not only determined by their data distribution, but also affected by how well the current model can converge on different classes.


This section describes how to effectively optimize neural network detection models via dynamic cost-sensitive learning.

we first propose a marginal utility based theoretical framework for measuring the importance of positive/negative instances.


In this section, we describe how to incorporate the above importance measures into the training procedure of neural networks, so that it can dynamically adjust weights of positive and negative instances regarding to F-measure.

given the current model of neural networks parameterized by θ, let w β (θ) denote the relative importance of negative instances to positive instances for F β -measure.


Besides, for T P and T N , we approximate them using their expectation on the current batch, which can produce a robust estimation even when the batch size is not large enough.

let i=1 } is N B negative instances in the batch, we estimate T P (θ) and T N (θ) as: T N B (θ) = Then we can compute the estimatorŵ β (θ) for w β (θ) as:ŵ whereŵ β (θ) is computed using only the instances in a batch, which makes it can be directly applied as a plug-in of conventional batch-based neural network optimization algorithm where the loss of negative instances in batch are scaled byŵ β (θ).


Many of these methods, though, were originally developed with in-domain performance in mind, so their effectiveness in a domain adaptation setting remains unexplored.

we re-evaluate three traditional bootstrapping methods, self-training (Yarowsky, 1995), tri-training (Zhou and Li, 2005), and tritraining with disagreement (Søgaard, 2010) for neural network-based approaches on two NLP tasks with different characteristics, namely, a sequence prediction and a classification task (POS tagging and sentiment analysis).


traditional tri-training is complementary to word embedding initialization, pushing the non-pre-trained baseline to the level of SRC with Glove initalization.


low that of their correct sentences, as Figure 2(a) shows.

we call the generated errorcorrected sentence pairs fluency boost sentence pairs because the sentence in the target side always improves fluency over that in the source side.


Since the less fluent sentences are generated by the error generation seq2seq model trained with error-corrected data, they usually do not change the original sentence's meaning; thus, they can be paired with their correct sentences, establishing fluency boost sentence pairs that can be used as training instances for error correction models, as Figure 3(a) shows.

we first train a seq2seq error generation model Θ gen with S * which is identical to S * except that the source sentence and the target sentence are interchanged.


As Figure 2(b) shows, fluency boost inference allows a sentence to be incrementally edited through multiround seq2seq inference as long as the sentence's fluency can be improved.

an error correction seq2seq model first takes a raw sentence x r as an input and outputs a hypothesis x o 1 .


Second, the diagnosis descriptions and the textual descriptions of ICD codes are written in quite different styles even if they refer to the same disease.

the textual description of an ICD code is formally and precisely worded, while diagnosis descriptions are usually written by physicians in an informal and ungrammatical way, with telegraphic phrases, abbreviations, and typos.


Contributions In this paper, we design a neural architecture to automatically perform ICD coding given the diagnosis descriptions.

we make the following contributions: • We propose a tree-of-sequences LSTM architecture to simultaneously capture the hierarchical relationship among codes and the semantics of each code.


In such situations, we need methods that can leverage the labeled and unlabeled data in a past event (we refer to this as a source domain), and that can adapt to a new event (we refer to this as a target domain) without requiring any labeled data in the new event.

we need models that can do domain adaptation to deal with the distribution drift between the domains and semi-supervised learning to leverage the unlabeled data in both domains.


The supervised component induces label information (e.g., relevant vs. non-relevant) directly in the network through the classification loss L C (Λ, Φ), which is computed on the labeled instances in the source event, D l S .

this branch of the network, as shown at the top in Figure 1, takes the shared representations z as input and pass it through a task-specific dense layer where V c is the corresponding weight matrix.


In adversarial training, we seek parameters (saddle point) such that which involves a maximization with respect to Ψ and a minimization with respect to {Λ, Φ, Ω}.

the updates of the shared parameters Λ = {U, V } for the discriminator work adversarially to the rest of the network, and vice versa.


It can be clearly observed that the graph-based semi-supervised approach outperforms the two baselines -supervised and self-training based semi-supervised.

the graph-based approach shows 4% to 13% absolute improvements in terms of F1 scores for the Nepal and Queensland datasets, respectively.


We also notice that our semi-supervised approach can perform above 90% depending on the event.

major improvements are observed from batch size 100 to 1,000, however, after that the performance improvements are comparatively minor.


To close this gap, a two-stage deep neural network (TDNN) is proposed.

in the first stage, using the rated essays for nontarget prompts as the training data, a shallow model is learned to select essays with an extreme quality for the target prompt, serving as pseudo training data; in the second stage, an end-to-end hybrid deep model is proposed to learn a prompt-dependent rating model consuming the pseudo training data from the first step.


Being aware of this difficulty, to this end, a twostage deep neural network, coined as TDNN, is proposed to tackle the prompt-independent AES problem.

to mitigate the lack of the prompt-dependent labeled data, at the first stage, a shallow model is trained on a number of rated essays for several non-target prompts; given a target prompt and a set of essays to rate, the trained model is employed to generate pseudo training data by selecting essays with the extreme quality.


Similar to (Tai et al., 2015), we opt for three stacked bi-LSTM, aiming at encoding individual phrases, sentences, and ultimately the whole essay in sequence.

according to the hierarchical structure from a parsing tree, the phrase-level bi-LSTM first encodes different phrases by consuming syntactic  Figure 2) from a lookup table of individual syntactic units in the tree; thereafter, the encoded dense layers in individual sentences are further consumed by a sentence-level bi-LSTM, ending up with sentence-level syntactic representations, which are ultimately combined by the essay-level bi-LSTM, resulting in − → e synt .


The RankSVM is also used for the prompt-independent stage in our proposed TDNN model.

the linear kernel RankSVM 2 is employed, where C is set to 5 according to our pilot experiments.


The combinations being considered are listed in the following.

the dimensions of POS tags and syntactic network are fixed to 50, whereas the sizes of the hidden units in LSTM, as well as the output units of the linear layers are tuned by grid search.


Therefore, one would like to understand how good the trained RankSVM is in feeding training data for the model in the second stage.

the precision, recall and F-score (P/R/F) of the essays selected by RanknSVM, namely, the negative ones rated between [0, 4], and the positive ones rated between [8,10], are displayed in Figure 4.


This model is later improved by employing attention layers.

the model learns text representation with LSTMs which can model the coherence and co-reference among sequences of words and sentences, and uses attention pooling to capture more relevant words and sentences that contribute to the final quality of essays .


To address this problem, we propose a novel controlled response generation mechanism to handle different utterance-response relationships in terms of specificity.

we introduce an explicit specificity control variable into a sequence-to-sequence model, which interacts with the usage representation of words through a Gaussian Kernel layer, to guide the model to generate responses at different specificity levels.


The encoder is to map the input utterance X into a compact vector that can capture its essential topics.

we use a bi-directional GRU (Cho et al., 2014) as the utterance encoder, and each word x i is firstly represented by its semantic representation e i mapped by semantic embedding  Figure 2: The overall architecture of SC-Seq2Seq model.


Finally, we use the final backward hidden state as the initial hidden state of the decoder.

the decoder is to generate a response Y given the hidden representations of the input utterance X under some specificity level denoted by the control variable s. at step t, we define the probability of generating any target word y t by a "mixture" of probabilities: where p M (y t ) denotes the semantic-based generation probability, p S (y t ) denotes the specificitybased generation probability, β and γ are the coefficients.


The decoder is to generate a response Y given the hidden representations of the input utterance X under some specificity level denoted by the control variable s. Specifically, at step t, we define the probability of generating any target word y t by a "mixture" of probabilities: where p M (y t ) denotes the semantic-based generation probability, p S (y t ) denotes the specificitybased generation probability, β and γ are the coefficients.

p M (y t ) is defined the same as that in traditional Seq2Seq model (Sutskever et al., 2014): where w is a one-hot indicator vector of the word w and e t−1 is the semantic representation of the t − 1-th generated word in decoder.


p S (y t ) denotes the generation probability of the target word given the specificity control variable s. Here we introduce a Gaussian Kernel layer to define this probability.

we assume that each word, beyond its semantic representation e, also has a usage representation u mapped by usage embedding matrix U.


Therefore, we use the inverse frequency of a response in a conversation corpus to indicate its specificity level.

we first build the response collection R by extracting all the responses from D. For a response Y ∈ R, let f Y denote its corpus frequency in R, we compute its Inverse Response Frequency (IRF) as: where |R| denotes the size of the response collection R. Next, we use the min-max normalization method (Jain et al., 2005) to obtain the NIRF value.


Hence, we can use the inverse corpus frequency of the words to indicate the specificity level of a response.

for a word y in the response Y, we first obtain its Inverse Word Frequency (IWF) by: where f y denotes the number of responses in R containing the word y.


We tuned the hyper-parameters via the development set.

we use one layer of bi-directional GRU for encoder and another uni-directional GRU for decoder, with the GRU hidden unit size set as 300 in both the encoder and decoder.


By conducting further analysis, we find that even though the conversation dataset is large, it is still limited and a general response could appear very few times in this corpus.

the inverse frequency of a response is very weakly correlated with its response spe-cificity.


Important matching information between segment pairs across multi-turn context and candidate response is then extracted via convolution with max-pooling operations, and further fused into one matching score via a single-layer perceptron, representing the matching degree between the response candidate and the whole context.

we use a shared component, the Attentive Module, to implement both selfattention in representation and cross-attention in matching.


If the number of clusters does not increase as fast as the increase in the number of segments, then the number of clusters per segment becomes smaller, leading to a smaller exponent.

the influence of each consecutive co-occurrence of a particular word decays slightly as the overall text size grows.


The height of the lowest common ancestor for two leaves (w i , w j ) is noted asd i j .

the syntactic distances of t can be any vector of scalars d induces the same ranking order as the quantitiesd j i computed between pairs of consecutive words in the sequence, i.e.


We show that LVGs can be seen as a special case of LVeGs.

we can use one-hot vectors in LVeGs to represent latent variables in LVGs and define weight functions in LVeGs accordingly.


We achieve this by defining weight functions that output zero if any of the input vectors is not one-hot.

we define the weight function of the production rule A BC as: where are one-hot vectors (which are zero everywhere with the exception of a single 1 at the x-th index of a x , the y-th index of b y , and the z-th index of c z ) and Θ A BC is multiplied sequentially by c, b, and a.


It is therefore not our goal to compete with the latest state-of-the-art approaches to tagging and parsing.

we currently do not incorporate contextual information of words and constituents during tagging and parsing, while such information is critical in achieving state-of-the-art accuracy.


To deal with the problem of unknown words in testing, we adopt the unknown word features used in the Berkeley parser and set the unknown word threshold to 1.

any word occurring less than two times is replaced by one of the 60 unknown word categories.


Because HMMs are equivalent to probabilistic regular grammars, we can extend HMMs with both LVGs and LVeGs.

the hidden states in HMMs can be seen as nonterminals in regular grammars and therefore can be associated with latent variables or latent vectors.


Taken together, this suggests a way to rapidly extend a newswire-trained parser to new domains.

we will show it is possible to achieve large out-of-domain performance improvements using only dozens of partially annotated sentences, like those shown in Figure 1.


Alternatively, we could just better align the model with the annotation task.

we could train a parser whose base model predicts exactly what we ask the annotator to annotate, e.g.


Domain adaptation has been recognized as a major NLP problem for over a decade (Ben-David et al., 2006;Daumé, 2007;Finkel and Manning, 2009).

domain adaptation for parsers (Plank, 2011;Ma and Xia, 2013) has received considerable attention.


The decision on the relation inventory is somewhat arbitrary, and subsequently, the inter-annotator agreement is relatively low (Kim and Baldwin, 2007).

a noun-compound may fit into more than one relation: for instance, in Tratz (2011), business zone is labeled as CONTAINED (zone contains business), although it could also be labeled as PURPOSE (zone whose purpose is business).


Subjective Corpus By extension, one may then deem a corpus subjective if its content does not at least meet Wikipedia's neutral point of view requirement.

if the content is replete with personal feelings and opinions.


In this paper, we propose an unsupervised metaphor processing model which can identify and interpret linguistic metaphors at the wordlevel.

our model is built upon word embedding methods (Mikolov et al., 2013) and uses WordNet (Fellbaum, 1998) for lexical re-lation acquisition.


We expect that using a combination of input and output vectors might work better.

we have experimented with four different model variants as shown below.


Usually, the lookup table can also be derived from the morphological lexicons for different languages, although it costs some time and manpower, we can create the lookup table once for all since it represents the common knowledge with respect to a certain language.

we first perform an unsupervised morpheme segmentation using Morefessor (Creutz and Lagus, 2007) for the vocabularies.


• A thorough discussions of issues encountered when training this model.

we motivate the use of KL scaling as introduced by Bowman et al.


Topological ordering algorithms for DAG (Kahn, 1962;Tarjan, 1976) are not good solutions, because the outputted ordering is not always optimal for machine translation.

a topo- (2005), for all the experiments in this paper, we fixed λ to log 2 600.


This indicates syntactic information contained in the constituent trees or forests is indeed useful for machine translation.


To this end, we construct a simple discourse-aware model, demonstrate that it achieves improvements over the discourse-agnostic baseline on an English-Russian subtitles dataset (Lison et al., 2018) and study which context information is being captured in the model.

we start with the Trans-former (Vaswani et al., 2017), a state-of-the-art model for context-agnostic NMT, and modify it in such way that it can handle additional context.


We study the distribution of attention to context and perform analysis on specific subsets of the test data.

the research questions we seek to answer are as follows: • For the translation of which words does the model rely on contextual history most?


However, it conditions the generation of a target word not only on the previously generated words and the current source sentence (as in the vanilla NMT model), but also on all the other source sentences of the document and their translations.

the generation process is as follows: where y t,j is the j-th word of the t-th target sentence, y t,<j are the previously generated words, and x −t and y −t are as introduced previously.


the highest possible score(s) using Geonames coordinates as the geocoding output.

it quantifies the minimum error for each dataset given the perfect location disambiguation.


There could be many ways to define the weights, but for now we use the following definition from (Zhu et al., 2003), where σ 2 is a hyper-parameter 2 : To assess the similarity between locations, we measure the cosine similarity between vectors of their co-occurrence frequencies with activities.

let matrix We use the same value σ 2 = 0.03 as (Zhu et al., 2003).


Instead, we need a modeling which is more restrictive than a general graph so that annotators can focus on relation annotation (rather than looking for pairs first), but also more flexible than a single axis so that ill-defined  relations are not forcibly annotated.

we need axes for intentions, opinions, hypotheses, etc.


5 In our multiaxis modeling, the strategy adopted by TB-Dense falls into a more general approach, "axis projection".

projecting events across different axes to handle the incomparability between any two axes (not limited to HYPOTHE-SIS/NEGATION).


Thus, instead of having the seq2seq model learn patterns of language only from aligned parallel corpora, we assist the model by providing it closely related (similar) samples from the training data that it can refer to while generating text.

given a context c, we retrieve a set  Table 1: Sample input contexts and corresponding gold responses from Ubuntu validation dataset along with similar contexts seen in training data and their corresponding responses.


We study the problem of building an interactive natural language interface to databases (INLIDB) for synthesizing SQL queries from natural language questions.

our goal is to design a dialogue system to extract and validate potential errors in generated queries by asking users multi-choice questions over multiple turns.


As the number of choices increases, the performance of DialSQL improves in all the cases.

for the SQLNet-DialSQL model we observe more accuray gain.


Starting from this data, we construct a setting that mitigates effects which may trivialize the task.

some topical contexts (such as politics and religion) are naturally more susceptible to antisocial behavior (Kittur et al., 2009;Cheng et al., 2015).


We mitigate topical confounds using matching, a technique developed for causal inference in observational studies (Rubin, 2007).

start-ing from our human-vetted collection of conversations, we pair each awry-turning conversation, with an on-track conversation, such that both took place on the same talk page.


Instead, we seek to identify pragmatic markers within the initial exchange of a conversation that might serve to reveal or exacerbate underlying tensions that eventually come to the fore, or conversely suggest sustainable civility.

in this work we explore how politeness strategies and rhetorical prompts reflect the future health of a conversation.


Hence, we may naturally expect certain politeness strategies to signal that a conversation is likely to stay on track, while others might signal derailment.

we consider a set of pragmatic devices signaling politeness drawn from Brown and Levinson (1987).


First, we derive reply-vectors of phrasings, which reflect their propensities to co-occur.

we perform singular value decomposition on a termdocument matrix R of phrasings and replies as R ≈R = U R SV T R , where rows of U R are lowrank reply-vectors for each phrasing.


10 Focusing on the first comment (represented as ♦s), we find a rough correspondence between linguistic directness and the likelihood of future personal attacks.

comments which contain direct questions, or exhibit sentenceinitial you (i.e., "2 nd person start"), tend to start awry-turning conversations significantly more often than ones that stay on track (both p < 0.001).


In this paper, we remove this limitation by proposing a novel encoder-decoder architecture with a structured fixed-size representation of the input that still allows the decoder to explicitly focus on different parts of the input.

our NMT system has both the capacity to attend to various parts of the input and to produce static representations of input sentences.


The main motivation for incorporating these multiple "views" of the state sequence is that it removes the need for the RNN cell to accumulate the representation of the whole sentence as it processes the input, and therefore it should have more capacity for modeling local dependencies.

we fix a number r, the number of attention heads, and compute an r ×T matrix A of attention weights α jt , representing the importance of position t in the input for the j th attention head.


To investigate this effect, we include baseline models where we replace all context vectors c i with the entire sentence embedding (indicated by the suffix "-CTX" in Table 1).

we provide either the flattened matrix M (for models with inner attention; ATTN or ATTN-CTX), the final state of the encoder (FINAL-CTX), or the result of mean-or max-pooling (*POOL-CTX) as a constant input to the decoder cell.


In addition to measuring metric reliability, MAEGE can also be used to analyze the sensitivities of the metrics to corrections of different types, which to our knowledge is a novel contribution of this work.

we find that not only are valid edits of some error types better rewarded than others, but that correcting certain error types is consistently penalized by existing metrics (Section 7).


To compute linear correlation with Pearson r, we make the simplifying assumption that all edits contribute equally to the overall quality.

we assume that a perfect correction (i.e., the top of a chain) receives a score of 1.


Our analysis reveals that there is still a room for improvement in the way statistical significance is used in papers published in our top-tier publication venues.

a large portion of the surveyed papers do not test the significance of their results, or use incorrect tests for this purpose.


CLT establishes that, in most situations, when independent random variables are added, their properly normalized sum tends toward a normal distribution even if the original variables themselves are not normally distributed.

accuracy measures in structured tasks tend to be normally distributed when the number individual predictions (e.g.


The null hypothesis for this test states that the marginal probability for each outcome (label one or label two) is the same for both algorithms.

when applying both algorithms on the same data we would expect them to be correct/incorrect on the same proportion of items.


While cross-validation is surely a desired practice, it is challenging to report statistical significance when it is employed.

the test sets of the different folds are obviously not independent -their content is even likely to overlap.


Such networks employ two (sometimes identical) RNNs, one for encoding (i.e., generating H (x) ) and another for decoding.

for decoding y j , the decoder RNN takes the embedding y j−1 to generate a hidden vector h j as a single vector which is mapped into an output space for a sof tmax operation (Eq.


This leverages the state-of-the-art neural seq2seq models to learn and dynamically generate them.

our bspan has an information field (marked with <Inf></Inf>) to store values of informable slots since only values are important for knowledge base search.


However, as we need to explicitly model the dependency on B t , we have copy-attention mechanism on B t instead of on X: treating all tokens of B t as the candidate for copying and attention.

we use hidden state generated by B t GRU, i.e., h m , to calculate copying using Eqs.


To conduct our investigation, we pick the food type slot to simulate unknown values.

we randomly select about 35% of the food types in the training set (26 out of 74) as unknown and discard all the training instances where the correct food type is one of the 26 unknown types that we selected.


On the other hand, Mem2Seq is able to produce the correct responses in this two examples.

in the navigation domain, shown in Table 1, Mem2Seq produces a different but still correct utterance.


We use both quantitative metrics and human judgements to evaluate the proposed MGL model and the CVaR model.

we use two kinds of metrics for quantitative comparisons.


For the diverse-requirement scenario, we define two measures to evaluate the performance.


That's because both MMI and Adver-REGS further consider some reward functions in the optimization process to encourage specific results.

mmI uses a predefined reward function to penalize generating common responses, and Adver-REGS uses a learned discriminator to define the reward function.


Here we show some generated responses for demonstration.

table 3 gives one example post and its ground-truth responses from Ubuntu.


First, we study the influences of different parameter α in CVaR.

we show the validation result with α ranging from 0 to 0.9 with step 0.1, to see the change of CVaR performances.


From the results, we can see MGL and CVaR models achieve comparable results, which are significantly better than baseline methods.

the averaged score of MGL and CVaR is 2.15 and 1.995, which is significantly higher than that of Adver-REGS and Mechanism, i.e., 1.83 and 1.775, respectively.


post and its three ground-truth responses from STC.


The knowledge base K is a collection of knowledge facts in the form of triplets (subject, relation, object).

both subjects and objects are entities in this work.


Accuracy measures how accurately the sentence function can be controlled.

we compared the prespecified function (as input to the model) with the function of a generated response, which is predicted by the self-attentive classifier (see Section 4.1).


To further analyze how our model realizes the global control of sentence function, we presented frequent words and frequent word patterns within each function.

we counted the frequency of a function-related word in the generated responses.


Recently, more and more work combines all the modules in an end-to-end training framework (Wen et al., 2016;Li et al., 2017;Williams et al., 2017;Liu and Lane, 2017a).


We used openSMILE  to extract acoustic features.

we used the paralinguistics configuration from Schuller et al.


Following these approaches, we treated the problem of dialog policy learning as a classification problem, which is to select actions among system action templates given conversation history.

we decided to adopt the framework of Hybrid Code Network (HCN) introduced in Williams et al.


When the dialogic features were all zero, in most cases it meant the user didn't express an obvious sentiment, we set the reward to be -1.

if there were repetitions, which correlate with negative sentiment (see Table 3), we assigned a penalty to that utterance.


Furthermore, reward shaping (Ng et al., 1999;Ferreira and Lefèvre, 2013) is an important technique in RL.

ferreira and Lefèvre (2013) talked about incorporating expert knowledge in reward design.


As the first answer to this question, we propose to measure the affinity in morphological type by considering the sets of morphological features attested within each of the UD treebanks.

3 for each pair of a source language set M S and a target language set M T , we estimate their Jaccard index, which is defined for two sets as the cardinality of their intersection divided by the cardinality of their union, as shown in Equation 4.


4 This tree edit distance operates on ordered trees with node (but not edge) labels, hence it is suited for delexicalised dependencies.

it is defined over a map M , which is a list of node pairs where the former belongs to S or (empty node), and the latter belongs to T or .


We report its performance by BLEU scores (Papineni et al., 2002).

we use an attentional encoder-decoder network that jointly learns to translate and align words (Bahdanau et al., 2015) implemented in the Nematus suite 12 (Sennrich et al., 2017).


The weights are learned by a multilayer perceptron that estimates the likelihood of the alignment between the predicted word and each of the input words: This model is enriched with additional linguistic features on input, as proposed by Sennrich and Haddow (2016).

we select the following which are proven as useful in prior work, and also relevant to our experiment: word form, POS tag, and dependency relations.


Since TreeLSTM is a lexicalised model, we employ multilingual word embeddings, such that the words of both languages lie in the shared cross-lingual semantic space.

we map English into Arabic through the iterative Procustes method devised by Artetxe et al.


A similar conclusion can be reached by comparing the performance of TreeLSTM-based models on the cross-lingual STS task, reported in Table 4.

the Pearson correlation score increases by 3.3 points and MSE decreases by 0.18 points when our tree processing algorithm is applied.


In order to circumvent the data scarcity issue, in this paper we propose the use of linguisticallymotivated synthetically generated CM data (as a supplement to real CM data) for development of CM NLP systems.

we use the Equivalence Constraint Theory (Poplack, 1980;Sankoff, 1998) for generating linguistically valid CM sentences from a pair of parallel sentences in the two languages.


Chinese NER is correlated with word segmentation.

named entity boundaries are also word boundaries.


External sources of information has been leveraged for NER.

lexicon features have been widely used (Collobert et al., 2011;Passos et al., 2014;Luo et al., 2015).


Different from the character-based model, however, the computation of c c j now considers lexicon subsequences w d b,e in the sentence.

each subsequence w d b,e is represented using x w b,e = e w (w d b,e ), where e w denotes the same word embedding lookup table as in Section 3.2.


(2017a) to automatically segment the development and test sets for word-based NER.

for the OntoNotes and MSRA datasets, we train the segmentor using gold segmentation on their respective training sets.


We pretrain word embeddings using word2vec (Mikolov et al., 2013) over automatically segmented Chinese Giga-Word 6 , obtaining 704.4k words in a final lexicon.

the number of single-character, twocharacter and three-character words are 5.7k, 291.5k, 278.1k, respectively.


Table 3 shows the values of hyper-parameters for our models, which as fixed according to previous work in the literature without grid-search adjustments for each individual dataset.

the embedding sizes are set to 50 and the hidden size of LSTM models to 200.


In this paper, we propose Nugget Proposal Networks (NPNs), which can solve the word-trigger mismatch problem by directly proposing entire trigger nuggets centered at each character regardless of word boundaries.

nPns perform event detection in a character-wise paradigm, where a hybrid representation for each character is first learned to capture both structural and semantic information from both characters and words.


Unfortunately, word-wise event detection models suffer from the word-trigger mismatch problem, because a number of triggers do not exactly match with a word.

a trigger can be part of a word or cross multiple words, which is impossible to detect using word-wise models.


Therefore, we propose to learn a hybrid representation which can summarize information from both characters and words.

we first learn two separate character-level and word-level representations using token-level neural networks.


General Hybrid aims to learn a shared hybrid representation for both trigger nugget proposal and event type classification.

we design a gated structure to model the information flow from f char and f word to the general hybrid feature representation f G : Here s is the sigmoid function, W GH ∈ R d ×d and U GH ∈ R d ×d are weight matrix, and is the bias term.


Such KG construc-tion methods are schema-guided as they require the list of input relations and their schemata (e.g., playerPlaysSport(Player, Sport)).

knowledge of schemata is an important first step towards building such KGs.


Each cell in a slice corresponds to an induced schema in terms of the latent factors from matrices A and B.

g 3 i,j,k is an induced binary schema for relation k involving induced categories represented by columns A i and B j .


This model induces schemata by grouping perrelation NP arguments from OpenIE extractions.

for each relation, all the Noun Phrases (NPs) in first argument form a cluster that represents the subject of the relation, all the NPs in the second argument form a cluster that represents object and so on.


If we rewrite the pairwise scores as we can see that Equation 3 is a special case of ment-norm when K = 1 and D 1 = 0.

ganea and Hofmann (2017) is our monorelational ment-norm with uniform α.


While methods to perform reasoning over such structures exist (Verhagen et al., 2007(Verhagen et al., , 2010UzZaman et al., 2013;Llorens et al., 2015;Pustejovsky et al., 2003), none of them have exploited advances in deep learning (Krizhevsky et al., 2012;Goodfellow et al., 2016).

recently proposed Graph Convolution Networks (GCN) (Defferrard et al., 2016;Kipf and Welling, 2017) have emerged as a way to learn graph representation while encoding structural information and constraints represented by the graph.


In order to capture nodes many hops away, multiple GCN layers may be stacked one on top of another.

h k+1 v , representation of node v after k th GCN layer can be formulated as: where h k u is the input to the k th layer.


For demonstrating the efficacy of GCNs and BiL-STM for the problem, we evaluate different ablated variants of NeuralDater on the APW dataset.

we validate the importance of using syntactic and temporal GCNs and the effect of eliminating BiLSTM from the model.


One limitation of sequence-to-sequence models, however, is that they require serialization of input AMR graphs, which adds to the challenge of representing graph structure information, especially when the graph is large.

closely-related nodes, such as parents, children and siblings can be far away after serialization.


State transition A recurrent neural network is used to model the state transition process.

the transition from g t 1 to g t consists of a hidden state transition for each node, as shown in Figure 2.


In addition to edge inputs, a cell also takes the hidden states of its incoming nodes and outgoing nodes during a state transition.

the states of all incoming nodes and outgoing nodes are summed up before being passed to the cell and gate nodes: Based on the above definitions of j , the state transition from g t 1 to g t , as repre-sented by h j t , can be defined as: where i j t , o j t and f j t are the input, output and forget gates mentioned earlier.


Since our graph encoder generates a sequence of graph states, only the last graph state is adopted in the decoder.

we make the following changes to the decoder.


(2017) adopt anonymization for dealing with the data sparsity problem.

they first replace the subgraphs that represent dates, numbers and named entities (such as "(q / quantity :quant 3)" and "(p / person :name (n / name :op1 "Ryan"))") with predefined placeholders (such as "num 0" and "person name 0") before decoding, and then recover the corresponding surface tokens (such as "3" and "Ryan") after decoding.


The proposed model differs from existing non-linear LSTM models such as Tree LSTM (Tai et al., 2015) and Graph LSTM (Liang et al., 2016) in addressing the mentioned problem.

tree LStM does not allow cycles, while the proposed model handles cycles by first using a combination of topological sort and breadth-first traversal over a graph, and then using an attention model to capture the global information of the knowledge graph.


Once all the communication models have been trained, we learn the combined decoding objective.

we learn the weight coefficients λ k in equation 1 to linearly combine the scoring functions, using a discriminative loss where A is the inference algorithm for beam search decoding.


Under the crowd-sourced evaluation (Table 2), on BookCorpus our model is consistently favored over the baselines on all dimensions of comparison.

our model tends to be much less repetitive, while being more clear and relevant than the baselines.


Hence, we conduct a human evaluation study for the best 2 neural (GAC,GAC-sparse) and best 2 non-neural methods (TEMP,NN).

setup: annotators are shown a chess move through previous board and resulting board snapshots, along with information on which piece moved (a snapshot of a HIT 7 is provided in the Appendix D).


Despite the distinct evaluation metrics for the aforementioned tasks, the standard training algorithm has been the same for all of them.

the algorithm is based on maximum likelihood estimation (MLE), which maximizes the log- * Equal contribution.


On the one hand, posing an over strong en- tropy regularization can easily cause the actor to diverge.

the model diverges when τ reaches 0.03 on the image captioning task or 0.06 on the machine translation task.


To verify the importance of explicitly modeling the entropy from future steps, we compared ERAC with the variant that only applies the entropy regularization to the actor but not to the critic.

the τ is set to 0 when performing policy evaluating according to Eqn.


However, many problems remain widely open.

the oracle Q-function Q φ we obtain is far from perfect.


Scientific progress on this endeavor is measured through multiple tasks such as machine translation, reading comprehension, questionanswering, and others, each of which requires the machine to demonstrate the ability to "comprehend" the given textual input (apart from other aspects) and achieve their task-specific goals.

reading Comprehension (rC) systems are required to "understand" a given text passage as input and then answer questions based on it.


For 4 out of the 8 questions (Q1, Q2, Q4, and Q7), though the answers extracted from the two plots are exactly the same, the analysis required to arrive at this answer is very different in the two cases.

for Q1 even though there is no explicit mention of the prisoner living in a subterranean shelter and hence no lexical overlap with the question, the workers were still able to infer that the answer is Philadelphia because that is the city to which James Cole travels to for his mission.


Another notable observation is that in many cases the answers to the same question are different in the two versions.

only 40.7% of the questions have the same answer in the two documents.


However, in our dataset, in many cases the answers do not correspond to an exact span in the document but are synthesized by humans.

for the Sel-fRC version of the dataset around 30% of the answers are synthesized and do not match a span in the document whereas for the ParaphraseRC task this number is 50%.


The purpose of this model is to establish whether even for instances where the answer matches a span in the document, our dataset is harder than the SQuAD dataset or not.

we want to explore the performance of state-of-the-art models (such as DCN (Xiong et al., 2016)), which exhibit near human results on the SQuAD dataset, on DuoRC (especially, in the ParaphraseRC setup).


For this, we employ a state-of-theart query based abstractive summarization model (Nema et al., 2017) as this task is very similar to our task.

in query based abstractive summarization the training data is of the form {query, document, generated summary} and in our case the training data is of the form {query, mini-document, generated answer}.


While addressing all of these challenges in their entirety is beyond the scope of a single paper, in the interest of establishing a good baseline for DuoRC, we additionally seek to address some of these challenges to a certain extent by using standard NLP techniques.

we look at the problems of paraphrase understanding, coreference resolution and handling long passages.


Next, we gather all the information on passages by a simple concatenation of its contextual information H p and its question-aware representation H q • C: ( 2)Typically, a passage may contain hundred of tokens, making it hard to learn the long dependencies within it.

inspired by (Lin et al., 2017), we apply a self-attended layer to rearrange the information U p as: we first obtain an n × n attention matrix with U p onto itself, apply dropout, then multiply this matrix with U p to obtain an updated U p .


In the proposed method, attention and fusion are conducted horizontally and vertically across layers at different levels of granularity between question and paragraph.

it first encode the question and paragraph with fine-grained language embeddings, to better capture the respective representations at semantic level.


Passage Advanced Representation As more focused information of the candidate C is available, we are provided with a better way to confirm its correctness by rereading its corresponding passage P .

we equip each position t in P with following advanced features: • Passage contextual representation: the former passage representation s t P .


As shown in Table 4, all these components modeling the selection procedure play important roles in our final architecture.

introducing the independent representation of the question and its common words with the passage seems an efficient way to consider the information of questions, which is consistent with previous work (Li et al., 2016b;.


Yet, learning the full context over the document is challenging and inefficient.

when the model is given a long document, or multiple documents, learning the full context is intractably slow and hence difficult to scale to large corpora.


We find that most questions can be answered using a few sentences, without the consideration of context over entire document.

we observe that on the SQuAD dataset (Rajpurkar et al., 2016), 92% of answerable questions can be answered using a single sentence.


First, the encoder computes sentence embeddings is the dimension of word embeddings, and L d and L q are the sequence length of the document and the question, respectively.

question-aware sentence embeddings are obtained as follows.


The experimental results on real-world datasets including Quasar-T, SearchQA and TriviaQA show that our system achieves significant and consistent improvement as compared to all baseline methods by aggregating extracted answers of all informative paragraphs.

we show that our model can achieve comparable performance by selecting a few informative paragraphs, which greatly speeds up the whole DS-QA system.


The equation 9is further defined using two ways: (1) Max.

we assume that only one token in the paragraph indicates the correct answer.


Here, R(P ) is defined as the KL divergence between Pr(p i |q, P ) and a probability distribution X where X i = 1 c P (c P is the number of paragraphs containing correct answer in P ) if the paragraph contains correct answer, otherwise 0.

r(P ) is defined as: To solve the optimization problem, we adopt Adamax to minimize the objective function as described in (Kingma and Ba, 2015).


In the experiments, we show that our models significantly and consistently outperforms state-of-the-art DS-QA models.

we demonstrate that the performance of our model is hardly compromised when only using a few topselected paragraphs.


The second example in Figure 2 shows another case where source words are translated into target items that are not their possible translations in that or in any other context.

冬奥 会/winter olympics is incorrectly translated into a target comma "," and 载誉/honors into have.


(2) Target-side bridging model: In a second more bold strategy, we seek to incorporate relevant source word embeddings more closely into the prediction of the next target hidden state.

the most appropriate source words are selected according to their attention weights and they are made to more closely interact with target hidden states.


This is par- tially inspired by the word alignments from SMT, where words from the two ends are paired as they are possible translational equivalents of each other and those pairs are explicitly recorded and enter into the system inner workings.

for a given target word, we explicitly determine the most likely source word aligned to it and use the word embedding of this source word to support the prediction of the target hidden state of the next target word to be generated.


as the backbone network for translation, ranging from recurrent neural networks (RNN) (Sutskever et al., 2014;Luong et al., 2015), convolutional neural networks (CNN) (Gehring et al., 2017a,b) to full attention networks without recurrence and convolution (Vaswani et al., 2017).

the neural Transformer, relying solely on attention networks, has refreshed state-of-the-art performance on several language pairs (Vaswani et al., 2017).


Differing noticeably from the self-attention in the Transformer, our AAN can be accelerated in the decoding phase via dynamic programming thanks to the simple average calculation.


Although different languages have different linguistic and syntactic structures, our model consistently yields rather competitive results against the Transformer on all language pairs in both directions.

on the De→En translation task, our model achieves a slight improvement of 0.10/0.07 case-sensitive/case-insensitive BLEU points over the Transformer.


Layers optionally take set of named arguments Layer definitions For each layer we have a corresponding layer definition based on the hidden states of the previous layer and any additional arguments.

each layer takes T hidden states h i 1 , ..., h i T , which in matrix form are H i ∈ R T ×d i , and produces a new set of hidden states h i+1 1 , ..., h i+1 T or H i+1 .


Note that we use a slightly updated residual structure as implemented by tensor2tensor 1 than proposed originally.

layer normalization is applied to the input of the residual block instead of applying it between blocks.


As we show next, abstract examples can be used to improve the process of training semantic parsers.

in sections 5.1-5.3, we use abstract examples in several ways, from generating new training data to improving search accuracy.


For our application of counterfactual learning to human bandit feedback, we found another deviation from standard counterfactual learning to be helpful: For humans, it is hard to assign a graded reward to a query at a sequence level because either the query is correct or it is not.

with a sequence level reward of 0 for incorrect queries, we do not know which part of the query is wrong and which parts might be correct.


The commitment of our parser to derive AMRs compositionally mirrors that of grammar-based AMR parsers (Artzi et al., 2015;Peng et al., 2015).

there are parallels between the types we use in the AM algebra and CCG categories (see Section 3 for details).


Each source in an as-graph specifies with its annotation the type of the as-graph which is plugged into it via APP.

for a source a, we may only a-apply G P with G A if the annotation of the a-source in G P matches the type of G A .


As above, this rule ensures that a well-typed AM dependency tree is generated by locally checking the types.

if all types τ 2 that can be derived for k are incompatible with τ 1 , we fall back to an item for k with τ 2 = ⊥ (which always exists), along with an IGNORE edge from i to k. The complexity of this algorithm is O(n , where d is the maximal arity of the nodes in t r .


We will also look into more principled methods for splitting the AMRs into elementary as-graphs to replace our hand-crafted heuristics.

advanced methods for alignments, as in Lyu and Titov (2018), seem promising.


To address the efficiency challenge in learning from all training samples, we devise a regression-based loss function for word embedding, which allows fast optimization with memorization strategies.

the acceleration is achieved by reformulating the expensive loss over all negative samples using a partition and a decouple operation.


To speed up the training process, it applies the negative sampling (Mikolov et al., 2013b) to approximate the full softmax.

each positive (w, c) pair is trained with n randomly sampled negative pairs (w, w i ).


r wc is the association coefficient between w and c, which is calculated from M wc .

we use r + wc to denote the ground truth value for positive (w, c) pairs and a constant value r − (e.g., 0 or -1) for negative ones since there is no interaction between w and c in negative pairs.


This means that they can be pre-calculated before training in each iteration.

we define p w dd , p c dd , q w d and q c d as the pre-calculated terms Then the computation ofL A can be simplified to .


Although baseline methods like SG and GloVe can also be parallelized, the stochastic gradient steps in these methods unnecessarily influence each other as there is no exact way to separate these updates for different workers.

the parallelization of SGD is not well suited to a large number of work- ers.


We then take a detailed look and categorize the changes in intermediate trees by their correlations with the semantic graphs.

when a modifier m's head is changed from h to h in the  Table 3: Syntactic parsing performance (in unlabeled attachment score, UAS) and DM semantic parsing performance (in labeled F 1 ) on different groups of the development data.


In this work, we formulate learning AL strategies as an imitation learning problem.

we consider the popular pool-based AL scenario, where an AL agent is presented with a pool of unlabelled data.


More concretely, for each x x x ∈ D pool rnd and its correct label y y y , the underlying model m φ φ φt is re-trained to get is a small subset of the current large pool of unlabeled data.

the expert action is then computed as: our algorithmic expert tries a subset of actions to roll-out one step from the current state, in order to efficiently compute a reasonable action.


We further investigate combining the transfer of the policy network with the transfer of the underlying classifier.

we first train a classi- fier on all of the annotated data from the source domain/language.


In this work, we tap into the power of natural language and allow annotators to provide supervision to a classifier via natural language explanations.

we propose a framework in which annotators provide a natural language explanation for each label they assign to an example (see Figure 1).


We went one step further: using the LFs that would be produced by a perfect semantic parser as starting points, we searched for "nearby" LFs (LFs differing by only one predicate) with higher endtask accuracy on the test set and succeeded 57% of the time (see Figure 5 for an example).

when users provide explanations, the signals they describe provide good starting points, but they are actually unlikely to be optimal.


Thus, our research studies methods for incorporating coreference information into the training of a question generation system.

we propose gated Coreference knowledge for Neural Question Generation (CorefNQG), a neural sequence model with a novel gating mechanism that leverages continuous representations of coreference clusters -the set of mentions used to refer to each entity -to better encode linguistic knowledge introduced by coreference, for paragraph-level question generation.


More specifically, given the input sentence S (containing an answer span) and the preceding context C, we first run a coreference resolution system to get the coref-clusters for S and C and use them to create a coreference transformed input sentence: for each pronoun, we append its most representative non-pronominal coreferent mention.

we apply the simple feedforward network based mention-ranking model of Clark and Manning (2016) to the concatenation of C and S to get the coref-clusters for all entities in C and S. The C&M model produces a score/representation s for each mention pair where W m is a 1 × d weight matrix and b is the bias.


To address this problem, we propose an end-to-end neural model that enables those answer candidates from different passages to verify each other based on their content representations.

we jointly train three modules that can predict the final answer based on three factors: the answer boundary, the answer content and the cross-passage answer verification.


Here, we propose a novel method that can obtain the representation of the answer candidates based on probabilities.

we change the output layer of the classic MRC model.


Equation 1 x 11 = to + go After solving these statements, we can query the values of all variables.

we are interested in S, which is related to the desired natural language expression John want to go 3 .


Second, every rule has no-copying, no-deleting statements.

all variables must be used exactly once in a statement.


In this work, we tackle the task of derived word generation.

given the word "run," we attempt to generate the word "runner" for "someone who runs."


The notation is specific to our model, but the argument is applicable to seq2seq models in general.

the goal of decoding is to find the most probable structureŷ conditioned on some observation x and transformation t. the problem is to solvê where Y is the set of valid structures.


Upon receiving (x, t) at test time, we compute f t (v x ) and find the most similar embeddings in Y V .

we find the top-k most similar embeddings, and take the most similar derived word that starts with the same 4 letters as the root word, and is not identical to it.


Thou art more lovely and more temperate: Rough winds do shake the darling buds of May, And summer's lease hath all too short a date: of stresses.

we focus on sonnets and generate quatrains in iambic pentameter (e.g.


Despite excellent form, the output of our model can easily be distinguished from humanwritten poetry due to its lower emotional impact and readability.

there is evidence here that our focus on form actually hurts the readability of the resulting poems, relative even to the simpler language models.


We then propose trading-day alignment: we reorganize our inputs, including the tweet corpora and historical prices, by aligning them to these T trading days.

on the tth trading day, we recognize market signals from the corpus the historical prices p t on d t−1 , for predicting the movement y t on d t .


To handle the circumstance that multiple stocks are discussed in one single message, in addition to text information, we incorporate the position information of stock symbols mentioned in messages as well.

the layer consists of a forward GRU and a backward GRU for the preceding and following contexts of a stock symbol, s, respectively.


Inspired by the news-level attention (Hu et al., 2018), we weight messages with their respective salience in collective intelligence measurement.

we first project M t non-linearly to u t , the normalized attention weight over the corpus, where ζ(•) is the softmax function and W m,u ∈ R dm×dm , w u ∈ R dm×1 are model parameters.


We use Convolutional Neural Networks (CNNs) (LeCun et al., 1989) to obtain the representations of images.

we use Residual Net (ResNet) (He et al., 2016), which achieves state-of-the-art on ImageNet (Russakovsky et al., 2015) detection, ImageNet localization, COCO (Lin et al., 2014) detection, and COCO segmentation tasks.


The first layer takes an average of each sample and its immediate neighbours (labelled and unlabelled) using weights inÂ, and performs a linear transformation using W and b followed by a nonlinear activation function (σ).

for user u i , the output of layer l is computed by: 1 Code and data available at https://github.com/ afshinrahimi/geographconv  Figure 1: The architecture of GCN geolocation model with layer-wise highway gates (W i h , b i h ).


There is a tendency for users to be wrongly predicted to be in CA, NY, TX, and surpris-ingly OH.

users from states such as TX, AZ, CO, and NV, which are located close to CA, are wrongly predicted to be in CA, and users from NJ, PA, and MA are misclassified as being in NY.


This consistent behavior validates the machine reading capabilities and the improved document representation with external features of our model for answer selection.

the combination of document reading and word overlap features is required to be done in a soft manner, using a classification technique.


Notably, the trade-off hyperparameter β = σ −2 /2 is closely related to the variance of the distribution p(x|z).

by controlling the variance σ, the model can adaptively explore different trade-offs between the rate and distortion objectives.


Inspired by the observations above, we propose to inject data-dependent noise into latent variable z, rather than to setting the variance term σ 2 to a fixed value (Dai et al., 2017;Theis et al., 2017).

log σ 2 is obtained via a one-layer MLP transformation from g φ (x).


To facilitate comparison with prior work (Wang et al., 2013;Chaidaroon and Fang, 2017), the performance is measured with precision.

during testing, for a query document, we first retrieve the 100 nearest/closest documents according to the Hamming distances of the corresponding hash codes (i.e., the number of different bits).


It is observed that for the encoder networks, increasing the non-linearity by stacking MLP layers leads to better empirical results.

endowing the hash function with more modeling capacity is advantageous to retrieval tasks.


However, due to the constrained space of possible questions defined by QA-SRL's slot format, accuracy-based metrics can still be informative.

we report the rate at which the predicted question exactly matches the gold question, as well as a relaxed match where we only count the question word (WH), subject (SBJ), object (OBJ) and Miscellaneous (Misc) slots (see Table 1).


Some more recent semantic resources require less annotator training, or can be crowdsourced (Abend and Rappoport, 2013;Reisinger et al., 2015;Basile et al., 2012;Michael et al., 2018).

the original QA-SRL (He et al., 2015) dataset is annotated by freelancers, while we developed streamlined crowdsourcing approaches for more scalable annotation.


Semantic role labeling (SRL), namely semantic parsing, is a shallow semantic parsing task, which aims to recognize the predicate-argument structure of each predicate in a sentence, such as who did what to whom, where and when, etc.

we seek to identify arguments and label their semantic roles given a predicate.


In prior work of SRL, considerable attention has been paid to feature engineering that struggles to capture sufficient discriminative information, while neural network models are capable of extracting features automatically.

syntactic information, including syntactic tree feature, has been show extremely beneficial to SRL since a larger scale of empirical verification of Punyakanok et al.


We then assign the REtag (s) (that are associated with the matching RE) to either the matched sentence (for intent detection) or some matched phrases (for slot filling).

our REtags for intent detection are the same as the intent labels.


Because of the aforementioned reason, we use a simplified version of the two-side attention, where all the slot labels share the same set of positive and negative attention.

to predict the slot label of word i, we use the following equations, which are similar to Eq.


To answer Q1 , we also exploit the full few-shot learning setting.

for intent detection, we randomly select 5, 10, 20 training instances for each intent to form the few-shot training set; and for slot filling, we also explore 5, 10, 20 shots settings.


Since most existing few-shot learning methods require either many few-shot classes or some classes with enough data for training, we also explore the partial few-shot learning setting for intent detection to provide a fair comparison for existing few-shot learning methods.

we let the 3 most frequent intents have 300 training instances, and the rest remains untouched.


Our hyper-parameters for the BLSTM are similar to the ones used by Liu and Lane (2016).

we use batch size 16, dropout probability 0.5, and BLSTM cell size 100.


We choose to control the RE complexity by modifying the number of groups.

we reduce the number of groups for existing REs to decrease RE complexity.


We show that by exploiting the implicit knowledge encoded within REs, one can significantly improve the learning performance.

we observe that using REs to guide the attention module works best for intent detection, and using REtags as features is an effective approach for slot filling.


Although sequencelevel smoothing is computationally efficient compared to reinforcement learning approaches (Ranzato et al., 2016;Rennie et al., 2017), it is slower compared to MLE.

we need to forward each of the samples y l through the RNN in teacher-forcing mode so as to compute its hidden states h l t , which are used to compute the sequence MLE loss as To speed up training, and since we already forward the ground truth sequence in the RNN to evaluate the MLE part of α Seq (y * , x), we propose to use the same hidden states h * t to compute both the MLE and the sequence-level smoothed loss.


(2014)), which contains a total of 9,927 sentence pairs.

the dataset has a split of 4500/500/4927 among training, dev, and test.


Last but not least, in trying to assess the impact of attention mechanisms (in particular the progressive attention model), we notice that the extra mileage gained on different structural encodings is different.

performance lift on Linear Bi-LSTM > performance lift on Constituency Tree-LSTM, and PA struggles to see performance lift on dependency Tree-LSTM.


In the semantic odd man out (SOMO) task, we modified sentences by replacing a random noun or verb o with another noun or verb r. To make the task more challenging, the bigrams formed by the replacement with the previous and following words in the sentence have frequencies that are comparable (on a log-scale) with those of the original bigrams.

if the original sentence contains bigrams w n−1 o and ow n+1 , the corresponding bigrams w n−1 r and rw n+1 in the modified sentence will have comparable corpus frequencies.


We showed that different encoder architectures trained with the same objective with similar performance can result in different embeddings, pointing out the importance of the architecture prior for sentence embeddings.

we found that BiLSTM-max embeddings are already capturing interesting linguistic knowledge before training, and that, after training, they detect semantic acceptability without having been exposed to anomalous sentences before.


However, the traditional setting of relation extraction cannot satisfy this condition: the input sentences are independent of each other.

we cannot merely use the information of the sentence being processed as the state.


Require: , the fixed number of removal γ t , γ v 1: Load parameters θ from pre-trained policy network 2: Initialize s * as the all-zero vector with the same dimension of s j 3: Randomly sample a j ∼ π(a| s j ; θ); compute p j = π(a = 0| s j ; θ) 7: if a j == 0 then Rank T based on p j from high to low, obtain T rank 12: end for 15:  19 : 22: end for the ability to distinguish the obvious false positive instances, which means the discrimination of the indistinguishable wrong-labeled instances are still ambiguous.

this indistinguishable part is the criterion to reflect the quality of the agent.


If F 1 score increases in the epoch i, it means the actions of the epoch i is more reasonable than that in the epoch i − 1.


In this work, in order to control the updates in detail to keep a balance, we modify (4) to use a a step counter τ r for each relation r, counting "number of updates" instead of data points 2 .

whenever M r gets a nonzero update from a gradient calculation, τ r increases by 1.


We aim to integrate this auxiliary task with distributed relation representation learning into a recursive neural network.

we generate a dependency tree for each sentence from the dependency parser and construct a unified RNN that integrates an auxiliary task into the computation of each node.


Hence, the auxiliary task guides the learning of relation vectors, which in turn affects their correspondingly interactive words.

in Figure 1, the relation vector for "amod" is computed from the features of its child and parent words, and also used to produce the hidden representation of its parent.


RNSCN is built on the dependency tree of each sentence, which is pre-generated from a dependency parser.

each node in the tree is associated with a word w n , an input word embedding x n ∈ R d and a transformed hidden representation h n ∈ R d .


For each cross-domain task, we conduct both inductive and transductive experiments.

we train our model only on the training sets from both (labeled) source and (unlabeled) target domains.


On this behalf, our proposed model shows clear advantage over other baselines for this more difficult transfer problem.

we achieve 6.77%, 5.88%, 10.55% improvement over the bestperforming baselines for aspect extraction in R→L, L→D and D→L, respectively.


To further verify that the autoencoder indeed reduces label noise when the parser is inaccurate, we generate new noisy parse trees by replacing some relations within each sentence with a random    relation.

in each source domain, for each relation that connects to any aspect or opinion word, it has 0.5 probability of being replaced by any other relation.


To qualitatively show the effect of the auxiliary task with auto-encoders for clustering syntactically similar words across domains, we provide some case studies on the predicted groups of some words in Table 5.

for each relation in the dependency tree, we use (4) to obtain the most probable group to assign the word in the child node.


Our approach is based on the Dyna-Q framework (Sutton, 1990) where planning is integrated into policy learning for task-completion dialogue.

we incorporate a model of the environment, referred to as the world model, into the dialogue agent, which simulates the environment and generates simulated user experience.


But the world model is not updated afterwards.

the world model learning part in Algorithm 1 (lines 19-22) is removed.


In this model, word type is latent because we do not need to specify the type of a word explicitly.

each word can belong to any of the three types, but with different probabilities given the current context.


This demonstrates that our decoders produce more appropriate questions, with richer topics.

our decoders have substantially better willingness scores, indicating that questions generated by our models are more likely to elicit further interactions.


booking an airline flight) than displaying a personality.

many of the tasks and datasets available are constrained to narrow domains .


As before, the dialogue history is encoded via LST M enc , the final state of which is used as the initial hidden state of the decoder.

each entry we weight words by their inverse term frequency: where tf is computed from the GloVe index via Zipf's law 4 .


Our model separates the responsibilities for utterance representation and domain classification between the shared encoder and the domain-specific classifiers.

the shared encoder needs to be retrained only if it cannot encode an utterance well (e.g., a new domain introduces completely new words) and the existing domain classifiers need to be retrained only when the shared encoder changes.


In this paper, we focus on a multimodal structure to leverage the advantages of each data source.

given an utterance, we consider the linguistic content and acoustic characteristics together to recognize the opinion or emotion.


We calculate the frame-level attention distribution f α ij by using f e ij as the input for equation 4.

we form the word-level acoustic vector f V i by taking a weighted sum of bidirectional contextual state f h ij of the frame and the corresponding framelevel attention distribution f α ij word-level Attention aims to capture the word-level acoustic attention distribution w α i based on formed word vector f V i .


While previous works have been successful with respect to accuracy metrics, they have not created new insights on how the fusion is performed in terms of what modalities are related and how modalities engage in an interaction during fusion.

to understand the fusion process one must first understand the n-modal dynamics .


In the tensor dot product W ⋅ Z, the weight tensor W can be then viewed as d h order-M tensors.

the weight W can be partitioned into h .


For binary classification and multiclass classification, we report F1 score and accuracy Acc−k where k denotes the number of classes.

acc−2 stands for the binary classification.


Our current experiments have provided some such evidence.

we have shown that participant responses to systematically manipulated passages involving discourse adverbials can be explained in terms of both the lexical semantics of discourse adverbials and properties of the passages that contain them.


We show that the method can capture discourse structures in narrative text across multiple genres, including biographies, stories, and newswire reports.

our method can capture biographical templates from Wikipedia, and is competitive with state-ofthe-art generative approaches on tasks such as predicting the outcome of a story, and sentence ordering.


Our method is scalable, and can incorporate a broad range of features.

the approach can work on simple tokenized text.


On the other hand, in order to induce spatial proximity in the grid to mimic the sequential flow of discourse in documents, we constrain the transition matrix P (which specifies transition preferences from one position in the grid to another) to a sparse banded matrix.

a position i = (i x , i y , i z ) in the grid can only transition to itself, its 4 neighbors in the same XY plane, and two cells in the succeeding two layers along the Z-axis ( .


This paper presents a joint inference framework for them using constrained conditional models (CCMs).

we formulate the joint problem as an integer linear programming (ILP) problem, enforcing constraints that are inherent in the nature of time and causality.


Assuming the availability of a temporal extraction system and a causal extraction system, the proposed joint framework combines these two using a constrained conditional model (CCM) (Chang et al., 2012) framework, with an integer linear pro-gramming (ILP) objective (Roth and Yih, 2004) that enforces declarative constraints during the inference phase.

these constraints include: (1) A cause must temporally precede its effect.


This paper also presents an empirical result of improving the temporal extraction component.

we incorporate explicit time expressions present in the text and high-precision knowledge-based rules into the ILP objective.


Besides the transitivity constraints represented by Y 1 above, we also propose to enforce prior knowledge to further constrain the search space for Y .

linguistic insight has resulted in rules for predicting the temporal relations with special syntactic or semantic patterns, as was done in CAEVO (a state-of-the-art method).


In the CCM framework, these rules can be represented as hard constraints in the search space for Y .

where J (rule) ⊆ MM is the set of pairs that can be determined by linguistic rules, and rule(j) ∈ R T is the corresponding decision for pair j according to these rules.


(3) can be written as: The prior knowledge represented as Y and W Y can be conveniently converted into constraints for this optimization problem.

y 1 has two components, symmetry and transitivity: wherer is the reverse relation of r (i.e.,b = a, i = ii,s = s, andv = v), and Trans(r 1 , r 2 ) is defined in Table 1.


The "causal only" baseline is to use s c (•) alone for the prediction of each pair.

for a pair i, if s c {i → causes} > s c {i → caused by}, we then assign "causes" to pair i; otherwise, we assign "caused by" to pair i.


When using gold temporal relations, causal accuracy went up to 91.9%.

91.9% of the C-Links satisfied the assumption that the cause is temporally before the effect.


This is extremely useful when the information need (i.e., the categories of interest) is dynamic and the text collection is large and temporally updated (e.g., the possible categories are hard to know).

we propose a novel deep relevance model for zero-shot document filtering, named DAZER.


To achieve this purpose, we extract the relevance signals based on the hidden feature interactions between the category and each word in the embedding space.

two element-wise operations are utilized in DAZER: element-wise subtraction and element-wise product.


Inspired by these observations, we use these two kinds of element-wise interactions to complement the representation of a word in a document.

for each word w in document d, we derive its interaction-based representation e c w towards category c as follows: where ⊕ is the vector concatenation operation.


We utilize m convolution filters to extract the relevance signals for each word based on its local window of size l in the document.

after calculating the interaction-based representation d = (e c 1 , e c 2 , ..., e c |d| ) for document d and category c, we apply the convolution operation as follows: where r i ∈ R m is the hidden features regarding the relevance signal extracted for i-th word, W 1 ∈ R m×3le(2l+1) and b 1 ∈ R m are the weight matrix and the corresponding bias vector respectively, e c i−l:i+l refers to the concatenation from e c i−l to e c i+l .


Otherwise, the proposed DAZER may not generalize well to unseen categories.

we introduce an category classifier over h c,d to calculate the probability that h c,d belongs to each category seen during the training: where W 4 ∈ R C×la and b 4 ∈ R C are the weight matrix and bias vector for the classifier, C is the number of categories covered by the training set.


The parameters Θ = {θ, φ, δ} are optimized via back propagation and stochastic gradient descent.

we utilize Adam (Kingma and Ba, 2014) algorithm for parameter update over minibatches.


First, we can see that either element-wise subtraction or product contributes signifcantly to the performance improvement.

from Table 6, we can see that both the element-wise subtraction and element-wise product play equally on Movie Review dataset.


By excluding e prod c,w , the MAP performance of DAZER is reduced from 0.782 to 0.045.

the element-wise product is more critical for extracting relevance signals for topical categorization.


One possible solution is to enrich the architecture of DAZER to allow few-shot document filtering.

the filtering decisions of high-confidence are utilized to derive more seed words for better filtering performance.


Phrases with the same k words will always have the same representation no matter where they are.

we incorporate the position-invariance into RNN by disconnecting the information flow of RNN.


It also provides a good empirical way for us to choose the optimal window size.

conducting experiments on a small dataset first to select the optimal window size.


We use • for function composition, and ∆ p for the set of one hot vectors in dimension p. Given a training set S = {(X n , y n )} N n=1 of pair-wise data, where X ∈ X is the text sequence, and y ∈ Y is its corresponding label.

y is a one hot vector in single-label problem and a binary vector in multi-label problem, as defined later in Section 4.1.


A simple way to measure the compatibility of label-word pairs is via the cosine similarity whereĜ is the normalization matrix of size K×L, with each element obtained as the multiplication of 2 norms of the c-th label embedding and l-th word embedding: To further capture the relative spatial information among consecutive words (i.e., phrases 1 ) and introduce non-linearity in the compatibility measure, we consider a generalization of (2).

for a text phase of length 2r + 1 centered at l, the local matrix block G l−r:l+r in G measures the label-to-token compatibility for the "label-phrase" pairs.


LEAM greatly improves the performance by considering nonlinear interaction between phrase and labels.

we note that the text embedding in PTE is similar with a very special case of LEAM, when our window size r = 1 and attention score β is uniform.


For the CNN, we assume the same size m for all filters.

h represents the dimension of the hidden units in the LSTM or the number of filters in the CNN; R denotes the number of blocks in the Bi-BloSAN; P denotes the final sequence representation dimension.


A given patient may have multiple diagnoses, and thus multi-label learning is required.

we consider an open-access dataset, MIMIC-III (Johnson et al., 2016), which AUC F1 Model Macro Micro Macro Micro P@5 Logistic Regression 0.829 0.864 0.477 0.533 0.546 Bi-GRU 0.828 0.868 0.484 0.549 0.591 CNN (Kim, 2014) 0.876 0.907 0.576 0.625 0.620 C-MemNN (Prakash et al., 2017) 0.833 ---0.42 Attentive LSTM (Shi et al., 2017) -0.900 -0.532 -CAML (Mullenbach et al., 2018) 0.875 0.909 0.532 0.614 0.609 LEAM 0.881 0.912 0.540 0.619 0.612 Table 5: Quantitative results for doctor-notes multi-label classification task.


We model the text quality rating on a scale of 1 to 10, using the three scores as input.

where Quality(T ) is the text quality rating of the text T .


Texts with poor coherence may lead readers to fixate more on different portions of text to understand them.

such gaze behaviour indirectly captures the effort needed by human readers to comprehend the text (Just and Carpenter, 1980), which, in turn, may influence the ratings given by them.


It can be viewed as a language model builds on a corpus with all entities replaced by their type.

assume the training corpus consists of x = {w 1 , w 2 , .., w n }.


The entity composite model predicts the next word based on modeling the conditional probability P (w|w, s(w), s(w), θ v ), which can be derived by where Ω(s(w)) is the set of words of the same type with w. To model the types of context word s(w) in P (w|w, s(w); θ v ), we consider learning a type embedding along with the word embedding by augmenting each word vector with a type vector when learning the underlying word representation.

we represent each word w as a vector of are the word vectors and type vectors learned by the model from the training corpus, respectively.


A blank place in this task will contain an ingredient and we check whether our model can predict it correctly.

we choose six ingredients from different frequency range (low, mid, high) based on how many times they have appeared in the training corpus.


In this section, we discuss the incorporation of basic, lexical-level semantic knowledge into neural NLI components.

we consider external lexical-level inference knowledge between word w i and w j , which is represented as a vector r ij and is incorporated into three specific components shown in Figure 1.


G is a nonlinear mapping function to reduce dimensionality.

we use a 1-layer feed-forward neural network with the ReLU activation function with a shortcut connection, i.e., concatenate the hidden states after ReLU with the input n j=1 α ij r ij (or m i=1 β ij r ji ) as the output a m i (or b m j ).