The results without position embeddings (words) show that position information is crucial for convolutional encoders.

for shallow models (Pooling and Convolutional), whereas deeper models are less effected.


2016, we choose another bidirectional approach to process the sequence in order to learn more temporal dependencies in this work.

an RNN layer processes the input sequence in forward direction.


About Length A more detailed comparison between DEEPLAU (4 layers encoder and 4 layers decoder), DEEPLAU(2 layer encoder and 2 layer decoder) and DEEPGRU (4 layers encoder and 4 layers decoder), suggest that with deep architectures are essential to the superior performance of our system.

we test the BLEU scores on sentences longer than {10, 20, 30, 40, 50, 60} on the merged test set.


In our work, we leverage the fact that we are generating rationales, where there is a sense of progression within the rationale.

we assume that the rationale solves the problem step by step.


To deal with this, we use a training method we call staged back-propagation which saves memory by considering slices of K tokens in z, rather than the full sequence.

to train on a minibatch where |z| = 300 with K = 100, we would actually train on 3 mini-batches, where the first batch would optimize for the first z 1:100 , the second for z 101:200 and the third for z 201:300 .


However, just training on a large corpus of poetry data is not enough.


The model is trained using stochastic gradient descent to predict the next phoneme given a sequence of phonemes.

we maximize a multinomial logistic regression objective over the final softmax prediction.


Orthographic decoding When decoding from phonemes back to orthographic symbols, the goal is to compute the most likely word corresponding to a sequence of phonemes.

we compute the most probable hypothesis word W given a phoneme sequence ρ: We can consider the phonetic encoding of plaintext to be a homophonic cipher; that is, a cipher in which each symbol can correspond to one or more possible decodings.


However, the fact that they require training data with internal poetic consistency implies that they do not generalise to other forms of poetry.

in order to generate poetry in Dactylic Hexameter (for example), a phonetic model must be trained on a corpus of Dactylic poetry.


These issues are circumvented by applying lightly supervised learning to determine the contextual stress pattern of any word.

we exploit the latent structure in our corpus of sonnet poetry, namely, the fact that sonnets are composed of lines in rigid Iambic Pentameter, and are therefore exactly ten syllables long with alternating syllabic stress.


However, this forces us to generate poetry according to discrete themes and styles from pretrained models, requiring a new training corpus for each model.

we would suffer from similar limitations as with the phonetic-level model, in that we require a dedicated corpus.


Since these devices can be orthographically described by the repetition of identical sequences of characters, we can apply the same heuristic to boost the probability of sampling character strings that have previously been sampled.

to sample a line with many instances of alliteration (multiple words with the same initial sound) we record the historical frequencies of characters sampled at the beginning of each previous word.


Another drawback is that, because the input representation derived from a text is relatively close to its surface form 2 , the NLG task is mostly restricted to surface realisation (mapping input to sentences).

these benchmarks give very limited support for learning models that can handle the interactions between micro-planning subtasks.


In contrast, input patterns are inputs where subject and object have been abstracted over.


Again, despite the much larger size of the RNNLG dataset, the number of input patterns in both datasets is almost the same.

the relative variety in input patterns is higher in WEBNLG.


In this paper, we propose an end-to-end question answering system called COREQA in sequence-to-sequence learning, which incorporates copying and retrieving mechanisms to generate natural answers within an encoder-decoder framework.

in COREQA, the semantic units (words, phrases and entities) in a natural answer are dynamically predicted from the vocabulary, copied from the given question and/or retrieved from the corresponding knowledge base jointly.


Therefore, facing the above challenges, this paper proposes a neural generative model called COREQA with Seq2Seq learning, which is able to reply an answer in a natural way for a given question.

we incorporate COpying and REtrieving mechanisms within Seq2Seq learning.


Therefore, we have adopted another two set of SUs X Q and X KB which cover words/entities in the source question and the partial KB.

we have adopted the instance-specific vocabulary V ∪ X Q ∪ X KB for each question.


And the three modes could compete with each other through a sof tmax function in generating target SUs with the shared normalization term (as shown in Figure 2.

the scoring functions of each mode are defined as follows: Predict-mode: Some generated words need reasoning (e.g.


Task: To test the performance of the proposed approach in open domains, we modify the task of GenQA (Yin et al., 2016) for supporting multifacts (a typical example is shown in Figure 1).

a natural QA system should generate a sequence of SUs as the natural answer for a given natural language question through interacting with a KB.


Combined with the originally published corpus, we create a lager and better-quality data for natural question answering.

an Integral Linear Programming (ILP) based method is employed to automatically construct "grounding" Q-A pairs with the facts in KB (inspired by the work of adopting ILP to parse questions (Yahya et al., 2012)).


We employ two annotators to rate such three aspects of Copy-Net, GenQA and COREQA.

we sample 100 questions, and conduct C 2 3 = 3 pair-wise comparisons for each question and count the winning times of each model (comparisons may both win or both lose).


Gold Answer Generated   In this paper, we propose an end-to-end system to generate natural answers through incorporating copying and retrieving mechanisms in sequenceto-sequence learning.

the sequences of SUs in the generated answer may be predicted from the vocabulary, copied from the given question and retrieved from the corresponding K-B.


In our QA task, we filter out the completely unrelated facts to save time.

we first collect all the topic entities of all the questions as initial set.


To employ the global information in our training process, we adopt a multi-task training strategy.

we perform KB-QA training and TransE training in turn.


For instance, from Figure 3, we can see that location.country is paying great attention to "Where", indicating that "Where" is much more important than the other parts in the question when dealing with this type.

the other parts are not that crucial since "Where" is strongly implying that the question is asking about a location.


Firstly, we consider the impacts of the different answer aspects when representing the question, and propose a novel cross-attention model for KB-QA.

we employ the focus of the answer aspects to each question word and the attention weights of the question towards the answer aspects.


Our goal in this work is to learn to translate between pairs of languages generated by different policies.

we assume that we have access to two policies for the same game: a "robot policy" π r and a "human policy" π h .


If tr respects the semantics of z r , then the bilingual pair a and b achieves only boundedly worse reward than the monolingual pair a and b.

if So as discussed in Section 4, even by committing to a semantic approach to meaning representation, we have still succeeded in (approximately) capturing the nice properties of the pragmatic approach.


As compared to Experiment 1 where models achieved similar performance, differences are more pronounced in the zero-shot setup, as shown in Table 4.

we find that the SIM-WAP model which induces individual predictors for words that have not been observed in the training data is clearly more successful than TRANS-FER or WAC that project predictions into the distributional space.


We use the Hierarchical Co-Attention model proposed by (Lu et al., 2016) that co-attends to both the image and the question to solve the task.

we evaluate the 'alternate' version, i.e.


The higher the values of these variables, the more the models tend to provide the wrong output.


the higher the frequency of both the original word and the foil one, the more the models fail.


Caption length turned out to be only a partially reliable predictor in the LSTM + norm I model, whereas it is a reliable predictor in HieCoAtt.

the longer the caption, the harder for the model to spot that there is a foil word that makes the caption wrong.


Our interpretation is that this behaviour is related to the discriminative/generative nature of our tasks.

t1 and t2 are discriminative tasks and LStM + norm I and HieCoAtt are discriminative models.


In this paper, we present an approach to acquire trivial physical knowledge from unstructured natural language text as an alternative knowledge source.

we focus on acquiring relative physical knowledge of actions and objects organized along five dimensions: size, weight, strength, rigidness, and speed.


(2013) show that allowing some degree of non-monotonicity, by using a limited set of non-monotonic actions that can repair past mistakes and replace previously-built arcs, can increase the accuracy of a transition-based parser.

they present a modified arc-eager transition system where the Left-Arc and Reduce transitions are non-monotonic: the former is used to repair invalid attachments made in previous states by replacing them with a leftward arc, and the latter allows the parser to link two words with a rightward arc that were previously left unattached due to an erroneous decision.


First of all, we adapt the computation of the set of individually unreachable arcs U(c, t G ) to the new algorithm.

if c has focus words i and j (i.e., c = λ 1 |i, λ 2 , j|B, A ), then an arc x → y is in U(c, t G ) if it is not in A, and at least one of the following holds: • j > max(x, y), (i.e., we have read too far in the string and can no longer get max(x, y) as Shift: Figure 2: Transitions of the non-monotonic Covington non-projective dependency parser.


We also carry out some error analysis to provide some insights about how non-monotonicity is improving accuracy with respect to the original Covington parser.

we notice that nonmonotonicity tends to be more beneficial on projective than on non-projective arcs.


On average, our non-monotonic algorithm obtains better performance than the monotonic version, regardless of which of the variants of the loss calculation is used.

one of the loss expressions developed proved very promising by providing the best average accuracy, in spite of being the farthest approximation from the actual loss.


We introduce a crowd representation in which a worker vector represents the noise associated with her labels.

the parameters in the original architecture learns the correct sequence labeling model while the crowd vectors add noise to its predictions to 'explain' the lower quality of the labels.


Several examples of these tasks exist in prior work: using labels to moderate politeness in machine translation results (Sennrich et al., 2016), modifying the output language of a machine translation system (Johnson et al., 2016), or controlling the length of a summary in summarization (Kikuchi et al., 2016).

however, we are motivated by the task of morphological reinflection (Cotterell et al.,1 An implementation of our model are available at https://github.com/violet-zct/ MSVED-morph-reinflection.


we can express z as a deterministic variable z = g φ ( , x) where is an independent Gaussian noise variable ∼ N (0, 1).


f (x; y, z, θ) calculates the likelihood of x, a function parametrized by deep neural networks.

we employ an RNN decoder to generate the target word conditioned on the lemma variable z and linguistic labels y, detailed in §5.


We use Monte Carlo methods to estimate the expectation over the posterior distribution q(z|x) and q(y|x) inside the objective function 14.

we draw Gumbel noise and Gaussian noise one at a time to compute the latent variables y and z.


It can be seen that most errors of MED and our models can be ascribed to either over-copy or under-copy of characters.

from the complete outputs we observe that our model tends to be more aggressive in its changes, resulting in  it performing more complicated transformations, both successfully (such as Maltese "ndammhomli" to "tindammhiex") and unsuccessfully ("tqożżx" to "qażżejtx").


This paper seeks to scale up Bayesian learning of RNNs to meet the challenge of the increasing amount of "big" sequential data in natural language processing, leveraging recent advances in stochastic gradient Markov Chain Monte Carlo (SG-MCMC) algorithms (Welling and Teh, 2011;Ding et al., 2014;Li et al., 2016a,b).

instead of training a single network, SG-MCMC is employed to train an ensemble of networks, where each network has its parameters drawn from a shared posterior distribution.


Given a test inputX (with missing outputỸ), the uncertainty learned in training is transferred to prediction, yielding the posterior predictive distribution: When the input is a sequence, RNNs may be used to parameterize the input-output relationship.


Long Short-Term Memory The LSTM architecture addresses the problem of learning longterm dependencies by introducing a memory cell, that is able to preserve the state over long periods of time.

each LSTM unit has a cell containing a state c t at time t. This cell can be viewed as a memory unit.


In word-level language modeling, the input to the network is a sequence of words, and the network is trained to predict the next word in the sequence with a softmax classifier.

for a length-T sequence, denote y t = x t+1 for t = 1, .


This only affects 172 samples across the whole dataset, and is only done during training.

we evaluate our models across all the test examples.


Encouragingly, our work proves to be fully competitive with the sequence-labeling approach by Bollmann and Søgaard (2016), without requiring a prior character alignment.

we demonstrated the aptitude of multi-task learning to mitigate the shortage of training data for the named task.


A straightforward modeling of such information has been obtained in statistical language learning with Tree Kernels (TKs) (Collins and Duffy, 2001), or by means of structured neural models (Hochreiter and Schmidhuber, 1997;Socher et al., 2013).

kernel-based methods (Shawe-Taylor and Cristianini, 2004) have been largely applied in language processing for alleviating the need of complex activities of manual feature engineering (e.g., (Moschitti et al., 2008)).


The latter is exploited through recent results in distributional models of lexical semantics, as proposed in word embedding methods (e.g., (Mikolov et al., 2013;Sahlgren, 2006).

we adopt the Smoothed Partial Tree Kernel (SPTK) described in Croce et al.


Their dimensionality depends on the dimensionality of the last hidden layer (called O −1 ) and the number |Y | of different classes, i.e., H O ∈ R h O −1 ×|Y | and b O ∈ R 1×|Y | , respectively.

this layer computes a linear classification function with a softmax operator so thatŷ = sof tmax( In order to avoid over-fitting, two different regularization schemes are applied.


Each node is considered as a candidate in covering a possible argument.


(2011) the authors propose a hybrid classifier, for bridging kernel methods and neural networks.

they use the output of a kernelized k-nearest neighbors algorithm as input to a neural network.


As shown in Figure 2, we take the learnt embeddings τ of reviews by the CNN as the tail part.

we denote the review text consisting of n words as {w 1 , w 2 , ..., w n }, the word embeddings e(w i ) ∈ R D , D is the word vector dimension.


yields 1 only when the condition described in the argument holds and 0 otherwise.

equation 5 computes the proportion of correct predictions to all predictions, assuming zero to be the decision boundary between pro and con.


Here the xaxis represents a threshold θ, which filters out users whose declarations of topic preferences are no greater than θ topics.

figure  3 shows prediction accuracy when we know user preferences for at least θ topics.


As the success of word embedding in capturing semantics of words (Turian et al., 2010), we employ word embedding to map the events in Freebase to frames in FrameNet.

we use the average word embedding of all words in i-th Freebase event type name e i and word embedding of k-th lexical units of j-th frame e j,k to compute the semantic similarity.


However, the method can only extract arguments of one plane crash type and need flight number strings as input.

the approach cannot extract whole event with different types automatically.


Based on the findings, we propose a type-based approach named SynTime 1 for time expression recognition.

we define three main syntactic token types, namely time token, modifier, and numeral, to group time-related token regular expressions.


All the findings relate to the principle of least effort (Zipf, 1949).

people tend to act under the least effort in order to minimize the cost of energy at both individual level and collective level to language usage (Zipf, 1949).


According to the findings we propose a typebased approach named SynTime ('Syn' stands for syntactic) to recognize time expressions.

we define three main token types, namely time token, modifier, and numeral, to group timerelated token regular expressions.


(The test for other languages needs only to construct a collection of token regular expressions in the target language under our defined token types.)

we evaluate SynTime against three state-of-the-art methods (i.e., HeidelTime, SUTime, and UWTime) on three datasets: TimeBank, WikiWars, and Tweets.


Although the texts are collected from different sources (i.e., news articles, Wikipedia articles, and tweets) and vary in sizes, the length  of time expressions follow a similar distribution.

the one-word time expressions range from 36.23% in WikiWars to 62.91% in Tweets.


This indicates that time tokens, which account for time expressions, are highly overlapped across the four datasets.

time expressions highly overlap at their time tokens.


The four findings all relate to the principle of least effort (Zipf, 1949).

people tend to act with least effort so as to minimize the cost of energy at both individual and collective levels to the language usage (Zipf, 1949).


The token regular expressions for initializing SynTime are collected from SUTime, 6 a state-of-the-art rule-based tagger that achieved the highest recall in TempEval-3 (Chang and Manning, , 2013.

we collect from SUTime only the tokens and the regular expressions over tokens, and discard its other rules of recognizing full time expressions.


The task of distantly supervised relation extraction is to extract knowledge triples, <subj, rel, obj>, from free text with the training data constructed by aligning existing KB triples with a large corpus.

given a triple in KB, DS works by first retrieving all the sentences containing both subj and obj of the triple, and then constructing the training data by considering these sentences as support to the existence of the triple.


To tackle the challenge of no direct guidance over the noise patterns, we implement a curriculum learning based training method to first train the model without considerations for noise.

we first focus on the loss from the prediction distribution p , and then take the noise modeling into account gradually along the training process, i.e., gradually increasing the importance of the loss from the observed distribution o while decreasing the importance of p. In this way, the prediction branch is roughly trained before the model managing to characterize the noise, thus avoids being stuck into poor local optimum.


We thus design to minimize the following loss function: where 0<α≤1 and β>0 are two weighting parameters, y i is the relation assigned by DS for the i-th instance, N the total number of training instances, o iy i is the probability that the observed relation for the i-th instance is y i , and p iy i is the probability to predict relation y i for the i-th instance.

initially, we set α=1, and train our model completely by minimizing the loss from the prediction distribution p. we do not expect to model the noise, but focus on the prediction branch at this time.


Moreover, as such a method does not rely on any extra assumptions, it can serve as our default training method for T. With Prior Knowledge of Data Quality On the other hand, if we happen to have prior knowledge about which part of the training data is more reliable and which is less reliable, we can utilize this knowledge as guidance to design the curriculum.

we can build a curriculum by first training the prediction branch on the reliable data for several epochs, and then adding the less reliable data to train the full model.


Furthermore, we can take better control of the training procedure with trace regularization, e.g., encouraging larger trace(T) for reliable subset and smaller trace(T) for less relaibale ones.

we propose to minimize: where β m is the regularization weight for the m-th data subset, M is the total number of subsets, N m the number of instances in m-th subset, and T mi , y mi and o mi,y mi are the transition matrix, the relation labeled by DS and the observed probability of this relation for the i-th training instance in the m-th subset, respectively.


We also use trace regularization for the most reliable subset, since there are still some noise annotations inevitably appearing in this split.

we expect its trace(T) to be large (using a positive β) so that the elements of T will be centralized to the diagonal and T will be more similar to the identity matrix.


As for the less reliable subset, we expect the trace(T) to be small (using a negative β) so that the elements of the transition matrix will be diffusive and T will be less similar to the identity matrix.

the transition matrix is encouraged to characterize the noise.


We choose to use time-related relations because time expressions speak for themselves in terms of reliability.

given a KB triple <e, rel, t> and its aligned sentences, the finergrained the time expression t appears in the sentence, the more likely the sentence supports the existence of this triple.


Evaluation Metric The performance is reported using the precision-recall (PR) curve, which is a standard evaluation metric in relation extraction.

the extraction results are first ranked decreasingly by their confidence scores, then the precision and recall are calculated by setting the threshold to be the score of each extraction result one by one.


We also compare our dynamic transition matrix method with the global transition matrix method, which maintains only one transition matrix for all training instances.

instead of dynam-ically generating a transition matrix for each datum, we first initialize an identity matrix T ∈ R |C|×|C| , where |C| is the number of relations (including no-relation).


(•) is a DNN with a single hidden layer.

if w i is an OOV word (e.g., the variable name my list), which is represented by a special <unk> token during encoding, we then directly copy the actual word w i from the input description to the derivation.


Finally, it is worth mentioning that, even if all the three language pairs show the same general behavior, there are clear differences in their absolute accuracy numbers, which can be attributed to the linguistic proximity of the languages involved.

the results for English-Finnish are about 10 points below the rest, which is explained by the fact that Finnish is a non-indoeuropean agglutinative language, making the task considerably more difficult for this language pair.


The word spans tagged by the SG network are used to determine the features for the other networks.

every span identified as a predicate will trigger the system to evaluate the Args network in order to determine the probabilities of outgoing predicate ARG relations.


Our constrained decoding algorithm described in Section 2.2 enables us to inject syntax as a decoding constraint without having to retrain the model.

if the decoded sequence contains k arguments that do not match any unlabeled syntactic constituent, it will receive a penalty of kC, where C is a single parameter dictating how much the model should trust the provided syntax.


These agents can perform simple tasks, answer factual questions, and sometimes also aimlessly chit-chat with the user, but they still lag far behind a human assistant in terms of both the variety and complexity of tasks they can perform.

they lack the ability to learn from interactions with a user in order to improve and adapt with time.


Neural Policy Network: For the neural approach, similar to (Williams and Zweig, 2016; Zhao and Eskenazi, 2016), we use an RNN to allow the network to maintain an internal state of dialogue history.

we use a GRU unit followed by a fully-connected layer and softmax nonlinearity to model the policy π over actions in During training, the agent samples its actions from the policy to encourage exploration.


Since it has more trainable components, this agent is also most prone to overfitting.

the vocabulary of the simulator it is trained against is quite limited (V n = 3078), and hence when real users provided inputs outside this vocabulary, it performed poorly.


The final matching degree is computed with the accumulation of the matching vectors.

for each utterance-response pair, the model constructs a word-word similarity matrix and a sequence-sequence similarity matrix by the word embeddings and the hidden states of a recurrent neural network with gated recurrent units (GRU) (Chung et al., 2014) respectively.


U ∈ R d×nu and R ∈ R d×nr are then used to construct a word-word similarity matrix M 1 ∈ R nu×nr and a sequence-sequence similarity matrix M 2 ∈ R nu×nr which are two input channels of a convolutional neural network (CNN).

the CNN distills important matching information from the matrices and encodes the information into a matching vector v. ∀i, j, the M 1 models the matching between u and r on a word level.


We publish it to research communities to facilitate the research of multi-turn response selection.

we crawled 1.1 million dyadic dialogues (conversation between two persons) longer than 2 turns from Douban group 3 which is a popular social networking service in China.


We use a constrained brute force ranking scheme to evaluate all possible groundings (with a restricted granularity) between an image and its caption.

we divide the image into a grid, and extract all of the image crops whose boundaries sit on the grid lines.


When computing the purity of a cluster, we count a cluster member as matching the cluster label as long as the overall cluster label appears in the member's label string.

an acoustic segment overlapping the words "the lighthouse" would receive credit for matching the overall cluster label "lighthouse".


47 cluster labels were found to have a similarity of 0.5 to some ILSVRC12 class, leaving 134 cluster labels whose highest similarity to any ILSVRC12 class was less than 0.5.

more than two thirds of the highly pure pattern clusters learned by our network were dissimilar to all of the 1,000 ILSVRC12 classes used to pretrain the VGG network, indicating that our model is able to generalize far beyond the set of classes found in the ILSVRC12 data.


Unlike the sole attention model, the forward-backward algorithm of CTC can enforce monotonic alignment between speech and label sequences during training.

rather than solely depending on data-driven attention methods to estimate the desired alignments in long sequences, the forward-backward algorithm in CTC helps to speed up the process of estimating the desired alignment.


We show that traces of the source language remain in the translation product to the extent that it is possible to uncover the history of the source language by looking only at the translation.

we automatically reconstruct phylogenetic language trees from monolingual texts (translated from several source languages).


The evident intra-family linguistic ties exposed by this experiment support the intuition that cross-linguistic transfer in translation is governed by typological properties of the source language.

translations from related sources tend to resemble each other to a greater extent than translations from more distant languages.


Both trees reasonably group individual languages into three language-family branches.

they cluster the Germanic and Romance languages closer than the Balto-Slavic.


First, drawing on previous work on ESL production, we observe that classifier uncertainty in NLIR correlates with global linguistic similarities across native languages.

the more similar are the languages, the more similar are the reading patterns of their native speakers in English.


Practically, they provide useful abstractions for performing NLIR over arbitrary textual input.

they enable performing this task using any textual input during both training and testing phases.


We note that the presented analysis is preliminary in nature, and warrants further study in future research.

reading times and classifier learned features may in some cases differ between the shared and the individual regimes.


The effectiveness of linguistically motivated criteria for fixation clustering and our subsequent analysis suggest that the ESL reading process is affected by linguistic factors.

we show that linguistic similarities between native languages are reflected in similarities in ESL reading.


To address this issue, we introduce a new framework (MORSE), the first to bring semantics into morpheme segmentation both on a local and a vocabulary-wide level.

when checking for the morphological relation between two words, we not only check for the semantic relatedness of the pair at hand (local), but also check if the difference vectors of pairs showing similar orthographic change are consistent (vocabulary-wide).


The pair of words is related via r on an orthographic level, but the score reflects the validity of the morphological relation via r on a semantic level.

this score is evaluated as: it is the fraction of pairs of words in the support set that exhibit a similar linear relation as (w 1 , w 2 ) in the vector space.


In this stage we perform morpheme segmentation using the knowledge gained from the first stage.

we begin with some notation: let R add = {r : we divide the rules to those where an affix is added (R add ) and to those where an affix is replaced (R rep ).


As for the inference stage of MORSE, the greedy inference approach limits its performance.

a wrong segmentation at the be-ginning will propagate and result in consequent wrong segmentations.


Downsampling with the number of feature maps fixed After each convolution block, we perform max-pooling with size 3 and stride 2.

the pooling layer produces a new internal representation of a document by taking the component-wise maximum over 3 contiguous internal vectors, representing 3 overlapping text regions, but it does this only for every other possible triplet (stride 2) instead of all the possible triplets (stride 1).


Here, pre-activation refers to activation being done before weighting instead of after as is typically done.

in the convolu-tion layer of DPCNN, Wσ(x) + b is computed at every location of each document where a column vector x represents a small region (overlapping with each other) of input at each location, σ(•) is a component-wise nonlinear activation, and weights W and biases b (unique to each layer) are the parameters to be trained.


Data preprocessing was done as in (Johnson and Zhang, 2016).

upper-case letters were converted to lower-case letters.


To facilitate comparison with ShallowCNN, we matched our unsupervised embedding setting exactly with that of (Johnson and Zhang, 2016).

we trained the same four types of tvembeddings, which are embeddings of 5-and 9word regions, each of which represents the input regions by either 30K-dim bow or 200K-dim  bags of {1,2,3}-grams, retaining only the most frequent 30K words or 200K {1,2,3}-grams.


However, words are difficult to divorce from their meanings, and sometimes repeating a word conflicts with repeating its referent.

pronouns often refer to different people depending on who uses the pronoun.


We begin with descriptive analyses of the behavior of pronouns, which are likely to reflect incorporation into the company.

we look at first-person singular (I), first-person plural (we), and second-person pronouns (you).


The coupling of entropy series in frequency space is negatively correlated with task success.

synchrony between interlocutors in terms of their information distribution hinders the success of collaboration.


While these parameters can be manually tuned to generate conversational text, the affect category can also be automatically inferred from preceding context words.

for model training, the affect category is derived from features generated using keyword spotting from a dictionary of emotional words, such as the LIWC (Linguistic Inquiry and Word Count) tool (Pennebaker et al., 2001).


The MTurk platform has been successfully used in the past for a wide range of perception experiments and has been shown to be an excellent resource to collect human ratings for large studies (Buhrmester et al., 2011).

we generated more than 200 sentences for four sentence beginnings (namely the three sentence beginnings listed in Table 2 Table 2: Example sentences generated by the model conditioned on different affect categories on the Fisher corpus was used for sentence generation.


The grammatical correctness deteriorates significantly with β.

a post-hoc Tukey test revealed that only the two highest β values show a significant drop in grammatical correctness at p<.05 (see Figure 3).


However, the training time for DA grows exponentially in the number of source domains.

when trained  Table 4: Intent classification accuracy (%) and slot tagging F1-score (%) of our baselines and variants of DE.


The dialog manager is one of the key components of dialog systems, which is responsible for modeling the decision-making process.

it typically takes a new utterance and the dialog context as input, and generates discourse-level decisions (Bohus and Rudnicky, 2003;Williams and Young, 2007).


Figure 1: Given A's question, there exists many valid responses from B for different assumptions of the latent variables, e.g., B's hobby.


Then we assume that the generation of x depends on c, z and y. y relies on z and c as shown in Figure 2.

during training the initial state of the response decoder is s 0 = W i [z, c, y] + b i and the input at every step is [e t , y] where e t is the word embedding of t th word in x.


The first decomposes a dialog system into a pipeline, typically including language understanding, dialog state tracking, action selection policy, and language generation (Levin et al., 2000;Singh et al., 2002;Williams and Young, 2007;Williams, 2008;Hori et al., 2009;Lee et al., 2009;Griol et al., 2008;Young et al., 2013;Li et al., 2014).

related to HCNs, past work has implemented the policy as feed-forward neural networks , trained with supervised learning followed by reinforcement learning .


If the system used for collection makes an error at turn t, the labeled dialog only includes the sub-dialog up to turn t, and the system being evaluated off-line is only evaluated on that sub-dialog.

in our case, reporting dialog accuracy would favor the HCN because it would be evaluated on fewer turns than the rule-based system.


Even though a linguistics-free sequence to sequence model in neural machine translation (NMT) has certain capability of implicitly learning syntactic information of source sentences, this paper shows that source syntax can be explicitly incorporated into NMT effectively to provide further improvements.

we linearize parse trees of source sentences to obtain structural label sequences.


Alternatively, we address how to incorporate explicitly the source syntax to improve the NMT translation accuracy with the expectation of alleviating the issues above in general.

rather than directly assigning each source word with manually designed syntactic labels, as Sennrich and Haddow (2016) do, we linearize a phrase parse tree into a structural label sequence and let the model automatically learn useful syntactic information.


To obtain syntactic knowledge, we linearize a parse tree into a structural label sequence and let the model automatically learn useful information through it.

we have described three different models to capture the syntax knowledge, i.e., Parallel RNN, Hierarchical RNN, and Mixed RNN.


In the first few hours, our approach demonstrates superior early detection performance than other models.

cPTK achieve 75% accuracy on Twitter15 and 73% on Twitter16 within 24 hours, that is much faster than other models.


To this end, we deploy the sparse graph regularized approach (Argyriou et al., 2007;Zhou et al., 2011) to encode the structure of the seven classes as a graph regularizer in a logistic regression framework.

we employed a multi-task learning paradigm, where each task is a one-vs-all classification.


Using the proposed model, by combining embedding vectors and attention at both word and character levels, we model all contextual words, including OOVs, in a unified context vector representation.

as we will discuss in Section 5, the character-level attention layer captures most useful information for correcting local errors that involve small edits in orthography.


To establish the potential of such models in word-based neural sequence-to-sequence systems, we integrate a web-scale count-based language model.

we use the modified Kneser-Ney 5-gram language model trained from Common Crawl (Buck et al., 2014), made available for download by Junczys-Dowmunt and Grundkiewicz (2016).


We analyze the impact of sub-word level information and the two nested levels of attention in more detail by looking at the performance of the models on different segments of the data.

we analyze the performance of the models on sentences containing OOV source words versus ones without OOV words, and corrections to orthographically similar versus dissimilar word forms.


Figure 6: Tryst relation between arab and islam using keywords to represent ideas (#2 in tryst): these two words tend to cooccur but are anti-correlated in prevalence over time.

islam was rarely used in coverage of terrorism in the 1980s.


Furthermore, latino holds head-to-head relations with two subgroups of Latin American immigrants: haitian and cuban.

the strength of the relation with haitian is ranked #18 in headto-head relations.


The first stage of automatic annotation is edit extraction.

given an original and corrected sentence pair, we need to determine the start and end boundaries of any edits.


Fortunately, one benefit of explicitly annotating system hypotheses is that it makes evaluation much more straightforward.

for each sentence, we only need to compare the edits in the hypothesis against the edits in each respective reference and measure the overlap.


To solve this problem, we can reprocess the references in the same way as the hypotheses.

we can apply ERRANT to the references such that each reference edit is subject to the same automatic extraction and classification criteria as each hypothesis edit.


[is a cat → are a cats] is selected over [is → are] and [cat → cats] even though these edits are best handled separately.

the M 2 scorer exploits its dynamic edit boundary prediction to artificially maximise true positives and minimise false positives and hence produce slightly inflated scores.


To demonstrate the value of ERRANT, we applied it to the data produced in the CoNLL-2014 shared task (Ng et al., 2014).

we automatically annotated all the system hypotheses and official reference files.


We revise a previous classification of pre-requisite skills for RC.

skills of knowledge reasoning are organized by using insights of entailment phenomena in NLP and human text comprehension in psychology.


They selected the sentences required to provide the answer, and then annotated them with appropriate prerequisite skills.

the datasets were annotated from the point of view of whether the context entailed the hypothesis constructed from the pair of the question and answer.


This means that our methodology cannot quantify the systems' competence in searching the context for necessary sentences and answer candidates.

our methodology can be only used to evaluate the competence of understanding RC questions as contextual entailments.


First, the readability of RC datasets does not directly affect the difficulty of their questions.

rC datasets that are difficult to read are not necessarily difficult to answer.


At the core of both of these components is the issue of span representation.

given that a span's correct label and its quality as a constituent depend heavily on the context in which it appears, we naturally turn to recurrent neural networks as a starting point, since they have previously been shown to capture contextual information suitable for use in a variety of natural language applications (Bahdanau et al., 2014;Wang et al., 2015) we run a bidirectional LSTM over the input to obtain context-sensitive forward and backward encodings for each position i, denoted by f i and b i , respectively.


Our model shares several features in common with that of Cross and Huang (2016).

our representation of spans and the form of our label scoring function were directly inspired by their work, as were our handling of unary chains and our use of an empty label.


The base case is again a singleton span (i, i+1), and follows the same form as the base case for the chart parser.

we select the label that satisfies omitting span scores from consideration since singleton spans cannot be split.


Inspired by the success of deep biaffine scoring in recent work by Dozat and Manning (2016) for dependency parsing, we also consider a split scoring function of a similar form for our model.

we let h ik = f left (s ik ) and h kj = f right (s kj ) be deep left and right span representations obtained by passing the child vectors through corresponding left and right feedforward networks.


In our "book model" of semantic dependency graph, the spine is made up of a sequence of words, and each half-plane contains a subset of dependency arcs.

the arcs on each page compose a noncrossing dependency graph, a.k.a.


The vertex set V consists of n vertices, each of which corresponds to a word and is indexed by an integer.

the arc set A represents the labeled dependency relations of the particular analysis G. an arc, viz.


A trade-off between rich features and exact decoding benefit tree parsing (McDonald and Nivre, 2011).

zhang and McDonald (2012) proposed a generalized higher-order model that abandons exact search in graph-based parsing in favor of freedom in feature scope.


Applying the above algorithm, we can obtain two corpora to train two noncrossing dependency parsing models.

we can learn two score functions f A and f B to score noncrossing dependency graphs.


Only the scores of factors are modified (see Line 3 and 4).

to modify the first order weights of edges, we take a subtraction of u ⊤ A in the first model and a substraction of u ⊤ B in the second one.


On top of the representation layer, we use a hidden layer to merge X W , X P and X C into a single vector (1) The hidden feature vector h is used to represent the state S = W, P, C , for calculating the scores of the next action.

a linear output layer with two nodes is employed: The first and second node of o represent the scores of SEP and APP given S, namely Score(S, SEP), Score(S, APP) respectively.


For the former, we follow prior methods (Xue et al., 2003;Pei et al., 2014) Figure 3, a multi-layer perceptron (MLP) is employed to derive a five-character window vector D C from single-character vector rep- For the latter, we follow recent work (Chen et al., 2015b;Zhang et al., 2016b), using a bidirectional LSTM to encode input character sequence.

3 the bi-directional LSTM ] of the next incoming character c 0 is used to represent the coming characters [c 0 , c 1 , ...] given a state.


We verify the utility of POS information for our segmentor by pretraining a classifier that predicts the POS on each character, according to the character window representation D C .

given , the POS of the word that c 0 belongs to is used as the output.


Table 6 shows the effectiveness of rich pretraining of D c on the development set.

by using punctuation information, the F-score increases from 95.86% to 96.25%, with a relative error reduction of 9.4%.


Time efficiency: The method should be able to train the parameters efficiently, and possible to perform decoding efficiently with choosing the candidate words from the full probability distribution.

the method should be performed fast on general CPUs to suppress physical costs of computational resources for actual production systems.


The proposed methods also improve actual computation time in both training and test.

on CPU, where the computation speed is directly affected by the size of the output layer, the proposed methods translate significantly faster than Softmax by x5 to x20.


Figure 2 shows that the gap between word-based and char-based representations increases as the frequency of the word in the training data decreases.

the more frequent the word, the less need there is for character information.


This implies that higher BLEU does not necessarily entail better morphological representations.

a better translation model learns more informative representations, but only when it is actually learning to translate rather than merely memorizing the data as in the autoencoder case.


Utterance-level sentiment analysis focuses on tagging every utterance of a video with a sentiment label (instead of assigning a unique label to the whole video).

utterance-level sentiment analysis is useful to understand the sentiment dynamics of different aspects of the topics covered by the speaker throughout his/her speech.


For extracting features from the textual modality, we use a CNN (Karpathy et al., 2014).

we first represent each utterance as the concatenation of vectors of the constituent words.


The features extracted by openSMILE consist of several low-level descriptors (LLD), e.g., MFCC, voice intensity, pitch, and their statistics, e.g., mean, root quadratic mean, etc.

we use IS13-ComParE configuration file in openS-MILE.


In our experiments, we obtained best results with 32 feature maps (f m ) with the filter-size of 5 × 5 × 5 (or f d × f h × f w ).


We hypothesize that, within a video, there is a high probability of inter-utterance dependency with respect to their sentimental clues.

we claim that, when classifying one utterance, other utterances can provide important contextual information.


LSTM (Hochreiter and Schmidhuber, 1997) is a kind of RNN, an extension of conventional feedforward neural network.

lSTM cells are capable of modeling long-range dependencies, which other traditional RNNs fail to do given the vanishing gradient issue.


The test set contains opinionated videos by rest 31 speakers.

1447 and 752 utterances are used in training and test, respectively.


Contributions We operationalize the sociolinguistic concept of stancetaking as a multidimensional framework, making it possible to measure at scale.

• we contribute a lexicon of stance markers based on prior work and adapted to the genre of online interpersonal discourse; • we group stance markers into latent dimensions; • we show that these stance dimensions are internally coherent; • we demonstrate that the stance dimensions predict and correlate with social phenomena.


Following this idea, we first present the model details, then discuss the optimization techniques for training.

energy function we stack all the concept projection matrices to a 3-dimensional tensor D ∈ R m×n×n , where m is the pre-specified number of concept projection matrices and n is the dimensionality of entity embeddings and relation embeddings.


To avoid adding another inner loop to our algorithm, we turn to a simple but fast approximation method based on the following single-matrix cost.

for each relation r, we consider the induced cost L H r,i where only a single projection matrix i is used for the head entity: is the corresponding energy function, and the subscript in P r and N r denotes the subsets with relation r. Intuitively, L H r,i measures, given the current tail attention vector α α α T r , if only one project matrix could be chosen for the head entity, how implausible D i would be.


2 As we can see, on WN18, ITransF outperforms STransE by a significant margin on rare relations.

in the last bin (rarest relations), the average Hits@10 increases from 55.2 to 93.8, showing the great benefits of transferring statistical strength from common relations to rare ones.


The probability cannot be too high to avoid generating negative samples that are actually correct, since there are generally a lot of facts missing in KBs.

let M H r = {h | ∃t(h, r, t) ∈ P } and M T r = {t | ∃h(h, r, t) ∈ P } denote the head or tail domain of relation r. Suppose N r = {(h, r, t) ∈ P } is the induced set of edges with relation r. We define the probability p r as Our motivation of such a formulation is as follows: Suppose O r is the set that contains all truthful fact triples on relation r, i.e., all triples in training set and all other missing correct triples.


We hypothesize that leveraging the interaction between content and discourse has the potential to yield better prediction performance on both phrase-based content selection and discourse relation prediction.

we utilize argumentative discourse relations as defined in Twente Argument Schema (TAS) (Rienks et al., 2005), where discussions are organized into tree structures with discourse relations labeled between nodes (as shown in Figure 1).


3 Our factor graph formulation draws from ideas previously used independently in parsing and argument mining.

maximum spanning tree (MST) methods for arc-factored dependency parsing have been successfully used by McDonald et al.


We devise a novel adversarial approach which enables a self-calibrated imitation mechanism.

we build a discriminator which distinguishes between the features by the two counterpart networks.


The connective-augmented recognizer is in analogy to an annotator with the help of connectives as in the human annotation process, and the implicit recognizer would be improved by learning from such an "informed" annotator.

we want the latent features extracted by the two models to match as closely as possible, which explicitly transfers the discriminability of the connective-augmented representations to implicit ones.


The discriminator is to differentiate between the reasoning behaviors of the implicit network i-CNN and the augmented network a-CNN.

d is a binary classifier that takes as inputs a la-tent feature H derived from either i-CNN or a-CNN given appropriate data (where implicit connectives is either missing or present, respectively).


In the adversarial training process, we found setting λ 2 = 0 gives stable convergence.

the connective-augmented features are fixed after the pre-training stage.


after the initialization stage the weights of the connective-augmented network a-CNN are fixed.


Using only one tree to represent a clustering makes the search for optimal mention clusters tractable.

structured prediction algorithms select h that maximizes the model learned at time t as shown in the next section.


However, we can use simple information about automatic mentions and how they relate to gold mentions and gold clusters.

we use four numbers: (i) correctly detected automatic mentions, (ii) links they have in the gold standard, (iii) gold mentions, and (iv) gold links.


We first experimented with the standard CoNLL setting to compare the LSP accuracy in terms of MELA using the three different loss functions, i.e., LSP F , LSP Y J and LSP ρ .

we used all the documents of the training set and all N ∼ 16.8M features from cort, and tested on the both dev.


In this paper, we focus instead on low-resource settings where task annotations are unavailable or scarce.

we use lexical resources to guide part-of-speech induction ( §4) and to bootstrap named-entity recognizers in low-resource languages ( §5).


Generalizing this a bit, we may suppose that one observation of string x in the lexicon is equivalent to c labeled tokens of x in a corpus, where the constant c > 0 is known as a pseudocount.

observing a lexicon of m y distinct types {x 1 , .


We employ SMC as a kernel in an MCMC sampler (Andrieu et al., 2010).

we use a block Gibbs sampler in which we iteratively resample the hidden labeling y of a sentence x conditioned on the current labelings for all other sentences in the corpus.


We evaluate our model on multi-lingual data released as part of the CoNLL 2007 and CoNLL-X shared tasks.

we use the same set of languages as Li et al.


Figure 1 shows that our generative model strongly beats the baseline in this low-data regime.

when there is little annotated training data, our proposed generative model can compensate by exploiting the lexicon, while the discriminative baseline scores terribly.


Using this dataset as a testbed, we focus on a recurring problem in models based on the encode-attend-decode paradigm.

it is observed that the summaries produced by such models contain repeated phrases.


We propose a model which explicitly prevents this by ensuring that successive context vectors are orthogonal to each other.

we subtract out any component that the current context vector has in the direction of the previous context vector.


To account for the history, we treat successive context vectors as a sequence and use a modified LSTM cell to compute the new state at each time step.

we use the following set of equations to compute a diverse context at time t: where 8; l 5 is number of hidden units in the LSTM cell.


SD 2 : This model again uses a relaxed version of the orthogonality constraint used in D 2 .

we define a gating parameter g t and replace (12) above by (14) as define below: where We compare with two recently proposed baseline diversity methods (Chen et al., 2016) as described below.


Further, we compare our diversity based attention models with existing models for diversity by suitably adapting them to this problem as described earlier.

we compare the performance of the following models: • Vanilla e-a-d: This is the vanilla encoderattention-decoder model adapted to the problem of abstractive summarization.


Effect of Diversity models: All the diversity models introduced in the paper (rows 7, 8, 9, 10) give significant improvement over the nondiversity models.

the modified LSTM based diversity model gives the best results.


Notice that the last example shows that our model is not very aggressive in dealing with the history and is able to produce valid repetitions (treated ... treated) when needed ber of sentences containing repeated words generated by different models.

for the 1268 test instances we counted the number of sentences containing repeated words as generated by different modes.


We found the pointer-generator model quicker to train, requiring less than 230,000 training iterations (12.8 epochs); a total of 3 days and 4 hours.

the pointer-generator model makes much quicker progress in the early phases of training.


Figure 7: Examples of abstractive summaries produced by our model (bold denotes novel words).

figure 6 shows that our final model copies whole article sentences 35% of the time; by comparison the reference summaries do so only 1.3% of the time.


However, ROUGE has been widely criticized for being too simplistic and not suitable for capturing important quality aspects we are interested in.

rOUGE does not capture sentences which are semantically equivalent but expressed with different words (Nenkova et al., 2007).


We search for such an approximate solution by employing a meta-heuristic solver introduced recently for extractive MDS by Peyrard and Eckle-Kohler (2016a).

we use the tool published with their paper.


Our model obtains remarkable improvements in performance over PageRank models that do not take into account word positions as well as over strong baselines for this task.

on several datasets of research papers, PositionRank achieves improvements as high as 29.09%.


(2010) extended topic-biased PageRank (Haveliwala, 2003) to kephrase extraction.

they decomposed a document into multiple topics, using topic models, and applied a separate topicbiased PageRank for each topic.


The idea of PositionRank is to assign larger weights (or probabilities) to words that are found early in a document and are frequent.

we want to assign a higher probability to a word found on the 2 nd position as compared to a word found on the 50 th position in the same document.


In this experiment, we analyze the influence that position-weighted frequent words in a document would have on the performance of PositionRank.

we compare the performance of the model that aggregates information from all positions of a word's occurrences, referred as PositionRank -full model with that of the model that uses only the first position of a word, referred as PositionRank -fp.


To our knowledge, we are the first to integrate the position information in novel ways in unsupervised keyphrase extraction.

unlike supervised approaches that use only the first position information, we showed that modeling the entire distribution of positions for a word outperforms models that use only the first position.


One of the most popular approaches for automatically evaluating the quality of dialogue responses is by computing their word overlap with the reference response.

the most popular metrics are the BLEU and METEOR scores used for machine translation, and the ROUGE score used for automatic summarization.


We therefore employ semi-supervised learning, and use a pre-training procedure to learn the parameters of the encoder.

we train the encoder as part of a neural dialogue model; we attach a third decoder RNN that takes the output of the encoder as input, and train it to predict the next utterance of a dialogue conditioned on the context.


When training ADEM, we also employ a subsampling procedure based on the model response length.

we divide the training examples into bins based on the number of words in a response and the score of that response.


studies (Liu et al., 2016;Galley et al., 2015).

this is the case for BLEU-4, which has frequently been used for dialogue response evaluation (Ritter et al., 2011;Sordoni et al., 2015b;Li et al., 2015;Galley et al., 2015;Li et al., 2016a).


Related to our approach is the literature on novel methods for the evaluation of machine translation systems, especially through the WMT evaluation task (Callison-Burch et al., 2011;Machácek and Bojar, 2014;Stanojevic et al., 2015).

(Albrecht and Hwa, 2007;Gupta et al., 2015) have proposed to evaluate machine translation systems using Regression and Tree-LSTMs respectively.


We adopt an encoder-decoder framework, which is widely used in machine translation (Bahdanau et al., 2014) and dialog systems (Mou et al., 2016), etc.

we use a hierarchical encoderdecoder framework similar to , as shown in Figure 1.


We use the Long Short-Term Memory (LSTM) (Hochreiter and Schmidhuber, 1997) as both the word encoder enc word and sentence encoder enc sent .

we adopt the variant of LSTM structure in (Graves, 2013).


• The events v i ∈ V and v j ∈ V are anticorrelated (when not independent).

while both vowels may individually have high probabilities (focalization), having either one in the inventory lowers the probability of the other (dispersion).


Our solution to this problem is to still construct our interpretable metric space R k , but then map that nonlinearly to R r for some large r. This latter map is constrained.


Our choice of a probabilistic model provides a natural test: how surprised is our model by held-out languages?

how likely does our model think unobserved, but attested vowel systems are?


As a preprocessing step, the first two formants values F 1 and F 2 are centered around zero and scaled down by a factor of 1000 since the formant values themselves may be quite large.

we use the development fold to select among the following combinations of hyperparameters.


In this paper, we propose an adversarial multicriteria learning for CWS by integrating shared knowledge from multiple segmentation criteria.

we regard each segmentation criterion as a single task and propose three different shared-private models under the framework of multi-task learning (Caruana, 1997;Ben-David and Schuller, 2003), where a shared layer is used to extract the criteria-invariant features, and a private layer is used to extract the criteria-specific features.


Chinese word segmentation task is usually regarded as a character based sequence labeling problem.

each character in a sentence is labeled as one of L = {B, M, E, S}, indicating the begin, middle, end of a word, or a word with single character.


Recently, neural networks are widely applied to Chinese word segmentation task for their ability to minimize the effort in feature engineering (Zheng et al., 2013;Pei et al., 2014;Chen et al., 2015a,b).

given a sequence with n characters X = {x 1 , .


We use a criterion discriminator which aims to recognize which criterion the sentence is annotated by using the shared features.

given a sentence X with length n, we refer to h (s) X as shared features for X in one of the sharing models.


Since the scale of each dataset varies, we use different training batch sizes for datasets.

we set batch sizes of AS and MSR datasets as 512 and 256 respectively, and 128 for remains.


Table 7 gives the results of Model-I on the NLPCC 2016 dataset under the help of the eight datasets in Table 2.

we firstly train the model on the eight datasets, then we train on the NLPCC 2016 dataset alone with shared parameters fixed.


However, for languages that do not contain word separation symbols, dependency parsers are used in pipeline processes with word segmentation and POS tagging models, and encounter serious problems because of error propagations.

chinese word segmentation is notoriously difficult because sentences are written without word dividers and chinese words are not clearly defined.


Different from classical methods, our method can extract triplets without knowing the information of entity types.

we did not use the label of entity types to train the model, therefore we do not need to consider the entity types in the evaluation.


As described above, our FOFE-based local detection approach for NER, called FOFE-NER hereafter, is motivated by the way how human actually infers whether a word segment in text is an entity or mention, where the entity types of the other entities in the same sentence is not a must.

the dependency between adjacent entities is fairly weak in NER.


The classification decision of the class of the target entity is mostly informed not by the whole sentence (or paragraph), rather it is a small and focused "predicate window" pointed to by the entity's head dependency.

most of the sentence is not only superfluous for the task, it actually lowers the accuracy of the model due to irrelevant input.


For our video captioning as well as entailment generation results, we use four diverse automatic evaluation metrics that are popular for image/video captioning and language generation in general: METEOR (Denkowski and Lavie, 2014), BLEU-4 (Papineni et al., 2002), CIDEr-D (Vedantam et al., 2015), and ROUGE-L (Lin, 2004).

mETEOR and CIDEr-D have been justified to be better for generation tasks, because CIDEr-D uses consensus among the (large) number of references and mETEOR uses soft matching based on stemming, paraphrasing, and Word-Net synonyms.


It is assumed that the source and target domains are balanced so that inf Θ error(Θ) ≤ 1/2 and the difference lies in [0,2].

we want to make error(Θ) as large as possible in order to generalize well to the target domain.


Thus we propose to instead optimize the shared parameters for random domain predictions.

the above loss is replaced with where d (i) is set to be src with probability 0.5 and tgt with probability 0.5.


0 is the best case where adversary training successfully fake shared encoder to predict domains.

thanks to adversary training our model make the domain-invariant features in shared encoder in order to generalize well to the target domain.


• Existing models use discrete representation for grammatical roles and features, which leads to the so-called curse of dimensionality problem (Bengio et al., 2003).

to model transitions of length k with R different grammatical roles, the basic entity grid model needs to compute R k tran-sition probabilities from a grid.


• Existing models compute feature representations from entity grids in a task-agnostic way.

feature extraction is decoupled from the target downstream tasks.


To compose higher-level features from the embedding vectors, we make the following modeling assumptions: • Similar to existing entity grid models, we assume there is no spatio-temporal relation between the entities in a document.

columns in a grid are treated independently.


We can observe that even without pre-training our models outperform existing models, and pretraining gives further improvements.

pre-trained Grid-CNN gives an improvement of 2.5% over the Grid model, and including entity features pushes the improvement further to 3.7%.


It fundamentally changes the way businesses and consumers behave (Qualman, 2012), it is instrumental to the success of individuals and businesses (Haenlein and Kaplan, 2009) and it also affects political regimes (Howard et al., 2011;Lamer, 2012).

automatic interaction in natural language in social media is now a common theme, as seen in the rapid popularization of chat applications, chat-bots, and "smart agents" aiming to conduct human-like interactions in natural language.


Finally, our human evaluations results show no learning effect.

human raters are unable to discover in the generated responses typical structures that would lead them to consider the responses machine-generated.


We then use a variation on the beam search algorithm (Reddy, 1977) and devise a methodology to select the k-best scoring trees to be carried on to the next iteration.

we use a Breadth-First algorithm for expanding the tree and define a dynamic programming algorithm that takes the score of a derivation tree of n − 1 expanded nodes, selects a new rule for the next non-expanded node, and from it, calculates the score of the expanded tree with now n nodes.


(ii) Sentiment Agreement measures whether the inferred sentiment of the response matches the input sentiment parameter used for generation.

we take the raw yield of the generated tree (a sentence) and run it through the sentiment classifier implemented in Socher et al.


C2T is usually addressed using template-based (Becker, 2002) or data-driven (Konstas and Lapata, 2013;Yuan et al., 2015) approaches.

researchers explored data-driven grammar-based approaches (Cahill and van Genabith, 2006), often assuming a custom grammar (Konstas and Lapata, 2013) or a closed-domain approach (DeVault et al., 2008).


Vanderwende (2008) point out that learning to ask good questions is an important task in NLP research in its own right, and should consist of more than the syntactic transformation of a declarative sentence.

a natural sounding question often compresses the sentence on which it is based (e.g., question 3 in Figure 1), uses synonyms for terms in the passage (e.g., "form" for "produce" in question 2 and "get" for "produce" in question 3), or refers to entities from preceding sentences or clauses (e.g., the use of "photosynthesis" in question 2).


In the exploratory phase, the minus term in the equation 9 helps to reduce the score of the sentences whose concepts have received feedback already.

it causes higher scores for sentences consisting of concepts which yet lack feedback.


To reduce the amount of feedback, we also consider content phrases to collect feedback.

syntactic chunks from the constituency parse trees consisting of non-function words (i.e., nouns, verbs, adjectives, and adverbs).


Due to the statistical nature of neural models, they pay much more attention to high-frequency patterns, whereas they ignore low-frequency ones.

the more regular and common the patterns, the better the neural model is good at learning them and tends to use them more frequently at run-time.


We argue that the essential problem is that statistical models are good at learning general rules (usage of regular words and their combinations) but are less capable of remembering special instances that are difficult to cover with general rules.

there is only rule-based reasoning, no instance-based memory.


In our study, the neural model is firstly trained, and then the memory is created by running f d (the decoder of the neural model).

for the k-th poem selected to be in the memory, the character sequence is input to the decoder one by one, with the contribution from the encoder set to zero.


From the first row of Table 2, we observe that the experts have a clear preference for the poems generated by the C 1 model, the one that can produce fluent yet uninteresting poems.

the 'aesthetic innovation' score for C ∞ is not better than C 1 , which was different from what we expected.


In comparison to sparking innovation, fluency and being meaningful is more important not only for non-expert readers, but also for professional poet-  Table 2: Preference ratios for systems with or without overfitting and with or without memory augmentation.

s. only meaningful innovation is regarded as innovation, and irrational innovation is simply treated as junk.


For the other automatic evaluation metric, we calculate F-measures for time-dependent expressions, using market comments written by humans as references, to investigate whether our model can correctly output timedependent expressions such as "open with" and describe how the price changes compared with the previous period referring to the series of preceding prices such as "continual fall".

we calculate F-measures for 13 expressions shown in Figure 3.


For informativeness, the judge used both generated comments and their input stock prices to rate the comments.

if the judge deem that a generated comment describes an important price movement or an outline of the movement properly, such comments are considered to be informative.


Furthermore, we found that the BLEU score of -multi, which did not use the multi-level representation of the data, was inferior.

incorporating the multi-level representation along with an output of an encoder into a decoder seems  to contribute to improving the automatic evaluation and producing a better representation of the input data.


On the other hand, a deletion-based method does not face such a problem in a cross-domain setting.

we propose two major changes to the model by Filippova et al.


We then use the hidden vectors to predict the label sequence.

label y i is predicted from h i as follows: where W ∈ R 2×h and b ∈ R 2 are a weight matrix and a weight vector to be learned.


, y n for a sentence, subject to some constraints.

the ILP problem consists of two parts: the objective function, and the constraints.


This observation motivates us to approach the reading comprehension problem in a constituentcentric manner, where the generation of candidate answers and their representation learning are both based upon constituents.

we design a Constituent-Centric Neural Network (CCNN) to perform end-to-end reading comprehension, where the inputs are the passage and question, and the output is a span in the passage that is mostly suitable to answer this question.


In this paper, we explore several aspects pertaining to counseling interaction dynamics and their relation to counselor empathy during motivational interviewing encounters.

we analyze aspects such as participants' engagement, participants' verbal and nonverbal accommodation, as well as topics being discussed during the conversation, with the final goal of identifying linguistic and acoustic markers of counselor empathy.


Similar analyses on the client side, shown in Figure 5, indicate significant differences in the use of linguistic markers by the client (except for articles and quantifiers).

during low empathy encounters, clients coordinate more on the use of conjunctions, adverbs, auxiliary verbs, prepositions, personal pronouns, and impersonal pronouns.


To address this issue, we introduce a knowledge sentinel that records the information of the current context and use a mixture model to allow for better tradeoff between the impact of background knowledge and information from the context.

we compute a sentinel vector s t as: where W b and U b are weight parameters to be learned.


We study the extent to which the prerequisite relation between knowledge concepts in Massive Open Online Courses (MOOCs) can be inferred automatically.

what kinds of information can be leveraged to uncover the potential prerequisite relation between knowledge concepts.


In this paper, we attempt to figure out what kinds of information in MOOCs can be used to uncover the prerequisite relations among concepts.

we consider it from three aspects, including course concept semantics, course video context and course structure.


Therefore, we improve this indicator to a more general one, which is also suitable for non-wiki concepts.

for a concept a ∈ K, let us denote the top-M most related wiki entities of a as Because concepts in K and entities in E are jointly embedded in the same vector space in Section 3.1, we can easily obtain R a with the semantic relatedness metric ω(•, •) in Feature 1.


Based on our observation, the chance that a prerequisite concept is frequently mentioned in its subsequent videos is larger than that a subsequent concept is talked about in its prerequisite videos.

if video V a is a precursor video of V b , and a is a prerequisite concept of b, then it is likely that f (b, V a ) < f (a, V b ), where f (a, V) denotes the term frequency of a in video V. We thus define another feature, namely distributional asymmetry distance (Dad), to calculate the extent that a given concept pair satisfies this distributional asymmetry pattern.


Complexity level of a course concept is implicit in its distribution in courses.

we observe that, for a concept in MOOCs, if it covers more videos in a course or it survives longer time in a course, then it is more likely to be a basic concept rather than an advanced one.


For example, the F 1 of our method on ML outperforms T-SRI and HPM by 10.5% and 43.6%, respectively.


The simplest scenario would require a single constant value for all elements in one L1 and another for all elements in the other L1.

using discrim(L1 x ) to denote "the ranked list of discriminative n-grams for L1 x ", we define and analogously for θ b [i].


CRFs learn a conditional probability model p λ (l|x) from a set of labeled training data, where x = (x 1 , ..., x n ) is a random sequence of input words, l = (l 1 , ..., l n ) is the sequence of label variables (NER tags) for x, and l has certain Markov properties conditioned on x.

a general-order CRF with order o assumes that label variable l i is dependent on a fixed number o of previous label variables l i−1 , ..., l i−o , with the following conditional distribution: where f k 's are feature functions, λ k 's are weights of the feature functions (parameters to learn), and Z λ (x) is a normalization constant.


Project the NER tags to the target-language sentence y using the alignment information.

if a sequence of English words (x i , ..., x i+p ) is aligned to a sequence of target-language words (y j , ..., y j+q ), and is recognized (by the English NER system) as an entity with NER tag l, then (y j , ..., y j+q ) is labeled with l 2 .


We create mapping functions which can map words in any language into English and we simply use the English NER system to decode.

by mapping all languages into English, we are using one universal NER system and we do not need to re-train the system when a new language is added.


The rank-based codecoding scheme is more effective for combining the two projection-based approaches.

the rank-based scheme that combines AP and NN2 achieves the highest F 1 score among all the weakly supervised approaches for Korean, German and Portuguese (second highest F 1 score for Japanese), and it improves over the best of the two  projection-based systems by 2.2 to 7.4 F 1 score.


, o k i as the class label, the highest probable class out of those corresponding to the applicable edit trees, is picked up.

the particular edit tree t j ∈ T is considered as the right candidate for w i , where In this way, we cancel out the non-applicable classes and focus only on the plausible candidates.


Note that each step of c covers a single constraint token.

constraints is an array of sequences, where individual tokens can be indexed as constraints ij , i.e.


Leveraging state-of-the-art machine learning approaches in Natural Language Processing and computer vision to detect and report advertisements suspected of trafficking is the main focus of our work.

we strive to find the victims and perpetrators of trafficking who hide in plain sight in the massive amounts of data online.


Recently, cybersecurity defense has also been recognized as one of the "problem areas likely to be important both for advancing AI and for its long-run impact on society" (Sutskever et al., 2016).

we feel that natural language processing (NLP) has the potential for substantial contribution in cybersecurity and that this is a critical research area given the urgency and risks involved.


We also observe that some signatures seem to benefit more from the annotations, such as persis-tence_autorun and has_pdb.

per-sistence_autorun has a direct parallel in attribute labels, which is MalwareCapability-persistence, showing that using MAEC vocabulary as attribute labels is appropriate.


For this first corpus, we define our unit of revision to be at the sentence level.

even if a pair of sentences contains multiple edits, the entire sentence pair will be annotated as one sentence revision.


Methods based on word sense induction try to induce sense representations without the need for any initial lexical resource by extracting semantic relations from text.

word sense induction (WSI) based on word ego networks clusters graphs of semantically related words (Lin, 1998;Pantel and Lin, 2002;Dorow and Widdows, 2003;Véronis, 2004;Hope and Keller, 2013;Pelevina et al., 2016;Panchenko et al., 2017a), where each cluster corresponds to a word sense.


First, we use a graph-based word sense induction method that is similar to the curvature-based approach of Dorow and Widdows (2003).

removal of the nodes participating in many triangles tends to separate the original graph into several connected components.


The resource is constructed using crowdsourcing and mostly covers general vocabulary.

non-expert users are allowed to edit synsets in a collaborative way loosely supervised by a team of project curators.


Considering the semantic relation between " (arrested)" and " (escaped)", we intuitively know that the person arrested by someone is likely to be the escaper.

information about one predicate-argument relation could help to identify another predicate-argument relation.


Performing experiments on the NAIST Text Corpus (Iida et al., 2007), we demonstrate that even without syntactic information, our neural models outperform previous syntax-dependent models (Imamura et al., 2009;Ouchi et al., 2015).

the neural model using Grid-RNNs achieved the best result.


In this work, we integrate residual connections (He et al., 2015;Wu et al., 2016) with our networks to form connections between layers.

the input vector h  Residual connections can also be applied to the single-sequence model.


This confirms that our neural models realize high performance, even without syntactic information, by learning contextual information effective for PAS analysis from the word sequence of the sentence.

for zero arguments (Zero), our models achieved a considerable improvement compared to the joint model in Ouchi et al.


The performance tends to improve as the RNN or Grid layers get deeper with residual connections.

the two models with eight layers and residual connections achieved considerable improvements of approximately 1.0 point according to the F-measure compared to models without residual connections.


Comparing the models using the NAIST Text Corpus 1.5, the single-and multi-sequence models outperformed the baseline models according to all metrics.

for the dative case, the two neural models achieved much higher results, by approximately 30 points.


In addition, for zero arguments (Zero), the neural models achieved better results than the baseline models.

for zero arguments of the nominative case (NOM), the multisequence model demonstrated a considerable improvement of approximately 2.5 points according to the F-measure compared with the joint model in Ouchi et al.


The experiments on the NAIST Text Corpus demonstrated that the models realize high performance without the need for syntactic information.

our multi-sequence model improved the performance of zero argument identification, one of the problematic issues facing Japanese PAS analysis, by considering the multi-predicate interactions with Grid-RNNs.


3 Section 4 shows that this assumption is valid over 75% of the time, making evidence documents a strong source of distant supervision for training machine reading systems.

we consider two types of distant supervision, depending on the source of our documents.


However, in Wikipedia we generally expect most facts to be stated only once, so we instead pool all of the evidence documents and never repeat the same question in the dataset (|D i | = 1.8 on average and q i = q j for all i, j).

each question (paired with the union of all of its evidence documents) is a single data point.


The mentions disambiguated first will be helpful for inducing the senses of the rest mentions.

how to choose the mentions disambiguated first will influence the performance.


Our approach explores the space of interactive question answering between humans and robots during the learning process.

motivated by previous work on robot learning (Cakmak and Thomaz, 2012), we designed a set of questions that are pertinent to verb semantic representations.


To enable human-robot communication and collaboration, recent years have seen an increasing amount of works which aim to learn semantics of language that are grounded to agents' perception (Gorniak and Roy, 2007;Tellex et al., 2014;Kim and Mooney, 2012;Matuszek et al., 2012a;Liu et al., 2014;Liu and Chai, 2015;Thomason et al., 2015Thomason et al., , 2016 and action (Matuszek et al., 2012b;Artzi and Zettlemoyer, 2013;Misra et al., 2014Misra et al., , 2015She and Chai, 2016).

for verb semantics, recent works explored the connection between verbs and action planning Misra et al., 2014Misra et al., , 2015She and Chai, 2016), for example, by representing grounded verbs semantics as the desired goal state of the physical world that is a result of the corresponding actions.


Inspired by these previous findings, our work explores interactive learning to acquire grounded verb semantics.

we aim to address when to ask what questions during interaction to improve learning.


In (She and Chai, 2016), instead of one hypothesis, it maintains a specific-to-general hypothesis space as shown in Figure 2 to capture all goal hypotheses of a particular verb frame.

it assumes that one verb frame may lead to different outcomes under different environments, where each possible outcome is represented by one node in the hierarchical graph and each node is a conjunction of multiple atomic fluents.


In this work, we formulate the choice of when to ask what questions during interaction as a sequential decisionmaking problem and apply reinforcement learning to acquire an optimal policy to manage interaction.

each of the execution and learning phases is governed by one policy (i.e., θ E and θ D ), which is updated by the reinforcement learning algorithm.


The SARSA algorithm with linear function approximation is utilized to update policies θ E and θ D (Sutton and Barto, 1998).

the objective of training is to learn an optimal value function Q(s, a) (i.e., the expected cu-  (s, a), where φ(s, a) is a feature vector and θ is a weight updated during training.


Vilnis and McCallum (2014) recently proposed an alternative view, where words are represented by a whole probability distribution instead of a deterministic point vector.

they model each word by a Gaussian distribution, and learn its mean and covariance matrix from data.


It is our contention that such flexibility is critical for both qualitatively learning about the meanings of words, and for optimal performance on many predictive tasks.

we model each word with a mixture of Gaussians (Section 3.1).


We represent each word w in a dictionary as a Gaussian mixture with K components.

the distribution of w, f w , is given by the density where K i=1 p w,i = 1.


We propose to use the expected likelihood kernel, which is a generalization of an inner product between vectors to an inner product between distributions (Jebara et al., 2004).

where •, • L 2 denotes the inner product in Hilbert space L 2 .


This metric incorporates the uncertainty from the covariance matrices in addition to the similarity between the mean vectors.

this metric measures the maximum similarity of mean vectors among all pairs of mixture components between distributions f and g. , which corresponds to matching the meanings of f and g that are the most similar.


We also use a separate output embeddings in addition to input embeddings, similar to word2vec implementation (Mikolov et al., 2013a,b).

each word has two sets of distributions qI and qO, each of which is a Gaussian mixture.


Based on this, we further show that by explicitly considering recursive architectures in both local inference modeling and inference composition, we achieve additional improvement.

incorporating syntactic parsing information contributes to our best result-it further improves the performance even when added to the already very strong model.


The previous work has included extensive research on recognizing textual entailment.

natural language inference (NLI) is concerned with determining whether a naturallanguage hypothesis h can be inferred from a premise p, as depicted in the following example from MacCartney (2009), where the hypothesis is regarded to be entailed from the premise.


To this end, we will also encode syntactic parse trees of a premise and hypothesis through tree-LSTM Tai et al., 2015;Le and Zuidema, 2015), which extends the chain LSTM to a recursive network (Socher et al., 2011).

given the parse of a premise or hypothesis, a tree node is deployed with a tree-LSTM memory block depicted as in Figure 2 and computed with Equations (3-10).


We describe the updating of a node at a high level with Equation (3) to facilitate references later in the paper, and the detailed computation is described in (4-10).

the input of a node is used to configure four gates: the input gate i t , output gate o t , and the two forget gates f L t and f R t .


incorporating syntactic parsing information contributes to our best result: it further improves the performance even when added to the already very strong model.


As mentioned above, several screenplays get revised during production.

character names get changed, sometimes even gender.


Of the three tests we perform, character gender distributions for writer and director genders are significantly different from the overall character gender distribution (p < 10 −10 and p < 10 −4 respectively; α = 0.05).

female writers and directors appear to produce movies with relatively balanced gender proportions (still slightly skewed towards the male side) compared to male writers Studies report potential biases in actor employment with age (Lincoln and Allen, 2004), particularly in female actors.


Several interesting patterns are revealed in the analysis.

movies with female writers and directors in the production team are observed to have balanced gender ratios in characters compared to male writers/directors.


Future work includes expanding the analyses to non-English movies and combining the linguistic metrics with character networks.

character network edges can be weighted using the psycholinguistic metrics to analyze the emotional patterns in inter-character interactions.


The forward and backward LSTMs are respectively formulated as follows: where g (LST M ) is the same as that in Eq (1).


Sarcasm is a form of speech in which speakers say the opposite of what they truly mean in order to convey a strong sentiment.

"Sarcasm is the giant chasm between what I say, and the person who doesn't get it.".


They contain comprehensive information about the companies' business as well as risk factors.

section Item 1A -Risk Factors of the reports contains information about the most significant risks for the company.


Context-aware Embeddings: Different from existing NRL models that learn context-free embeddings, CANE learns various embeddings for a vertex according to its different contexts.

for an edge e u,v , CANE learns context-aware embeddings v (u) and u (v) .


As stated before, we assume that a specific vertex plays different roles when interacting with others vertices.

each vertex should have its own points of focus about a specific vertex, which leads to its context-aware text embeddings.


(2) What calls for special attention is that, both CENE and TADW exhibit unstable performance under various training ratios.

cENE performs poorly under small training ratios, because it reserves much more parameters (e.g., convolution kernels and word embeddings) than TADW, which need more data for training.


In this paper, we propose the concept of Context-Aware Network Embedding (CANE) for the first time, which aims to learn various context-aware embeddings for a vertex according to the neighbors it interacts with.

we implement CANE on text-based information networks with proposed mutual attention mechanism, and conduct experiments on several real-world information networks.


This overall approach is illustrated in Figure 1.

we train a basic Singlish parser with the best off-the-shelf neural dependency parsing model using biaffine attention (Dozat and Manning, 2017), and improve it with knowledge transfer by adopting neural stacking (Chen et al., 2016;Zhang and Weiss, 2016) to integrate the English syntax.


2 https://github.com/wanghm92/Sing_Par Neural networks have led to significant advance in the performance for dependency parsing, including transition-based parsing (Chen and Manning, 2014;Zhou et al., 2015;Weiss et al., 2015;Andor et al., 2016), and graph-based parsing (Kiperwasser and Goldberg, 2016;Dozat and Manning, 2017).

the biaffine attention method of Dozat and Manning (2017) uses deep bi-directional long short-term memory (bi-LSTM) networks for highorder non-linear feature extraction, producing the highest-performing graph-based English dependency parser.


Topic-prominence: This type of sentences start with establishing its topic, which often serves as the default one that the rest of the sentence refers to, and they typically employ an object-subjectverb sentence structure (Leimgruber, 2009(Leimgruber, , 2011.

three subtypes of topic-prominence are observed in the Singlish dependency treebank and their annotations are addressed as follows: First, topics framed as clausal arguments at the beginning of the sentence are labeled as "csubj" (clausal subject), as shown by "Drive this car" of (1) in Figure 2, which is consistent with the dependency relations in its Chinese translation.


1 A polytree (Rebane and Pearl, 1987) is a multitree whose underlying graph is a tree.

the out property  An ordered digraph is weakly projective (PROJ W ) if for all vertices i, j and k, if k → j → i, then either {i, j} < k or {i, j} > k. the constraint, aka the outside-to-inside constraint (Yli-Jyrä, 2005), states that no outgoing arc of a vertex properly covers an incoming arc.


Our combined system, TagLM, uses the LM embeddings as additional inputs to the sequence tagging model.

we concatenate the LM embeddings h LM with the output from one of the bidirectional RNN layers in the sequence model.


ferent locations in the baseline sequence tagger.

we used the LM embeddings h LM k to: • augment the input of the first RNN layer; i.e., x • augment the output of the first RNN layer; i.e., • augment the output of the second RNN layer; i.e., Table 5 shows that the second alternative performs best.


The attention value of each word in a given sentence is calculated by an operation between the current word and the target trigger candidate.

in training procedure, we first construct gold attentions for each trigger candidate based on annotated arguments.


This paper uses the learned word embeddings as the source of basic features.

we use the Skip-gram model (Mikolov et al., 2013) to learn word embeddings on the NYT corpus 3 .


To achieve this goal, we propose two strategies to construct gold attention vectors: S1: only pay attention to argument words.

all argument words in the given context obtain the same attention, whereas other words get no attention.


The relation between topics is captured through a copula that provides a joint probability for all the topics used in a segment.

to generate words in a segment, one first jointly generates all the word specific topics z via a copula, and then generates each word in the segment from its word specific topic and the word-topic distribution φ.


Figure 5 illustrates another aspect of our model, namely the possibility to detect topically coherent segments.

as one can note, the sentence is segmented in six parts by our model, the first one is a NP, Ralph Borsodi where one single topic is assigned to both words.


Conditions enforce additional constraints on which cells in the answer column can be chosen; a select statement without any conditions indicates that an entire column of the table is the answer to the question.

each condition contains a column name as the condition column and an operator with zero or more arguments.


The p-values are 0.319 for CNN, <0.00001 for DailyMail, 0.028 for CBT-NE, and <0.00001 for WDW.

gA statistically significantly outperforms all other baselines on 3 out of those 4 datasets at the 5% significance level.


We compare the GA Reader as described here to a model which is exactly the same in all aspects, except that it passes document embeddings D (k) in each layer directly to the inputs of the next layer without using the GA module.


(D6) Ohsumed and Ohsumed-full: Documents are medical abstracts from the MeSH categories of the year 1991.

there are 23 cardiovascular diseases categories.


bustly superior performances among all ten methods.

it ranks first in three datasets, and second in the other one.


We are also interested in how word embedding can help group domain-specific texts into clusters.

does the semantic knowledge "embedded" in words provides enough clues to discriminate fine-grained concepts?


We evaluate the pipeline in two downstream NLP applications: polarity detection and topic categorization.

we use a classification model based on Convolutional Neural Networks which has been shown to be very effective in various text classification tasks (Kalchbrenner et al., 2014;Kim, 2014;Johnson and Zhang, 2015;Tang et al., 2015;Xiao and Cho, 2016).


It has been shown by Xiao and Cho (2016) that a recurrent layer can replace multiple layers of convolution and be beneficial, particularly when the length of input text grows.

we use a Long Short-Term Memory (Hochreiter and Schmidhuber, 1997, LSTM) as our recurrent layer which was originally proposed to avoid the vanishing gradient problem and has proven its abilities in capturing distant dependencies.


Finally, our full system is evaluated using multiple benchmarks.

we show that performance is improved across all datasets through the use of multitask learning and distant supervision compared to single task training.


Our method works as follows: Paragraph encoding We first represent all tokens p i in a paragraph p as a sequence of feature vectorsp i ∈ R d and pass them as the input to a recurrent neural network and thus obtain: where p i is expected to encode useful context information around token p i .

we choose to use a multi-layer bidirectional long short-term memory network (LSTM), and take p i as the concatenation of each layer's hidden units in the end.


• Aligned question embedding: Following (Lee et al., 2016) and other recent works, the last part we incorporate is an aligned question embedding , where the attention score a i,j captures the similarity between p i and each question words q j .

a i,j is computed by the dot products between nonlinear mappings of word embeddings: , and α(•) is a single dense layer with ReLU nonlinearity.


For the task of evaluating open-domain question answering over Wikipedia, we use the SQuAD development set QA pairs only, and we ask systems to uncover the correct answer spans without having access to the associated paragraphs.

a model is required to answer a question given the whole of Wikipedia as a resource; it is not given the relevant paragraph as in the standard SQuAD setting.


Table 3 compares the performance of the two approaches described in Section 3.1 with that of the Wikipedia Search Engine 5 for the task of finding articles that contain the answer given a question.

we compute the ratio of questions for which the text span of any of their associated answers appear in at least one the top 5 relevant pages returned by each system.


Therefore, we formulate it as a reinforcement learning problem and apply policy gradient method to train the model.

we need to maximize a reward function over θ a which can be constructed as follows.