To explore the differences between low-quality and high-quality counseling we start by analyzing the counselor and client dialog interaction.

we analyze their turn-by-turn interaction by examining the average number of words used by each speaker as well as their word ratio.


We consider a toy classification task: determining whether a conversation is taken by a new coun-selor who has experienced less than 20 conversations (S 0 ), versus a tenured counselor who has taken between 100 and 120 (S), on the basis of the language in the counselor's messages.

we formulate a paired prediction problem of distinguishing between two conversations taken by the same counselor at S 0 and S. Over a random subset of 10% of counselors in our data-comprising 3,075 conversation pairs-we train a logistic regression model using bigrams in counselor messages as features; we perform 10fold cross-validation and ensure that no counselor spans multiple folds.


Future work could adapt our framework to examine more complex forms of conversational development.

we observed that counselors diversify linguistically, perhaps signaling a benefipopulation-wide changes in how counselors use a subset of core words, so it cannot characterize changes outside this general vocabulary.


11 Let (q i , [a i0 , ..., a im ]) be a question with its possible answers, we first create a set of m queries of the form [q i + a i0 , ..., q i + a im ], which will be sent separately to a search engine.

we use the DrQA's Document Retriever (Chen et al., 2017), which scores the relation between the queries and the articles as TF-IDF weighted bagof-word vectors, and also takes into account word order and bi-gram counting.


For the task of learning pointwise evidence convincingness scores from a data set of evidence pairs, we bring ideas from the field of learning to rank.

in our model, we take inspiration from the training procedure provided by RankNet (Burges et al., 2005) of a Siamese network.


We propose a multimodal method that is inspired by the way humans process emotions in a conversation.

lexical and acoustic information is simultaneously perceived at every word step.


We induce lexical information from our multimodal model into the acoustic network during training while still providing a lexical-independent acoustic model for testing or deployment.

our acoustic model learns to capture semantic and contextual information without relying on explicit lexical inputs such as ASR or transcripts.


We use word embeddings to represent the lexical information.

we employ deep contextualized word representations using the language model ELMo (Peters et al., 2018).


The idea of the modality-based attention is to prioritize one of the modalities at the word level.

when the lexical features are more relevant to capture emotions (i.e., informative words are used), the model should prioritize such features and vice versa (i.e., arousal and pitch levels increase).


We split the dataset using the one-speaker-out experimental setting.

we take four sessions for training, and the remaining session is split by speakers into the validation and test sets.


Compared with Indep, Inter-EC gets great improvements on the ECPE task as well as the two sub-tasks.

we find that the improvements are mainly in the recall rate on the cause extraction task, which finally lead to the great improvement in the recall rate of ECPE.


Inter-CE also gets significant improvements on the ECPE task compared to Indep.

we find that the improvements are mainly in the precision score on the emotion extraction task, emotion extraction cause extraction emotion-cause pair extraction P R F 1 P R F 1 P R F 1 Inter-CE-Bound #0.9144 #0.8894 #0.9016 #1.0000 #1.0000 #1.0000 #0.8682 #0.8806 #0.8742 Inter-EC-Bound #1.0000 #1.0000 #1.0000 #0.7842 #0.7116 #0.7452 #0.7610 #0.7084 #0.7328  Table 4: Experimental results of all proposed models and variants using precision, recall, and F1-measure as metrics on the ECPE task with or without the pair filter.


These results demonstrate the effectiveness of the pair filter.

by introducing the pair filter, some of the candidate emotion-cause pairs in P all are filtered out, which may result in a decrease in the recall rate and an increase in precision.


We do this in a framework of certain types of motions (section 3).

we define several commonplace themes which are likely to be a point of contention -that is, where arguments of opposing stance can be made around this theme 2 .


Unlike these efforts, the taxonomy suggested here is of recurring principled semantic themes.

arguments which in this work would be categorized as belonging to a specific argument theme could be of various topoi and follow different argumentation schemes.


For the sake of simplicity and brevity (and with some abuse of notation) we will ignore this distinction.

one of the allowed "actions" is brings more harm than good, which would usually be considered as indicating an analysis motion.


Having a sizable dataset of (motion, CoPA) pairs, we examined several classifiers over it.

given a motion and a CoPA, the classifier aims to determine whether they match.


Since the CoPAs are quite varied, we examined various classifiers, some focused on a motion's action, some on its topic, and some on a combination of both.

we examined the following classifiers: By action (BA-k): Some actions are strongly indicative for (some of) the CoPAs a motion belongs to.


This vector is then used as a feature vector for a logistic regression classifier.

each CoPA is assigned the classification score of the classifier so trained.


All classifiers (except one 9 ) were evaluated in a leave-one-motion-out framework, over all motions and over relevant CoPAs.

each classifier was trained and tested 689 times -in each iteration it was trained over 688 motions and the relevant CoPAs, and then predicted whether the leftout motion matched these CoPAs.


In the complete dataset, 87% of the motions belong to at least one CoPA, and on average each motion belongs to 1.95 CoPAs.

while this is certainly only a first step toward modeling principled recurring arguments, the suggested CoPAs are indeed a concise set that covers distinct argumentative themes and offers a good coverage w.r.t.


This is probably due to the rather generic phrasing of the claims, which in the first place were constructed to be applicable "as-is" in multiple contexts.

this annotation not only confirms that the CoPA claims convey arguments actually alluded to by humans, but that they do so at a rather high level, and so capture arguments that are not only plausible for a motion but also probable.


Thus, one could phrase more specific backlash arguments for the CoPA Subsidies or Coercion (with different phrasings), and use them when the CoPA is matched to a motion.

we can expand the set of CoPA arguments to include more than just 2 claims; it could include further instances of principled arguments, each perhaps tailored to the specific CoPA.


using the Carneades Argumentation System of Walton and Gordon (2012).

instead of having such data per topic, as is currently the case in Araucari-aDB, having such data for commonplace principled arguments facilitates their use over a wide range of topics.


The framework suggested here may provide a richer and more stable basis for argumentative text generation.

a CoPA may include structured data which describes its principal theme.


Rhetoricians have characterized the art of convincing as starting from general and basic views, facts and opinions accepted by everyone (Perelman, 1971;Kock, 2009).

an efficient argument starts not from the most original and unseen premises, but from what the audience takes as consensual, and only then progresses to what is controversial.


Recent years have also seen considerable interest in neural approaches to summarization.

neural extractive approaches focus on learning vector representations of source sentences; then based on these representations they determine if a source sentence is to be included in the summary (Cheng and Lapata, 2016;Yasunaga et al., 2017;Nallapati et al., 2017;Narayan et al., 2018).


det(L Y ) also has a particular geometric interpretation as the squared volume of the space spanned by sentence vectors i and j, where the quality measure indicates the length of the vector and the similarity indicates the angle between two vectors ( Figure 1).

we adopt a feature-based approach to compute sentence quality: x i is a feature vector for sentence i and θ are the feature weights to be learned during training.


We compare our system with a number of strong summarization baselines (Table 1 and 2).

sumBasic (Vanderwende et al., 2007) is an extractive approach assuming words occurring fre- Opinosis (Ganesan et al., 2010) 25.15 5.12 8.12 Extract+Rewrite  29.07 6.11 9.20 Pointer-Gen (see et al., 2017) 31.44 6.40 10.20 sumBasic (Vanderwende et al., 2007) 31.58 6.06 10.06 KLsumm (Haghighi et al., 2009) 31.23 7.07 10.56 LexRank (Erkan and Radev, 2004) 33.10 7.50 11.13 DPP (Kulesza and Taskar, 2011)   quently in a document cluster are more likely to be included in the summary; KL-sum (Haghighi and Vanderwende, 2009) is a greedy approach adding a sentence to the summary to minimize KL divergence; and LexRank (Erkan and Radev, 2004) is a graph-based approach computing sentence importance based on eigenvector centrality.


Our DPP-based framework belongs to a strand of optimization-based methods.

iC-SiSumm (Gillick et al., 2009) formulates extractive summarization as integer linear programming; it identifies a globally-optimal set of sentences covering the most important concepts of the source documents; DPP (Kulesza and Taskar, 2011) selects an optimal set of sentences that are representative of the source documents and with maximum diversity, as determined by the determinantal point process.


In the training environment, we design different summarization models to analyze how they influence the performance.

these models differ in the types of architectures (Encoders: CNN, LSTM, Transformer (Vaswani et al., 2017); Decoders: auto-regressive 2 , non auto-regressive), external transferable knowledge (GloVe (Pennington et al., 2014), BERT (Devlin et al., 2018), NEWSROOM (Grusky et al., 2018)) and different learning schemas (supervised learning and reinforcement learning).


Extractive Summarization Most of recent work attempt to explore different neural components or their combinations to build an end-to-end learning model.

these work instantiate their encoder-decoder framework by choosing recurrent neural networks (Cheng and Lapata, 2016;Nallapati et al., 2017;Zhou et al., 2018) as encoder, auto-regressive decoder (Chen and Bansal, 2018;Jadhav and Rajan, 2018;Zhou et al., 2018) or non auto-regressive decoder (Isonuma et al., 2017;Narayan et al., 2018;Arumae and Liu, 2018) as decoder, based on pre-trained word representations (Mikolov et al., 2013;Pennington et al., 2014).


3, we can observe that models with pointer-based decoder are prone to achieving better performance against SeqLabbased decoder.

among these eight datasets, models with pointer-based decoder outperform SeqLab on six domains and achieves comparable results on the other two domains.


However, when the models are equipped with BERT, we are excited to observe that the performances of all types of architectures are improved by a large margin.

the model CNN-LSTM-Pointer has achieved a new state-of-the-art with 42.11 on R-1, surpassing existing models dramatically.


Building on this, Tsvetkov (2014) argued that the original theory of Shannon can operate at the semantic level by relying on semantic units.

existing semantic information theories imply the existence of semantic units in formal semantics (Carnap and Bar-Hillel, 1953), which treat natural languages as formal languages (Montague, 1970).


A summarizer that minimizes the KL divergence minimizes Redundancy while maximizing Relevance.

in fact, this is an instance of the Kullback Minimum Description Principle (MDi) (Kullback and Leibler, 1951), a generalization of the Maximum Entropy Principle (Jaynes, 1957): the summary minimizing the KL divergence is the least biased (i.e., least redundant or with highest entropy) summary matching D. this summary fits D while inducing a minimum amount of new information.


For the background knowledge K, a promising direction would be to use the framework to actually learn it from data.

one can apply supervised techniques to automatically search for K, α and β: finding the values of these parame-ters such that θ I has the best correlation with human judgments.


To further verify that the perturbations in the adversarial examples are hard for humans to recognize, we find six workers on Amazon Mechanical Turk to evaluate the examples generated by PWWS.

we select 100 clean texts in IMDB and the corresponding adversarial examples generated on word-based CNN.


The paper is not well written either.https://openreview.net/forum?id=HyWDCXjgx&noteId=B1Mp8grVl 4 3 0.01 Transformation based Models of Video Sequences -While I agree with the authors on these points, I also find that the paper suffer from important flaws.

: -the choice of not comparing with previous approaches in term of pixel prediction error seems very "convenient", to say the least.


Unlike conventional speech recognition models, our model learns longer conversational-context information that spans across sentences and is consequently better at recognizing long conversations.

we propose to use text-based external word and/or sentence embeddings (i.e., fast-Text, BERT) within an end-to-end framework, yielding significant improvement in word error rate with better conversational-context representation.


In this paper, we create a conversational-context aware end-to-end speech recognizer capable of incorporating a conversational-context to better process long conversations.

we propose to exploit external word and/or sentence embeddings which trained on massive amount of text resources, (i.e.


At each step i, the hidden state from the upper layer and the state from the lower layer are actually concatenated and attended together.

they use the same attention weight,  i .


Oftentimes the author uses emojis and emoticons to convey messages to a potential customer.

these types of advertisements may also contain equivocal words, e.g., roses as a substitute for dollars.


(3) For a compact representation, unnecessary jumps should be avoided.

for at least one complete path the positions should increase by exactly 1 across all adjacent succeeding lattice nodes.


We conduct a manual inspection and showcase several common patterns in which the lattice input helps improve translation quality, as well as one counter example.

we compare the outputs of the sequential and lattice models according to the 3rd and the last row in Table 1, on Fisher.


However, isolated parallel sentences are a lot easier to acquire and hence only a fraction of the parallel data will be at the document level in any practical scenario.

a context-aware model trained only on documentlevel parallel data is highly unlikely to outperform a context-agnostic model estimated from much larger sentence-level parallel corpus.


We manually gather examples with such structures in a source sentence and change the morphological inflection of the relevant target phrase to create contrastive translation.

we focus on noun phrases where the verb is elided, and the ambiguity lies in how the noun phrase is inflected.


Experimental results demonstrate that our model can significantly outperform the strong standard baseline multilingual systems and achieve even better performance than individually trained models on most of the language pairs.

our contributions are three-fold in this work: (1) We present a universal representor to replace encoder and decoder, leading to a compact translation model, which fully explores the commonality between languages.


In previous methods, UBWE is first trained using nonparallel monolingual corpora and then this pre-trained UBWE is used to initialize the word embedding in the encoder and decoder of UNMT.

the training of UBWE and UNMT are separate.


This pretrained UBWE is then used to initialize the word embedding in the encoder and decoder of UNMT.

the training of UBWE and UNMT take place in separate steps.


Table 1 shows that the performance of the Baseline-fix system is quite similar to that of the original baseline system.

baseline-fix prevents the degradation of UbWE accuracy; however, the fixed embedding also prevents UbWE from further improving UNMT training.


W 1 is trained to confuse the discriminator D 1 by making W 1 enc x and dec y increasingly similar.

we train D 1 to maximize the probability of choosing the accurate language between the original word embedding and samples from G 1 .


One shared model for multiple language pairs inherently limits the modeling capacity for each task.

if one language pair has much smaller training data than the other, oversampling the lowresource portion is not enough to compensate the scale discrepancy in multilingual training.


For a different notation, we use (x i , y j ) where i, j ∈ {0, ..., K} to represent a pair of sentences translating from a source language i to a target language j. K + 1 languages are considered in total.

a multilingual model is usually trained by maximizing the likelihood over training sets D i,j of all available language pairs S. : where we denote L j θ (x i , y j ) = log p(y j |x i , j; θ).


A multilingual model is usually trained by maximizing the likelihood over training sets D i,j of all available language pairs S. That is: where we denote L j θ (x i , y j ) = log p(y j |x i , j; θ).

the target language ID j is given to the model so that it knows to which language it translates, and this can be readily implemented by setting the initial token y 0 = j for the target sentence to start with.


(2017) showed that a trained multilingual NMT system could automatically translate between unseen pairs without any direct supervision, as long as both source and target languages were included in training.

a model trained for instance on English & French and Spanish & English is able to directly translate from Spanish to French.


This allows us to change the target language by setting j to a desired language, and is equivalent to ignoring the correlation between x i and z.

the mutual information between the source language ID i and the output language ID z -I(i; z) -is 0.


Although it is undesirable for the model to capture the dependency between i (Es) and z (En), MLE does not have a mechanism to prevent it (i.e., I(i; z) > 0) from happening.

we cannot explicitly control the trade off between I(i; z) and I(j; z) with MLE training.


Once the model captures the correct dependency that guides the model to output the desired language, it is more likely for the model to ignore the spurious correlation during standard NMT training.

we pre-train the decoder as a multilingual language model.


Our key insight is that we can use syntactic information as a proxy to the learned discrete latent space of the LT.

instead of producing a sequence of latent discrete variables, our model produces a sequence of phrasal chunks derived from a constituency parser.


In this section, we perform several analysis and ablation experiments on the IWSLT En-De dev set to shed more light on how SynST works.

we explore common classes of translation errors, important factors behind SynST's speedup, and the performance of SynST's parse decoder.


This paper further visualizes the translation through the word alignment induced by NMT.

it analyzes the effect of alignment errors on translation errors at word level and its quantitative analysis over many testing examples consistently demonstrate that alignment errors are likely to lead to translation errors measured by different metrics.


From the table, it is clear that wrong alignment is more likely to cause a translation error while correct alignment is likely to make a correct translation decision.

compared with right alignment, when alignment is wrong, the forced decoding translation error rate of CFS words increases much more than CFT words (∆).


Analyze translation errors in real decoding Besides the forced decoding translation error, we care more about search-aware statistics in real decoding.

we identify words in the reference which are recalled through the real translation, and those unrecalled words are called real decoding translation errors defined as {y} \ {ŷ}, the difference between the two sets where {y} is the set of words in y.


Then it suggests prediction difference is better for understanding NMT and visualizes NMT from word alignment induced by prediction difference.

it quantitatively analyzes that alignment errors which are likely to lead to translation errors at word level measured by different metrics.


Nevertheless, the autoregressive property of the NMT decoder has been a bottleneck of the translation speed.

the decoder, whether based on Recurrent Neural Network (RNN) (Hochreiter and Schmidhuber, 1997;Cho et al., 2014) or attention mechanism (Vaswani et al., 2017), sequentially generates words.


It is straightforward to bring the imitation learning idea for boosting the performance of NAT.

we introduce a knowledgeable AT demonstrator to supervise each decoding state of NAT model.


Specifically, we introduce a knowledgeable AT demonstrator to supervise each decoding state of NAT model.

in such case, We propose to employ a knowledgeable AT demonstrator to supervise every decoding state of NAT across different time steps and layers, which works pretty well practically.


The negative loglikelihood loss function for NAT model become is then defined as: The approach breaks the dependency among the target words across time, thus the target distributions can be computed in parallel at inference time.

the encoder stays unchanged from the original Transformer network.


We bring the intuition of imitation learning to nonautoregressive NMT and adapt it to our scenario.

the NAT model can be regarded as a learner, which will imitate a knowledgeable demonstrator at each decoding state across layers and time steps.


We propose an imitation learning framework for non-autoregressive neural machine translation to bridge the performance gap between NAT and AT.

we propose to employ a knowledgeable AT demonstrator to supervise every decoding state of NAT across different time steps and lay-ers.


To do so, we must consider every possible path through which the model could assign attention to a given encoder state.

we can compute the attention distribution induced by MILk by The first summation reflects the fact that h j can influence c i as long as k ≥ j, and the term inside the summation reflects the attention probability associated with some monotonic probability α i,k and the soft attention distribution.


We propose instead to enforce a minimum delay for writing any target token.

we model each target token as taking at least 1 γ units of time to write, mirroring the speed of the ideal simultaneous translator in AL's Equation (18).


In this work, we take the GloRE approach further and apply it to large-scale, domainindependent data labeled with distant supervision, with the goal of learning general-purpose textual relation embeddings.

we create the largest ever distant supervision dataset by linking the entire English ClueWeb09 corpus (half a billion of web documents) to the latest version of Freebase (Bollacker et al., 2008), which contains 45 million entities and 3 billion relational facts.


Experiment results on ACE 2005 Chinese and English corpus show that our proposed approach can successfully address the data imbalance problem and significantly improve the performance, outperforming the state-of-the-art models in terms of F1-score.

we find BIO tag embeddings very effective, which we believe could be used as a general part of character/word representation.


By inspecting the performance of existing models  on ACE05, we find that, for many entities, their spans are correctly identified, but their entity types are wrong.

the F1 of extracting typed entities is about 83%, while the F1 of extracting entity spans is about 90%.


given all detected entity spans in a sentence, we define an entity-relation bipartite graph.


To further utilize the structure of the graph, we also propose assigning different weights on graph edges.

we introduce a binary relation classification task, which is to determine whether the two entities form a valid relation.


We evaluate the performances using precision (P), recall (R) and F1 scores following (Miwa and Bansal, 2016;.

an output entity (e, y) is correct if its type y and the region of its head e are correct, and an output relation r is correct if its (e 1 , y 1 , e 2 , y 2 , l) are correct ( i.e., exact match).


After introducing graph convolutional networks, all three GCN-based models improve per-  formances of entity and relation.

the "GCN (static)" has been slightly improved on relations.


In general, the performances on four tasks are insensitive to the number of GCN layers 7 .

the performances on entity span, entity and relation fluctuate at 1.0 points, and the binary relation fluctuate at 1.4 points.


This work presents a solution that can resolve the inefficient multiple-passes issue of existing solutions for MRE by encoding the input only once, which significantly increases the efficiency and scalability.

the proposed solution is built on top of the existing transformer-based, pretrained general-purposed language encoders.


Compared to standard BERT's self-attention, ij , a K ij ∈ R dz are extra, which could be viewed as the edge representation between the input element x i and x j .

we devise a V ij and a K ij to encourage each token to be aware of the relative distance to different entity mentions, and vice versa.


In the supervised setting, neural network models have demonstrated substantial improvement over approaches using hand-crafted features.

piecewise convolutional neural networks (PCNN, Zeng et al., 2015) are now widely used as a basis for other improvements, such as the instance-level selective attention mechanism of Lin et al.


We proposed two losses (named RelDist) to effectively train expressive relation extraction models by enforcing the distribution over relations to be uniform -note that other target distributions could be used.

we were able to successfully train a deep neural network classifier that only performed well in a supervised setting so far.


while forcing an uniform distribution with the distance loss L D might be meaningful with a low number of predicted clusters, it might not generalize to larger number of relations.


ARNOR assumes that a trustable relation label should be explained by the neural attention model.

our ARNOR framework iteratively learns an interpretable model and utilizes it to select trustable instances.


ARNOR aims to train a neural model which is able to clearly explain the relation patterns through Attention Regularization (AR), and at the same time reduce noise based on an assumption: the clearer the model explain the relation in an instance, the more trustable this instance is.

our ARNOR framework iteratively learns the interpretable model and selects trustable instances.


Since we predict relations based on word pairs, the ground truth should likewise be based on word pairs.

word United has a HasPresident relation to both word Barack and word Obama, as does word States.


• easing the pattern-writing work of human experts by generating patterns automatically; • enabling the quick generalization to new relation types by only requiring a small number of human annotations; • presenting both significant and interpretable performance improvements as well as intuitive diagnostic analyses.

for one relation with severe false negative noises, we improve the F1 score by about 0.4.


Given an instance s with T tokens 2 , a common input representation of NRE models is x denotes the embedding of token i and d x is the token embedding size.

x i is the concatenation of the word embedding, , to be aware of both semantics and entity positions, where Given the relation type r, NRE models perform different types of tensor manipulations on x and obtain the predicting probability of r given the instance s as P φ (r|x), where φ denotes model parameters except for the input embedding tables.


We employ the REINFORCE algorithm (Williams, 1992) and policy gradient methods (Sutton et al., 2000) to optimize parameters of the agent network, where the key step is to rewrite the gradient formulation and then apply the back-propagation algorithm (Rumelhart et al., 1986) to update network parameters.

we define our objective as: where x denotes the input representation of the instance s. By taking the derivative of J(Θ) with respect to Θ, we can obtain the gradient Besides, we utilize the -greedy trick to balance exploration and exploitation.


(2009), 2) Gold Label Mix , mixing human-annotated highquality labels with DS-generated noise labels, and 3) RLRE (Feng et al., 2018), building an instanceselection agent to select correct-labeled ones by only interacting with NRE models trained on noise labels.

for Gold Label Mix, we use the same 200 labels obtained at the patternrefinement stage as the high-quality labels.


One issue of these hypergraph approaches is the spurious structures of hypergraphs as they enumerate combinations of nodes, types and boundaries to represent entities.

these models are specially designed for the nested named entities and are not suitable for the non-overlapping named entity recognition.


An overview of the proposed MGNER framework for multi-grained entity recognition, is illustrated in Figure 2.

mGNER consists of two sub-networks: the Detector and the Classifier.


For each token position, R entity proposals with the length varies from 1 to the maximum length R are generated.

it is assumed that an input utterance consists of a sequence of N tokens (t 1 , t 2 , t 3 , t 4 , t 5 , t 6 , ..., t N ).


We mainly evaluate our framework on ACE-2004and ACE-2005(Doddington et al., 2004 with the same splits used by previous works (Luo et al., 2015;) for the nested NER task.

seven different types of entities such as person, facility, weapon and vehicle, are contained in the ACE datasets.


As the original experimental setting of FewRel is few-shot learning, we rearrange the FewRel dataset for the common relation classification setting.

we sample 100 instances from each class for the training set, and sample 200 instances for the development and test respectively.


Between KGs, each channel transfers structure knowledge via shared parameters.

for KG completion, we first employ AMIE+ (Galárraga et al., 2015) on each KG to induce rules, then transfer them between KGs towards consistent completion.


to find as many alignments as possible A e = {(e, e ) ∈ E × E |e ↔ e } for which an equivalent relation ↔ holds between e and e .

e and e are in different KGs, but denote the same thing.


We utilize two channels of GNN encoder for each KG, so as to reconcile the two types of differences separately.

we generate two adjacency matrices for each channel: A 1 based on KG selfattention and A 2 based on cross-KG attention.


As shown in Figure 4, there is a performance drop in MuGNN w/o SelAtt and MuGNN w/o CroAtt as compared to MuGNN, which demonstrates the effectiveness of both channels.

the performance decrease more with the loss of cross-KG attention channel than that of KG self-attention, which implies the importance of utilizing cross-KG information for entity alignment.


Corresponding entities in Chinese DBPZH shall have a similar structure with their counterparts in English DBPEN, thus similar embeddings.

muGNN indeed reconciles structural differences by rule transfer, and learns alignment-oriented embeddings.


The overall architecture of our model is shown in Figure 2.

our model is comprised of a multi-digraph, an adapted GGNN embedding layer and a BiLSTM-CRF layer.


Our proposed Past Decode Regularization method builds on the work of using sophisticated regularization techniques to train LSTMs for language modeling.

the AWD-LSTM model achieves state-of-the-art performance with a single softmax on the four datasets considered in this paper (Merity et al., 2018b,a).


We regard the first 14 layers as the phrase generator and the last 4 layers as the word generator.

the context-phrase alignment is trained with the outputs of the 14th layer.


Putting aside the Quaterion unit basis [1, i, j, k] , W ⊗ Q can be expressed as: where We highlight that, there are only 4 distinct parameter variable elements (4 degrees of freedom), namely W r , W x , W y , W z , in the weight matrix (left) of (3), as illustrated by Figure 1; while in real-space feed-forward, all the elements of the weight matrix are different parameter variables (4 × 4 = 16 degrees of freedom).

the degrees of freedom in Quaternion feedforward is only a quarter of those in its real-space are used in 16 pairwise connections between components of the input and output Quaternions.


Recall the definition of a partial derivative, Denote by p ⋆ := π Ω (z).

we will show that for any j such that p ⋆ j = 0, and any ε ≥ 0, we consider only one side of the limit, namely subtracting a small non-negative ε.


This work examines the robustness of selfattentive neural networks against adversarial input perturbations.

we investigate the attention and feature extraction mechanisms of state-of-the-art recurrent neural networks and self-attentive architectures for sentiment analysis, entailment and machine translation under adversarial attacks.


This type of models, including Transformer (Vaswani et al., 2017) and "Bidirectional Encoder Representations from Transformers," shortened as BERT (Devlin et al., 2019), rely on the attention mechanism (Luong et al., 2015) to learn a context-dependent representation; compared to recurrent neural networks (RNN), these self-attention-based models have faster encoding speed and the capacity of modeling a wider context.

bERT is recently proposed to extend the directionality of the Transformer model, and "pre-trained" using multiple objectives to strengthen its encoding capability.


LIST employs a list of semantically similar words (i.e., synonyms), and manages to replace a word in the input sentence with another from the list to construct adversarial examples.

the list is used to replace a word with one of its synonyms; this process is repeated for every word in the input sentence until the target model makes an incorrect prediction.


In other words, the list is used to replace a word with one of its synonyms; this process is repeated for every word in the input sentence until the target model makes an incorrect prediction.

for every sentence, we start by replacing the first word with its synonyms, each forming a new adversarial example.


We implement LSTM and Transformer machine translation models using OpenNMT-py 2 .

for the LSTM model, we train it with 453 thousand pairs from the Europarl corpus of German-English WMT 15 Task 3 , common crawl, and news-commentary.


For doing this, we conduct a typo-based attack (Belinkov and Bisk, 2018).

we randomly select one word in each sentence and change it to a typo predefined in a common typo list.


To further validate our analysis, we calculate the ratio of the L2 norms of embeddings variation.

let z and z adv denote the embeddings of the original sentence and adversarial input, respectively.


, |x|} |y| to be set of all alignments from x to y where an alignment aligns each target character y i to exactly one source character in x.

it allows zero-to-one 2 or many-to-one alignments between x and y.


2 Zero in the sense of non-character like BOS or EOS The model above is exactly an 0 th -order neuralized hidden Markov model (HMM).

p(y i | a i , y <i , x) can be regarded as an emission distribution and p(a i | y <i , x) can be regarded as a transition distribution, which does not condition on the previous alignment.


In this work, we have addressed the latter issue.

we extended existing capsule networks into a new framework with advantages concerning scalability, reliability and generalizability.


First, for MTL, we propose Task-Adaptive Representation learning using Soft-coding, namely TARS, wherein shared and private features are both mixtures of features.

as shown in Figure 1c, TARS begins as a generic sharing framework using one common shared encoder, but also adopts its paired task-specific layers to feed a Mixture-of-Experts (MoE) module (Shazeer et al., 2017;Guo et al., 2018) which captures soft-private features with a weighted combination of all task-dependent features, where a gating network G in Figure 1c, decides on output weights for each task.


Instead, for an additional challenge of refining low-quality input, we add Refiner.

once the source language is translated into the target language, CASE moves the noisy representation on the target side towards a direction of space on the source side back in a form of adversarial example, and uses this as an additional training data to task classifier.


Thus, our objective is to learn a sentence (or pair) representation x k per task k, but take into account the correlation among related tasks.

given an input sequence x k = {w k 1 , w k 2 , ..., w k T } with length T , we aim to learn a sentence representation x k for the entire sequence as follows, x k = Encoder({w k 1 , w k 2 , ..., w k T }).


To be specific, TARS feeds the shared features s k into individual P-Encoder for each task, to encode task-dependent features as follows: Simultaneously, a gating network decides on output weights for each expert (i.e., individual P-Encoder).

the gating network G, parameterized by θ g , is used to map the shared representation of current task into the correct expert, and each expert is thus learning task-dependent features for that task, estimating task label of s k : where W g and b g is a trainable weight matrix and a bias, respectively.


Specifically, the gating network G, parameterized by θ g , is used to map the shared representation of current task into the correct expert, and each expert is thus learning task-dependent features for that task, estimating task label of s k : where W g and b g is a trainable weight matrix and a bias, respectively.

based on above, the final softprivate representation p(s k ) is a mixture of all expert outputs with respect to s k as the following: Soft-Shared Module In order to learn taskinvariant features, inspired by (Liu et al., 2017), TARS adopts an adversarial network, which contains a feature extractor and a task discriminator D. The basic idea is to learn features that cannot be distinguished by D. D aims to discriminate which task the feature comes from, while the feature extractor (e.g., S-Encoder) tries to fool D so that it cannot identify the task of the feature and is hence task-invariant.


However, trying to keep the shared space too pure inevitably leads to sparseness, for which we additionally introduce the density constraint L dense for this purpose.

the objective of the density constraint L dense is to push the soft-private features from the private embeddings closer to the shared ones, such that the shared space is encouraged to being dense rather than being too sparse, resolving the sparseness of the shared space.


For which, based on the basic architecture of variational auto-encoder (VAE) (Kingma and Welling, 2013), CASE models a neural refiner to refine low-quality representations.

as shown in Figure 1d, CASE first encodes pseudoparallel sentences into shared space, e.g., (x,x).


As a result,z is fed into the classifier, and the overall loss of CASE is defined as follows: Selective Refinement Lastly, CASE aims to refine only when the perturbation can refine the translation.

if the translation is already good, CASE avoids a refinement, by parameterizing refinement with α set to be near zero.


This suggests that the synergetic effect of soft-private and -shared modules in TARS is critical to outperform other baselines.

aFS and aSP show a "negative transfer", which is an inherent challenge of MTL.


As ASP requires target task labels to train its private encoders, we compare TARS only with AFS and MoE, where TARS shows the best performance in MTL.

as shown in Figure 3, we observe that when TaRS covers sentences in CNLI and LCQMC, using its gating network that identifies that the unknown target tasks are the most similar to SNLI and QQP, respectively: highest weights are assigned to these two, but other source tasks also contribute, with non-zero weights.


• (RQ3) Does our model improve robustness?

in this paper we focus on Chinese, as we could hire native speaking volunteers to validate whether automatically generated perturbations indeed preserve semantics.


On average, the feature-aware models assigned a log likelihood of −20.98 to the words in the test set, and the feature-naive models assigned an average log likelihood of −20.07.

the feature-naive models assigned over twice the probability mass to the test set compared to the feature-aware models, in terms of raw (non-log) probability.


Other work has looked into how the architecture of language models can be improved for morphologically-rich languages (Gerz et al., 2018a).


When analyzing the effect of LM training dataset size on Czech and Russian, we also train models on the training portion of a larger version of the MWC corpus, MWC-large, which contains approximately twice as much training data as the standard MWC dataset.

mWC-large contains 10.2m training characters for Czech and 18.2m for Russian.


This enables us to optimize for delayed rewards that are non-decomposable.

we directly minimize a distance function between strings, e.g., Levenshtein distance, by using negative distance as a delayed reward: 1 R(Ŷ ) = −Levenshtein(Y,Ŷ ).


In text classification with neural networks, sentence representation is calculated based on tokens that compose the sentence.

a sentence is first tokenized into meaningful units such as characters, words, and subwords (Zhang et al., 2015;Joulin et al., 2017).


After sampling, the language model is updated by adding the token counts in a currently tokenized sentence.

count(t) in (4) is reduced for every token t included in a sentence.


This is because the Chinese vocabulary is larger than the Japanese and English vocabularies.

chinese datasets have a larger number of types of n-grams.


In the figure, "INITIAL" means a prediction by a model inputted into a sentence tokenized by an initial language model.

"INITIAL" shows the prediction by the model without updating the language model.


However, Gonen and Goldberg (2019) note that the above method fails to capture the full picture of gender bias in vector spaces.

even after the projections of word embeddings representing gender-neutral words onto the gender subspace have been removed, word embeddings representing words with similar biases still cluster together.


(2018) propose a variation on the traditional generative adversarial network (Goodfellow et al., 2014) by having the generator learn with respect to a protected gender attribute.

the generator attempts to prevent the discriminator from identifying the gender in a given task such as analogy completion.


Taking gender-bias as a working example, we propose a debiasing method that preserves non-discriminative gender-related information, while removing stereotypical discriminative gender biases from pre-trained word embeddings.

we consider four types of information: feminine, masculine, gender-neutral and stereotypical, which represent the relationship between gender vs. bias, and propose a debiasing method that (a) preserves the genderrelated information in feminine and masculine words, (b) preserves the neutrality in genderneutral words, and (c) removes the biases from stereotypical words.


From Table 1, we see that the best performances (highest accuracy on Definition and lowest accuracy on Stereotype) are reported by GP (GN-GloVe), which is the application of the proposed method to debias word embeddings learnt by GN-GloVe.

in both SemBias and SemBias-subset, GP (GN-GloVe) statistically significantly outperforms GloVe and Hard-Glove according to Clopper-Pearson confidence intervals (Clopper and Pearson, 1934).


Considering that a significant number of words in English are gender-neutral, it is essential that debiasing methods do not adversely change their orientation.

the proposed method's ability to debias stereotypical words that carry unfair gender-biases, while preserving the genderorientation in feminine, masculine and neutral words is important when applying the debiased word embeddings in NLP applications that depend on word embeddings for representing the input texts We proposed a method to remove gender-specific biases from pre-trained word embeddings.


As a result, we take an unsupervised 3 approach using dependency trees, lemmata, part-of-speech (POS) tags, and morpho-syntactic tags from Universal Dependencies corpora (UD; Nivre et al., 2018).

we propose the following four-step process: 1.


We use the unary factors only to force or disallow particular tags when performing an intervention.

we define where α > 1 is a strength parameter that determines the extent to which m i should remain unchanged following an intervention.


The algorithm is a generalization of the forward-backward algorithm for hidden Markov models (Rabiner and Juang, 1986).

we pass messages from the leaves to the root and vice versa.


We therefore annotated Spanish and Hebrew sentences ourselves, with annotations made by native speakers of each language.

for each language, we extracted sentences containing animate nouns from that language's UD treebank.


Second, through an annotation study, we introduce a way of mitigating annotator bias through dialect and race priming.

by designing tasks that explicitly highlight the inferred dialect of a tweet or likely racial background of its author, we show that annotators are significantly less likely to label an AAE tweet as offensive than when not shown this information ( §4).


Figure 2 (middle and right) shows that the proportions of tweets classified as toxic also differ by group in these corpora.

in DEMOGRAPHIC16, AAE tweets are more than twice as likely to be labelled as "offensive" or "abusive" (by classifiers trained on DWMW17 and FDCL18, respectively).


we use simple linear models to look for influential terms.

we train l 2 -regularized multiclass logistic regression classifiers operating on unigram features for each of DWMW17 and FDCL18 (tuning the regularization strength on validation data).


We represent n via a multi-hot vector f n ∈ {0, 1} T of its lexical features-i.e., its genderless lemma, its gender (male or female), and its number (singular or plural).

f n always has exactly three non-zero entries; for example, the only non-zero entries of f stewardesses are those corresponding to STEWARD, +FEM, and +PL.


Given an additional distribution q(s | ν) that provides external information about the sentiment of neighbor ν, we regularize p(s | ν), as defined by our model, to be close (in the sense of KL-divergence) to q(s | ν).

we construct the following posterior regularizer: where H(q) is constant and p(s | ν) is defined as We use the combined sentiment lexicon of Hoyle et al.


This proposition therefore provides insight into how our model builds on PMI.

in contrast to PMI, 1) our model can consider lexical features other than gender, 2) our model is regularized to avoid the pitfalls of maximumlikelihood estimation, and 3) our model cleanly incorporates latent sentiments, relying on posterior regularization to ensure that the p(s | ν) is close to the sentiment lexicon of Hoyle et al.


(2014), which categorizes adjectives into thirteen senses: BEHAVIOR, BODY, FEELING, MIND, etc.

each adjective has a distribution over senses, capturing how often the adjective corresponds to each sense.


We report results under the framework of (Lample et al., 2018).

we form sentence embeddings using the average of the tf-idf weighted word embeddings in the bag-of-words representation of the sentence.


As we can see, both methods experience a decrease in overall performance when compared to models fitted on just a pair of languages, however MBFA performs better overall.

the direct approaches preserve their performance on translation to and from English, but translation from French to Italian decreases significantly.


Table 2 shows the accuracy of our nominal and verbal lemmatizers.

verbal lemmatization appears to be a more difficult task than its nominal equivalent.


( , 2014 modeled ITG-based reordering rules in the translation by using neural networks.

the NN-based reordering models can not only capture semantic similarity but also ITG reordering constraints (Wu, 1996(Wu, , 1997 in the translation context.


These results indicate that our method can effectively improve the NIST ZH-EN and WAT JA-EN translation tasks.

our approach is a universal method for improving the translation of other language pairs.


This indicates that +Both REs (base) dealt better than the vanilla baseline with this scenario.

the learned REs retrained part of reordering information in a sentence.


More recently, the system based on multi-layer self-attention (call it Transformer) has shown strong results on several large-scale tasks (Vaswani et al., 2017).

approaches of this kind benefit greatly from a wide network with more hidden states (a.k.a.


In this way, we can see that both post-norm and pre-norm are special cases of the general residual unit.

the post-norm Transformer is the special case when: while for pre-norm Transformer, it is: Here we take a stack of L sub-layers as an example.


We show that a naïve method to create compressed phoneme-like speech representations is far more effective and efficient for translation than traditional frame-level speech features.

we generate phoneme labels for speech frames and average consecutive frames with the same label to create shorter, higher-level source sequences for translation.


Inspired by these works, we explore higher-level continuous speech embeddings for end-to-end speech translation.

we use alignment methods to generate phoneme labels, and average consecutive frames with the same label to create phonemelike feature vectors from variable numbers of frames.


Therefore, for Mboshi, we use a method that does not require language-specific data to generate a phoneme-like sequence.

we apply a Connectionist Temporal Classification (CTC) model trained with 6000 hours of English data (notably, not a related language), as described in Dalmia et al.


However, all of these approaches are based on linguistic annotations.

they operate on the part-of-speech tags of words instead of word tokens.


Next, given the visual grounding of constituents, Θ is optimized for producing constituents that can be better matched with images.

we optimize textual representations and the visual-semantic embedding space using a hinge-based triplet ranking loss: where i and k index over all image-caption pairs in the data set, while j and enumerate all constituents of a specific caption (c (i) and j } is the set of textual representations of all constituents, and δ is a constant margin, The loss L extends the loss for image-caption retrieval of Kiros et al.


Intuitively, the following objective encourages adjectives to be associated (combined) with the corresponding nouns, and verbs/prepositions to be associated (combined) with the corresponding subjects and objects.

we use REINFORCE (Williams, 1992) as the gradient estimator for Θ.


Thus, we impose an inductive bias to guide the learner to correctly associate prepositions with their complements, determiners with corresponding noun phrases, and complementizers with the corresponding relative clauses.

we discourage abstract constituents (i.e., constituents that cannot be grounded in the image) from being combined with a preceding constituent, by modifying the original reward definition (Equation 2) as: where λ is a scalar hyperparameter, v (i) is the image embedding corresponding to the caption being parsed, and abstract denotes the abstractness of the span, defined analogously to concreteness (Equation 1): The intuition here is that the initial heads for prepositional phrases (e.g., on) and relative clauses (e.g., which, where) are usually abstract words.


Based on the concreteness score of words, we introduce another baseline similar to VG-NSL.

we recursively combine two consecutive constituents with the largest average concreteness, and use the average concreteness as the score for the composed constituent.


We also test the effectiveness of using pretrained word embeddings.

for VG-NSL+HI+FastText, we use a pre-trained FastText  embedding (300-D, Joulin et al., 2016), concatenated with a 212-D trainable embedding, as the word embedding.


Here, we evaluate the consistency between word-level concreteness estimates induced by VG-NSL and those produced by other methods (Turney et al., 2011;Brysbaert et al., 2014;Hessel et al., 2018).

we measure the correlation between the con-Model/method VG-NSL (+HI) Turney et al.


Meanwhile, it is worth noting that even if we select VG-NSL by R@1, it shows better stability compared with PRPN and ON-LSTM (Table 1), in terms of the score variance across different random initialization and self F 1 .


There are also limitations to the idea of grounded language acquisition.

the current approach has thus far been applied to understanding grounded texts in a single domain (static visual scenes for VG-NSL).


Our experiments show that interleaving behavioral cloning and policy gradient training phases improves performance on the validation set.

we interleaved each policy gradient update batch with K behaviour cloning batches, with the value of K decaying exponentially, such that the training strategy asymptotically becomes only policy gradient updates.


In this paper, we address a novel task, namely weakly-supervised spatio-temporally grounding natural sentence in video.

given a natural sentence and a video, we localize a spatio-temporal tube in the video that semantically corresponds to the given sentence, with no reliance on any spatio-temporal annotations during training.


In this paper, we introduce a novel task, referred to as weakly-supervised spatio-temporally grounding sentence in video (WSSTG).

given a natural sentence and a video, we aim to localize a spatio-temporal tube (i.e., a sequence of bounding boxes), referred to as an instance, in the video which semantically matches the given sentence (see Fig.


To facilitate our proposed WSSTG task, we contribute a new grounding dataset, called VIDsentence, by providing sentence descriptions for the instances of the ImageNet video object detection dataset (VID) (Russakovsky et al., 2015).

7, 654 instances of 30 categories from 4, 381 videos in VID are extracted.


Different from Zhao et al., 2018) which used only the last hidden state h q tq as the feature embedding for the query sentence, we generate visually guided sentence features H qp = {h qp t } tp t=1 by exploiting their fine-grained relationships based on H q and H p .

given the i-th visual feature h p i , an attention mechanism (Xu et al., 2015) is used to adaptively summarize where .


After obtaining a set of visually guided sentence features H qp = {h qp t } tp t=1 , we characterize the fine-grained matching behaviors between the visual and sentence features.

the matching behavior between the i-th visual and sentence features is defined as The instantiation of φ can be realized by different approaches, such as multi-layer perceptron (MLP), inner-product, or cosine similarity.


We, therefore, resort to MIL to optimize the proposed network based on the obtained matching behaviors of the instance-sentence pairs.

our objective function is defined as where L rank is a ranking loss, aiming at distinguishing aligned video-sentence pairs from the unaligned ones.


Thus, it is desirable to have a diverse distribution of the matching behaviors n=1 , we propose a diversity loss L div to strengthen the matching behaviors between reliable instance-sentence pairs and penalize the unreliable ones during training.

we first normalize {s(q, p n )} N n=1 by softmax and then penalize the entropy of the distribution of {s (q, p n )} N n=1 by defining the diversity loss as Note that the smaller L div is, the more diverse {s(q, p n )} N n=1 will be, which implicitly encourages the matching scores of semantically aligned instance-sentence pairs being larger than those of the misaligned pairs.


For simplicity, we use the same CNN architecture to encode the functions f used in the prior q and in the inference networks p, in each case with different parameters.

in prior q, the embedding sizes of domain and label are set to 16 and 4, respectively.


Our goal is to show that an accurate entity linker can be created relying solely on naturally occurring data.

our approach relies only on Wikipedia and a collection of unlabeled texts.


This approach is reminiscent of constraintdriven learning (Chang et al., 2007), as well as of multi-instance learning methods common in relation extraction (Riedel et al., 2010;Surdeanu et al., 2012).

we minimize where Θ is the set of model parameters, δ is a margin, and We rely primarily on Wikipedia to produce weak supervision.


In order to test how far we can go without using the disambiguation model, we combine together the signals we relied on in the previous section.

rather than using r wiki alone, we linearly combine the Levenstein edit distance (Levenshtein, 1966), with the scores p wiki and r wiki .


Our final model is the sum ofŝ and s c (i.e.,ŝ + s c ) where s c is computed by a linear combination of: • d(e i , m i ), the string similarity score between the title of e i and m i , using Levenshtein algorithm, • p wiki (e i |m i ), and .

we have: We tune α, β, γ on the development set.


With this view in mind, we allow our instruction following agent to observe the environment and build a representation of it prior to seeing any linguistic instructions.

we adopt a semisupervised setup with two phases, as shown in Figure 2: an environment learning phase where the system sees samples of language-free state transitions from actions in the environment, and a language learning phase where instructions are given along with their corresponding effects on the en- vironment.


Thus, previous studies conduct domain adaptation through selecting relative and informative source data according to the nature of the target domain, via entropy-based methods , Bayesian optimization (Ruder and Plank, 2017), etc.

for NLP, TDS are proved to be effective in various tasks, such as in language modeling (Moore and Lewis, 2010), word segmentation , machine translation (Chen et al., 2016;van der Wees et al., 2017), and multilingual NER (Murthy et al., 2018).


In this way, the aspect information can be utilized at each time step for generating an entire sketch.

we set the initial hidden vector for the j-th sketch as the last embedding of the previous sketch: we have h S 1,0 = v c for initialization.


Incorporating intentional rhetorical embellishments is essential to achieving the desired stylistic aspects of impassioned modern Chinese poetry.

the use of metaphor and personification, both frequently used forms of rhetoric, are able to enrich the emotional impact of a poem.


In particular, the use of metaphor and personification, both frequently used forms of rhetoric, are able to enrich the emotional impact of a poem.

a metaphor is a figure of speech that describes one concept in terms of another one.


In recent years, deep learning-driven methods have shown significant success in poetry generation, and topic-based poetry generation systems have been introduced (Ghazvininejad et al., 2017(Ghazvininejad et al., , 2018Yi et al., 2018b).

zhang and Lapata (2014) propose to generate Chinese quatrains with Recurrent Neural Networks (RNNs), while Wang et al.


We introduce an Addressing Function as α = A(M, q), which calculates the probabilities of each slot of the memory being selected and invoked.

we define: where σ defines a non-linear layer, q is the query vector, b is the parameter, M is the memory to be addressed, M k is the k-th slot of M , and α k is the k-th element in vector α.


The Rhetoric-F1 score is used to measure the rhetorically controlled accuracy of the generated poem sentences.

if the rhetoric label of the generated sentence is consistent with the ground truth, the generated result is right, and wrong otherwise.


Overall, our model generates poems better than other baselines in terms of fluency, coherence, meaningfulness, and rhetorical aesthetics.

these results show that a rhetorically controlled encoder-decoder can generate reasonable metaphor and personification in poems.


To remedy this, we propose a novel Seq2SentiSeq model.

the numeric sentiment intensity value is incorporated into the decoder via a Gaussian kernel layer to finely control the sentiment intensity of the output.


Second, parallel data 3 is unavailable in practice.

we can only access the corpora which are labeled with fine-grained sentiment ratings or intensity values.


A pretrained sentiment scorer is used to evaluate how well the sampled sentenceŷ matches the target sentiment intensity value v y .

the reward for sentiment transformation is formulated as: where ϕ refers to the pre-trained sentiment scorer which is implemented as LSTM-based linear regression model.


In order to expose the decoder to the original sentence from the training corpus, we borrow ideas from back-translation (Lample et al., 2018a,b).

the model first generates a sequenceŷ based on the input text x and the target sentiment intensity value v y , and then reconstructs the source input x based onŷ and the source sentiment intensity value v x .


Letŷ be the gold output and y the model output.

relation Generation measures how factual y is compared to input r. it measures the precision and number of relations extracted from y which are also found in r. Content Selection measures the precision and recall of relations betweenŷ and y.


To assess the impact of hierarchical attention (Section 4.2) over ED+CC, we report the performance of a stripped-down variant of our model without dynamic entity memory.

the entity memory was kept static and set to u t=−1,k (see Equation (5)).


Therefore, we acquire head-modifier templates by using a dependency parser provided by Stanford CoreNLP (Manning et al., 2014).

a type description is formed by head words (or heads), modifier words (or modifiers) and conjunctions.


Note that aside from ground truth, infobox is also a reliable source to provide candidate types.

in Wikidata, the values in instance of (P31) and subclass of (P279) are usually suitable types for an entity, though not every entity has these properties and these types could be too coarse-grained like human.


Our method outperforms other baselines in term of Grammar Accuracy, which demonstrates that the model benefits from the head-modifier templates in term of readability by knowing "how to say it".

the templates improves the Grammar Accuracy substantially compared with Ptr-Gen.


Inspired by these work, we break up the table-to-text generation into two stages, each of which is performed by a model trainable with only a few annotated data.

it first predicts the key facts from the tables, and then generates the text with the key facts, as shown in Figure 1.


The parameters of the Gaussian distribution are computed from z x with a conditional prior network parameterized by ψ.

µ and σ 2 are the outputs of multilayer perceptron (MLP) networks taking z x as the input.


This is because SIVAE-c considers the dependency between the sentence prior and the syntactic tree prior, so it can more efficiently prevent the mismatch between two latent variables.

sIVAE-c learns more robust syntactic representations, but this advantage is not reflected on the reconstruction evaluation.


However, the generated paraphrases are very similar, indicating that the variance of the conditional prior network is small.

given a sentence latent representation, the range for SIVAE-c selecting a possible syntactic tree representation is small, so it tends to generate similar paraphrases.


In this paper, we propose to leverage several multi-level structures to learn a VAE model for generating long, and coherent text.

a hierarchy of stochastic layers between the encoder and decoder networks is employed to abstract more informative and semantic-rich latent codes.


Recently, NL generation models have been proposed to automatically construct humanreadable descriptions from MRs, for code summarization (Hu et al., 2018;Allamanis et al., 2016;Iyer et al., 2016) that predicts the function of code snippets, and for AMR-to-text generation (Song et al., 2018;Konstas et al., 2017;Flanigan et al., 2016).

a common objective that semantic parsers aim to estimate is p θ (y|x), the conditional distribution between NL input x and the corresponding MR output y, as demonstrated in Fig.


Sequence-to-sequence (seq2seq) models have achieved competitive results on both semantic parsing and generation (Dong and Lapata, 2016;Hu et al., 2018), and without loss of generality, we adopt it as the basic framework for both tasks in this work.

for both p θ (y|x) and q φ (x|y), we use a two-layer bidirectional LSTM (bi-LSTM) as the encoder and another one-layer LSTM as the decoder with attention mechanism (Luong et al., 2015).


be both improved during the update of I p e (x,y) .

we rewrite E p e (x,y) log p e (x, y) as E p e (x,y) log p e (y)p e (x|y), where p e (y) and p e (x|y) are referred as the dual task distributions.


Here we study whether DIM helps enhance the lower bounds of the expected joint distributions of NL and MRs.


The model begins to select the subject entity and update the memory states if the subject entity will change.

the model first calculates the probability of selecting an entity: where E t−1 is the set of entities that have already been referred to by time step t, and s is defined as s = max{s : s ≤ t − 1, e = e s }, which indicates the time step when this entity was last mentioned.


If the salient data record is the same as the previous one (Z t = 0), it predicts the next word y t via a probability over words conditioned on the context vector h t : Subsequently, the hidden state of language model h LM is updated: where y t is the embedding of the word generated at time step t. 4 We also incorporate the information about the writer of the summaries into our model.

instead of using Equation 9, we concatenate the embedding w of a writer to to construct context vector h t : where W H is a new weight matrix.


We speculate this is mainly due to the reduced amount of our training data (Section 3).

their model might be more data-hungry than other models.


We speculate that the tracking model successfully memorized which input records have been selected in h ENT s .

the description about DERRICK ROSE's relations, "15 points, four assists, three rounds and one steal in 33 minutes.


2We then augment our model with two or three layers of reasoning but without the decision maker p d .

we directly use the hidden states of the integration LSTM as the input to the next reasoning layer (formally, U j =Ũ j ).


To tackle the above difficulties, we propose a Transformerbased encoder-decoder framework with two novel extensions for abstractive document summarization.

(1) to encode the documents comprehensively, we design a focus-attention mechanism and incorporate it into the encoder.


In this paper, we propose the Extended Transformer model for Abstractive Document Summarization (ETADS) to tackle the above issues.

we design a novel focusattention mechanism and saliency-selection network equipped in the encoder and decoder respectively: (1) To comprehensively encode the documents, we design a focus-attention mechanism, where a learnable Gaussian focal bias is employed as a regularization term on attention scores.


The calculation of BATT(Z, t) is similar to the Eq.(1).

for the l-th bridge sub-layer in the decoder, key K l and value V l are obtained by linear projections from Z.


To reduce the amount of calculation, we only utilize the query matrices Q l to compute the position vector and coverage scope vector.

for the i-th encoding step in l-th layer, the center position scalar µ i l ∈ R and the coverage scope scalar σ i l ∈ R are calculated by two linear projections, namely: where W p ∈ R dm×dm , and W g ∈ R dm×dm are two shared weight matrices.


The experimental results demonstrate that our model is competitive with or outperforms other unsupervised approaches.

for relatively long reviews, it achieves a competitive or better performance than supervised models.


The contributions of this work are three-fold: • We propose a novel unsupervised end-to-end model to generate an abstractive summary of a single product review while inducing a latent discourse tree • The experimental results demonstrate that our model is competitive with or outperforms other unsupervised models.

for long reviews, it achieves a competitive or better performance than the supervised models.


We denote a ij as the marginal probability of dependency where the i-th sentence is the parent node of the j-th sentence.

a 0j denotes the probability that a root node is the parent (see Figure 2).


The experimental results demonstrated that our model is competitive with or outperforms other unsupervised approaches.

for relatively long reviews, our model achieved competitive or better performance compared to supervised models.


We develop an abstractive meeting summarizer from both videos and audios of meeting recordings.

we propose a multi-modal hierarchical attention mechanism across three levels: topic segment, utterance and word.


We study the use of natural language inference (NLI) (Bowman et al., 2015), also known as textual entailment (Dagan et al., 2006), to detect factual errors.

we test whether entailment predictions of NLI models can be used to rerank generated summaries such that more correct ones are preferred.


For every summary sentence, a worker assigns the label correct, if the information is entailed by the document, incorrect, if it contradicts the document or contains information not present 2 , or unclear, if the worker cannot  decide.

as we cannot assume that crowdworkers are familiar with the term entailment, we ask them whether a summary sentence is "correct given the information in the article".


The knowledge learned about the document during the pre-training process will be transferred and benefit on the summarization task.

the Mask task randomly masks some sentences and predicts the missing sentence from a candidate pool; the Replace task randomly replaces some sentences with sentences from other documents and predicts if a sentence is replaced.


Mask Similar to the task of predicting missing word, the Mask task is to predict the masked sentence from a candidate pool.

we first mask some sentences within a document with the probability P m and put these masked sentences ( into a candidate pool T m .


Replace The Replace task is to randomly replace some sentences (with probability P r ) in the document with sentences from other documents, and then predict if a sentence is replaced.

we use sentences from 10, 000 randomly chosen documents to form a candidate pool T r .


We explore data augmentation methods and evaluate state-of-the-art neural abstractive models on this new task.

we show that semantic augmentation from question datasets improves the overall performance, and that pointer-generator networks outperform sequence-to-sequence attentional models on this task, with a ROUGE-1 score of 44.16%.


This paper focuses on the topic of inferential machine comprehension, which aims to fully understand the meanings of given text to answer generic questions, especially the ones needed reasoning skills.

we first encode the given document, question and options in a context aware way.


Based on above observations, we here propose an end-to-end approach to mimic human process for deducing the evidence chain.

we first encode the given document, question and options by considering contextual information.


The cell includes the memory and three operating units that work in tandem.

master unit derives a series of attention-based operations based on the question; reader unit extracts relevant document content on the operation; and writer unit performs the operation to deduce a result and update the memory.


Practically, we utilize position and POS embedding (Li et al., 2018b) generated by word embedding tool.

the embedding layer is a W ∈ R d×v , where d is the dimension and v denotes the number of instances.


Based on the memory state, three operational units work together in series to accomplish the reasoning process.

master unit analyzes the question details to focus on certain aspect via self-attention; reader unit then extracts related content, guided by the question aspect and text context; and the writer unit iteratively integrates the content with preceding results from the memory to produce a new intermediate result.


To dynamically adjust to such depth, a terminated gate is designed by considering two conditions.

the correlation between the intermediate result m t−1 and the reasoning operation a t−1 in previous step, as well as m t−1 and candidate answer options o l | L l=1 .


(2), so as to restrict the space of the valid reasoning operations and boost the convergence rate.

we calculate the correlation ac t,s and pass it through a softmax layer to yield a distribution av t,s over the question words.


Lastly, we use soft attention to select content that is relevant to the reasoning operation a t and candidate options o l | L l=1 .

we unify the a t and o l | L l=1 by a linear transformation to obtain where the options size L is fixed and predefined.


We then leverage a classifier to predict an answer by referring to the question q and options o l | L l=1 .

we first measure the correlation of m t−1 against q and o l | L l=1 , to get m t−1 q, m t−1 o l .


As elaborated in Figure 6, our approach outperformed the individual baselines on all three data sets 1 .

for RACE data set, our approach achieved the best performance and outperformed the second one (i.e.


In this work, we adopt 1D convolution as the local encoder.

let X ∈ R D×L be the in-put matrix of an L-token sequence, and each token embedding is D-dimensional.


Experiments show that the regularization can produce small gains in performance.

we jointly optimize the following regularization term when training the model.


This layer consists of N stacked DynSA blocks.

as is shown in Figure 2, we concatenate the vector sequences of all passages end to end, and then stack N layers of DynSA blocks on top of this long vector sequence.


We also conduct experiments to show the computational costs of the proposed model and other baseline models.

we replace the DynSA blocks in Figure 2 with Bi-LSTM (Hochreiter and Schmidhuber, 1997), full SAN, and Bi-BloSAN (Shen et al., 2018b) respectively.


Considering the requirements of our MRC model, we only extract the positional information of such inter-word semantic connections.

for each word w, we extract a set E w , which includes the positions of the passage words that w is semantically connected to (if w itself is a passage word, we will exclude its own position from E w ).


We use the pre-extracted general knowledge to construct the enhanced context embeddings.

for each word w, whose context embedding is c w , to construct its enhanced context embedding c * w , first recall that we have extracted a set E w , which includes the positions of the passage words that w is semantically connected to, thus by gathering the columns in C P whose indexes are given by E w , we obtain the matching context embeddings Z ∈ R d×|Ew| .


Thus we use the preextracted general knowledge to guarantee that the fusion of coarse memories for each passage word will only involve a precise subset of the other passage words.

for each passage word p i , whose coarse memory is g p i , to perform the fusion of coarse memories, first recall that we have extracted a set E p i , which includes the positions of the other passage words that p i is semantically connected to, thus by gathering the columns in G whose indexes are given by E p i , we obtain the matching coarse memories Z ∈ R d×|Ep i | .


The passages in the adversarial sets contain misleading sentences, which are aimed at distracting MRC models.

each passage in AddSent contains several sentences that are similar to the question but not contradictory to the answer, while each passage in AddOneSent contains a human-approved random sentence that may be unrelated to the passage.


We compare KAR with other MRC models in both performance and the robustness to noise.

we not only evaluate the performance of KAR on the development set and the test set, but also do this on the adversarial sets.


We compare KAR with other MRC models in the hunger for data.

instead of using all the training examples, we produce several training subsets (i.e.


In this paper, we innovatively integrate the neural networks of MRC models with the general knowledge of human beings.

inter-word semantic connections are first extracted from each given passage-question pair by a WordNet-based data enrichment method, and then provided as general knowledge to an end-to-end MRC model named as Knowledge Aided Reader (KAR), which explicitly uses the general knowledge to assist its attention mechanisms.


Our ensemble model of six training runs, where each model was trained with the two answer styles, achieved state-of-theart performance on both tasks in terms of ROUGE-L.

for the NLG task, our single model outperformed competing models in terms of both ROUGE-L and BLEU-1.


Current state-ofthe-art models use the pointer-generator mechanism (See et al., 2017).

content selection approaches, which decide what to summarize, have recently been used with abstractive models.


The key to its success is transferring the style-independent NLG capability to the target style by use of the question-passages reader and the conditional pointer-generator decoder.

the capability of copying words from the question and passages can be shared among the styles, while the capability of controlling the mixture weights for the generative and copy distributions can be acquired for each style.


Unlike previous pipelined approaches that contain separate models, we integrate an early-stopped retriever, a distantly-supervised reader, and a span-level answer reranker into a unified network.

we encode segments of text with pre-trained Transformer blocks (Devlin Model Retrieve Read Rerank Architecture DS-QA (Lin et al., 2018) 3 3 7 Pipeline R 3 (Wang et al., 2018a) 3 3 7 Pipeline* Extract-Select (Wang et al., 2018d) 7 3 Pipeline* V-Net (Wang et al., 2018c) 7 3 3 Unified Re-Ranker (Wang et al., 2018b) 3 3 3 Pipeline RE 3 QA 3 3 3 Unified Table 1: Comparison of RE 3 QA with existing approaches.


Therefore, we propose to rank all segments using early-summarized hidden representations as a mechanism for efficiently retrieving few top-ranked segments.

let h J denote the hidden states in the J-th block, where J < I.


Inspired by the non-maximum suppression (NMS) algorithm (Rosenfeld and Thurston, 1971) that is used to prune redundant bounding boxes in object detection (Ren et al., 2015), we present a span-level NMS (Algorithm 1) to alleviate the problem.

span-level NMS starts with a set of candidate answers A with scores S. After selecting the answer a i that possesses the maximum score, we remove it from the set A and add it to B.


Finally, we define the following reranking objective: 3.6 Training and Inference Rather than separately training each component, we propose an end-to-end training strategy so that downstream components (e.g., the reader) can benefit from the high-quality upstream outputs (e.g., the retrieved segments) during training.

we take a multi-task learning approach (Caruna, 1993;Ruder, 2017), sharing the parameters of earlier blocks with a joint objective function defined as: Algorithm 2 details the training process.


To promote diversity, we over-generate prospective candidates (Heilman and Smith, 2010) for every answer span and later prune them.

we use beam search with a beam size of 3 to generate three highly-probable question candidates.


Although our model does not use any pre-trained language model such as  BERT (Devlin et al., 2019) for encoding, it outperformed the methods that used BERT such as DFGN + BERT and BERT Plus.

the improvement in the evidence EM score was +37.5 points against the baseline and +5.4 points against GRN.


Figure 4 reveals the reason for the high EM and precision scores; QFE rarely extracts too much evidence.

it predicts the number of evidence sentences more accurately than the baseline.


The FEVER score was used as a metric accounting for both A T and E. The FEVER score of a sample is 1 if the predicted evidence includes all gold evidence and the answer is correct.

the FEVER score emphasizes the recall of extracting evidence sentences over the precision.


It takes as input passage P and question Q, and computes for each token a context-aware representation.

given passage P = {p i } m i=1 and question Q = {q j } n j=1 , we first pack them into a single sequence of length m + n + 3, i.e., where SEP is the token separating Q and P , and CLS the token for classification (will not be used in this paper).


It takes as input the BERT representations {h L i } output from the previous layer, and enriches them with relevant KB embeddings, which makes the representations not only context-aware but also knowledge-aware.

for each token s i , we get its BERT representation h L i ∈ R d 1 and retrieve a set of potentially relevant KB concepts C(s i ), where each concept c j is associated with KB embedding c j ∈ R d 2 .


KB Embedding In contrast to directly encoding KBs as symbolic (subject, relation, object) facts, we choose to encode them in a continuous vector space.

given any triple (s, r, o), we would like to learn vector embeddings of subject s, relation r, and object o, so that the validity of the triple can be measured in the vector space based on the embeddings.


Our experiments demonstrate that translationbased methods do not perform well in crosslingual OpenQA task.

we observe a large gap between the results of multilingual BERT and translate-test BERT for Chinese and Tamil.


All models soundly outperform right branching baselines, and we find that the neural PCFG/compound PCFG are strong models for grammar induction.

the compound PCFG outperforms other models by an appreciable margin on both English and Chinese.


While distributed representations promote parameter sharing, we were unable to obtain improvements through more factorized parameterizations that promote even greater parameter sharing.

for rules of the type A → BC, we tried having the output embeddings be a function of the input embeddings (e.g.


Clark (2001) and Klein and Manning (2002) were some of the first successful statistical approaches to grammar induction.

the constituent-context model (CCM) of Klein and Manning (2002), which explicitly models both constituents and distituents, was the basis for much subsequent work (Klein and Manning, 2004;Huang et al., 2012;Golland et al., 2012).


The curves on both PB and ZX clearly show that corpus weighting is extensively helpful, and the performance gap between a good weight factor and a bad one can be large for certain target domains and methods.

for PB as the target domain, it seems sufficient to use the same weight for the source and target domains (i.e., M = 1), and choosing a proper larger M leads to less than 1% improvement.


In standard HPSG relating to HFP, the HEAD value of any headed phrase is structure-shared with the HEAD value of the head daughter.

the phrase in our simplified HPSG tree may be exactly the same as that in a constituent tree and the head word of the phrase corresponding to the parent of the head word of its children in dependency tree 3 .


After that, consecutive words being predicted as positive class by the classifier of the same type are treated as an entity.

for sequence s = {w 1 , w 2 , w 3 , w 4 , w 5 }, if its predicted labels by the classifier of a given type are L = {1, 1, 0, 0, 1}, then we treat {w 1 , w 2 } and {w 5 } as two entities of the type.


The key idea of the proposed method is to adaptively enrich the named entity dictionary.

we first train a PU learning classifier f and use it to label the unlabeled dataset.


There are several overlaps between these linguistic formalisms, and we show below that parsers, using multi-task learning strategies, can take advantage of these overlaps or synergies during training.


In this paper, we present a global context enhanced deep transition architecture to eliminate the mentioned limitations of BiLSTMs.

we base our network on the deep transition (DT) RNN (Pascanu et al., 2014), which increases the transition depth between consecutive hidden states for richer representations.


Previous studies have shown the superiority of this architecture on both language modeling (Pascanu et al., 2014) and machine translation (Barone et al., 2017; Meng and Zhang, 2019).

meng and Zhang (2019) propose to maintain a linear transformation path throughout the deep transition procedure with a linear gate to enhance the transition structure.


We adopt an independent deep transition RNN named global contextual encoder (right part in Figure 1) to capture global features.

we transform the hidden states of global contextual encoder into a fixed-size vector with various strategies, such as mean pooling, max pooling and self-attention mechanism (Vaswani et al., 2017).


In this experiment, we investigate the effects of locations on the global contextual embedding in our hierarchical model.

we use the global embedding g to augment: • input of final softmax layer ; x • input of sequence labeling encoder; x encoder k = [w k ; c k ; g] Table 5 shows that the global embedding g improves performance when utilized at the relative low layer (row 3) , while g may do harm to performances when adapted at the higher layers (row 0 vs. row 1 & 2).


These experiments can be done with PCFGs randomly sampled from any distribution, but the fact that maximizing data likelihood as the objective can give better models than arbitrary random models ensures that evaluations are tractable and meaningful.

this model defines a Chomsky normal form (CNF) PCFG as a matrix G of binary rule probabilities which is first drawn from the Dirichlet prior with a concentration parameter β: trees for sentences 1..N are then generated by drawing from a PCFG: each tree τ is a set {τ , τ 1 , τ 2 , τ 11 , τ 12 , τ 21 , ...} of category node labels τ η where η ∈ {1, 2} * defines a path of left or right branches from the root to that node.


The main difference is that a great number of entities in the CBS News dataset are closely related to the domain of science and technology.

for the MISC category, more technology terms such as Space X, bitcoin and IP are included, as compared with the CoNLL data set.


We investigate supervised transfer from CoNLL to 13PC and 13CG, comparing our model with a range of baseline transfer approaches.

three sets of comparisons are made, including (1) a comparison between our method with other supervised domain adaptation methods, such as MULTITASK(NER) 4 and ELMo, (2) a comparison between the use of different subsets of data for transfer under our own framework and (3) a comparison with the current state-of-the-art in the literature for these datasets.


Following this line of thought, it is natural to ask whether we can introduce more structured knowledge into node representations.

if biaffine mappings encode the first order parent-children relations, can we incorporate other high-order relations (such as grandparents and siblings)?


To encode the surface form of s, we can use recurrent neural networks (Kiperwasser and Goldberg, 2016b).

we apply a bidirectional long short-term memory network (biLSTM, (Hochreiter and Schmidhuber, 1997)  where x i is the input of a LSTM cell which includes a randomly initialized word embedding e(w i ), a pre-trained word embedding e ′ (w i ) from Glove (Pennington et al., 2014) and a trainable embedding of w i 's part-of-speech tag e(pos i ), Then, a context-dependent node representation of word i is the concatenation of the two hidden vectors, With the node representations, we can define the score function σ using a multi-layer perceptron σ(i, j) = MLP(c i ⊕ c j ) (Pei et al., 2015), or using a normalized bilinear function (A, b 1 , b 2 are parameters), x 1 x 2 x 3 x 4 Figure 1: The GNN architecture.


Concretely, based on c i , we use two multilayer perceptrons to generate two different vectors, The score funcion in Equation 2 now becomes The main task we will focus on in following sections is to further encode deep structure of s to node vectors h i and d i .

besides the parent-child relation, we would like to consider high-order dependency relations such as grandparents and siblings in the score function σ.


First, for incorporating grandparent information (Figure 2.a), we expect σ t (j, i), which depends on the head vector of j and the dependent vector of i, not only considers the parent-child pair (j, i), but also consults the ("soft") parent of j suggested by the previous layer (denoted by k).

the new head representation of node j should examine representations of its neighbors when they act as parents of j.


In general, our proposed method achieves significant improvements over our baseline biaffine parser and matches state-of-the-art models.

it achieves 0.29 percent UAS and 0.35 percent LAS improvement over the baseline parser, and 0.1 percent UAS and 0.12 percent LAS improvement over the strong transition-based parser (Ma et al., 2018).


Our results also present a consistent conclusion.

on UAS, LAS and LA, the 2-layer model has the highest sum of scores.


In this work, we study the prediction of keyphrases, generally formed with words or phrases reflecting main topics conveyed in input texts .

we focus on producing keyphrases for social media language, proven to be beneficial to a broad range of applications, such as instant detection of trending events (Weng and Lee, 2011), summarizing public opinions (Meng et al., 2012), analyzing social behavior (Ruths and Pfeffer, 2014), and so forth.


(2017) based on variational autoencoder (Kingma and Welling, 2013), which consists of an encoder and a decoder to resemble the data reconstruction process.

the input x bow is first encoded into a continuous latent variable z (representing x's topic) by a BoW encoder.


Then the BoW decoder, conditioned on z, attempts to reconstruct x and outputs a BoW vector x bow .

the decoder simulates topic model's generation process.


Analogous to LDA-style topic models, it is assumed that there are K topics underlying the given corpus C. Each topic k is represented with a topic-word distribution φ k over the vocabulary, and each post x ∈ C has a topic mixture denoted by θ, a K-dim distributional vector.

in neural topic model, θ is constructed by Gaussian softmax (Miao et al., 2017).


The decoder hence takes the following steps to simulate how each post x is generated: is also a ReLU-activated neural perceptron for inputs.

we employ the weight matrix of f φ (•) as the topic-word distributions (φ 1 , φ 2 , ..., φ K ).


Its target is to output a word sequence y as x's keyphrase.

for a source post with multiple gold-standard keyphrases, we follow the practice in Meng et al.


(2017), which allows keywords to be directly extracted from the source input.

we adopt a soft switcher λ j ∈ [0, 1] to determine whether to copy a word from source as the j-th target word: with W λ and b λ being learnable parameters.


Posts without such keyphrase tags are hence removed from the datasets.

for Stack-Exchange, we concatenate the question title together with its description as the source input.


However, the system is unable to pick up this nuance because of lack of information about current events.

the system does not have prior information on who Dr. Ford is.


We evaluate the performance by the top k (k = 1, 2) accuracy (A@1, A@2), average token-level F 1 score (F 1 @1), and mean reciprocal rank (MRR).

the accuracy and MRR are calculated at the segmentation-level, which means that an output segmentation is considered correct if and only if it fully matches the human segmentation.


We introduce a novel approach to analyzing entities that maps contextualized embeddings to interpretable dimensions.

we propose using pre-trained embeddings to extract affect information about target entities.


We use the Standford Natural Language Inference (SNLI) dataset (Bowman et al., 2015) to pretrain the parameters of entailment-based attention model.

we train a model for Recognizing Textual Entailment (RTE) as follow: whereȳ is the entailment class label, i.e., entails, contradicts, or neutral, h RT E has the same form as Eq.


The focus of our work is on identifying the most important stylistic parameters to predict socioeconomic group.

we show the effectiveness of morpho-syntactic features as stylistic predictors of socioeconomic group, in contrast to lexical features, which are good predictors of topic.


While successful, a system assigning high weights to features strongly associated with cheap/expensive food, will limit the scope of our conclusions on stylistic variation.

the features identified are more related to the restaurants themselves than to the writing characteristics of their authors.


We experiment with this idea under the assumption that authors belonging to different groups will show a difference in the formality of their writing, and that a bleached representation is well suited for capturing such a difference.

we hypothesise that some of our target classes are typified by certain writing styles which differ in their formality and the extent to which they approach informal speech.


In a ranking task, we need to rank comments for each article.

when we label 1,000 comments, there are many choices, e.g., 200 articles with 5 comments or 10 articles with 100 comments.


Our conditions consist of a precondition for maintaining decency and relevance and four main conditions for representing typical cases of being constructive.

a constructive comment is defined as one satisfying the precondition and at least one of the main condition in Table 1.


The key concept of this model is similar to that of RankSVM, i.e., solving the ranking problem as a classification problem of whether a comment is more constructive than another one.

the model is constructed to predict the ranking score of a comment and trained so that, given two comments, the magnitude relation of the predicted scores corresponds to that of the true constructiveness scores, via cross entropy loss.


For example, 81% of the adults in the USA spend on average two hours on social media and collectively publish 170 million tweets 1 every day on their feelings and observations (Wu et al., 2018).

social media users can be considered as "social sensors" to perceive environmental changes and real-world events.


The inputs are the adjacency matrix of all the network IDs and a feature matrix of the Bio's bagof-words.

we keep the input feature vectors corresponding to the main IDs as null (all zeros), since their Bios were discarded.


We use the simulated data generated by SimDial (Zhao and Eskénazi, 2018).

we use three domains: restaurant, weather, and bus information search, as source data and test the meta-learned parameter initialization against the target domain, movie information search.


We propose to apply meta-learning algorithm on top of the sequicity model to achieve dialog domain adaptation.

we chose the recently introduced algorithm, model-agnostic meta-learning(MAML) (Finn et al., 2017), because it generalizes across different models.


We find that our approach substantially improves story generation.

we show that generating the action sequence first makes the model less prone to generating generic events, leading to a much greater diversity of verbs.


Recent progress in sequence-to-sequence (seq2seq) text generation models has delivered both fluent and content rich outputs by explicitly conducting content selection and ordering (Gehrmann et al., 2018;Wiseman et al., 2018), which is a promising avenue for enabling end-to-end counter-argument construction (Le et al., 2018).

our prior work (Hua and Wang, 2018) leverages passages retrieved from Wikipedia to improve the quality of generated arguments, yet Wikipedia itself has the limitation of containing mostly facts.


We can find that Seq2Seq+aug+iter improves 2.59% accuracy on exact match over Slug2Slug.

slug2slug augments original training data by only deleting additional slot values not realized in the utterance with an aligner, which is not capable of the situation where the given utterance contains incorrect or additional slot values and leads more con-Utterance: Located in riverside, near Caf sicilia, is the Phoenix, a French pub that is family-friendly and has average prices and an average rating.


While giving accurate answers, our framework further provides explainable reasoning paths.

our implementation 1 based on BERT and graph neural network (GNN) efficiently handles millions of documents for multi-hop reasoning questions in the HotpotQA fullwiki dataset, achieving a winning joint F 1 score of 34.9 on the leaderboard, compared to 23.6 of the best competitor.


Our proposed HDE graph has the following advantages: • Instead of graphs with single type of nodes (Song et al., 2018;De Cao et al., 2018), the HDE graph contains different types of queryaware nodes representing different granularity levels of information.

instead of only entity nodes as in (Song et al., 2018;De Cao et al., 2018), we include nodes corresponding to candidates, documents and entities.


We next introduce an Answer Proposer that performs query-context reasoning at the word-level on the retrieved chain and predicts an answer.

it encodes the leaf document of the reasoning chain while attending to its ancestral documents, and outputs ancestor-aware word representations for this leaf document, which are compared to the query to propose a candidate answer.