A recent trend is to produce personalized responses by conditioning the generation on a persona profile to make the response more consistent through the dialogue (Li et al., 2016b).

personaChat (Zhang et al., 2018b;Kulikov et al., 2018) dataset was created, and then extended in ConvAI 2 challenge (Dinan et al., 2019), to show that by adding persona information as input to the model, the produced responses elicit more consistent personas.


To enrich word embedding with concept representations, we propose a dynamic context-aware affective graph attention mechanism to compute the concept representation for each token.

the concept representation c(t) ∈ R d for token t is computed as where c k ∈ R d denotes the concept embedding of c k and α k denotes its attention weight.


We propose a hierarchical self-attention mechanism to exploit the structural representation of conversations and learn a vector representation for the contextual utterances X i j−1 , ..., X i j−M .

the hierarchical self-attention follows two steps: 1) each utterance representation is computed using an utterance-level self-attention layer, and 2) a context representation is computed from M learned utterance representations using a context-level self-attention layer.


The state-of-the-art DialogueRNN model performs the best overall among all baselines.

dialogueRNN performs better than our model on IEMOCAP, which may be attributed to its detailed speaker information for modelling the emotion dynamics in each speaker as the conversation flows.


In recent years, deep neural network models have been widely used for text classification.

the attention-based recurrent neural networks (RNNs) (Schuster and Paliwal, 2002;Yang et al., 2016) prevail in text classification.


In this section, we introduce the pipeline to extract high quality justifications from raw user reviews.

our goal here is to identify review segments that can be used as justifications and build a personalized justification dataset upon them.


The hidden state at time-step t is updated via the GRU unit based on the previous hidden state and the input word.

: where e u ls and e i ls are the last hidden states of the user and item encoder outputê u andê i .


Unlike their approach, we adopt a higher rate to mask fine-grained aspects since they are more important in justifications.

if we encounter a fine-grained aspect, we will replace it with a [MASK] token 30% of the time; while for other words, we will replace them with a [MASK] token 15% of the time.


However, these models are trained based on plain textual data which are in a much simpler form than our multi-turn dialogue structure.

customer service dialogue has unique characteristics.


In contrast, we consider complex multi-turn interactions within dialogues and explore context clue matching between customer utterances and server utterances for multi-tasking in the SSA task.

we improve the basic MIL models by proposing a position-guided automatic context clue matching mechanism (CCMM) to conduct customer utterance and context clues alignments for better sentiment classification to boost satisfaction classification.


In this work, we adopt a standard LSTM model (Hochreiter and Schmidhuber, 1997) to learn a fixed-size utterance representation v u i ∈ R k , where k is the size of LSTM hidden state.

we first convert the words in each utterance u i to the corresponding word embeddings E u i ∈ R d×|u i | which are then fed into a LSTM for obtaining the last hidden state as the v u i , where d is the dimensionality of word embeddings.


The hyper-parameters are tuned on the validation set.

the initial learning rate is fixed as 0.1, the dropout rate is 0.2, the batch size is 32 and the number of epochs is 20.


We make use of the same graph neural network encoder structure to represent dependency syntax information for both the baseline and our model.

a graph recurrent neural network architecture (Beck et al., 2018; is used, which has been shown effective in encoding graph structures , giving competitive results with alternative graph networks such as graph convolutional neural networks Bastings et al., 2017).


To this end, we slightly adjust the GRN encoding process without introducing additional parameters.

we enhance the original message sum function (Equations 5 and 6) by applying the edge probabilities in calculating weighted message sums: where ✏ (instead of a triple) is used to represent an edge for simplicity, and p ✏ is the parser probability for edge ✏.


However, using a much larger K (such as 100) is not practical due to dramatically increased computation time.

the running time of KBESTEISNER with K = 10 is already much longer than that of EDGEWISE.


Both EDGEWISEPS and KBESTEIS-NERPS are still more effective than DEPTREE.

kBESTEISNERPS significantly improves TEXTONLY with 5k dependency trees, and EDGEWISEPS is helpful even with 1k dependency trees.


To bridge the gap, we propose Relational Siamese Networks (RSNs) to learn transferable relational knowledge from supervised data for OpenRE.

rSNs learn relational similarity metrics from labeled data of pre-defined relations, and then transfer the metrics to measure the similarity of unlabeled sentences for open relation clustering.


For labeled data, we have in which D KL indicates the Kullback- indicates a new distance estimation with perturbations t 1 and t 2 on both input instances respectively.

t 1 and t 2 are worst-case perturbations that maximize the KL divergence between with a limited length.


Empirically, we approximate the perturbations the same as the original paper (Miyato et al., 2016).

we first add a random noise to the input, and calculate the gradient of the KL-divergence between the outputs of the original input and the noisy input.


Here, we propose Distantly-supervised RSN, which can learn from both distantly-supervised data and unsupervised data for relational knowledge transfer.

we use the following loss function: which treats auto-labeled data as labeled data but removes the virtual adversarial loss on the autolabeled data.


Experimental results show that the sample ratio decides RSN's tendency to predict larger or smaller clusters.

it controls the granularity of the predicted relation types.


At the same time, major hyperparameters are selected with grid search according to the model performance on a validation set.

the validation set contains 10,000 randomly chosen sentence pairs from the unlabeled set (i.e.


RW-HAC and VAE both rely on external linguistic tools to extract rich features from plain texts.

we first align entities to Wikidata and get their KB types.


First, we report the result of supervised RSN with different clustering methods.

sN represents the original RsN structure, HAC and L indicate HAC and Louvain clustering introduced in sec.


Compared with RW-HAC, SN-HAC achieves better clustering results because of its supervised relational representation and similarity metric.

unsupervised baselines mainly use sparse one-hot features.


This is because aligned entities tend to have some relations in common, and similar relations should have similar categories of head and tail entities.

we first pre-train the entity alignment model (Section 4.2) until its entity alignment performance has converged to be stable.


Like entity alignment, we also observe that joining entity and relation alignments improves relation alignment, as evidenced by the better performance of HGCN-JR and GCN-JR over HGCN-PR and GCN-PR, respectively.

joint modeling produces better entity representations, which in turn provide better relation approximations.


The statistics of the two datasets are shown in Table 1.

the average numbers of words in the textual descriptions have a large variation between two datasets.


trait is removed in the baseline "Ours -trait" and TC-VAE is removed in the baseline "Ours -TCVAE" respectively.


To take best advantages of all WL data, we then propose a unified neural framework that solves name tagging from two perspectives: sequence labeling and classification for two types of data, respectively.

the classification module focuses on noisy data to efficiently pretrain the tag classifier by capturing textual context semantics.


To make notations consistent, we useỸ = Y {UN,B-NT,I-NT} to denote the label set of WL data, where UN indicates that the word is unlabeled, and NT denote only the type is unlabeled.

the word with UN may be any one of the label in Y, and the word with NT may be any type.


To deal with the issue of limited annotations, we construct WL data D = {(X,Ỹ )} based on Wikipedia anchors and taxonomy, whereỸ An anchor m, e ∈ A links a mention m to an entity e ∈ E, where m contains one or several consecutive words of length |m|.

we define A(X) as the set of anchors in X.


To empower the linking model with associative ability, aside from previously linked entities, we also incorporate entities that are closely associated with entities in S t .

for eachê i ∈ S t , we collect its neighborhood N (ê i ) consisting of Wikipedia entities that have inlinks pointing toê i .


In this section, we explore different learning strategies for the linking model.

we present a Supervised Learning model, where the model is given all gold entities for training, and a Reinforcement Learning model, where the model explores possible linking results by itself in a longterm planning task.


After the model training, the generator G learns the mapping function between the document-event distribution and the document-level event-related word distributions (entity, location, keyword and date).

with an event distribution Algorithm 1 Training procedure for AEM  10: 11: 12: end for 14: 15: end for Sample m noise end while ⃗ θ ′ as input, G could generate the corresponding entity distribution, location distribution, keyword distribution and date distribution.


To address the above two challenges, we propose a novel bootstrapping method combining the Monte Carlo Tree Search (MCTS) algorithm with a deep similarity network, aiming to effectively evaluate the delayed feedback of patterns and adaptively score entities given sparse supervision signals.

our method tackles the delayed feedback problem by enhancing the traditional bootstrapping method using the MCTS algorithm, which effectively estimates each pattern's delayed feedback via efficient multi-step lookahead search.


To address the above two problems, this paper enhances the bootstrapping system using the Monte Carlo Tree Search (MCTS) algorithm for lookahead search, combined with a pattern mover similarity network (PMSN) for better entity scoring.

we use MCTS for the efficient lookahead search by pruning bad patterns.


Besides, for each (s, p) pair, we store a cumulative reward Q(s, p) and a visit count N (s, p) during tree search for the subsequent reward function defined in Section 3.3.

each MCTS simulation contains four stages, as demonstrated in Figure 2 Selection.


We describe this model in detail in Section 4.

for a pattern p linked from a tree node s, the PMSN is used as the prior policy network to compute the prior probability by: where E is the set of known entities included in node s, and SIM(p, E) is the similarity of the pattern p to the entity set E using the PMSN.


The pattern mover similarity network (PMSN) is a unified model for adaptively scoring the similarity of entities or patterns to seed entities.

the pattern mover similarity network contains two components: 1) the adaptive distributional pattern embeddings (ADPE) that adaptively represent patterns, entities, and entity sets in a unified way; 2) the pattern mover similarity (PMS) measurement that estimates the similarity of two ADPEs.


As a basic step of our PMSN model, we first embed a context pattern of an entity as a single embedding vector.

we use the average word embeddings, from the pre-trained GloVe (Pennington et al., 2014) embeddings, of a pattern surface text as the pattern embeddings.


In addition, we estimate the basic DPE for a single pattern and a whole entity set.

a single pattern can be regarded as a special "entity", whose context pattern is only the pattern itself.


4) SetExpan (Shen et al., 2017): corpus-based entity set expansion method, which adaptively selects context features and unsupervisedly ensembles them to score entities.

we compare baselines (1)-(3) and our method on Google Web 1T; we compare baseline (4) and our method on APR and Wiki 1 .


In this paper, we propose a deep similarity network-based model combined with the MCTS algorithm to bootstrap Entity Set Expansion.

we leverage the Monte Carlo Tree Search (MCTS) algorithm to efficiently estimate the delayed feedback of each pattern in the bootstrapping; we propose a Pattern Mover Similarity Network (PMSN) to uniformly embed entities and patterns using a distribution on context pattern embeddings; we combine the MCTS and the PMSN to adaptively learn a better embedding for evaluating both patterns and entities.


Third, the formerly-detected concepts, attribute names, and phrases are absolutely useful for tagging the slots of subjects and objects.

the tags "B/I-c" and "B/I-a" in the CAP sequence are strongly associated with the target tags "B/I-XYc" and "B/I-XYa", respectively.


Despite their capability of learning from large scale data, we show in this paper that these DS relation extraction strategies fail to adequately model the characteristic of the noise in the data.

most of the works fail to recognize that the labels can be noisy in a bag and directly use bag labels as training targets.


This aforementioned observation has inspired us to study a more realistic setting for DS relation extraction.

we treat the bag labels as the noisy observations of the ground-truth labels.


Let set R contain all relation labels of interest.

each label r in R corresponds to a candidate relation in which any considered pair (e, e ) of entities may participate.


Specifically, each label r in R corresponds to a candidate relation in which any considered pair (e, e ) of entities may participate.

since R cannot contain all relations that are implied in a corpus, we include in R an additional relation label "NA", which refers to any relation that cannot be regarded as the other relations in R. Any subset of R will be written as a {0, 1}valued vector of length |R|, for example, z, where each element of the vector z corresponds to a label r ∈ R. if and only if label r is contained in the subset, its corresponding element z[r] of z equals 1.


A key modelling assumption in nEM is that random variables X, Y and Z form a Markov chain X → Y → Z.

the dependency of noisy labels Z on the bag X is modelled as The conditional distribution p Z|Y is modelled as .


For the j th word in a sentence, Word-Position Embedding generates a vector representation w j as concatenated three components w w j , w p1 j , w p2 j .

w w j is the word embedding of the word and w p1 j and w p2 j are two position embeddings.


For this reason, we annotate a subset of the original test data for evaluation purpose.

we annotate a bag x b by its all correct labels, and if no such labels exist, we label the bag NA.


For each bag x b in the training set, we generate a noisy label vectorz b from the observed ground-truth label vector y b .

,z b is generated by flipping each element y b [r] from 0 to 1 or 1 to 0 with a probability p f .


Given the necessity of applying EE on the financial domain, the specific characteristics of financial documents as well as those within many other business fields, however, raise two critical challenges to EE, particularly arguments-scattering and multi-event.

the first challenge indicates that arguments of one event record may scatter across multiple sentences of the document, while the other one reflects that a document is likely to contain multiple such event records.


Therefore, we evaluate DEE by directly comparing the predicted event table with the groundtruth one for each event type.

for each document and each event type, we pick one predicted record and one most similar ground-truth record (at least one of them is non-empty) from associated event tables without replacement to calculate event-role-specific true positive, false positive and false negative statistics until no record left.


For better comparison, we use the same data split as previous works (Chen and Ji, 2009;Zeng et al., 2016;Feng et al., 2018;Lin et al., 2018).

the ACE2005 (LDC2006T06) contains 697 articles, with 569 articles for training, 64 for validating and the rest 64 for testing.


Hard parameter sharing greatly reduces the risk of overfitting (Baxter, 1997) and increases the correlation of our boundary detection module and categorical label prediction module.

the hidden state of bidirectional LSTM can be expressed as following: where x t i is the token representation which is mentioned in section 3.1.


1 It takes three-time steps to generate one triplet.

in time step t (t = 1, 2, 3, ..., T ), if t%3 = 1, we predict the relation.


the word embedding dimension is 100, the position embedding dimension is 5, we use 128 filters and the filter size is 3.


In this paper, we propose a novel model that employs a collaborative curriculum learning framework to reduce the effects of mislabelled data.

we firstly propose an internal self-attention mechanism between the convolution operations in convolutional neural networks (CNNs) to learn a better sentence representation from the noisy inputs.


After learning representations of all sentences in an entity pair bag, we then use two approaches to form the entity pair bag representation, as illustrated in Figure 3(b), i.e., we employ a mechanism of averaging all sentences using attention scores (NetAtt) (Lin et al., 2016), and a maximum probability sentence selection method (NetMax) (Zeng et al., 2015), respectively, to learn the bag representation.


All the parameters fine-tuned in our experiments are shown in Table 1.

the best performance of [CNN+ATT/ONE+SelfAtt] with training parameters of learning rate:0.4 and CNN filters:230; the best performance of [PCNN+ATT/ONE+SelfAtt] with training parameters of learning rate:0.4 and CNN filters:128; and the best performance of [PCNN+ATT/ONE+SelfAtt+CCL] with training parameters of learning rate:0.1 and CNN filters:256.


When using ELMo, we first encode the sentence: , and we use the average of contextualized hidden states corresponding to the entity span as the contextualized entity representation.

f ELMo (w 1: With BERT, following Onoe and Durrett (2019), we concatenate the full sentence with the entity mention, starting with [CLS] and separating the two by [SEP], i.e., [CLS] We encode the full sequence using BERT and use the output from the [CLS] token as the entity mention representation.


We define a hyperlink-based training objective and add it to ELMo.

we use contextualized entity representations to decode the hyperlinked Wikipedia description, and also use the descriptive entity representations to decode the linked context.


i.e., event extraction and temporal relation classification, and they also assume that gold events are given when training the relation classifier (Verhagen et al., 2007(Verhagen et al., , 2010UzZaman et al., 2013;Ning et al., 2017;Meng and Rumshisky, 2018).

they built end-toend systems that extract events first and then predict temporal relations between them (Fig.


3 Therefore, we develop a two-state learning approach which first trains a pipeline version of the joint model without feedback from global constraints.

the local neural scoring functions are optimized with cross-entropy loss using gold events and relation candidates that are constructed directly from the outputs of the event model.


Experiments and comparative studies on two benchmark datasets show that the proposed model is effective for end-to-end event temporal relation extraction.

we improve the performances of previously published systems by 10% and 6.8% on the TB-Dense and MATRES datasets, respectively.


Experimental results on five public datasets from different domains show that combining the base models with HiLAP yields an average performance improvement of 33.4% in Macro-F1 over corresponding flat classifiers and outperforms state-ofthe-art HTC methods by a large margin.

ablation study shows that HiLAP is especially beneficial to those unpopular labels at the bottom levels.


In contrast, we learn a policy that (1) makes consistent, inter-dependent predictions by traversing the label hierarchy and maintaining state representation; (2) measures the holistic quality of label assignment via reinforcement learning.

the policy P puts x i at the root label in the beginning.


Instead of waiting until the end of the label assignment process and comparing the predicted labels with the gold labels, we use reward shaping (Mao et al., 2018), i.e., giving intermediate rewards at each time step, to accelerate the learning process.

we set the reward r of x i at time step t to be the difference of Example-based F1 scores between current and the last time step: .


At each time step t, the object goes down one level on the hierarchy and the labels under the same parent are discriminated locally.

the local per-parent label probabil- where σ denotes the sigmoid function, and C t ∈ R |C(lt)|×C denotes the candidate embeddings of HiLAP-SL, i.e., an embedding matrix consisting of the children of current label l t , rather than all the labels where x i has been placed.


Another difference is that in HiLAP the actions are sampled and thus might place the objects to incorrect labels, while in HiLAP-SL only the ground-truth positions are traversed during training.

if there are K(≥ 1) groundtruth labels at the same level, the object embedding e d would be copied K times and losses on the K different paths would be measured independently (see Fig.


We further add a flat component to HiLAP as a regularization of the base model.

the flat component is a feed-forward network that projects the object embedding e d to a label probability distribution p Flat of all the labels on the hierarchy: The combination of the base model and the flat component functions the same as a flat model and ensures that the object representation e d has the capability of flat classification.


Different from most of existing global HTC methods that rely on pre-specified features (Gopal and Yang, 2013) as input or build on specific models (Cai and Hofmann, 2004;Vens et al., 2008;Silla Jr and Freitas, 2009), our framework is trained in an end-to-end manner by leveraging a differentiable feature representation learning model as the base model.

we use TextCNN (Kim, 2014), HAN (Yang et al., 2016), bow-CNN (Johnson and Zhang, 2014) on the three textual datasets, and a feed-forward network on the two nontextual datasets.


A major line of work for HTC is Support Vector Machines (SVM) and its hierarchical variants.

sVM performs standard multi-label classification using one-vs-the-rest (OvR) strategy.


As one can see, HiLAP outperforms all the baselines on both datasets by a large margin.

we observe significant improvement on Macro-F1 over the best baseline (47.9% and 53.9%, respectively), which shows that our method is especially better at classifying sparse labels than previous approaches.


Many previous studies (Liu et al., 2005;Sun and Lim, 2001) train a set of local classifiers and make predictions in a top-down manner.

bi and Kwok (2015) develop bayesoptimal predictions that minimize the global risks but their model is still locally trained.


Our HCapsNet achieves the best results on 5 out of 6 datasets, which verifies the effectiveness of our model.

hCapsNet outperforms vanilla capsule network Capsule-B ] by a remarkable margin, which only utilizes the dynamic routing mechanism without hyperplane projecting.


We analyze the attended primary capsule representation for the polysemic words in brackets.

we report the output vectors of the projected primary capsule, which is mostly attended by the routing mechanism.


As mentioned above, the multi-label document may be tagged by more than one label, and each document should have the most relative contexts with its corresponding labels.

each document may contain multiple components, and the words in one document make different contributions to each label.


Obviously, the same dimension belonging to different classes has different importance when we calculate the euclidean distance.

some feature dimensions are more discriminative for distinguishing specific class in the feature level space, and other features are confusing and useless at the same time.


The order seems to depend on both the task and the model: even within SST, the similarity between built-in and LIME can rank as third, second, or first.

post-hoc methods generate more similar important features when we compare different models, but that is not the case when we fix the model.


We believe that the reason is that type-token ratio is strongly negatively correlated with length (the Spearman correlation for Yelp, SST and deception dataset is -0.92, -0.59 and -0.84 respectively).

type-to-token ratio becomes redundant to length and fails to capture text complexity beyond length.


This paper proposes a novel Encoder1-Encoder2 architecture, where global information is incorporated into the procedure of local feature extraction from scratch.

en-coder1 serves as a global information provider, while encoder2 performs as a local feature extractor and is directly fed into the classifier.


The identical input sequence will be encoded twice by two encoders respectively, but only the output of Encoder2 is used directly for the classifier.

the Encoder1 serves as a pioneer for providing global information, while the Encoder2 focuses on extracting better local features by incorporating the former into the local extraction procedure.


According to different local contexts, the guidances from En-coder1 can be be more targeted.


It makes sense since the classifier could use features from CNN and RNN simultaneously and different model structures complement each other for classification.

our architecture performs best for both cases.


In this section, we generalize our architecture to semi-supervised domain which is equipped with pre-trained word embeddings and then compare with popular pre-trained based models.

we use GloVe vectors 2 with 300 dimensions to initialize the word embeddings in our architecture.


Instead of traditionally exploiting deeper and complicated operations in upper neural layers, our work innovatively provides another lightweight way for improving the ability of neural model.

we propose a novel architecture named Encoder1-Encoder2 with two Interaction Modes for their interacting.


In this section, we study the aggregation function φ with a routing scheme, which has achieved good results in other NLP tasks (Zhao et al., , 2019.

we introduce a nonparametric clustering with Kernel Density Estimation (KDE) for routing since KDE bridges a family of kernel functions with underlying empirical distributions, which often leads to computational efficiency , defined as: where d(•) is a distance function, γ ij denotes the underlying closeness between the aggregated vector v j and vector z i in the i-th layer, and k is a kernel function.


Reinforce-select (RS) (Ling and Rush, 2017;Chen and Bansal, 2018) utilizes reinforcement learning to approximate the marginal likelihood.

it is trained to maximize a lower bound of the likelihood by applying the Jensen inequalily: The gradient to γ is approximated with Monte-Carlo sampling by applying the REINFORCE algorithm (Williams, 1992;Glynn, 1990).


In our VRS model, we can easily adjust the degree of controllability by limiting an upper bound of the conditional mutual information (CMI) I(β, Y |X) (Zhao et al., 2018).

we can change our objective into: λ is a fixed lagrangian multiplier.


We then extract keyphrases from the retrieved passages based on topic signature words (Lin and Hovy, 2000)   words, together with their related terms from WordNet (Miller, 1994), are used to determine whether a phrase in the passage is a keyphrase.

a keyphrase is (1) a noun phrase or verb phrase that is shorter than 10 tokens; (2) contains at least one content word; (3) has a topic signature or a Wikipedia title.


For the multilingual experiments, we use the dependency-based annotations, given that there is labeled data available in different languages on this format.

we use the English and German portions of CoNLL-09 and the automatically annotated French SRL corpus of van der Plas et al.


In this paper, we focus on analyzing the relationship between the syntactic dependency information and the SRL performance.

we investigate the following four aspects: 1) Quality of the syntactic information: whether the performance of the syntactic parser output affects the SRL performance; 2) Representation of the syntactic information: how to represent the syntactic dependencies to better preserve the original structural information; 3) Incorporation of the syntactic information: at which layer of the SRL model and how to incorporate the syntactic information; and 4) the Relationship with other external resources: when we append other external resources into the SRL model, whether their contributions are orthogonal to the syntactic dependencies.


Similar to prior work on sentence embeddings (Kiros et al., 2015;Hill et al., 2016), we use an encoded sentence representation to predict its surrounding sentences.

we predict the immediately preceding and succeeding sentences.


• The NL or SPP loss alone has complex effects across tasks in DiscoEval, but when they are combined, the model achieves the best performance, outperforming our baseline by 0.6% on average.

it yields 39.3% accuracy on PDTB-I, outperforming Skip-thought by 0.6%.


However, some tasks are unsolvable with this simple architecture.

the DC tasks have low accuracies with all models unless a hidden layer is used.


We also introduce a simple yet effective strategy to incorporate visual information into any neural network beyond weights from pretrained networks.

we consider the tags identified in an image as an additional textual input, and leverage pretrained word embeddings as usually done with regular text.


The first one-weights from InceptionNet-is common in previous work, but the second one is novel.

the second component considers the objects and events identified in an image as an additional textual input.


devices (e.g., watch, guitar, cell phone) yield more possessions (alienable and control labels) and most of the possessions are alienable.

people tend to tweet about devices they own, and rarely about devices they only have control over.


We have presented a corpus of 5,000 tweets and experimental results to extract possession relations.

we work with text and images in order to reveal the possesees of the author of a tweet.


We show that in addition to using past dialogue, these models are able to effectively use the state of the underlying world to condition their predictions.

we show that grounding on the details of the local environment, including location descriptions, and the objects (and their affordances) and characters (and their previous actions) present within it allows better predictions of agent behavior and dialogue.


This is done by evaluating state-of-theart models on our task and evaluating the effects of providing additional grounding.

we adapt the BERT contextual language model (Devlin et al., 2018) to the task of dialogue in two ways: as a bi-ranker, which is fast and practical as a retrieval model, and as a cross-ranker which is slower at inference time but allows more feature cross-correlation between context and response.


We observe that BERT-based models exhibit good transfer ability relative to other models, but the gap between their performance and human performance increases from the seen test set to the unseen one.

there is a 21 point gap on the unseen dialogue test set compared to an 11 point gap on the seen test set, making this a significant challenge for future methods.


ANNA is a simulation of human assistants who do not necessarily know themselves how to optimally accomplish the agent's goal: they are only familiar with scenes along certain paths in the environment, and thus give advice to help the agent make partial progress.

the assistance from ANNA is modeled by a set of language-assisted routes R = {r 1 , r 2 , .


We denote it g(s) for simplicity, where g is a multi-layer perceptron with input s = F t (S; θ t ) and parameters θ i Cluster information (C g ) The cluster information leverages the fact that two sentences describe, or not, the same underlying reality.

the goal is to measure if two sentences are visually equivalent (assumption (1) in Section 3.1) without considering the content of related images.


In this paper we pursue an alternative approach, which is to embrace different perspectives of the EL task, proposing a fine-grained categorization of different types of EL mentions and links, and then reannotating three existing datasets with respect to these categories, allowing us to compare the performance of EL tools within the different categories.

our contributions are as follows (indicating also the relevant section): § 2 We design and present the results of a questionnaire addressed to authors of EL papers intended to understand the consensus (or lack thereof) regarding the goals of the EL task.


Here we define an instance of µ G * based on the questionnaire results shown in Figure 2.

we consider annotations with Proper Forms, Noun Phrases, No Overlap and Direct Reference -which consistently score greater than 0.9 in Figure 2 -to have a membership degree of 1; we consider these to be strict annotations.


In addition to the Open IE results, we extract part-of-speech (pos) tags, syntactic role and dependency parse tree.

we consider dependency parse tree based on the one-hop neighbors (i.e., parent and children) of a word in the dependency tree.


In this paper, we propose a new method for cross-lingual ED, demonstrating a minimal dependency on parallel resources.

to construct a lexical mapping between different languages, we devise a context-dependent translation method; to treat the word order difference problem, we propose a shared syntactic order event detector for multilingual cotraining.


In the monolingual embedding layer, each word is assigned to a distributed vector as its representation.

we first train English/Chinese word embeddings on the corresponding Wikipedia dumps via Skip-gram model (Mikolov et al., 2013) with a dimensional size d = 300.


Let X and Y be the English and Chinese embedding spaces.

in order to achieve multilingual alignment, we learn a mapping W ∈ R d×d from X to Y via a seed dictionary with a size of m, by optimizing: where  Next, we retrieve translation candidates for each token w i in s. we first project w i into the aligned embedding space (i.e., by applying W on x i ), and then we explore its neighborhood to find the nearest Chinese words as its translation candidates.


For each token w i , after obtaining its translation candidates list T (w i ) and contextual representation c i , we impose a selective attention mechanism to automatically weigh each candidate.

the weight of the jth candidate T is computed as: where m j measures the semantic relatedness of c i and T , which is computed by: where [;] indicates the concatenation operations; y denotes the Chinese word embedding of ; W r ∈ R d×1 and b r ∈ R are parameter matrix and bias respectively.


This allows our model to train with the translated data t and the other labeled data in Chinese indiscriminately.

for each token w i , our model computes a graph convolution feature vector based on its immediate neighbors in the syntactic graph.


Our model incorporates a logistic regression classifier to predict w i 's event type.

we compute a prediction vector for w i by taking f i as the input: where W o ∈ R d×c and b o ∈ R c are parameters, and c is the total number of event types (i.e., 34 in this study).


From the results, our approach demonstrates a definite advantage over monolingual ED models in the annotation-poor scenario.

when there is no Chinese training document available (i.e., in the unsupervised cross-lingual transfer scenario), our model achieves a performance of 27.0% on F1 in ACE, and 28.7% on F1 in TAC KBP, while supervised ED methods completely fail.


In this paper, we propose a new cross-lingual approach for event detection, which demonstrates a minimal dependency on parallel resources.

we propose a context-dependent lexical mapping method to obtain content-dependent translation, and we devise a shared syntactic order event detector to explore the syntactic similarity for multilingual co-training.


We follow ACE's reasonable reader rule, which states that a fact should only be annotated when there is no reasonable interpretation of the sentence in which the fact does not hold.

annotators are asked to only annotate facts that are either explicitly stated in the sentence or inferred with absolute certainty from the sentence alone (i.e., without using external world knowledge).


Entity Based Attention Second, we explore a more fine-grained way of using the GPT model outputs.

we use bilinear attention between g e and the transformer output for the process tokens X to get a contextual representation c e,t for a given entity.


In this paper, we propose a novel cross-lingual pre-training method for unsupervised machine translation by incorporating explicit cross-lingual training signals.

we first calculate cross-lingual n-gram embeddings and infer an n-gram translation table from them.


Then, we pre-train cross-lingual BERT language models with our proposed Cross-lingual Masked Language Model (CMLM) objective.

we randomly choose n-grams in the monolingual sentences and predict corresponding translation candidates in the n-gram translation table inferred in the first step.


Based on that, we calculate the similarity score of n-grams x and y in two languages respectively with the marginal-based scoring method (Conneau et al., 2017;Artetxe and Schwenk, 2018).

given the crosslingual embeddings of x and y, denoted as e x and e y , the similarity score is calculated as: cos(e y , e z ) 2n ) (1) where margin(a, b) is a marginal scoring function and NN n (x) denotes x's k-nearest neighbors in the other language.


Our proposed method breaks the limitation that masked n-grams and predicted ones have to be the same length during BERT training.

according to IBM Model 2, given a source n-gram x l 1 and a target one y m 1 , where l and m are the numbers of tokens in the source and target n-grams respectively, the translation probability from x l 1 to y m 1 is calculated as: where = p(m|x l 1 ), i.e.


Different from them, we step further and make another iteration with updated n-gram translation tables.

we translate the monolingual sentences with our latest translation model and run GIZA++ (Och and Ney, 2003) on the generated pseudo parallel data to extract updated n-gram translation pairs, which are used to tune the encoder as Section 3.3, together with the back-translation within a multi-task learn-ing framework.


Our implementation is based on the released code of XLM 1 (Paszke et al., 2017).

we use a Transformer architecture with 1024 hidden units, 8 heads, GELU activations (Hendrycks and Gimpel, 2016), with a dropout rate of 0.1.


We also conduct experiments on the cross-lingual classification task  using the cross-lingual language inference (XNLI) dataset .

we add a linear classification layer on top of the first hidden state of our pre-trained model and fine-tune its parameters on the English NLI dataset.


However, because the latent states in LaSyn don't directly depend on each other, we can exhaustively search over the latent states.

we can show that when y is fixed (observed), the {z n } N n=1 variables are d-separated (Bishop, 2006) i.e., are mutually independent.


In terms of complexity, LaSyn only adds a linear term (in POS tag size to the decoding complexity.

its decoding complexity is where B is beam size, m is a constant proportional to the tag set size and N is output size.


Given the authentic bilingual corpus D b , Bayesian neural networks aim at finding the posterior distribution over model parameters P (θ y→x |D b ).

with a target sentence y in the monolingual corpus D m and its translationx, the translation probability is given by we are interested in calculating the variance of the distribution P (x|y, θ y→x ) that reflects our ignorance over model parameters, which is referred to as model uncertainty.


Likewise, remember from above, the lower level capsule will send its output to the higher level capsule with similar output.


Performance on long sentences A more detailed comparison between CAPSNMT and Transformer can be seen in Figure 4.

we test the BLEU scores on sentences longer than {0, 10, 20, 30, 40, 50}.


Instead of real-valued vectors, ComplEx gives complexvalued vectors to entities and relations.

for each entity e or relation r, let w e ∈ C k or w r ∈ C k be the corresponding complex-valued vector, i.e.


We regard alignment prediction as a ranking problem.

suppose we need to align the entities in KG 1 to KG 2 , for each entity e i in KG 1 , we rank each entity e j in KG 2 based on the similarity between e i and e j .


After the first window, we keep the last 64 subwords from the previous window as context.

for a non-first window, only (up to) 64 new subwords are added for prediction.


To this end, we propose an iterative dual domain adaptation framework for NMT.

we first pretrain in-domain and out-of-domain NMT models using their own training corpora respectively, and then iteratively perform bidirectional translation knowledge transfer (from indomain to out-of-domain and then vice versa) based on knowledge distillation until the indomain NMT model convergences.


However, it is difficult to achieve this goal due to domain differences.

when the domain difference is significant, such conventional brute-force transfer may be unsuccessful, facing the similar issue as the domain adaptation for other tasks (Pan and Yang, 2010).


In this way, both in-domain and out-of-domain NMT models are expected to constantly reinforce each other, which is likely to achieve better NMT domain adaptation.

we employ a knowledge distillation (Hinton et al., 2015;Kim and Rush, 2016) based approach to transfer translation knowledge.


Furthermore, we extend our framework to the scenario of multiple out-of-domain corpora.

we introduce knowledge distillation into the domain adaptation for NMT and experimental results demonstrate its effectiveness, echoing its successful applications on many tasks, such as speech recognition (Hinton et al., 2015) and natural language processing (Kim and Rush, 2016;Tan et al., 2019).


To better describe our framework, we summarize the training procedure of our framework 848 in Algorithm 1.

we first individually train the initial in-domain and out-of-domain NMT models, respectively denoted by θ in and θ 0out , via minimizing the negative likelihood of their own training corpora D in and D out (Line 3): Then, we iteratively perform bidirectional translation knowledge transfer to update both in-domain and out-of-domain NMT models, until the maximal iteration number K is reached (Lines 5-14).


To deal with this issue, we introduce knowledge distillation (Kim and Rush, 2016) to conduct the translation knowledge transfer.

during the transfer process from θ in , we first initialize θ in not only to match the references of D in , but also to be consistent with probability outputs of the previous best in-domain NMT model θ * in , which is considered as the teacher model.


Finally, we used IWSLT tst2010, tst2011 and tst2012 as in-domain test sets.

in order to verify whether our framework can enable NMT models of two domains to benefit each other, we also tested the performance of outdomain NMT model on NIST 2003NIST , 2004NIST , 2005NIST , 2006 For the English-German translation task, our training corpora totally include one in-domain dataset: 200K TED Talk sentence pairs provided by IWSLT2015, and two out-of-domain datasets: 500K sentence pairs (News topic) extracted from WMT2014 corpus, and 500K sentence pairs (Medical topic) that are sampled from OPUS EMEA corpus 2 .


In this group of experiments, we investigated the impacts of out-of-domain corpus size on our proposed framework.

we inspected the results of our framework using different sizes of out-of-domain corpora: 50K, 200K and 1.12M, respectively Figure 5 shows the comparison results on the average BLEU scores of all IWSLT test sets.


this is the first animal to walk upright KD this is the first growing primate WDCD this is the first primate walking around IDDA-1 this is the first upright -walking primate IDDA-2 this is the first upright -walking primate IDDA-3 this is the first primates walking upright IDDA-4 this is the first upright primate IDDA-5 this is the first upright primate IDDA-6 this is the first upright primate IDDA-7 this is the first upright primate example provides the insight into the advantage of our proposed framework to some extent.

we observe that MFT, KD, WDCD are unable to correctly understand the meaning of "zhílì xíngzǒu de língzhǎnglèi dòngwù" and thus generate incorrect or incomplete translations, while IDDA successfully corrects these errors by gradually absorbing transferred translation knowledge.


In this paper, we propose a universal yet effective learning method for training multiple agents.

each agent learns advantages from the ensemble model when necessary.


In this work, we present multi-granularity self-attention (MG-SA): a neural network that combines multi-head selfattention and phrase modeling.

we train several attention heads to attend to phrases in either n-gram or syntactic formalism.


Accordingly, we spare several attention heads for modeling phrase patterns for SANs.

we use two representative types of phrases that are widely-used in SMT models: n-gram phrases (Koehn et al., 2003) to use surface of adjacent words, and syntactic phrases (Liu et al., 2006) induced from syntactic trees to represent well-formed structural information.


In the standard MH-SA, it jointly attends to information from different representation subspaces at different positions.

mH-SA transform input layer H = h 1 , ..., h n ∈ R n×d into h-th subspace with different linear projections: where are respectively the query, key, and value representations of the h-  Finally, the output states are concatenated to produce the final state.


We introduce various phrase granularity over the conventional word-level memory to generate phrase level memory.

we first transform the input layer H to a phrase level memory by function F h in certain attention head: where H g is the generated phrase level memory, h denotes the h-th head which is used to generate a certain granularity of phrase memory, and F h is a representation function with its own trainable parameters.


Since recurrence has proven important for capturing structure information (Tran et al., 2018;Hao et al., 2019b), we propose to introduce recurrence to interact phrases and further model latent structure among phrases.

we apply the recurrence function REC(•) on the output of phrase composition G x = (g 1 , ..., g M ) in order to model the latent structure of the phrase sequence.


Results on the model ratio show that self-attention sublayer has a (near) increasing effect (β > 1) that intensifies error signal, while feed-forward sublayer manifests a decreasing effect (β < 1).

though the encoder-decoder attention sublayer and the self-attention sublayer share the same attention formulation, the model ratio of the former is smaller.


Our model performs on par with current SOTA, the dynamic convolution model (DCNN) (Wu et al., 2019).

though DCNN achieves encouraging performance on newstest2014, it falls behind the baseline on other test sets.


These findings illustrate the benefit of supervised training for alignment, suggesting further annotation of gold-standard alignments is warranted, with an emphasis on alignment quality.

5 Figure 3 indicates that labelled alignment data is far more valuable than bitext, meaning that efforts should be focused on annotating existing bitext for alignment; FastAlign's performance (when trained on the entire 4 million-sentence training bitext and the alignment bitext) can be surpassed after pretraining the translation model on only 400K sentences of unlabelled bitext and the full alignment set (4871 sentences).


In this paper, we propose a unified and discourseaware ZP translation approach for neural MT models.

we jointly learn to predict and translate ZPs in an end-to-end manner, allowing both components to interact with each other.


Similarly, we expect that ZP prediction and translation could interact with each other: prediction offers more ZP information beyond 1-best result to translation and translation helps prediction resolve ambiguity.

we first cast ZP prediction as a sequence labeling task with a neural model, which is trained jointly with a standard neural machine translation (NMT) model in an end-to-end manner.


Integrating Discourse into ZP Prediction We directly feed the discourse-level context to the reconstructor to improve ZP prediction.

we combine the context vector and the reconstructor state: Here f c (•) is a function for combining reconstructor states and the context vector, which is a simple concatenation (CONCAT) in this work.


The proposed models consistently outperform other models in all cases, demonstrating the superiority of the joint learning of ZP prediction and translation.

the "Joint Model" (Row 4) significantly improves translation performance by +4.24 over baseline model.


, y T by maximizing the conditional probabilities p(y|x) = T t=1 p(y t |y <t , x).

an encoder first maps the source sentence into a sequence of encoded representations: where f e is the encoder's transformation function.


Our systematic contribution lies on annotated data, supervised model and discussions of its potential to help existing machine translation evaluation task.

: • We propose a task of adequacy-oriented error detection for machine translation evaluation and contribute a high-quality corpus TransErr annotated by professional annotators for machine translation.


In this work, we propose to address this gap by focusing on understanding the input-output behavior of NMT models.

we measure the word importance by attributing the NMT output to every input word through a gradient-based method.


In this paper, we focus on the second thread and try to open the black-box by exploiting the gradients in NMT generation, which aims to estimate the word importance better.

we employ the integrated gradients method (Sundararajan et al., 2017) to attribute the output to the input words with the integration of first-order derivatives.


Here the operation number denotes the number of perturbed words in a sentence.


Figure 2 shows that two black-box methods (i.e., Content, Frequency) perform only slightly better than the Random method.

the Frequency method demonstrates even worse performances under the Mask perturbation.


There exists another straightforward method, Erasure (Alvarez-Melis and Jaakkola, 2017; Arras et al., 2016;Zintgraf et al., 2017), which directly evaluates the word importance by measuring the translation performance degradation of each word.

it erases (i.e., Mask) one word from the input sentence each time and uses the BLEU score changes to denote the word importance (after normalization).


In this section, we analyze the linguistic characteristics of important words identified by the attribution-based approach.

we investigate several representative sets of linguistic properties, including POS tags, and fertility, and depth in a syntactic parse tree.


We study two methods for language clustering: (1) using prior knowledge, where we cluster languages according to language family, and (2) using language embedding, in which we represent each language by an embedding vector and cluster them in the embedding space.

we obtain the embedding vectors of all the languages by training a universal neural machine translation model.


Most existing dictionaries used for BLI evaluation do not account for the full spectrum of linguistic properties of language.

as we demonstrate in §2, they omit most morphological inflections of even common lexemes.


8 The sizes of dictionaries vary across different language pairs and so does the POS distribution.

while Slavic dictionaries are dominated by nouns and adjectives, verbs constitute the majority of pairs in Romance dictionaries.


(2018)-three of the best performing BLI models, which induce a shared crosslingual embedding space by learning an orthogonal transformation from one monolingual space to another (model descriptions are given in the supplementary material).

10 the last two employ a self-learning method in which they alternate between a mapping step and a word alignment (dictionary induction) step in an iterative manner.


tactic categories and more and less frequent lexemes, similar to that observed for related language pairs.

we observe that 2;IMP;PL;Vthe category difficult for Polish-Czech BLI is also among the most challenging for Polish-Spanish.


To further introduce structural feature transfer, we transform a source treebank into a code-mixed treebank by considering word alignments between a source sentence and its machine translation.

source words with highly confident target alignments are translated into target words by consulting the machine translation output, so that target word vectors can be directly used for learning target syntax.


We remove such words to produce dependency trees that are close in syntax to the target language.

we accumulate the probabilities of p(e i |f j ) for the source word e i who has no aligned target word: where we traverse all target words to sum their alignment probabilities with e i .


Figure 8 show the F-scores of labeled dependencies by different arc distances.

we treat the root type as one special case.


Given sentence s, the probability distribution of POS tags for word w is obtained using a one hidden-layer neural network applied to the corresponding encoder output h w 2 : For dependency parsing, words in a sentence are treated as nodes in a graph.

each word w in sentence s receives exactly one in-going edge (u, w, r) from head word u to its dependent w with relation r. We use a graph-based dependency parser similar to the one presented in , which treats dependency parsing as a classification task and its goal is to predict which in-going edge (u, w, r) connects to each word w. Mathematically, the probability of an edge is: where "score" is the scoring function: The bilinear classifier uses a weight matrix W r specific to the candidate relation and a weight matrix W shared across all dependency relations.


The auxiliary prediction modules take − → h t pri and ← − h t pri as input.

we add the following four auxiliary prediction modules to the model: The "forward" module makes predictions without seeing the right context of the current word.


In this section, we propose another approach to directly align the distributions of internal representations between different languages.

we leverage the mean and variance of internal distributions for alignment.


To demonstrate the strengths of the proposed cross-lingual contextualized representations (CLCRs) over cross-lingual word embeddings (CLWEs), we also report the results of using CLWEs for direct model transfer in Table 1.

we compare with the unsupervised method MUSE from Conneau et al.


Hence, character-level Chinese NER using lexicon features to better leverage word information has attracted research attention (Passos et al., 2014;Zhang and Yang, 2018).

zhang and Yang (2018) introduced a variant of a long short-term memory network (latticestructured LSTM) that encodes all potential words matching a sentence to exploit explicit word information, achieving state-of-the-art results.


In the gold or automatic segmentation settings, the char and bichar features boost the performance of word-level methods.

with the gold-standard segmentation, these methods are able to achieve competitive state-of-the-art results on the dataset .


The F1 score decreases 1.20% on average against the best results when the step number is less than 3.

the F1 score of the OntoNotes and Weibo datasets even suffered a serious reduction around 1.5% and 1.8%, respectively.


We also find that lexicons play an important role in character-level NER.

the performance of the OntoNotes, MSRA and Weibo datasets are seriously hurt by over 3.0% without lexicons.


The main idea is to directly capture semantic relationships among words, slots and intents, which is conducted simultaneously at each word position in a collaborative manner.

we alternately perform information exchange among the task-specific features referred from memories, local context representations and global sequential information via the well-designed block, named CM-block, which consists of three computational components: • Deliberate Attention: Obtaining slotspecific and intent-specific representations from memories in a collaborative manner.


At each word position, we take the hidden state h t as query, and obtain slot feature h slot t and intent feature h int t from both memories by the deliberate attention mechanism, which will be illustrated in the following.

for the slot feature h slot t , we firstly get a rough intent representation h int t by the wordaware attention with hidden state h t over the intent memory M int , and then obtain the final slot feature h slot t by the intent-aware attention over the slot memory M slot with the intent-enhanced representation [h t ; h int t ].


We extend the S-LSTM with slot-specific features H slot t and intentspecific features H slot t retrieved from memories.

at each input position t, we take the local window context ξ t , word embedding x t , slot feature h slot t and intent feature h int t as inputs to conduct combinatorial calculation simultaneously.


We conduct ablation experiments to investigate the impacts of various components in our CM-Net.

we remove one component among slot memory, intent memory, local calculation and global recurrence.


We find that without the constraint of the softmax operation in (6) the model prefers to link all words together and assign all words to the same constituency.

giving both s i,i+1 and s i,i−1 large values, so the attention head freely attends to any position without restriction of constituent prior, which is the same as the original Transformer.


As mentioned in Section 3, constituents in the lower layer merge into larger one in the higher layer.

once two words belong to the same constituent in the lower layer, they would still belong to the same constituent in the higher layer.


However, current state-ofthe-art SRL systems use powerful sentence encoders (e.g., layers of LSTMs He et al., 2017) or multi-head self-attention (Strubell et al., 2018)) and factorize over small fragments of the predicted structures.

most modern models process individual arguments and perform predicate disambiguation independently.


Our structured refinement network is simple but encodes non-local dependencies.

it takes into account the information about the role distributions on the previous iteration aggregated over the entire sentences but not the information what the other arguments are.


R ∈ ∆ n r−1 is a matrix of size n × r such that each row sums to 1, corresponding to a probability distribution over roles.

r i,0 is the probability of i-th word not being an argument of the predicate.


However, for the smaller data set we use, we find this not sufficient.

we use Gumbel-Softmax instead of Softmax.


The Gumbel noise helps for all languages except for Czech and English, the two largest datasets.

we observe almost no improvement on the Spanish dataset without using the Gumbel noise.


This suggests that the refinement model benefits from accessing information about other arguments when doing refinement.

modeling argument interaction appears genuinely useful.


This paper focuses on cross-lingual approaches to Named Entity Recognition (NER), owing to NER's importance as a core component in information retrieval and question answering systems.


After token-level matching, we construct a list of potential entity spans in the target sentence that match with a given source entity e S by grouping adjacent target tokens for which a token-level matching score s e (e S , •) is above a threshold δ (to remove spurious matches).

we construct the following set: is the phrase spanning tokens indexed from q to r (1 ≤ q, r ≤ L T ) in the target sentence.


Our technique outperforms previous state-of-theart cross-lingual methods on Spanish, German, Chinese, Hindi and Tamil and performs competitively on Dutch (Table 1).

our method shows marked improvements over BWET, a wordby-word translation baseline, for languages such as German, Hindi, Tamil and Chinese that differ markedly in word ordering (with respect to English), demonstrating the impact of improved machine translation quality on final NER tagging accuracy.


Since we use exact span matches to compute true and false positives, these small errors reduce the scores substantially.

we find a large number of missing B tags at the start of spans and misplaced O tags that split spans in the middle.


For each training sentence, we replace each content word-nouns, verbs, or adjectivewith an alternative word having the same universal POS, morphological features, and dependency label.

2 for each content word, we first stochastically choose whether to replace it; then, if we have chosen to replace it, we uniformly sample the replacement word type meeting the corresponding constraints.


In the monolingual experiments, we observe that both dependency tree morphing (MORPH) and nonce sentence generation (NONCE) improve performance, indicating the strong benefits of data augmentation when there are no other resources available except the target treebank itself.

when the number of training data is the lowest (T 10 ), data augmentations improves performance up to 9.3% LAS.


Looking at the produced parse trees, one striking difference is that monolingual model sometimes predicts a "rootless" tree.

it fails to assign a head of any word with index '0' and label the dependency with a root label.


In cases when both both models predict the correct heads, we observe that cross-lingual training gives further improvements in predicting most of the labels.

regarding the "rootless" trees discussed before, we see evidence that cross-lingual training helps in predicting the correct root index, and the correct root label.


In this paper, we show that weak supervisions of linguistic knowledge for the target languages can improve a cross-lingual graph-based dependency parser substantially.

we explore several types of corpus linguistic statistics and compile them into corpus-wise constraints to guide the inference process during the test time.


In this paper, we propose a complementary approach that studies how to leverage the linguistic knowledge about the target languages to help the transfer.

we use corpus linguistic statistics of the target languages as weak supervision signals to guide the test-time inference process when parsing with a graph-based parser.


• Binary constraints consider statistics regarding a pair of POS tags (P OS 1 , P OS 2 ).

a unary constraint specifies the ratio r of the heads of a particular P OS appears on the left of that P OS.


The best parse tree is given by arg max Y q ⇤ (Y).

we define the feasible set as: To measure the distance between two distribution, we use the KL-divergence, and find the best feasible distribution q ⇤ (Y): If the feasible set has the expectation form: Eq.


Due to our selection of memory (one example for each grammar rule in G), for a given x there is a unique sequence of retrieval and adaptation actions that lead to the production of the correct y.

letting a i be the ith action in the correct sequence of l actions, we define the probability of producing y given x as the probability of taking the sequence of actions a 1 , ..., a l .


Using a multi-task strategy to combine HSCRF and CRF, the HSCRF model exhibits state-of-the-art results without using external resources.

hSCRF resolves the segmentation problem by matching entity lengths.


But it also studies the potential cognitive interpretation of the underlying representations built by the search method (beam activity) through analysis of neuro-imaging signal through correlations with the neuro-imaging signal during naturalistic text listening The paper provides a preliminary investigation of how the improvements in natural language parsing techniques can be used for modelling brain activity during online sentence comprehension.

we focus on the RNNG generative parsing framework (Dyer et al., 2016), that allows us to extract information-theoretic measures that are relevant for cognitive studies such as surprisal or entropy (Hale, 2016).


Instantiating syntactic processing in terms of parser's analyses has already highlighted the language brain network to a large extent in the neuroimaging literature, we therefore expect that the goodness of fit of our parsing model with fMRI signal can be taken as an index of cognitive relevance.

we aim at using our parsers' beam activity as a proxy modelling the kind of processes that might take place during online sentence comprehension, and thus investigate the potential cross-fertilization between computational solutions and cognitive-neuro imaging approaches to human parsing.


Observe also that the set of actions differs from the standard top down formulation of RNNG (Dyer et al., 2016) and from the inorder transition system of (Liu and Zhang, 2017) only because of the definition of the Open action: our formulation pops from the stack the left corner constituent S 0 and pushes back on the stack the predicted category •N before S 0 .

during parsing, the stack keeps the same internal structure as in the top down version (Dyer et al., 2016) but the tree traversal is different.


We therefore study another solution to this problem that is inspired by Sequential importance sampling methods (Doucet and Johansen, 2011;Buys and Blunsom, 2015) and that can be interpreted as a variable beam search strategy where the search method carefully avoids to compare apples and oranges within the same beam.

the method ensures that comparisons between derivations are made only if derivations have an identical number of generated lexical elements.


To counter this effect, we reweight the branches by synchronizing the search process on word generation events.

each derivation (x i+1 , y i+1 ) ∈Ỹ(x i+1 ) that successfully generated the word x i+1 is reweighted by normalizing its importance weight: We subsequently reassign each derivation with a number of particles π (x i+1 , y i+1 ) proportional to its normalized weight.


As our exploration method does not directly samples K individual derivations but clusters the K particles on a smaller number of derivations with the recurrence (1).

we reformulate (8) not directly as (7) but rather as: we approximate the sum on the full set of parses Y(x i ) by the sum of the set of parses found in the beamỸ(x i ) and sampled with the importance distribution P * (x, y).


Such predictors are here to ensure that conclusions about parsing difficulty would be specific to the processes they instantiate, as opposed to more general aspects of speech perception.

lexical frequency was added as a covariate of noninterest, to statistically factor out effects of general word frequency (generally used in psycholinguistics), that may correlate with other types of expectations.


Second, the blank-out and selection can be done in phrase or sentence level.

if phrase boundaries are provided, a phrase-level aggregation can avoid a selector from breaking compound nouns or meaningful phrases (e.g., "Los Angeles", "not bad").


By constructing a candidate piece of knowledge as a sentence, we can use a language model to approximate the likelihood of this text as a proxy for its truthfulness.

we use a masked language model to estimate point-wise mutual information between entities in a possible relation, an approach that differs significantly from fine-tuning approaches used for other language modeling tasks.


Given a commonsense head-relation-tail triple x = (h, r, t), we are interested in determining the validity of that tuple as a representation of a commonsense fact.

we would like to determine a numeric score y ∈ R reflecting our confidence that a given tuple represents true knowledge.


In §4 we show that some of our sentiment analysis models rely exclusively on as few as three WFSAs.

2 2 a rational RNN with hidden size 3. x 7 !


Compared to the baselines, training with group lasso provides a better tradeoff between performance and number of transitions.

our heavily regularized models perform substantially better than the unigram baselines, gaining between 1-2% absolute improvements in four out of five cases.


Meta-learning algorithms aim at learning good initializations that can be useful for fine-tuning on various tasks with minimal training data, which makes them appealing alternatives to multi-task learning.

we investigate the recently proposed model-agnostic metalearning algorithm (MAML) (Finn et al., 2017) and its variants, namely first-order MAML and Reptile (Nichol et al., 2018), for NLU tasks.


In this paper, we mainly focus on optimization-based meta-learning algorithms, which achieve the goal by adjusting the optimization algorithm.

we investigate MAML, one of the most representative algorithms in this category, and its variants for NLU tasks.


We experiment with three different choices of the task distribution p(T ).

we propose the following options: • Uniform: sample tasks uniformly.


Figure 2 reveals that our model consistently outperforms the strong MT-DNN baseline across different settings, indicating the learned representations are more effective for transfer learning.

the algorithm is more effective when less data are available, especially compared to BERT, suggesting the metalearning algorithms can indeed be helpful for lowresource tasks.


While contextualized embeddings are useful, we observe that a language model-based embedding model, ELMo (Peters et al., 2018), cannot accurately capture the semantic equivalence of contexts.

in cases where the contexts of a word have equivalent or similar meanings but are changed in sentence formation or word order, ELMo may assign very different representations to the word.


To address these challenges, we propose a simple and effective paraphrase-aware retrofitting (PAR) method that is applicable to arbitrary pretrained contextualized embeddings.

pAR prepends an orthogonal transformation layer to a contextualized embedding model.


We use boldfaced w ∈ R k to denote a k-dimensional input word embedding, which can be pre-trained or derived from a character-level encoder (e.g., the character-level  PAR learns an orthogonal transformation M ∈ R k×k to reshape the input representation into a specific space, where the contextualized embedding vectors of a word in paraphrased contexts are collocated, while those in unrelated contexts are differentiated.

given two contexts S 1 and S 2 that both contain a shared word w, the contextual difference of a input representation w is defined by the L2 distance, Let P be the set of paraphrases on the training corpus, we minimize the following hinge loss (L H ).


Sallee (2004) show that just as arithmetic coding is optimal for data compression, it is also optimal for steganography.

given any goal distribution p s (y) that one wants to sample from, starting with a uniform m and applying the deterministic steganography procedures yields a distribution q = p s or equivalently D KL (q||p s ) = 0 for long sequences.


To get a dense low-dimenional representation of texts, we test seven representations covering sentence embedding, variants of un-contextualized word embedding and variants of contextualized word embedding.

: (i) Two Google universal sentence encoders: (Cer et al., 2018), an encoder (enc-2) based on deep averaging net (Iyyer et al., 2015) and an encoder (enc-3) based on transformer (Vaswani et al., 2017).


Since we use the reference summaries as target sequences to train the model and assume that they are the gold standard, we give both articles and reference summaries to the annotator to score the generated summaries.

relevance  we compare the generated summaries against the reference ones and the original article to obtain the (relative) scores in Table 3.


Knowledge bases Knowledge bases have been leveraged across multiple tasks across NLP (Bordes et al., 2011;Chang et al., 2014;Lin et al., 2015;Toutanova et al., 2015;Yang and Mitchell, 2017).

for coreference resolution, Prokofyev et al.


We justify the reliability of the proposed settings by demonstrating their strong performance on benchmark datasets.

without any task-specific techniques or architecture, our model outperforms not only all previous single-model results but also all ensemble results except for the ensemble result by Grundkiewicz et al.


DIRECTNOISE Whereas BACKTRANS (NOISY) and BACKTRANS (SAMPLE) generate ungrammatical sentences with a reverse model, DIRECT-NOISE injects noise "directly" into grammatical sentences (Edunov et al., 2018;Zhao et al., 2019).

for each token in the given sentence, this method probabilistically chooses one of the following operations: (i) masking with a placeholder token mask , (ii) deletion, (iii) insertion of a random token, and (iv) keeping the original 3 .


Nevertheless, Gigaword consistently outperforms the other two corpora.

DIRECTNOISE with Gigaword achieves the best value of F 0.5 among all the configurations.


We summarize these findings and attempt to combine PRETRAIN and JOINT.

we pretrain the model using 70M pseudo data of BACKTRANS (NOISY).


(2019) proposed the method of probabilistically injecting character-level noise into the source sentence of pseudo data D p .

one of the following operations is applied randomly at a rate of 0.003 per character: deletion, insertion, replacement, or transposition of adjacent characters.


In case of the Dirichlet distribution, it is a conjugate prior for the Multinomial distribution.

the probability density function of Dirichlet distribution is given by: where parameter α i > 0 is a K dimension vector and B(α) is a Beta function: When when all the factors in α equal 1, Dirichlet distribution becomes a uniform distribution.


Dir-VHRED achieves the best performance among the three word-embedding metrics on 1-turn and 3-turn responses of the two datasets.

the responses generated by Dir-VHRED own the highest topic similarity with the ground truth.


We can observe that the dimension 1 (dim1) is inclined to generate the shortest responses which, to a certain extent, shows the attitude of the speaker.

it prefers to generate the responses like "thanks", "no" or "yes".


Note that this context gate is crucial to avoid retraining on all data or repeatedly duplicating a large base model every time.

the lateral connections transfer the hidden states t−1 i for each word i from M t−1 to the newly expanded component M t .


Decay-hard sampling: to imitate the behavior that human gradually increase the difficulty of their learning materials at different stages, we propose a decay-hard sampling strategy which decays the margin in Equation (4) with the training process going on.

we consider two methods, namely exponential decay and linear decay.


We observe a similar trend for both models: they first increase monotonically until the margin reaches 0.07, and then drop as the margin increases.

the performance of both models with the semi-hard sampling strategy is worse than that with the random sampling strategy when the margin reaches 0.5.


The Dialog State Tracking Challenge (DSTC) datasets are another related resource.

dSTC 1 (Williams et al., 2013), dSTC 2 (Henderson et al., 2014a), and dSTC 3 (Henderson et al., 2014b) contain "chatbot style" queries, but the datasets are focused on state tracking.


For instance, CJK Symbols and Punctuation has a dedicated Unicode block in range U+3000...U+303F.

we count the appearances of characters in each of the 300 blocks.


Empirically, we drop the zero-count dimensions directly during training and assign conceptually low scores 3 when a non-zero count is seen during scoring.

we use normalized feature vectors e = e 1 , e 2 , ..., e B , where 1, 2, ..., B are dimensions in B whose original counts are non-zero, for training.


Therefore, we perform LM experiments on MT corpora.

we take monolingual data from parallel corpora of WMT19 (Barrault et al., 2019) Chinese-English (zh-en) and Russian-English (ru-en) as training data.


The main idea is to apply CSLS to select those languages that would be helpful for translating the desired source word.

a language m is selected as an auxiliary language only if the translated word y m is closer than the target word y j to the source word x in the shared space.


ON-LSTM is better at modeling hierarchical structure by introducing a syntax-oriented inductive bias, which enables RNNs to perform tree-like composition by controlling the update frequency of neurons.

we stack SANs encoder on top of ON-LSTM encoder (cascaded encoder).


Our method is based on this observation: if the rank of the next ground-truth target word is high enough in the prediction of the model, then this implies that there is enough source-side information for the model to make a correct prediction.

we sequentially input the source words to the pre-trained model, and use it to predict next target word.


The multilingual version of BERT (which is trained on Wikipedia articles from 100 languages and equipped with a 110,000 shared wordpiece vocabulary) has also demonstrated the ability to perform 'zeroresource' cross-lingual classification on the XNLI dataset (Conneau et al., 2018).

when multilingual BERT is finetuned for XNLI with English data alone, the model also gains the ability to handle the same task in other languages.


In spite of their success, the input representation only involves static order dependencies based on discrete numerical information.

any word in the entire vocabulary has the same PE on the same position index.


3: Moreover, self-attention can be further refined as multi-head self-attention to jointly attend to the information from different representation subspaces at different positions.

q, K, and V are linearly projected H times with different, learned linear projections to d k and d v dimensions, respectively.


We propose a recurrent positional embedding approach based on part of word embedding instead of numerical indices of words to capture order dependencies between words in a sentence.

the embedding of each word x j is divided into two parts x p j and x r j , and their dimensions are respectively.


7: Therefore, the final sentence representation S T is formally represented as: Compared with the RPEHead model, the MPRHead model applies the multi-head mechanism to the RPEs.

rPEs are encoded into the sentence representation from different vector sub-spaces, as shown in Fig 1(c).


For +RPEHead (or +MPRHead), with the increasing in the dimension of d r , BLEU scores are gradually increasing, but BLEU scores begin to decrease when d r is more than 320 (or 256).

+RPEHead and +MPRHead achieve the highest BLEU scores at d r =320 and d r =256, respectively.


The reason may be that adding RPEs into variant heads can encode order dependencies based word vectors from variant vector sub-spaces, which is one of the advantages of the multi-head mechanism.

+MPRHead (base/big) is slightly better than Transformer (base/big)+BiARN.


Reordering can be done either using rules based on linguistic knowledge (Isozaki et al., 2010b;Collins et al., 2005) or learning from aligned parallel data (Xia and McCord, 2004;Habash, 2007), and in principle our pseudo-corpus creation paradigm is compatible with any of these methods.

in this work we utilize rule-based methods, as our goal is to improve translation of low-resource languages, where large quantities of high-quality parallel data do not exist and we posit that current data-driven reordering methods are unlikely to function well.


Beam search is universally used in fullsentence translation but its application to simultaneous translation remains non-trivial, where output words are committed on the fly.

the recently proposed wait-k policy (Ma et al., 2019a) is a simple and effective method that (after an initial wait) commits one output word on receiving each input word, making beam search seemingly impossible.


In this work, we propose to augment SANs with structural position representations to model the latent structure of the input sentence, which is complementary to the standard sequential positional representations.

we use dependency tree to represent the grammatical structure of a sentence, and propose two strategies to encode the positional relationships among words in the dependency tree.


Absolute Structural Position We exploit the tree depth of the word in the dependency tree as its absolute structural position.

we treat the main verb (Tapanainen and Jarvinen, 1997) of the sentence as the origin and use the distance of the dependency path from the target word to the origin as the absolute structural position where x i is the target word, tree is the given dependency structure and the origin is the main verb of the tree.


This paper focuses on translation from English (En) to N different languages.

we consider exploiting two types of corpora.


Because we have no in-domain parallel data, we learn DAFEs via an auxiliary task, namely language modeling.

our proposed model consists of a base network, whose parameters are shared across settings, as well as both domain and task embedding learners.


DAFE disentangles hidden states into different parts so that the network can learn representations for particular domains or tasks, as illustrated in Figure 1.

it consists of three parts: a base network with parameters θ base that learns common features across different tasks and domains, a domain-aware feature embedding learner that generates embeddings θ τ domain given input domain τ and a task-aware feature embedding learner that outputs task representations θ γ task given input task γ.


The attention weights of ambiguous nouns are lower than those of nouns in general.

more attention is distributed to context words, which implies that self-attention recognizes ambiguous nouns and distributes more attention to the context.


In Korean, morpheme processing and POS tagging affect each other.

correct morpheme analysis helps POS tagging, and POS tags are helpful in analyzing morphemes.


However, if morpheme processing obtains some information from POS tagging, it can decompose ambiguous eojeols correctly.

the first '나는' is decomposed into '나 (I)' and '는 (topical postposition)', while the second '나는' is into '날 (fly)' and '는 (verbal ending)'.


The key distinguishing factor between Chinese and Korean mor-phological analysis is that lemmatization that recovers a base form for every morpheme as well as morpheme segmentation is a must in Korean morphological analysis, while Chinese morphological analysis requires only word segmentation.

in Korean, n i=1 where is a concatenate operator.


The remaining but majority errors are related to morpheme recovery.

these errors occur when the proposed method fails in recovering morphemes from eojeols accurately.


To encourage Q-BOT to ask diverse questions, we propose a simple auxiliary loss that penalizes similar dialog state embeddings.

given Q-BOT states -s Q t´1 , s Q t -in addition to maximizing likelihood of question (during supervised pretraining), or image-guessing reward (during selftalk RL finetuning), we maximize a smooth-L1 penalty on resulting in ř N t"2 f p∆ t q as an additional term in the overall objective (N " no.


Meanwhile, the latter is usually used when the amount of source data is significantly larger than the amount of target data.

the SLU model is pre-trained on a large amount of supervised source data, and then either tested directly on the target language (e.g.


With the aim of promoting and understanding the multilingual version of image search, we leverage visual object detection and propose a model with diverse multi-head attention to learn grounded multilingual multimodal representations.

our model attends to different types of textual semantics in two languages and visual objects for finegrained alignments between sentences and images.


Multi-head attention with diversity: We employ K-head attention networks to attend to the visual objects in an image as well as the textual semantics in a sentence then generate fixedlength image/sentence representations for alignment.

the k-th attended German sentence representation g k is computed by: where a k i is the attention weight, W k g ∈ R H×Hattn , W k cg ∈ R Hc×Hattn is the learnable transformation matrix for German.


In this work, we take a step in this direction by examining the effect of decoupling box proposal and featurization on downstream vision and language tasks.

we consider a two-stage object detector and set a goal of pushing the "featurization" aspect of the task further than before.


Second, this approach is better suited to downstream tasks whose domains are different from the one the object detector is trained on.

it allows us to benefit from transfer learning, which is a great advantage given the relatively modest amount of available supervised data for these downstream vision and language tasks.


We empirically demonstrate the abovementioned advantages through a focused study of the effect of improved featurization on image captioning and VQA in transfer learning settings.

we (i) leverage ultra-fine-grained semantic labels (e.g., "golden gate bridge" vs. "bridge") for featurization (Juan et al., 2019); and, (ii) focus on scenarios in which object detection modules trained on Visual Genome (VG) (Krishna et al., 2017) are applied to out-of-domain images: image captioning on the Conceptual Captions dataset (Sharma et al., 2018), and VQA on the VizWiz dataset (Gurari et al., 2018).


To this end, we exploit a graph-based, semi-supervised representation learning approach called Graph-Regularized Image Semantic Embedding (Graph-RISE) (Juan et al., 2019).

graph-RISE is based on ResNet-101 where the 10x10x2K feature map is first average pooled to 4x4x2K, and then flattened and projected to a 64-dimensional embedding before the softmax layer.


Nevertheless, these features are considerably complementary.

box features B-Ultra complements the Base G, pushing the score from 0.868 to 0.912.


Model We follow the setting described in Pythia v0.1 (Jiang et al., 2018), the winning entry to the VQA challenge 2018.

the architecture is a simplified "up-down" model from (Anderson et al., 2018) 7 .


The second stage is to measure three metric scores.

we measure relevance using standard cosine similarity.


Following Lee et al., we leverage a pretrained Stacked Cross Attention Neural Network (SCAN) to build a multi-modal semantic space.

we obtain word features by averaging the forward and backward hidden states per word from a bidirectional GRU (Bahdanau et al., 2014), where τ = {C, R} denotes either a candidate C or a reference R sentence of M words.


Notice that a small distance value indicates that the irrelevant context vector a C i⊥ is closed to the original context vector a C i .

the original context contains a large amount of extra information.


The first challenge is generalizing grounded natural language instruction understanding from seen to unseen environments.

in the R2R task, only 69% of bigrams are shared between training and evaluation.


which action should we use to interact with the environment to elicit the next state?


We have chosen 12 gaze features and based on the previous work (Barrett et al., 2016; Hollenstein and Zhang, 2019) we have divided them into 4 groups.

we explore the informativeness of the basic gaze features: total fixation duration on a word w (total fix dur), mean fixation duration on w (mean fix dur, num-ber fixations on w (n fix) and fixation probability on w (fix prob).


We will compare results on the development and test set to verify whether the improvements are consistent.

we carry out two experiments: • Experiment 1 (on parallel data): Tests how eye-tracking data influences the performance when both gaze features and syntactic dependencies are extracted from the same data (in this case the Dundee treebank).


on parallel data), suggesting that the degree of informativeness of the gaze features that a parser can leverage differ among each other.

we observe that the improvements across different models and the development and test sets are unstable.


In this work, we show that we can effectively recover these types of labels using the learned phrase vectors from deep inside-outside recursive autoencoders (DIORA).

we cluster span representations to induce span labels.


In this method, the context sentences are considered in the form of hierarchical attentions retrieved by the current generation.

it utilizes a wordlevel attention to represent a sentence and then a sentence-level attention to represent all the involved context.


The right part in Figure 1 shows details of the Sub-layerexpanded Transformer.

the equation of multi-head attention computing procedure is as following: where W i Q , W i K and W i V are the parameter matrices, Q, K and V indicates the query, key and value representations.


We performed the same configuration on our models according to the settings of the Maruf and Haffari (2018).

for the Transformer, we set the hidden size and point-wise FFN size as 512 and 2048 respectively.


From the results, the performance of our model is significantly improved by 0.93 BLEU points (the first two rows in Table 4) when the largescale sentence-level parallel corpus is used for the pre-training process.

when we use the mixed data of both sentence-and document-level parallel corpora 2 to first pre-train our model, the performance of our model is significantly improved by 5.26 BLEU points (the last row in Table 4).


As we do not exploit any training data in the target language, we could regard this approach as a zero-shot cross-lingual baseline system.

we use Google Neural Machine Translation (GNMT) system for source-to-target and target-to-source translations.


Moreover, as Transformers are fundamental components of BERT, multi-head self-attention layer (Vaswani et al., 2017) is used to extract useful information within the input sequence.

in our model, to enhance the target representation, we use a multi-head self-attention layer to extract useful information in source BERT representation B S .


This paper considers the reading comprehension task in which some discrete-reasoning abilities are needed to correctly answer questions.

we focus on a new reading comprehension dataset called DROP (Dua et al., 2019), which requires Discrete Reasoning Over the content of Paragraphs to obtain the final answer.


To obtain a universal representation for both the question and the passage, we utilize BERT (Devlin et al., 2019), a pre-trained deep bidirectional Transformer model that achieves state-of-the-art performance across various tasks, as the encoder.

we first tokenize the question and  Figure 2: An illustration of MTMSN architecture.


(2017) to predict the starting and ending positions across the entire sequence.

we first compute three vectors, namely g Q 0 , g Q 1 , g Q 2 , that summarize the question information among different levels of question representations: where g Q 0 and g Q 1 are computed over Q 0 and Q 1 respectively, in a similar way as described above.


As a result, an arithmetic expression that has a number as the final answer can be obtained and easily evaluated.

for each number mentioned in the passage, we gather its corresponding representation from the concatenation of M 2 and M 3 , eventually yielding where N numbers exist.


Therefore, in order to further validate the prediction, it is necessary to rank several highly confident expression candidates using the representation summarized from the expression's context.

we use beam search to produce top-ranked arithmetic expressions, which are sent back to the network for reranking.


We use simple rules to search over all mentioned numbers to find potential negations.

if 100 minus a number is equal to the answer, then a negation occurs on this number.


First, we investigate the effects of some "global vectors" used in our model.

we find that removing the question and passage vectors from all involved computation leads to 1.3 % drop on F1.


The output is a label identifying whether clarification is needed.

the input is {Q a , E 1 , E 2 } for single-turn cases or {Q p , R p , Q a , E 1 , E 2 } for multi-turn cases.


We implement several widely-used sequence classification models, including Convolutional Neural Network (CNN) (Kim, 2014), Long-Short Term Memory Network (LSTM) (Schuster and Paliwal, 1997), and Recurrent Convolutional Neural Network (RCNN) (Lai et al., 2015) Structured models use separate structures to encode different source information and adopt an additional structure to model the inter-relation of the source information.

we use two representative neural networks, Hierarchical Attention Network (HAN) (Yang et al., 2016) and Dynamic Memory Network (DMN) (Kumar et al., 2016), as our structured baselines.


The linguistic subtlety of microaggressions in communication has made it difficult for researchers to analyze their exact nature, and to quantify and extract microaggressions automatically.

the lack of a corpus of real-world microaggressions and well-defined criteria for annotating them have prevented researchers from addressing these problems at scale.


However, while they were comprehensive in the clinical domain, here, we aim to create a more general framework for classifying MAS in online interactions.

our new typology builds upon the work of Sue et al.


We compare our framework (R 2 HP) with a wide range of prior arts.

we re-implement the methods of learning from deep neural networks and with hand-crafted features.


In this experiment we draw episodes from the first half of 2016-09 as queries and the second half of 2016-09 as targets.

for  each of 25,000 randomly selected users from the 2016-09 split we randomly draw a query episode of length 16 from among those posts published by that user before 2016-09-15.


Despite the broad definition of style, this work mainly focuses on one specific aspect of style, pragmatics aspects in group characteristics of speakers, which is also called persona.

we look at multiple types of group characteristics in conjunction, such as gender, age, education level, and more.


PASTEL can be used in many style related applications including style classification, stylometry (Verhoeven and Daelemans, 2014), style transfer (Fu et al., 2018), visually-grounded style transfer, and more.

we chose two applications, where PASTEL helps design appropriate experiment and evaluation: controlled style classification ( §5.1) and supervised style transfer ( §5.2).


Recent years have seen increased interest in the automatic analysis of long and complex narratives.

machine Reading Comprehension (mRC) and Question Answering (QA) tasks are transitioning from investigating single short and clean articles or queries (Rajpurkar et al., 2016;Nguyen et al., 2016;Trischler et al., 2016) to large scale datasets that consist of complex stories (Tapaswi et al., 2016;Frermann et al., 2018;Kočiskỳ et al., 2018;Joshi et al., 2017) or require reasoning across multiple documents (Welbl et al., 2018;Wang et al., 2018;Dua et al., 2019;.


Next, we compute the semantic similarity of the current sentence with each context representation.

we consider the element-wise product b i , cosine similarity c i and pairwise distance u i as similarity metrics: The interaction representation of sentence cp i with its left context is the concatenation of cp i , f l i , and the above similarity values (i.e., b i , c i , u i ): The interaction representation f r i for the right context rc i is computed analogously.


Topic-Aware Model (TAM) TAM enhances our screenplay encoder with information about topic boundaries.

we compute the representations of the left lc i and right rc i context window of the i th scene in the screenplay as described in Section 4.1.


Inference During inference in our first task (i.e., identification of TPs in synopses), we select one sentence per TP.

we want to track   the five sentences with the highest posterior probability of being TPs and sequentially assign them TP labels based on their position.


Table 4a reports our results on the development set (we extracted 20 movies from the original training set) which aim at comparing various model instantiations for the TP identification task.

we report the performance of a baseline model which is nei- ther context-aware nor utilizes topic boundary information against CAM and TAM.


Aside from the theory and distribution-based baselines, we also experimented 3 with a com- mon IR baseline which considers TP synopsis sentences as queries and retrieves a neighborhood of semantically similar scenes from the screenplay using tf*idf similarity.

we compute the maximum tf*idf similarity for all sentences included in the respective scene.


However, they are applicable to a small group of people.

for the people who are suffering but tend to hide inmost thoughts and refuse to seek helps from others, the approaches cannot function (Essau, 2005;Rickwood et al., 2007;Zachrisson et al., 2006).


After that, four doctoral students major in computational mental healthcare were employed to annotate users that whether they are at risk of suicide or not.

we only decide the user "at suicide risk" based on self-report of his/her tree hole posts.


While outperforming end-to-end dialogue models on bAbI Dialog Tasks in a zero-shot setup  due to its prior linguistic knowledge in the form of a dialogue grammar, this method inherited the limitations of it as well.

it's limited to a single domain until a wide-coverage grammar is available.


We first describe the task we are addressing in this paper, and the corresponding base model.

we have a set of dialogues in source domains and just a few seed dialogues in the target domain.