Therefore, in this work, we introduce a novel approach to directly address the lack of data.

we propose a simple yet effective round-trip translation strategy to obtain cross-lingual documentsummary pairs from existing monolingual summarization datasets (Hermann et al., 2015;Zhu et al., 2018;Hu et al., 2015).


CLS+MT: It trains CLS and MT tasks via alternating training strategy.

we optimize the CLS task in a mini-batch, and we optimize the MT task in the next mini-batch.


In this way, the training pairs are distantly-labelled for training the model.

the representations of the reference summaries and the testing set are computed by summing all the involved word embeddings.


Here we use an oracle setting which ignores contractions.

we compare the WO+MR output not with the final sentence but with the sentence before contraction applies (the ability to handle contractions is investigated in the next section).


We propose an iterative matching and translation algorithm composed of the following three steps: ( Step 1) Matching In iteration t = 0, we construct a large pseudo-parallel corpusX andŶ (0) by pairing sentences from X with those from Y .

we calculate the semantic cosine similarity score (detailed in Section 4.4) between a sentence x i ∈ X and every sentence y ∈ Y , select the one with the highest score asŷ i , and only keep the sentence pair if the similarity exceeds a threshold γ.X denotes the subset of the original corpus X for which we find matches.


( Step 3) Refinement In this step, we refine thê Y (t) obtained in step 1 with the trained translation model M (t) in step 2.

we apply the model M (t) to translate each sentence i , and form a temporary corpus Trans (t) .


MLP with BERT as encoder has the best overall performance.

bERT+MLP+Pref significantly outperforms (p < 0.05) all the other models that do not use bERT+MLP, as well as the metrics that rely on reference summaries (see Table 1).


We experiment on question generation and abstractive summarization tasks and show that our method achieves the best trade-off between accuracy and diversity over previous models on SQuAD (Rajpurkar et al., 2016) and CNN-DM (Hermann et al., 2015;See et al., 2017) datasets.

compared to the recently-introduced mixture decoder (Shen et al., 2019) that also aims to diversify outputs by creating multiple decoders, our modular method not only demonstrates better accuracy and diversity, but also trains 3.7 times faster.


For each task, we compare our method with other techniques which promote diversity at the decoding step.

we compare with recent diverse search algorithms including Truncated Sampling (Fan et al., 2018), Diverse Beam Search (Vijayakumar et al., 2018), and Mixture Decoder (Shen et al., 2019).


The bottom rows  of Tables 1 and 2 show the upper bound performance of SELECTOR by feeding focus guide to generator during test time.

we assume that the mask is the ground truth overlap between input and target sequences at test time.


The third family is the generative architecture that we focus on.

applies the conditional variational autoencoders (CVAEs) (Sohn et al., 2015) with LSTM encoderdecoders in generating paraphrases, expecting RNN's advantage of modeling local dependencies to be a good complement to the CVAE's power at learning global representations.


We propose the first paraphrase generative model via adversarial training to tackle the above problems, which we believe is a natural answer to the aforementioned challenges.

we formalize CVAE as a generator of the Generative Adversarial Network (GAN) (Goodfellow et al., 2014) and tailor a discriminator to CVAE.


All the annotations toward one image describe the most prominent object or action in this image, which is suitable for the paraphrase generation task.

the dataset has two divisions: training dataset ("Train2014", over 82K images) and validation dataset ("Val2014", over 40K images).


The first layer is used to learn the representation of the table cell.

we employ three self-attention models to obtain three representations of the table cell in its row, column and time dimension.


In multi-document summarization, a set of documents to be summarized is assumed to be on the same topic, known as the underlying topic in this paper.

the underlying topic can be collectively represented by all the documents in the set.


The representations for news sentences S and news documents D are learned in a hierarchical structure, as shown in Figure 2.

the embeddings of words w in each sentence are fed into a bidirectional RNN encoder-decoder framework , which inherently encodes word sequence information in sentence representations.


For subtopic representation, we also utilize S, and incorporate the comment sentence embedding Y .

we only take a single sentence without context information to build subtopic representations, as in a typical topic model.


We showed participants the output of our model and a comparison model, and asked them to make a pairwise preference judgment between the two passages.

we randomly selected 100 samples from the Random test set.


In recent years, sequence to sequence learning techniques have achieved improvements in a variety of NLP tasks, such as machine translation ), text summarization (Nallapati et al., 2016) and dialogue systems (Serban et al., 2017).

the paraphrase generation can also be formulated as a sequence to sequence problem.


(2017)  Given a source sentence X, we aim to generate multiple target sentences S = {Y 1 , Y 2 , ..., Y k }.

our goal is to learn a model which can generate S such that every Y i (i=1..k) has the same meaning as X and obvious differences between each other.


The i-th generator can get a high reward only when sentence Y i is recognized as a paraphrase of X by the paraphrase discriminator and the differences between Y i and sentences generated by other generators is so large that generator discriminator takes Y i as a sentence generated by the i-th generator.

we use the self-critical algorithm (Rennie et al., 2017) as our reinforcement learning method.


Since G t and G b capture distinct perspectives of the graph structure, the information flow is prop-agated throughout TD and BU directions, respectively.

for each node v i , the GE receives the node embeddings of v i and its neighbors, and computes its node representation: where Each node v i is represented by two different hidden states, h t i and h b i .


For each AMR graph, we generate a node sequence by depth-first traversal order.

given a representation sequence from r 1 to r n , the hidden forward and backward states of r i are defined as: where LSTM f is a forward LSTM and LSTM b is a backward LSTM.


We explore distinct strategies for neighborhood aggregation, adopting three GNNs: Gated Graph Neural Networks GGNNs employ gated recurrent units to encode node representations, reducing the recurrence to a fixed number of steps.

the l-th layer of a GGNN is calculated as: where N (i) is the immediate neighborhood of v i , W 1 is a parameter and GRU is a gated recurrent unit (Cho et al., 2014).


Therefore, the attention weights among tasks generally maintain a hierarchical relation.

the higher tasks (headline generation and summarization) need to focus on a wider range of words, and the lower tasks (classification) focus on the range on which the higher tasks focused.


rors are one of the most frequent error types in learner English (Tetreault and Chodorow, 2008); (b) they are relatively explainable (c.f., article errors); and (c) it would be a good starting point for feedback comment generation research considering the potential difficulty of the task.

we have annotated learner corpora with feedback comments, written in English and Japanese, on preposition use; we have released a part of the dataset to the public.


This makes feedback comments more flexible.

the word(s) inside << and >> in a generated feedback comment may be replaced with another word(s) in the commented sentence.


In addition, the cosine similarity should be a good indicator to determine whether the retrieved feedback comments are appropriate or not, and thus retrieved feedback comments are discarded if their similarity is lower than a certain threshold.

we do not generate feedback comments if we cannot find similar feedback comments in the training data.


To address this issue, we propose a method to jointly model the unstructured sentence and the structured answer-relevant relation (extracted from the sentence in advance) for question generation.

the structured answer-relevant relation acts as the to the point context and it thus naturally helps keep the generated question to the point, while the unstructured sentence provides the full information.


We find that the performance drops at most 36% when the relative distance increases from "0 ∼ 10" to "> 10".

when the useful context is located far away from the answer, current proximity-based answer-aware approaches will become less effective, since they overly emphasize neighboring words of the answer.


With the help of such structured information, our model is less likely to be confused by sentences with a complex structure.

we firstly extract multiple relations with an off-the-shelf Open Information Extraction (OpenIE) toolbox (Saha and Mausam, 2018), then we select the relation that is most relevant to the answer with carefully designed heuristic rules.


In the evaluations on the SQuAD dataset, our system achieves significant and consistent improvement as compared to all baseline methods.

we demonstrate that the improvement is more significant with a larger relative distance between the answer and other non-stop sentence words that also appear in the ground truth question.


It contains an additional sub-layer that performs multi-head attention over the outputs of the encoder block.

a decoding layer d l i is the result of multi-head attention over the outputs of the encoder which in turn is a function of the encoder memory and the outputs from the previous layer: where, the keys and values are the encoder outputs and the queries correspond to the decoder input.


Researchers have been utilizing multiple versions of Seq2Seq (Sutskever et al., 2014) framework to generate online product reviews of good quality, including Aspect-Aware Representations (Ni and McAuley, 2018), Gated Contexts to Sequences (Tang et al., 2016), RNN (Yao et al., 2017), Aspect-Sentiment Score (Zang and Wan, 2017), Generative Concatenative Nets (Lipton et al., 2015) and Sentiment Units (Radford et al., 2017).

(Li et al., 2015) proposed a two-stage LSTM neural network to construct a hierarchical autoencoder for long-text representation and gen-eration.


To demonstrate that our purposed model indeed achieves the state-of-the-art review generation performance, we implement various evaluation metrics, including distribution-based Log-Likelihood and Perplexity, coherence-based Word Mover Distance (WMD) (Kusner et al., 2015), ngram-based BLEU (Papineni et al., 2002) and ROUGE (Lin, 2004), contextual label accuracy and human evaluation to measure the performance of review generation.

following the same metric as (Dong et al., 2017), we use sentiment accuracy, the ratio of the reviews whose sentiment matches with the given label, as an important indication of the personalization ability of the generator.


ROUGE Arguably the most popular metric for summarization at the moment, it provides a set of measures to compare automatically generated texts against one or more references (Lin, 2004).

rOUGE-N is based on the count of overlapping n-grams, while rOUGE-L accounts for the longest common sub-sequence between the candidate and its corresponding reference(s).


1 greedily by computing g t = argmax gt P (g t |g <t , x, z p ) with the plan decoder (a GRU).

at time step t, the plan decoder makes a binary prediction for each input item by estimating P (d i ∈ g t |g <t , x, z p ): where σ denotes the sigmoid function, h i is the vector of input item d i , and h p t is the hidden state of the plan decoder.


Second, to capture the dependencies on s <t , we explicitly model the dependencies among local latent variables by inputting z s t−1 to the sentence decoder, so that z s t is conditioned on z s <t and is expected to model smooth transitions in a long text.

we initialize the hidden state h s 0 by encoding the input x, the global planning latent variable z p and the planning result g: where h g T is the last hidden state of GRU g that encodes the planning result g. word-level Generation The word decoder (a GRU) conducts word-level generation; it decodes a sentence s we sample word w t k of s t as follows: we train our model end-to-end.


We made some modifications to the original specification tables.

if some attribute-value pairs from a table do not occur in the corresponding text, the pairs are removed from the table.


Since we do not have a parallel corpus, both GSTs are trained to minimize the reconstruction loss.

for a sentence x, the model learns to reconstruct y = x given c x , its own attributes a x (only for G-GST) and its own style s src (only for B-GST).


We then proceed to obtain human evaluations on these models along with ours through Amazon Mechanical Turk 7 .

we ask annotators to rate each pair of generated sentences given the source sentence, on content preservation, style transfer strength, fluency, and overall success.


We run off-the-shelf coreference resolution system from Stanford CoreNLP (Manning et al., 2014) on the input articles to extract entities, each represented as a cluster of mentions.

from each input article, we extract the coreferenced entities, and construct the mention clusters for all the mentions of each entity in that article.


For this, we focus on the non-restrictive appositives and relative clauses, which often carry noncritical information (Conroy et al., 2006;Wang et al., 2013) and can be automatically detected based on comma usage patterns.

a sentence contains a non-restrictive appositive if i) it contains two commas, and ii) the word after first comma is a possessive pronoun or a determinant (Geva et al., 2019).


Typically, an encoder encodes the input x i to a semantic representation c i , while a decoder controls or modifies the stylistic property and decodes the sentence x i based on c i and the pre-specific style l i .

we denote an encoder-decoder model as (E, D).


An analysis of existing AQG models shows that they produce questions which do not adhere to one or more of the abovementioned qualities.

the generated questions look like an incomplete draft of the desired question with a clear scope for refinement.


We observe that the increase in Q-Metric for refined questions is because the RefNet model can correct/add the relevant Named Entities in the question.

we observe that the Named Entity component score in Q-Metric increases from 32.42 for the first draft to 37.81 for the refined draft.


Alternative approaches for enrichment exists, of course, but we wonder how worthwhile further efforts would be.


In this paper, we introduce a novel interactive approach to translate a source language into two different languages simultaneously and interactively.

the generation of one language relies on not only previously generated outputs by itself, but also the outputs predicted in the other language.


(2) Extensive experiments show the superiority of our proposed method.

this synchronous approach can significantly outperform both the conventional NMT model and the multilingual NMT model.


Hypotheses are called complete if they end with < /s > and partial if they do not.

beam search is the ubiquitous decoding algorithm for NMT, but it is prone to search errors as the number of active hypotheses is limited by n. beam search never compares partial hypotheses of different lengths with each other.


This paper systematically studies this temporal commonsense problem.

we define five classes of temporal commonsense, and use crowdsourcing to develop a new dataset, MCTACO , that serves as a test set for this task.


Our first contribution is that, to the best of our knowledge, we are the first to systematically study and quantify performance on a range of temporal commonsense phenomena.

we consider five temporal properties: duration (how long an event takes), temporal ordering (typical order of events), typical time (when an event happens), frequency (how often an event occurs), and stationarity (whether a state holds for a very long time or indefinitely).


In LC , because all answer word representations are expected to carry the information of each other and their contexts, we choose to maximize averaged MI between the sampled answer word representations and the whole answer sequence with its context words.

a positive sample is obtained by pairing the sampled answer word representation x 2 r a = {r p m , .


In Original ADDSENT ADDONESENT BiDAF-S (Seo et al., 2016) 75.5 34.3 45.7 ReasoNet-S  78.2 39.4 50.3 Reinforced Mnemonic Reader-S (Hu et al., 2017) 78.5 46.6 56.0 QANet-S (Yu et al., 2018) 83.8 45.2 55.7 GQA-S (Lewis and Fan, 2018) 83.7 47.3 57.8 FusionNet-E (Huang et al., 2017) 83.6 51.4 60.7 BERT-S (Devlin et al., 2018) 88.5 51.0 63.4 BERT-S + QAInfomax 88.6 54.5 † 64.9 † the experiments, we use S(r a ) = ( 1 M P r a i ) as our summarization function, where is the logistic sigmoid nonlinearity.

a positive sample here is the pair of a answer summary vector s = S(r a ) and all other word representations in r. Negative samples are provided by sampling question, passage and answer representations {r q ,r p ,r a } from an alternative training example.


In this paper, we propose a meta-based algorithm for multi-hop reasoning (Meta-KGR) to address the above problems, which is explainable and effective for few-shot relations.

in Meta-KGR, we regard triple queries with the same relation r in KGs as a task.


To solve the above MDP problem, we need a model which has policy to choose actions at each state.

different from normal RL problems, we apply a policy network considering the search history over G. Formally, after embedding all entities and relations in G as e ∈ R d and r ∈ R d respectively, each action a t = (r t+1 ,ê t+1 ) ∈ A t is represented as a t = [r t+1 ;ê t+1 ].


We use two typical datasets FB15K-237 (Toutanova et al., 2015) and NELL-995 (Xiong et al., 2017) for training and evaluation.

we set K = 137 and K = 114 to select few-shot relations from FB15K-237 and NELL-995 respectively.


We do so by systematically examining threats to the validity of experiments involving recent CSR models.

any study aiming to show a conclusion (e.g., that a particular model can perform CSR) is subject to threats to its internal and exter- Examples Proportion Non-Assoc.


Based on the above motivations, we introduce Generative Adversarial Net (Goodfellow et al., 2014) into pun generation task.

the generator can be any model that is able to generate a pun sentence containing a given word with two specific senses.


We have shown that autoregressive predictions expressed in the observation model instead of hidden states deliver better results on a simple corpus.

mistakes at the bigram-level, such as repetitions, are avoided and we suspect that more densely connected CRFs allow to extend these promising results to more complex patterns found in more complex corpora.


Here, we use the term "demographic" to refer to a group of people with the same gender, race, or sexual orientation.

we examine the groups female and male for gender, Black and White for race, and gay and straight for sexual orientation.


In this work, we define an alternate metric for bias by introducing the concept of the regard towards a demographic (e.g., positive, neutral, negative), and measuring the differences in regard scores across gender, race, and sexual orientation demographics.

we specifically design regard to measure bias.


11 Overall, GPT-2 text generations exhibit different levels of bias towards different demographics.

when conditioning on context related to respect, there are more negative associations of black, man, and gay demographics.


While the technique of delexicalization/masking has been used before (Zeman and Resnik, 2008, e.g.,), here we expand it by incorporating semantic information.

we first replace named entities with their corresponding semantic tags from Stanford's CoreNLP .


which consistently achieves near state-of-the-art performance on RTE tasks 1 .

we use the AllenNLP 2 implementation of DA, which was provided by the FEVER task organizers.


This also allows us to evaluate using the official scoring measures of the target domain.

when training on FEVER and testing on FNC, the data points in FEVER that belong to the class supports were relabeled as agree, and those in refutes as disagree.


Also the distribution of labels was made similar to that of FNC.

16.86% of the total 35,639 data points that originally belonged to the class NEI in FEVER were thus assigned the label discuss and the rest (83.14%) were assigned the label unrelated.


We use the word-level attention weights (Bahdanau et al., 2014) learned by DA to perform an error analysis in the cross-domain evaluation.

we first trained two equivalent DA models, one on FEVER and the other on FNC.


Overlap Aware NER (OA-NER): This technique additionally captures the lexical overlap between the claim and evidence sentences with entity ids.

the first instance of a given entity in the claim is tagged with c1, where the c denotes With Singapore Airlines, the Airbus A380 entered commercial service.


Idiosyncrasies Distorting Performance We investigate the correlation between phrases in the claims and the labels.

we look at the n-gram distribution in the training set.


The dataset should enforce models to validate the claim with respect to the retrieved evidence.

the truth of some claims might change as the evidence varies over time.


We reweight the instances in the dataset to flatten the correlation of claim n-grams with respect to the labels.

for 'give-away' phrases of a particular label, we increase the importance of claims with different labels containing those phrases.


However, the tree-LSTM also heavily emphasizes the root node in the tree to build sentence representations.

words that are closed to the root will be given higher priority than words that are far away from the root.


We use existing pretrained architectures for Text Recognition, and add a shallow deep-network that performs a postprocessing operation to re-rank the proposed candidate texts.

we re-rank the candidates using their semantic relatedness score with other visual information extracted from the image (e.g.


To test this hypothesis, we investigate if a more reliable estimator of topic quality can be obtained by combining these two classes of methods in a supervised approach.

we train a support vector regression (SVR) estimator (Joachims, 2006) with the estimations of these methods as features, including all the topic quality measures introduced in Section 2 along with our proposed variability method.


Second, relevance judgments in nearly all newswire test collections are annotations on documents, not on individual sentences or passages.

given a query, we only know what documents are relevant, not spans within those documents.


Our second innovation involves the aggregation of sentence-level evidence for document ranking.

given an initial ranked list of documents, we segment each into sentences, and then apply inference over each sentence separately, after which sentence-level scores are aggregated to yield a final score for ranking documents.


To determine document relevance, we apply inference over each individual sentence in a candidate document, and then combine the top n scores with the original document score as follows: where S doc is the original document score and S i is the i-th top scoring sentence according to BERT.

the relevance score of a document comes from the combination of a document-level term-matching score and evidence contributions from the top sentences in the document as determined by the BERT model.


On Core17 and Core18, we observe significant increases in AP as well (and other metrics in some cases).

relevance models learned from tweets 3 Even though Lin (2018) and Yang et al.


tune the pre-trained BERT model and achieve new state-of-the-art results on WSD task.


There is a strong connection between the Korean and Chinese languages due to cultural and historical reasons (Lee and Ramsey, 2011).

a set of logograms with very similar forms to the Chinese characters, called Hanja 1 , served in the past as the only medium for written Korean until Hangul, the Korean alphabet, and Jamo came into existence in 1443 (Sohn, 2001).


(2017), the scoring function incorporates subword information.

given all possible n-gram character sets G = {g 1 , .


While not strictly proven, we conjecture one possible reason for the unsatisfying performance is the low reliability of the Hanja tagger we leveraged, which is not guaranteed to work well with spoken language.

we have manually inspected the tagging results and observed that there are some errors which can lead to performance degradation.


Several studies on knowledge-based algorithms have indicated that the LKB structure is of vital importance in determining the accuracy of sense disambiguation.

it has been demonstrated that WSD performance improves 1 A syntagmatic relation exists between two words which co-occur in spoken or written language more frequently than would be expected by chance and which have different grammatical roles in the sentences in which they occur (Saussure, 1916).


In this paper, we develop a novel Sparse Self-Attention Fine-tuning model (referred as SSAF) which integrates sparsity into selfattention mechanism to enhance the finetuning performance of BERT.

sparsity is introduced into the self-attention by replacing softmax function with a controllable sparse transformation when fine-tuning with BERT.


To overcome the above limitation, in this work, we develop a novel Sparse Self-Attention Finetuning model (referred as SSAF) by integrating sparsity into self-attention to refine the attention distribution when fine-tuning the pretrained BERT.

we propose sparse self-attention mechanism (SSAM) to induce sparsity by replacing the traditional softmax function with a sparse transformation in self-attention networks.


In this part, we propose a sparse self-attention fine-tuning model (SSAF).

this finetuning model with BERT is composed of N sparse self-attention layers, where each layer learns a representation by taking the output from the previous layer:h where SSAM is adopted to replace the traditional self-attention mechanism, h 0 = embed(x) denotes the representation for the input sequence x which is the sum of token embeddings and the position embeddings, and LN is the layer normalization operation.


To demonstrate that SSAF truly improves the finetuning performance, we compare SSAF against BERT BASE from (Devlin et al., 2018) on all the tasks.

for sentiment analysis, we follow (Ambartsoumian and Popowich, 2018) to make comparisons with the following representa-1 https://github.com/google-research/bert  tive methods: Ave, LSTM, BiLSTM, and CNN from (Barnes et al., 2017); SSAN from (Ambartsoumian and Popowich, 2018).


In crosslanguage tasks, these vector-space representations have recently emerged as a tool to transfer knowledge from one language to another.

several studies have suggested forming crosslingual embeddings by learning a linear mapping from a source-language embedding space to a target-language one and demonstrated the benefits of this approach for word translation (Mikolov et al., 2013;Klementiev et al., 2012).


The rationale behind this addition is to adopt the examples from other tasks as simulated OOD examples for the current meta-train tasks.

we first define a hinge loss on x out j and the closest ID supporting set in S in , then we push the examples from another task away from the prototypical vectors of ID supporting sets.


Since a parallel corpus, albeit small, is available, formality style transfer usually takes a seq2seq-like approach (Rao and Tetreault, 2018;Niu et al., 2018a;Xu et al., 2019b).

this paper focuses on harnessing pre-trained neural networks with rulebased systems.


In this paper, we study differentiable neural architecture search (NAS) methods for natural language processing.

we improve differentiable architecture search by removing the softmax-local constraint.


More active units usually indicates richer latent representations (Burda et al., 2016).

a dimension is "active" when it is sensitive to the change in observations x.


We test for copying behaviour on the PTB dataset.

we sampled 300 sentences from the prior and retrieved their nearest neighbors in the training set.


We investigate parsing Chinese questions to SQL by creating a first dataset, and empirically evaluating a strong baseline model on the dataset.

we translate the Spider (Yu et al., 2018b) dataset into Chinese.


(2018a) as the baseline model, which can be regarded as a sequence-to-tree model.

the input question is encoded using an LSTM sequence encoder, and the output is a SQL query in its syntactic tree form.


The detailed results are shown in Table 4.

the char-based methods achieve around 41% on SELN and SEL (SELECT), which are about 5% higher compared to the word-based methods.


Figure 3 shows the negative influence of segmentation errors.

the incorrect segmentation of the word "店名" (shop name) leads to incorrect SQL for the whole sentence, since the   character "店" (shop) can typically be associated with "店长" (shop manager).


Figure 4 shows the sensitivity of our model to sentence patterns.

the wordbased model gives incorrect predictions for many question sentences frequently.


Figure 5 shows the sensitivity of the model to Chinese linguistic patterns.

the first sentence has a zero pronoun "各党的" (in each party), which is omitted later.


This was done by converting the DB schema into a graph, where nodes are DB constants, and edges connect tables and their columns, as well as primary and foreign keys (Figure 2 To focus the GCN's capacity on important nodes, a relevance probability ρ v was computed for every node, and used to "gate" the input to the GCN, conditioned on the question.

given a learned embedding r v for every database constant, the GCN input is h Then, the GCN recurrence is applied for L steps.


Here, we conduct an empirical study of transductive learning for neural models and demonstrate its utility in syntactic and semantic tasks.

we fine-tune language models (LMs) on an unlabeled test set to obtain test-set-specific word representations.


In transductive learning (Vapnik, 1998), an unlabeled test set is given in the training phase.

the inputs of the test set, i.e., the raw texts, can be used during training, but the labels are never used.


After the pioneering work of Joachims (1999), who proposed a transductive support vector machine for text classification, transductive methods for linear models have been investigated in only a few tasks, such as lexical acquisition (Duh and Kirchhoff, 2006) and machine translation (Ueffing et al., 2007).

transductive learning with neural networks is underexplored.


Here, we investigate the impact of transductive learning on state-of-the-art neural models in syntactic and semantic tasks, namely syntactic chunking and semantic role labeling (SRL).

inspired by recent findings that language model (LM)-based word representations yield large performance improvement (Devlin et al., 2019), we fine-tune Embeddings from Language Models (ELMo)  on an unlabeled test set and use them in each task-specific model.


We present a simple transductive method for neural models.

we finetune an LM on an unlabeled test set.


In addition, we control the sizes of the target domain unlabeled data and test sets.

we use the same number of sentences in the unlabeled data of each target domain as in each test set.


In this study, we investigated the impact of transductive learning on state-of-the-art neural models in syntactic and semantic tasks.

we fine-tuned an LM on an unlabeled test set.


The word vectors are generated randomly from a standard normal distribution with d = 10. d-dimensional word embeddings in an N × d matrix, then apply DCT along the rows.

each feature in the vector space is compressed independently, and the resultant DCT embeddings summarize the feature patterns along the word sequence.


structures such as E2,E3 because one of their arguments, in this case E1, is an event.

e1 is a flat event since its argument is an entity.


In this work, we show that pretrained language models, BERT (Devlin et al., 2018) in particular, can be used for this task to capture contextual dependencies without the need for hierarchical encoding nor a CRF.

we construct a joint sentence representation that allows BERT Transformer layers to directly utilize contextual information from all words in all sentences.


In contrast, we propose an input representation which allows the Transformer layers in BERT to directly leverage contextualized representations of all words in all sentences, while still utilizing the pretrained weights from BERT.

we represent all the sentences in the document as one long sequence of words with special delimiter tokens in between them.


We first examine whether it is sufficient to have just two autonomous agents to develop a common communication protocol.

we ask whether a symmetric language emerges when there are only two agents in a linguistic community, or that they learn to speak distinct idiolects.


We next examine what happens when we expose different linguistic communities to each other.

we consider two linguistic communities of population sizes N 1 and N 2 , which are trained independently from each other as fully-connected communities and have developed separate communication protocols.


This results from the fact that members of both communities are rewarded for cooperating and playing the game well (via the bridge agents).

the agents prefer to integrate or assimilate rather than segregate, similar to how it has been found that minority groups shift toward the use of dominant language (Fase et al., 1992).


The topic and context token representations are combined with the sentence-level sentence representations through an attention mechanism.

an attention score is computed between each h i and the topic and context representations.


In Table 5 we examine whether the combination of different learning rates (lr E andlr D ) is indeed beneficial.

we report model perplexity on the CNN/DailyMail validation set for varying encoder/decoder learning rates.


Moreover, we also assessed the overall quality of the summaries produced by abstractive systems which due to their ability to rewrite content may produce disfluent or ungrammatical output.

we followed the Best-Worst Scaling (Kiritchenko and Mohammad, 2017) method where participants were presented with the output of two systems (and the original document) and    Table 6: QA-based evaluation.


Our contributions can be summarized as follows: • We propose to use prototype information to help generate better summaries with patterns.

• we propose to generate the summary incorporating the prototype summary pattern and extracted facts from input document.


Core to our approach is the principle of the Information Bottleneck (Tishby et al., 1999), producing a summary for information X optimized to predict some other relevant information Y.

we map (conditional) language modeling objectives to the Information Bottleneck principle to guide the unsupervised model on what to keep and what to discard.


To make the process tractable, we introduce a two-stage approach that features abstract programs.

we decompose semantic parsing into two steps: 1) a natural language utterance is first mapped to an abstract program which is a composition of high-level operations; and 2) the abstract program is then instantiated with low-level operations that usually involve relations and entities specific to the knowledge base at hand.


Executable programs z that can query tables are defined according to a language.

the search space of programs is constrained by grammar rules so that it can be explored efficiently.


As an evaluation proxy, we measure the coverage of our grammar in terms of consistent programs.

we exhaustively search for all consistent programs for each question in the training set.


To understand how well structured attention can help a parser differentiate between correct and spurious programs, we analyzed the posterior distribution of consistent programs given a denotation: WIKISQL includes gold-standard SQL annotations, which we do not use in our experiments but exploit here for analysis.

we converted the annotations released with WIKISQL to programs licensed by our grammar.


We further manually inspected the output of our structured-attention model and the standard attention baseline in WIKITABLEQUESTIONS.

we randomly sampled 130 error cases independently from both models and classified them into three categories.


It is based on the observation that the abstract programs which can be successfully instantiated into correct programs are usually not very complex.

the consistent programs that are instantiated by long abstract programs are very likely to be spurious.


For instance, programs like "select( previous( next( previous(argmax [all rows], column:silver) column:bronze)" are unlikely to have a corresponding question.

we set the maximal number of production rules for generating abstract programs to 6 and 9, which leads to search time of around 7 and 10 hours for WIKISQL and WIKITABLEQUES-TIONS respectively, using a single CPU.


If a row slot is aligned with this token, then it is expected to be instantiated with the special condition 'all rows'.

this special token is mapped to a learnable vector during instantiations.


2: an encoder first encodes the input text into hidden states; and then conditioned on the hidden states, at each decoding time step, a decoder takes the previous semantic relation as input, and outputs a new semantic relation, which includes a target node, a relation type, and a source node.

there a significant difference between Zhang et al.


It can be considered as another attention layer over the existing graph, yet the attention weights explicitly indicate the likelihood of the new node being attached to a specific node.

its aim is to answer the question of where to expand.


Rather than iterating over all possible graphs, we adopt a beam search to approximate the best graph.

for each partially constructed graph, we only consider the top-K concepts obtaining the best single-step probability (a product of the corresponding concept, arc, and relation label probability), where K is the beam size.


We refine the ordinary Smatch metric to take into consideration the notion of core semantics.

we compute: • Smatch-weighted: This metric weights different triples by their importance of composing the core ideas.


Specifically, we compute: • Smatch-weighted: This metric weights different triples by their importance of composing the core ideas.

the root distance d of a triple is defined as the minimum root distance of its involving nodes, the weight of the triple is then computed as: the weight has a linear decay in root distance until d thr .


• Smatch-core: This metric only compares the subgraphs representing the main meaning.

we cut down AMR graphs by setting a maximum root distance d max and only keep the nodes and edges within the threshold.


As seen, both parsers make some mistakes.

our method fails to identify the concept generated-01.


To solve the above challenges, we propose a character-based Collaborative Graph Network, including an encoding layer, a graph layer, a fusion layer and a decoding layer.

there are three word-character interactive graphs in the graph layer.


Since the scale of each dataset varies, we set different training batch size for different datasets.

we set batch sizes of MSRA, OntoNotes and Weibo NER as 64, 20 and 10.


We can clearly see that removing any graph causes obvious performance degradation, but the importance of different graphs varies from dataset to dataset.

on OntoNotes and MSRA, 'w/o T-graph' obtains worse performance than others, showing that Tgraph is important.


In this paper, we study the problem what limits the performance of DS-trained neural models, conduct thorough analyses, and identify a factor that can influence the performance greatly, shifted label distribution.

we found this problem commonly exists in real-world DS datasets, and without special handing, typical DS-RE models cannot automatically adapt to this shift, thus achieving deteriorated performance.


It modifies the bias term in softmax classifiers and explicitly fits models along the shift.

the proposed method estimates the label distribution of target domain with a small development set sampled from test set, and derives the adapted predictions under reasonable assumptions.


We select three popular relation extraction datasets as benchmarks.

two of them are distantly supervised and one is human-annotated.


We consider two classes of relation extraction methods, i.e., feature-based and neural models.

feature-based models include Logistic Regression (Mintz et al., 2009), CoType-RM  and ReHession (Liu et al., 2017).


Meanwhile, we observe such performance boost starts to diminish on distantly supervised datasets.

we list the perfor- Bi-GRU outperforms ReHession with a significant gap on TACRED, but only has comparable performance (with Re-Hession) on KBP and NYT.


Max threshold and entropy threshold are designed to identify the "ambiguous" relation mentions (i.e., predicted with a low confidence) and label them as the NONE type Liu et al., 2017).

referring to the original predictions as r * = arg max r i p(y = r i |h), we formally introduce these two threshold techniques: • Max Threshold introduces an additional hyperparameter T m , and adjusts the prediction as : Otherwise .


It is apparent that the strategy of rejecting "ambiguous" predictions would guide the model to predict more NONE types, leading the predicted label distribution towards a favorable direction.

in the train set of KBP, 74.25% instances are annotated as NONE and 85.67% in the test set.


At the same time, the same technology fails to achieve performance improvements on TACRED, the human annotated dataset.

we found that, by explicitly adapting the model along label distribution shift, consistent improvements can be achieved on distant supervision but not on human annotations.


In this paper, we focus on sentence-level RE.

prediction should be purely based on the information provided within the sentence, instead of external knowledge or commonsense.


The former is designed to encourage the model to select candidates meeting the domain requirements of relations, and the latter is designed to give consideration to decisions produced by sentence level extractors.

although the sentence level extractors may make wrong decisions, the global statistics of their decisions are reliable, and should be preserved.


The first example in Figure 1 illustrates the relationship between a dependency structure and a named entity.

the word "premises", which is a named entity of type LOC (location), is characterized by the incoming arc with label "pobj" (prepositional object).


Similar to the work by Miwa and Bansal (2016), we concatenate the word representation together with the corresponding head word representation and dependency relation embedding as the input representation.

given a dependency edge (x h , x i , r) with x h as parent, x i as child and r as dependency relation, the representation at position i can be denoted as: where w i and w h are the word representations of the word x i and its parent x h , respectively.


In general, we can stack more interaction functions and BiLSTMs to enable deeper reasoning over the dependency trees.

the hid- den states of the (l + 1)-th layer H (l+1) can be calculated from the hidden state of the previous layer H (l) : where p i indicates the parent index of the word represents the interaction functions between the hidden state at the i-th and p ith positions under the dependency edges ( The number of layers L can be chosen according to the performance on the development set.


The last two columns of Table 2 show the relationships between the dependency trees and named entities with length larger than 2 for the complete dataset.

the penultimate column shows the percentage of entities that can form a complete subtree (ST) under their dependency tree structures.


The heatmap table in Figure 3 shows the correlation between the entity types and the dependency relations in the OntoNotes English dataset.

each entry denotes the percentage of the entities that have a parent dependency with a specific dependency relation.


Similarly, several works have studied crosscultural differences between Japanese and English (both American and British variants) (Koga and Pearson, 1992;Minami, 1994;Martinsons, 2001;Adachi, 2010).

adachi (1996) and Ziv (1988) investigated sarcasm in Japanese and the latter showed that american students are more sarcastic than their Japanese counterparts.


Cross-cultural differences have been studied extensively in the social science literature (Shweder, 1991;Weber and Hsee, 1998;Liu and McClure, 2001;Boroditsky et al., 2003;Yin et al., 2011;Hong et al., 2012).


A supervised learning model is trained to differentiate between a pair of languages based on their cross-cultural differences, as manifested in the available text data.

the training data contains a corpus of text documents collected from two distinct languages, with the assumption that the language of each document is known.


However, the texts need not be aligned beyond this coarse level.

full alignment between specific documents across the two languages is not required.


Train a binary classifier, denoted h, to discriminate document representations in C T 0 from those in C T 1 .

the classifier is trained to classify representations of documents as originating from either L 0 or L 1 .


In our application we seek a document embedding to assist in solving a binary classification task.

we seek a context vector c d that is most discriminative, taking advantage of additional information about some binary label.


(2) How does the choice of transformation impact the performance of the approach?

we consider the choice of machine translation and document representation.


Figure 2 provides visual explanations of documents from the evaluation datasets of both the formality classification, and sarcasm detection tasks and their corresponding LIME interpretation.

figure 2a showcases documents from the Amazon fashion and the New York Times datasets classified as informal, respectively, while figure 2b showcases documents from the Sarcasm Corpus dataset classified as sarcastic.


To compensate, we adjusted the class rationale prototype representation to: where T all is the collection of all training instances across all classes.

we subtract the entire training-set prototype vector from each class rationale prototype.


The rationalebiased representation appears to discriminate better between positive and negative instances.

we notice an increase in the distance between the representations of class prototypes and a clearer clustering of the development set instances around their respective class prototypes.


For larger training sets of size 60 and more, all systems perform better, as expected.

the more complex CNN and BERT-based models have significantly improved performance with more supervision.


The output of the Relation Module is called the relation score, representing the correlation between c i and e q , which is a scalar between 0 and 1.

we use the neural tensor layer (Socher et al., 2013) in this module, which has shown great advantages in modeling the relationship between two vectors (Wan et al., 2016;Geng et al., 2017).


Using these two facts, we can infer that P contradicts Z.

these three decisions are not independent of each other.


Annotation Consistency For labeled examples, we expect that a model should predict what an annotator specifies.

we require where Y represents the ground truth label for the example (P, H).


Symmetry Consistency Given any premisehypothesis pair, the grounds for a model to predict Contradiction is that the events in the premise and the hypothesis cannot coexist simultaneously.

a (P, H) pair is a contradiction if, and only if, the (H, P ) pair is also a contradiction: Transitivity Consistency This constraint is applicable to any three related sentences P , H and Z.


Table 4 shows the coverage of the three unlabeled datasets during the first training epoch.

we count the percentage of unlabeled examples where the symmetry/transitivity loss is positive.


Moreover, the numerator is the number of examples where the label for an example is not Y .

the value of ρ and τ represents the fraction of examples in D that are mislabeled.


This section shows the loss associated with the transitivity consistency in the NLI case study.

for an individual example (P, H, Z), applying the product t-norm to the definition of the transitivity consistency constraint, we get the loss the total transitivity loss L tran is the sum of this expression over the entire dataset.


In this paper, we propose a meta-learning approach to learn text emotion distributions from a small sample.

we propose to learn low-rank sentence embeddings by tensor decomposition to capture their contextual semantic similarity, and use K-nearest neighbors (KNNs) of each sentence in the embedding space to generate sample clusters.


For each sentence s in D, we count the word-word co-occurrence in a small window H, and build a binary matrix V s ∈ [0, 1] V ×V .

v s (i, j) = 1 indicates word w i and w j co-occurs in s within a small window H at least once.


With the tensor decomposition, we can find low-rank embeddings of sentences that capture the similarity of contextual patterns (Hosseinimotlagh and Papalexakis, 2018).

, is the embedding vector of the sentence s. We measure the similarity of two sentences i, j by Euclidean distance: , where c(r) denotes the r-th element of c. We can train a meta-learner on the generated T train .


The CNN learner has the same architecture with (Zhang et al., 2018).

given the input sentence s, we first stack a matrix X with the word vector of each word (Mikolov et al., 2013).


Similarly to (Zhang et al., 2018), we also optimize the classification accuracy of the dominant emotion by a cross-entropy (CE) objective, which is shown to improve EDL performance.

1(y j ) = 1 if y j is the maximal value of y, otherwise 1(y j ) = 0.


We can see that EDL-Meta shows better results than other methods on all metrics.

eDL-Meta can usually outper-   form other methods by about 5% on most metrics.


To investigate the effect of the proposed metatraining and adapting procedures, we conduct another experiment.

with the same ex-EDL-CNN EDL-Meta Figure 3: The performance of EDL-CNN and EDL-Meta on different training percentages.


1 Motivated by the intuition that a dialog intent is not only expressed in the user query utterance but also captured in the rest of the dialog, we split a conversation into two independent views and exploit multi-view clustering techniques for inducing the dialog intent.

we propose alternating-view k-means (AV-KMEANS) for joint multi-view representation learning and clustering analysis.


AV-KMEANS uses different neural encoders to embed the inputs corresponding to the two views and to encourage the representations learned by the encoders to yield similar cluster assignments.

we alternatingly perform k-means-style updates to compute the cluster assignment on one view and then train the encoder of the other view by predicting the assignment using a metric learning algorithm (Snell et al., 2017).


We further use two neural network encoders f φ 1 and f φ 2 to transform the two views into vec- We are interested in grouping the data points into K clusters using the multi-view feature representations.

the neural encoders corresponding to the two views are jointly optimized so that they would commit to similar cluster assignments for the same instances.


multi-view spectral clustering (MVSC) (Kanaan-Izquierdo et al., 2018) is a competitive standard multi-view clustering approach.

8 we carry out clustering using the query-view and content-view representations learned by the representation learning methods (k-means only requires query-view representations).


(2017) for training the prototypical networks.

we fix the number of query examples and the number of support examples to 15 and 5.


The Layer-wise Relevance Propagation assigns to each dimension, or feature, x d a relevance score R < 0 correspond to evidence in favor or against, respectively, the output classification.

lRP allows to identify fragments of the input playing key roles in the decision, by propagating relevance backwards.


Regarding the generation capability, we observe that VAE-MINE and LaggingVAE exhibit similar performance in terms of PPL and FPPL (sample quality) and outperform the other baselines by a significant margin.

vAE-MINE performs significantly better than others in terms of RPPL (sample diversity).


This problem can be mitigated by creating smaller representative datasets with active learning which can be used for training DNNs to achieve similar test accuracy as that using the full training dataset .

the smaller sample can be considered a surrogate for the full data.


the setup chosen while training the algorithm, in the context of deep active text classification on large datasets.

we consider two sampling biases: label and distributional bias, three algorithmic factors: initial set selection, query size and query strategy along with two trained models and four acquisition functions on eight large datasets.


(iii) We demonstrate that the actively acquired training datasets can be utilized as small, surrogate training sets with a 5x-40x compression for training large, deep text classification models.

we can train the ULMFiT (Howard and Ruder, 2018) model to state of the art accuracy at 25x-200x speedups.


for each example, a new class distribution,p i , is computed by combining p i and b i .


Our simplest ensemble is a product of experts (Hinton, 2002): Probabilistic Justification: For a given example, x, let x b be the bias of the example.

it is the features we will use in our bias-only model.


Data: We experiment with a synthetic dataset built by modifying MNLI (Bowman et al., 2015).

we add a feature that is correlated with the class label to the train set, and build an out-of-domain test set by adding a randomized version of that feature to the MNLI matched dev set.


To simulate a case where this independence is broken, we experiment with adding an additional feature that is correlated with the bias feature, but is not treated as being part of the bias (i.e., it is not used by the bias-only model).

80% of the time a token is added to the start of the hypothesis that matches the label with 90% probability, and the "0" token is appended to the end of the hypothesis.


Inspection shows that, when the indicator is 1, the bias product model is anticorrelated with the bias.

it assigns an average of 22.5% probability to the class indicated by the bias, where an unbaised model would assign an average of 33% since the bias is random.


This specification may equivalently be formulated as a set of half-space constraints on the logits: for each class y where e i is a one-hot vector with 1 in the i-th position.

the true class logit should be greater or equal than those for all other classes y, which means the prediction remains constant.


The key idea is to force the generator to explicitly understand what to generate rationales for.

we make the label that would be predicted with the full text as an additional input to the generator thereby ensuring better overall performance.


3 This problem happens because they have no control of the words unselected by R. Intuitively, in the presence of degeneration, some key predictors in X will be left unselected by R. Thus by looking at the predictive power of R c , we can determine if degeneration occurs.

when degeneration is present, all the useful information is left unselected by R, and so H(Y |R c ) is low.


Figure 1(c) illustrates the model with the introspection generator.

the improved generator still fits into the basic three-player framework in Section 3.1.


All hyper-parameters are tuned on the development sets according to predictive accuracy.

all the models are tuned without seeing any rationale annotations.


Since the average sparsity level of human-annotated rationales is about 18%, we consider the following evaluation settings.

we compare the generated rationales to human annotations by measuring the precision when extracting 10% of words and the recall for 20% 6 .


In summary, our three-player framework consistently improves the quality of extracted rationales on both of the original (Lei et al., 2016) and the introspective framework.

without controlling the unselected words, the introspection model experiences a serious degeneration problem as expected.


To this question, we investigate a neural network model for output label sequences.

we represent each possible label using an embedding vector, and aim to encode sequences of label distributions using a recurrent neural network.


BiLSTM-LAN can be viewed as a form of multi-layered BiLSTM-softmax sequence labeler.

a single-layer BiLSTM-LAN is identical to a single-layer BiLSTM-softmax model, where the label embedding table serves as the softmax weights in BiLSTM-softmax, and the label attention distribution is the softmax distribution in BiLSTM-softmax.


As mentioned in the introduction, a singlelayer BiLSTM-LAN is identical to a single-layer BiLSTM-softmax sequence labeling model.

the BiLSTM-softmax model is given by Eq 1.


Let The extreme values of the product occur at one of the four points corresponding to the products of the extreme values of the inputs.

propagating intervals through multiplication nodes therefore requires four multiplications.


Second, loss landscapes of fine-tuning partially explain the good generalization capability of BERT.

pre-training obtains more flat and wider optima, which indicates the pre-trained model tends to generalize better on unseen data (Chaudhari et al., 2017;Li et al., 2018;Izmailov et al., 2018).


As shown in Figure 1, the fine-tuning path from the start point to the end point on the loss landscape is more smooth than training from scratch.

the training loss of fine-tuning BERT tends to monotonously decrease along the optimization direction, which eases optimization and accelerates training convergence.


Our goal is to train a classifier f , parameterized by θ, which learns to accurately predict the target label, while ignoring superficial topical correlations present in the training set.

for a text x we wish to predictŷ = f θ (x) which doesn't encode any information about t. Following Ganin et al.


This training setup usually proves too difficult to optimize, and often results in poor performance.


In recent years, the fields of machine learning and NLP have seen an renewed interest in incorporating natural language supervision in models of machine intelligence (Narasimhan et al., 2015;Elhoseiny et al., 2013;Goldwasser and Roth, 2014;Fried et al., 2018;Wang et al., 2016).

methods such as Bab-bleLabble (Hancock et al., 2018) and LNL (Srivastava et al., 2017) show progress towards realistic Figure 1: Question-Answer dialog can enable learning from a mix of strategies, including label observations (traditional supervised learning), explanations and clarifications (to overcome parsing limitations).


Seeking labels for specific examples: This is similar to traditional active learning.

we can have a different action corresponding to every active learning criterion, which chooses which example to label next.


A notable issue in learning from explanations, which we do not model here is that a teacher's multiple explanations of a concept can have a large variance in their utility to a learner.

we might expect that teachers would be more likely to provide the most useful explanations first, and minor explanations subsequently.


From a qualitative analysis, learned policies are seen to be intuitive and interpretable.

the policies overwhelmingly seek clarifications when confidence in the parser is low.


guage ID is not mandatory, but helps in cases in which a word is found in the vocabulary of both languages and is pronounced differently), compatible pronunciation dictionaries and unigram word probabilities for each of the languages.

4 we compose the following FSTs: (a) an FST for converting a sentence into a sequence of phonemes, (b) an FST that allows minor changes in the phoneme sequence, (c) an FST for decoding a sequence of phonemes into a sentence, the inverse of (a).


By using MCA to scale Seq2Seq models to encode the entire graph, further gains can be seen.

in information synthesis tasks, prior work has shown the importance of reading more information.


Among them, one of the most cited approach is back translation (briefly, BT) (Sennrich et al., 2016a), which leverages the target-side monolingual data.

a target-to-source translation model (trained on the genuine bitext) is used to translate target-side monolingual sentences into the source domain to generate a set of synthetic bitext, which is then used together with the genuine bitext to train a source-to-target NMT model.


Different from the view of model architecture, in this paper, we study the NMT training from the data aspect.

we study the effect of both source-side and target-side monolingual data at scale and investigate how to make the best utilization of the monolingual data.


We have similar observation as that for En$De translations.

the noised training can improve WMTPC by 1.9 and 2.2 points on De!Fr and Fr!De.


We first investigate the effect about different combinations of monolingual data.

we compare three different ways to use monolingual data, including leveraging source-side monolingual data only (i.e.,B s ), target-side monolingual data only (i.e.,B t ), and the monolingual data from both two sides (i.e.,B s andB t ).


With the same data size, on average, the model trained with combined synthetic dataset outperforms those trained withB s only andB t only by 2.6 and 3.5 points respectively.

on newstest2018, the advantage of the combined dataset is nearly 5 points compared toB t , which is extremely significant.


In Table 4, our model performs better with all evaluation metrics on both datasets.

for 1-shot link prediction, MetaR increases by 33%, 28.1%, 29.2% and 27.8% on MRR, Hits@10, Hits@5 and Hits@1 on NELL-One, and 41.4%, 18.8%, 37.9% and 62.2% on Wiki-One, with average improvement of 29.53% and 40.08% respectively.


To remedy this without the knowledge of the test distribution, we propose an approach which trains a model that performs well over a wide range of potential test distributions.

we derive a new distributionally robust optimization (DRO) procedure which minimizes the loss of the model over the worst-case mixture of topics with sufficient overlap with the training distribution.


Yelp reviews in a Yelp-newswire mixture).

1 adding data from topics outside the test topics should not hurt.


As discussed in § 7, prior work has generally treated tagging the PPCEME as a problem of domain adaptation, with a post-processing stage to map deterministically between the tagsets.

we train on the Penn Treebank (PTB) corpus of 20th century English (Marcus et al., 1993), and then evaluate on the PPCEME test set, using a mapping between the PPCHE and PTB tagsets defined by Moon and Baldridge (2007).


The problem of processing historical English can be treated as one of unsupervised domain adaptation.

we assume that labeled data is available only in the "source domain" of contem-porary modern English, and adapt to the "target domain" of historical text by using unlabeled data.


To address this issue, we propose the AdaptaBERT model for unsupervised domain adaptation, which adds an additional fine-tuning objective: masked language modeling in the target domain.


Lalor et al., 2016;Sakaguchi and Van Durme, 2018; Kaushik and Lipton, 2018).

a common finding is that different examples within the same class have very different qualities such as difficulty, and these differences affect models' performance.


The goal here is not to build an ensemble of DNNs to surpass current classification state of the art results, but instead to test our hypothesis to determine if machine RPs can fit IRT models that can benefit NLP tasks.

we trained 1000 LSTM models for NLI classification using the SNLI data set and 1000 LSTM models for binary SA classification using the SSTB data set (Bowman et al., 2015;Socher et al., 2013).


To examine these more closely we identified those examples from the data sets where the rank order was most different between the human-and machineresponse pattern models (Table 3).

we calculated the absolute difference in ranking between the human model and the DNN model, and selected those where that value was highest.


Often examples are re-weighted according to some notion of difficulty, or model uncertainty (Chang et al., 2017).

the internal uncertainty of the model is used as the basis for selecting how training examples are weighted.


In this paper, we propose a simple, efficient, and effective model for non-autoregressive sequence generation using latent variable models.

we turn to generative flow, an elegant technique to model complex distributions using neural networks, and design several layers of flow tailored for modeling the conditional density of sequential latent variables.


To make P (Y 1 |X 1 ) and P (Y 2 |X 2 ) generalize well at test time, we hope to remove unnecessary information from X 1 and X 2 .

we want X 1 (X 2 ) to be a minimal sufficient statistic for Y 1 (Y 2 ).


(2017); Logeswaran and Lee (2018)); and (iii) contextualized word representation (e.g., Cove (McCann et al., 2017), ELMo (Peters et al., 2018).

eLMo (Peters et al., 2018) learns high-quality, deep contextualized word representation using bidirectional language model, which can be directly plugged into standard NLU models for performance boosting.


On the other hand, fine-tuning approaches mainly pre-train a language model (e.g., GPT (Radford et al., 2018), BERT (Devlin et al., 2018)) on a large corpus with an unsupervised objective, and then fine-tune the model with in-domain labeled data for downstream applications (Dai and Le, 2015;Howard and Ruder, 2018).

bERT is a large-scale language model consisting of multiple layers of Transformer blocks (Vaswani et al., 2017).


To mitigate this issue, instead of forcing the student to learn only from the logits of the last layer, we propose a "patient" teacher-student mechanism to distill knowledge from the teacher's intermediate layers as well.

we investigate two patient distillation strategies: (i) PKD-Skip: the student learns from every k layers of the teacher (Figure 1: Left); and (ii) PKD-Last: the student learns from the last k layers of the teacher (Figure 1: Right).


When the teacher changes from BERT 12 to BERT 24 (i.e., Setting #1 vs. #2), there is not much difference between the students' performance.

bERT 12 teacher performs better on SST-2, QQP and QNLI, while bERT 24 performs better on MNLI-m, MNLI-mm and RTE.


This is equivalent to maximizing the following evidence lower bound ELBO, In this case, Mean-field (Kingma and Welling, 2014) assumption is often used for simplicity.

we assume that the members of variational family Q are dimensional-wise independent, meaning that the posterior q can be writ- The simplicity of this form makes the estimation of ELBO very easy.


Now, to calculate the log copula density, we only need to do, To make sure that our model maintains the dependency structure of the latent codes, we seek to maximize both the ELBO and the joint log posterior likelihood log q(z|x) during the training.

we maximize the following modified ELBO, where L is the original ELBO.


For clarification, we are not attacking the claim made by the prior work (Vaswani et al., 2017;Shaw et al., 2018;Huang et al., 2018b;Dai et al., 2019;Child et al., 2019), but we aim at providing a new look at the order-invariance problem when considering the attention mechanism with masks (masks refer to the set filtering function in our kernel formulation).

previous work did not consider the Table 3: Order-Invariance in Attention.


State-of-the-art performance is usually obtained by fine-tuning the pre-trained model on the specific task.

bERT-based models are currently dominating the leaderboards for SQuAD 1 (Rajpurkar et al., 2016) and GLUE benchmarks 2 (Wang et al., 2018).


large number of such relations could be investigated, we chose to examine semantic role relations defined in frame semantics, since they can be viewed as being at the intersection of syntax and semantics.

we focused on whether BERT captures FrameNet's relations between frame-evoking lexical units (predicates) and core frame elements (Baker et al., 1998), and whether the links between them produce higher attention weights in certain specific heads.


To test this hypothesis we checked whether there are vertical stripe patterns corresponding to certain linguistically interpretable features, and to what extent such features are relevant for solving a given task.

we investigated attention to nouns, verbs, pronouns, subjects, objects, and negation words 6 , and special BERT tokens across the tasks.


For example, the English topical words "politician", "United States" and "president" will appear in the English page for "Donald Trump", and similar topical words in French will appear in the corresponding French page.

if we translate an article from English to French sentence-by-sentence, the word distribution of the translated article should be generally similar to the word distribution of the corresponding article in French.


(2018) and use this model to mine the sentence pairs.


Fine-tuning is performed on Multi30k Task 1 (Elliott et al., 2016).

importantly, there is no overlap in the pre-training data and the fine-tuning data.


While PG+LM achieves slightly lower LM NLL than PG+LM+G, its communication protocol drifts much more from English (middle plot).

for PG+LM, syntactic conformity is obtained at the expense of semantic preservation.


Rather than measuring the quality of representations obtained from a particular model on some auxiliary task, we characterize how the learning objective determines the information flow in the model.

we consider how the representations of individual tokens in the Transformer evolve between layers under different learning objectives.


Then we use cluster labels instead of the continuous representations in the MI estimator.

we take only representations of the 1000 most frequent (sub)words.


As the first step in our analysis, we measure the difference between representations learned for different tasks.

we compare representations for v m 1 ,l and v m 2 ,l at different layers l. Here the data is all tokens of 5000 sentences.


However, they still rely on one critical assumption which leads them to degraded performance for distant language pairs: they assume approximate isomorphism (Barone, 2016; between monolingual embedding spaces to learn the initial seed dictionary.

they assume very similar geometric constellations between two monolingual spaces: due to the Zipfian phenomena in language (Zipf, 1949) such near-isomorphism can be satisfied only for similar languages and for similar domains used for training monolingual vectors.


In this work, we investigate embedding-based approaches to encode entities from multilingual KGs into the same vector space, where equivalent entities are close to each other.

we apply graph convolutional networks (GCNs) to combine multiaspect information of entities, including topological connections, relations, and attributes of entities, to learn entity embeddings.


As shown in Figure 1, the connected relations and attributes of two equivalent entities, e.g., "University of Toronto" (English) and "ト ロ ン ト 大 学" (Japanese), have a lot of overlap, which can benefit cross-lingual entity alignment.

they share the same relation types, e.g., "country" and "almaMater", and some attributes, e.g., "foundDate" and "創立 年".


Therefore, we re-gard relations and attributes of entities as bag-ofwords features to explicitly model these two aspects.

we construct count-based Nhot vectors X r and X a for these two aspects of features, respectively, where the (i, j) entry is the count of the j-th relation (attribute) for the corresponding entity e i .


(2018), we propose the Multi-Aspect Alignment Network (MAN) to capture the three aspects of entity features.

three l-layer GCNs take as inputs the tripleaspect features (i.e., X t , X r , and X a ) and produce the representations H a according to Equation 1, respectively.


(2018), our margin-based ranking loss function is defined as: where x}, I denotes the set of negative entity alignment pairs constructed by corrupting the gold pair (e 1 , e 2 ) ∈ I.

we replace e 1 or e 2 with a randomly-chosen entity in E 1 or E 2 .


The PAIRWISEBERT model consists of two components, each of which takes as input the description of one entity (from the source or target language), as depicted in Figure 3 (right).

the input is designed as , which is then fed into PAIRWISEBERT for contextual encoding.


Following this idea, the GCN-based models (i.e., MAN and HMAN) are used to generate a candidate pool whose size is much smaller than the entire universe of entities.

gCN-based models provide top-q candidates of target entities for each source entity (where q is a hyperparameter).


Recently, BERT (Devlin et al., 2019) has advanced the state-of-the-art on multiple natural language understanding tasks.

multilingual BERT enables learning representations of sentences under multilingual settings.


We study cross-lingual stance detection, which aims to leverage labeled data in one language to identify the relative perspective (or stance) of a given document with respect to a claim in a different target language.

we introduce a novel contrastive language adaptation approach applied to memory networks, which ensures accurate alignment of stances in the source and target languages, and can effectively deal with the challenge of limited labeled data in the target language.


We conducted our experiments on English (as source language) and Arabic (as target language).

we used the Fake News Challenge dataset (Hanselowski et al., 2018) as source data and an Arabic benchmark dataset  as target data.


GIZA++ estimates IBM1-5 models and a first-order hidden Markov model (HMM) as introduced in (Brown et al., 1993) and (Vogel et al., 1996), respectively.

we perform 5 iterations of IBM1, HMM, IBM3 and IBM4.


We further establish SOCIAL IQA as a resource that enables transfer learning for other commonsense challenges, through sequential finetuning of a pretrained language model on SOCIAL IQA before other tasks.

we use SOCIAL IQA to set a new state-of-the-art on three commonsense challenge datasets: COPA (Roemmele et al., 2011) (83.4%), the original Winograd (Levesque, 2011) (72.5%), and the extended Winograd dataset from Rahman and Ng (2012) (84.0%).


We train our models on the 33k SOCIAL IQA training instances, selecting hyperparameters based on the best performing model on our dev set, for which we then report test results.

we perform finetuning through a grid search over the hyper-parameter settings (with a learning rate in {1e−5, 2e−5, 3e−5}, a batch size in {3, 4, 8}, and a number of epochs in {3, 4, 10}) and report the maximum performance.


Modules: As shown in the 6th to 8th row, removing either the Compare, Relocate or NoOp module also causes drops in the metric scores.

removing the Relocate module causes significant degrade in bridge-type questions, which solidifies our claim that relocating the attention based on the inferred bridge entity is important for compositional reasoning.


Building upon the state-ofthe-art framework for predicting effects of actions, we present a new model, called XPAD ("eXPlaining Action Dependencies") that also considers the purpose of those effects.

xPAD biases those predictions towards those that (1) explain more of the actions in the paragraph and (2) are more plausible with respect to background knowledge.


Dependency Explanation Graph: In addition to being able to predict when entities undergo state changes, the system also needs to understand the dependency relationships between steps in a procedure.

for a step s i to be executable, the system must be able to identify which previous steps are required to have been completed first.


In the context of our data generation process, we distinguish between the knowledge base, R, which contains a finite number of predicates and rules specifying how kinship relations in a family resolve, and a particular kinship graph G, which contains a grounded set of atoms specifying the particular kinship relations that underlie a single story.

r contains the logical rules that govern all the generated stories in CLUTrr, while G contains the grounded facts that underlie a specific story.


To circumvent this issue and increase the flexibility of our benchmark, we reuse and compose AMT paraphrases to generate longer stories.

we collected paraphrases for stories containing k = 1, 2, 3 supporting facts and then replaced the entities from these collected stories with placeholders in order to re-use them to generate longer semi-synthetic stories.


Most prominently, CLUTRR allows us to explicitly evaluate a model's ability for systematic generalization.

we rely on the following hold-out procedures to test systematic generalization: • During training, we hold out a subset of the collected paraphrases, and we only use this held-out subset of paraphrases when generating the test set.


• We also hold out a subset of the logical clauses during training (for clauses of length k > 2).

3 during training, the model sees all logical rules but does not see all combinations of these logical rules.


We also find that Taskmaster-1 is  more realistic than MultiWOZ.

the two-person dialogs in Taskmaster-1 involve more real-word entities than seen in MutliWOZ since we do not restrict conversations to a small knowledge base.


However, learning from purely human-human based corpora presents challenges of its own.

human conversation has a different distribution of understanding errors and exhibits turn-taking idiosyncrasies which may not be well suited for interaction with a dialog system (Williams and Young, 2007;Serban et al., 2017).


Following the recent line of work on generative dialog systems (Vinyals and Le, 2015), we treat the problem of response generation given the dialog history as a conditional language modeling problem.

we want to learn a conditional probability distribution P θ (U t |U 1:t−1 ) where U t is the next response given dialog history U 1:t−1 .


During data collection, our data associates report on the quality of each conversation.

the data associates grade the conversation on a scale from "Unusable", "Poor", "Good", to "Excellent".


(2018); throughout the rest of the paper, we use the BERTbased architecture in our experiments.

we used this baseline trained on WTC to bootstrap our approach, to be described subsequently.


Further, note that the adversarial task becomes harder with each subsequent round.

the performance of the standard models S i rapidly deteriorates between round 1 and round 2 of the adversarial task.


Trees Aiming to address the limitations of existing approaches (as discussed in previous sections), we leverage a graph convolutional network over dependency trees of sentences.

after the dependency tree 3 of the given sentence is constructed, we first attain an adjacency matrix A ∈ R n×n according to the words in the sentence.


By doing so we aim at reducing the noise and bias that may have naturally arisen from the dependency parsing process.

the function F(•) is: where q i ∈ R is the position weight to i-th token.


To better understand how ASGCN works, we present a case study with several testing examples.

we visualize the attention scores offered by MemNet, IAN, ASCNN and ASGCN-DG in Table 4, along with their predictions on these examples and the corresponding ground truth labels.


Our ASGCN-DG correctly handles all the three samples, implying that GCN effectively integrates syntactic dependency information into an enriched semantic representation.

aSGCN-DG makes correct predictions on the second and the third sample, both having a seemingly biased focus.


Word embeddings e 1 , e 2 , ..., e |x| are processed into hidden states via recurrently exploring word co-occurrence with left neighbors.

for word w n , its hidden states h n is: where f LST M (•) refers to an LSTM unit.


Here f * (•) is a ReLU-activated neural perceptron described above.

the weight matrix of f φ (•) (with the softmax normalization) are employed as the aspect-word distributions (distributional word clusters), φ, used to represent the latent aspect factors and serves as our main output.


As such, an aspect tagger trained on a source domain can work well when applied to a target domain.

at the final hop, we adopt a domain discriminator for each word with a gradient reversal layer (Ganin et al., 2016) to perform domain adversarial learning over its correlation vector (alignment).


We link these two LSTM layers so that the hidden representations generated by the LSTM B can be fed to LSTM U as the guidance information.

their hidden representations are calculated as follows: The probability scores z B i ∈ R |Y B | over the aspect boundary tags Y B ={B, I, E, S, O} are calculated by a fully-connected softmax layer: Similarly, the scores z U i ∈ R |Y U | over the unified tags Y U defined in Section 2 are obtained as: Before detailing the DMI module, we firstly introduce Global-Local Memory Interaction (GLMI) that describes the interaction between a local memory h i ∈ R dim h and a global memory m ∈ R dim h .


Formally, we parameterize the GLMI f (h i , m; Θ, G), with Θ = {W, b} and G, which consists of a residual transformation and a tensor product operation.

we firstly incorporate the global memory information m into each local position with a residual transformation where [:] denotes the vector concatenation.


For the auxiliary opinion detection task, we feed r L o,i into a softmax layer for predicting the probability scores z O i ∈ R |Y O | over the opinion labels 2 Y O , i.e., a word is an opinion word or not, as: To adapt the captured relations to be domaininvariant, we propose a Selective Adversarial Learning (SAL) method to dynamically align the words with high probability to fall into the aspect boundaries, i.e., being an aspect word.

we introduce a domain discriminator for each word, which aims to identify the domain label y D i ∈ R |Y D | of the input word, i.e., the word in the sentence is from the source or the target domain.


In this paper, we propose constrained attention networks (CAN), a simple yet effective solution, to regularize the attention for multiaspect sentiment analysis, which alleviates the drawback of the attention mechanism.

we introduce orthogonal regularization on multiple aspects and sparse regularization on each single aspect.


To alleviate the above issue, we propose a model for multi-aspect sentiment analysis, which regularizes the attention by handling multiple aspects of a sentence simultaneously.

we introduce orthogonal regularization for attention weights among multiple non-overlapping as-pects.


In summary, the main contributions of our work are as follows: • We propose CAN for multi-aspect sentiment analysis.

we introduce orthogonal and sparse regularizations to constrain the attention weight allocation, helping learn better aspect-specific sentence representations.


We follow the approach in the work (Bahdanau et al., 2015) to compute the attention.

given the sentence S with K aspects, A s = {A s 1 , ..., A s K }, for each aspect A s k , its attention weights are calculated by: where u s k is the embedding of the aspect A s k , e L ∈ R L is a vector of 1s, u s k ⊗ e L is the op-eration repeatedly concatenating u s k for L times.


Detailed statistics are summarized in Table  1.

85.23% and 83.73% of the multiaspect sentences are non-overlapping in Rest14 and Rest15, respectively.


Firstly, we observe that by introducing attention regularizations (either R s or R o ), most of our proposed methods outperform their counterparts.

aT-CaN-R s and aT-CaN-R o outperform aT-LSTM in all results; aTaE-CaN-R s and aTaE-CaN-R o also outperform aTaE-LSTM in 15 of 16 results.


We propose constrained attention networks for multi-aspect sentiment analysis, which handles multiple aspects of a sentence simultaneously.

we introduce orthogonal and sparse regularizations on attention weights.


To address the shortcomings of previous seed word-based approaches, we propose a novel weakly supervised approach, which uses the available seed words in a more effective way.

we consider a student-teacher framework, according to which a bag-of-seed-words classifier (teacher) is applied on unlabeled segments to supervise a second model (student), which can be any supervised model, including neural networks.


4 Unfortunately, unsupervised topic models are not effective when used directly for aspect detection.

in ABAE, the K topics learned to reconstruct the segments are not necessarily aligned with the K aspects of interest.


We thus change the teacher to consider seed word quality.

we replace Equation (1) by: whereẑ j is the current estimate of z j .


(2018) and estimate z j via Maximum Likelihood Estimation using the student's predictions as the current estimate of the ground truth labels.

we assume that the prediction of the student for a training segment s i is t i = argmax k p k i .


Content representation is to reflect how words appear therein, where we employ a Convolutional Neural Network (CNN) (Kim, 2014) encoder to model a turn's word sequence.

given a turn t in conversation c, we first map each word in t into a word embedding layer (initialized with pre-trained word vectors) to explore deep word semantics.


It indicates the usefulness to encode user interactions for conversation recommendation.

cONVMF is able to encode turns' temporal orders yet ignores how they reply with each other in conversation history.


To address the above problems, we design a sifted multi-task learning model with filtering mechanism (Figure 1(b)) to detect fake news by joining stance detection task.

we introduce a selected sharing layer into each task after the shared layer of the model for filtering shared features.


Figure 2: The architecture of the sifted multi-task learning method based on shared-private model.

the two blue boxes represent selected sharing layers of stance detection and fake news detection and the red box denotes shared layer between tasks.


Therefore, the embeddings of a sentence are represented as l×(dp+dw) .

we adopt one-hot encoding to embed positions of tokens, rather than sinusoidal position encoding recommended in BERT model (Devlin et al., 2018).


Attention Sharing Cell To focus on helpful shared features that are beneficial to specific tasks from upstream shared layer, we devise an attention sharing cell based on attention mechanism.

this cell utilizes input embeddings of the specific task to weight shared features for paying more attention to helpful features.


Learned from Two Tasks We visualize shared features learned from the tasks of fake news detection and stance detection.

we first look up these elements with the largest values from the outputs of the shared layer and the selected shared layer respectively.


Our approach is based on the premise that people's moral sentiments are reflected in natural language, and more specifically, in text (Bloom, 2010).

we know that books are highly effective tools for conveying moral views to the public.


However, scientists and journalists have been found to inappropriately use causal language in research publications and news articles.

causal interpretation of correlational findings from observational studies has been a major type of misinformation in science communication (Cofield et al., 2010;Sumner et al., 2014;Chiu et al., 2017;Boutron and Ravaud, 2018).


A sample of 30 abstracts were randomly selected for the inter-coder reliability test.

two annotators labelled the relation type for each sentence from the 30 conclusion subsections, and also highlighted the linguistic indicators.


This is largely caused by the confounding language cues in the no relationship examples.

a no relationship sentence that describes implications for future studies or functions of certain assessments may contain causal markers, such as "is critical to achieving", and "improved" in the following Examples 1 and 2, and thus mislead the prediction model.


To better understand what claims are about, we conduct entity linking for all claims.

mentions of people, places, organisations, and other named entities within a claim are recognised and linked to their respective Wikipedia pages, if available.


To distill the complementary information from the structural feature s i , inspired by LSTM, an input gate is designed to eliminate the information in s i that has already appeared in t i .

the gate is designed as where σ(•) is the sigmoid function; denotes the element-wise multiplication; P and b g are used to align the structural feature s i to the space of textual features t i .


Given the raw textual features r i and r j of two neighboring nodes i and j, we diminish the information that are not consistent in the two raw features.

the final textual features are computed for nodes i and j as where σ(•) serves as the role of gating.


In this paper, we advance this line of research.

we address a common shortcoming in current models by using state-of-the-art graph neural networks to encode and leverage homophily relations.


Our framework, equipped with our best model, significantly outperforms existing domain adaptation algorithms on our tasks.

adaptive ensembling achieves gains of 11.4 and 10.1 macro-averaged F1 on the binary and multi-label tasks, respectively.


Both networks share the same base model architecture and initial parameter values, but follow different training paradigms (Laine and Aila, 2017).

the student network is updated via backpropagation, then the teacher network is updated with an exponential average of the student network's parameters (Tarvainen and Valpola, 2017).


To ease the difficulty of domain adaptation, we exploit the temporal information in our datasets to introduce a curriculum (Bengio et al., 2009).

each article comes with metadata that includes the year in which the article was published.


In order for political scientists to effectively study latent variables-such as political polarization-over time, the extracted political subcorpus must contain a similar integrity as the original corpus.

the subcorpus' distribution of documents across years and sources must relatively match that of the original corpus.


We experimented with a range of hyperparameters to ensure the Latent Dirichlet Allocation (LDA) model was best optimized for our datasets, leveraging the Gensim 3 library.

we removed all stopwords, extremely rare words (tail 10-20% from a unigram distribution), and set the number of topics to 50.


In this section, we provide an ablation study to examine the contribution of each model component.

we remove the character-level word embedding, the word-level attention, the fieldlevel attention, the transformer encoders, and the country supervision signal one by one at a time.


To evaluate the impact of the hierarchical memory, we implement a simplified version of CRAFT where the memory size of the context encoder is zero (CRAFT − CE), thus effectively acting as if the pre-training component is a vanilla seq2seq model.

this model cannot capture inter-comment dependencies, and instead at each step makes a prediction based only on the utterance encoding of the latest comment.


We take a topic modeling approach to explore whether CS productions are characterized by a unique topical flavor, when compared to monolingual messages.

we compare two topic models, one created using English text from the CodeSwitch-Reddit corpus, and a second us- ing the monolingual dataset from the same subreddits (see Section 2.4).


Using the same sets of posts as in Section 3.1, we tested our hypothesis by inspecting these markers of informality in CS posts compared to monolingual posts by the same set of authors.

each author is assigned two measurements, reflecting the frequency of the informality markers in the entire collection of their code-switched posts and of their monolingual English posts.


The l th GCN layer (l ∈ [1, L]) computed over the entire conversation structure can be written as an efficient matrix operation: where H (l−1) ∈ R |C|×d l−1 and H (l) ∈ R |C|×d l denote the input and output features of all tweets in the conversation C respectively.

the first GCN layer takes the content features of all tweets as input, i.e., H (0) = (c 1 , c 2 , .


It integrates both content features and stance features learned from the bottom Conversational-GCN to facilitate the veracity prediction of the rumor.

given a conversation thread C = {t 1 , t 2 , .


We then transform the sequence to a vector v by a max-pooling function that captures the global information of stance evolution, and feed it into a one-layer feed-forward neural network (FNN) with softmax normalization to produce the predicted veracity distributionv: The loss function of C for veracity prediction is also computed by cross-entropy criterion: where v denotes the veracity label of C. To leverage the interrelation between the preceding task (stance classification) and the subsequent task (veracity prediction), we jointly train two components in our framework.

we add two tasks' loss functions to obtain a joint loss function L (with a trade-off parameter λ), and optimize L to train our framework: In our Hierarchical-PSV, the bottom component Conversational-GCN learns content and stance features, and the top component Stance-Aware RNN takes the learned features as input to further exploit temporal evolution for predicting rumor veracity.


(1), named Original-GCN) on stance classification task.

we cluster tweets in the test set according to their depths in the conversation threads (e.g., the cluster "depth = 0" consists of all source 4794   tweets in the test set).


In Section 5.4.2, to demonstrate the effectiveness of modeling the temporal dynamics of stance evolution, we replace the GRU layer by a CNN layer that only captures local temporal information.

this CNN layer consists of a 1D convolution layer and a max-pooling function (Kim, 2014).


To overcome the limitations of IntentCapsNet, we propose to reconstruct capsule networks for zeroshot intent classification.

we introduce two modules to capsule networks: (1) a dimensional attention module that helps to extract more representative semantic capsules and (2) a new method for constructing the transformation matrices to improve the model generalization ability to unseen intents.


To tackle the first problem, we propose to learn a Mahalanobis distance metric to measure the relationship between unseen and seen intents.

given the embeddings of an unseen intent l and a seen intent k, their squared Mahalanobis distance is given by: where Ω is a learnable covariance matrix which models the correlation between dimensions of the embedding.


Based on this, to solve the second problem, we propose to derive the transformation matrices of unseen intents using the transformation matrices of seen intents and the similarities between unseen and seen intents, and then make predictions for unseen intents with the transformation matrices.

given a matrix Q ∈ R L×K that encodes the similarities between unseen and seen intents, for an unseen intent l, we propose to construct the transformation matrix W lr for the r-th semantic capsule with respect to the l-th unseen intent by: where q lk is the element in the l-th row and the kth column of Q, W kr is the transformation matrix for the r-th semantic capsule with respect to the kth seen intent.


Since the original amount of recruitment data is huge, we randomly sample a fraction of the entire data, containing job postings and resumes within a time period of six months.

the dataset has covered six job domains: two major domains have a large number person-job pairs and four Train shared .


In this paper, we propose a novel heterogeneous graph neural network based method for semi-supervised short text classification, leveraging full advantage of few labeled data and large unlabeled data through information propagation along the graph.

we first present a flexible HIN (heterogeneous information network) framework for modeling the short texts, which can integrate any type of additional information as well as capture their relations to address the semantic sparsity.


In this work, we propose a novel heterogeneous graph neural network based method for semisupervised short text classification, which makes full use of both limited labeled data and large unlabeled data by allowing information propagation through our automatically constructed graph.

we first present a flexible HIN framework for modeling the short texts, which is able to incorporate any additional information (e.g., entities and topics) as well as capture the rich relations among the texts and the additional information.


To further enrich the semantics of short texts and advance the information propagation, we consider the relations between entities.

if the similarity score (cosine similarity) between two entities, computed based on their embeddings, is above a predefined threshold δ, we build an edge between them.


Unfortunately, GCN cannot be directly applied to the HIN for short texts due to the node heterogeneity issue.

in the HIN, we have three types of nodes: documents, topics and entities with different feature spaces.


Given a specific node v, the type-level attention learns the weights of different types of neighboring nodes.

we first represent the embedding of the type τ as  where μ τ is the attention vector for the type τ , || means "concatenate", and σ(•) denotes the activation function, such as Leaky ReLU.