In world (a), Event-TransR can precisely predict events in three out of four relations.

we can contrast the fact that "John died" with "John survived," which has not been addressed in previous works.


Copy mechanism (See et al., 2017) is incorporated to directly copy words from inputs, because words in paragraphs or source questions are of great value for unanswerable question generation.

we use s t and c t to produce a gating probability g t : where W g and b g are learnable parameters.


The Levenshtein distance between the answerable and unanswerable questions in pairs is 3.5 on average.

the 17, 475 pairs extracted from the SQuAD 2.0 training set are used to train generation models.


We first generate unanswerable questions using the trained generation model.

we use EM / F1 BERT Base 74.3/77.4 -+ TFIDF 75.0/77.8 +0.7/+0.4 + RULE 75.6/78.5 +1.3/+1.1 + SEQ2SEQ 75.5/78.2 +1.2/+0.8 + PAIR2SEQ 76.4/79.3 +2.1/+1.9 Table 5: Results using different generation methods for data augmentation. "


We also fine-tune the model using incorrect paragraphs selected by the retrieval system.

we retrieve 30 paragraphs and select the eight paragraphs with the lowest y empty scores predicted by the trained model.


Next, we use single-paragraph BERT to adversarially select the eight distractor paragraphs from these 50 candidates.

we feed each paragraph to the model and select the paragraphs with the lowest y empty score (i.e., the paragraphs that the model thinks contain the answer).


We suspect this is also due to the insufficiencies of standard TF-IDF retrieval for multi-hop questions.

table 5 shows that single-paragraph BERt achieves 53.12 F1 even when using 500 distractors (rather than eight), indicating that 500 distractors are still insufficient.


Since a question needs to be matched with different relations and each relation is only described by part of the question, instead of matching the relations with a single question vector, we calculate the matching score in a more finegrained way.

we first use r to attend each question token and then model the matching s r by a dot product as follows: In addition to the question-relation similarities, we find another binary indicator feature derived from the topic entity is very useful.


To simulate the real-world scenarios, we test our models following the settings of (Sun et al., 2018), where the KB is 10%   (Miller et al., 2016) is a simple baseline that treats KB triples and documents as memory cells.

we consider its two variants, KV-KB and KV-KB+Text.


Eyeballing the results, there were a fair number of false positives (words and phrases that were found by spaCy but were not actually named entities), especially in Illegal Onion sites.

slang terms for drugs, as well as abbreviated drug terms, for example "kush" or "GBL", were being falsely picked up by spaCy.


In order to isolate what factors contribute to the classifiers' performance, we experiment with four manipulations to the input data (in training, validation and testing).

we examine the impact of variations in the content words, function words and shallow syntactic structure (represented through POS tags).


In this section, we present an unsupervised embedding based algorithm that iteratively generates ranked concept candidates from an external knowledge base KB.

we build a boundary to avoid semantic drift based on the following concept space assumption.


The whole process is summarized in Algorithm 1.

due to the huge number of operations for finding nearest concepts, we use K-D Tree 8 to store concept vectors, which greatly improves the time efficiency.


Taking "Floyd Algorithm" as an example, its search path, "BFS → InstanceOf → Graph Algorithms → Instance → Floyd Algorithm", indicates that "Floyd Algorithm is a sibling of course concept "BFS".

to make effective use of this semantic information, we employ an RNN encoder-decoder neural network (Cho et al., 2014) to encode path(e) for candidate e. we train the neural network to take path(e) as input and output the same sequence.


We define a threshold α ∈ [0, 1] to control the reranking range and adjust rankings after α * |E c |.

we sort positive and negative results separately according to their confidence score s and then place the positive results before negative ones.


To ensure data quality, we avoid users' irresponsible deletion by employing a group-vote scoring mechanism.

when a user deletes the expanded concept e i , he/she gets a score Q = del(e i )/ max e j ∈Ec del(e j )− 1 2 , i.e., every user operation corresponds to a score 9 S ∈ − 1 2 , 1 2 based on all existing deletion data, which means that irresponsible operations subject to a penalty.


The results demonstrate extra information provided by classifier effectively lifts the recall in low-confidence area (latter 70% in average).

our model uses a gamified form to build a human interface.


We tuned the hyperparameters in our approach on the validation set via a grid search.


As we see, the introduction of our method to the standard backbone model (Trans.-Base) leads to substantial improvements across the validation and test sets.

our approach achieves an average gain of 2.25 BLEU points and up to 2.8 BLEU points on NIST03.


The following systems are involved: RNNsearch: Our implementation of an improved model as described in Section 2, where the decoder employs two GRUs and an attention.

equation 6 is substituted with: Besides, in equation 3, s j−1 is replaced withs j−1 .


While our approach OR-NMT can outperform the baseline system RNNsearch and the competitive comparison systems by directly incorporate the sentencelevel oracle and noise perturbations for relieving the overcorrection problem.

our OR-NMT significantly outperforms the RNNsearch by +2.36 BLEU points averagely on four test datasets.


Figure 7 shows the BLEU scores of generated translations on the MT03 test set with respect to the lengths of the source sentences.

we split the translations for the MT03 test set into different bins according to the length of source sentences, then test the BLEU scores for translations in each bin separately with the results reported in Figure 7.


However, BLEU is not without problems when used as a training criterion.

since BLEU is based on n-gram precision, it aggressively penalizes lexical differences even when candidates might be synonymous with or similar to the reference: if an n-gram does not exactly match a sub-sequence of the reference, it receives no credit.


To define an exact metric, we reference the burgeoning field of research aimed at measuring semantic textual similarity (STS) between two sentences (Le and Mikolov, 2014;Pham et al., 2015;Wieting et al., 2016;Hill et al., 2016;Conneau et al., 2017;Pagliardini et al., 2017).

we start with the method of Wieting and Gimpel (2018), which learns paraphrastic sentence representations using a contrastive loss and a parallel corpus induced by backtranslating bitext.


Hence, combining objects and actions with a small set of specific semantic relations allows the representation of a broad range of domains with a simple computational structure.

in the health domain, research in knowledge discovery techniques has steadily increased motivated by the potential impact in the quality of human life.


∆-representation learning aims to decouple, learn, and fuse alterable ∆parts for event representation, instead of learning a single comprehensive representation.

we decouple an event representation r ed into three parts r ed = r w r d r g (Section 2), where r w is the pre-trained word representation of trigger words, r d is the lexical-specific event representation which captures discrimination knowledge for distinguishing ambiguous triggers, r g is the lexical-free event representation which captures generalization knowledge for detecting unseen/sparse triggers, and is the fusion function to fuse different parts.


Based on this observation, our system needs to rely more on r d to detect frequent candidate fired, but more on r g to detect the OOV candidate hacked.

we need to adaptively fuse different representations for different words, rather than simply concatenate them.


To achieve this goal, we adopt adversarial techniques and design a lexical-adversarial ∆-learning algorithm.

we remove the lexical-specific part in Figure 2 and show the lexical-adversarial ∆learning framework in Figure 3 (b).


The above adversarial loss leads two different optimized directions for Θ g and Θ DeLexi , which can be implemented by a gradient reversal layer (Ganin et al., 2016) during backpropagation.

l minmax is jointly optimized with the main ED task objective l event , while gradients from adversarial loss are reversed with the factor λ adv when they reach r g .


This paper proposes a new representation learning framework -∆-learning, which can distill both discrimination and generalization knowledge for event detection.

two effective ∆-learning algorithms are proposed to distill discrimination and generalization knowledge independently, and a lexical gate mechanism is designed to fuse different knowledge adaptively.


Although the lattice LSTM model can exploit word and word sequence information, it still could be severely affected by the ambiguity of polysemy.

these models cannot handle the polysemy of words with the change of language situation.


In addition, we leverage position embeddings to specify entity pairs, which are defined as the relative distances from the current character to head and tail entities (Zeng et al., 2014).

the relative distances from the i-th character c i to the two marked entities are denoted as p 1 i and p 2 i respectively.


This situation indicates that considering multiple senses only in the pre-trained stage would add noise to the word representations.

the word representation tends to favor the commonly used senses in the corpora, which will disturb the model when the correct sense of the current word is not the common one.


Consequently, we argue that the syntactic and semantic information should be allowed to interact earlier in the modeling process to produce more effective representations for EFP.

we propose a novel method to integrate syntactic and seman-tic structures of the sentences based on the graph convolutional neural networks (GCN) (Kipf and Welling, 2016) for EFP.


BERT is a pre-trained language representation model with multiple computation layers that has been shown to improve many NLP tasks.

the sentence (x 1 , x 2 , ..., x n ) would be first fed into the pre-trained BERT model from which the contextualized embeddings of the words in the last layer are used for further computation.


In order to learn the weight matrix A, as presented in the introduction, we propose to leverage both semantic and syntactic structures of the input sentence.

for the semantic structure, we use the representation vectors from LSTMs for x i and x j (i.e., h i and h j ) as the features to compute the contribution score in the cell a sem ij of the semantic weight matrix A sem = (a sem ij ) i,j=1..n : Note that we omit the biases in the equations of this paper for convenience.


Finally, the weight matrix A for GCNs would be the linear combination of the sematic structure A sem and the syntactic structure A syn with the trade-off λ: Given the weight matrix A, the GCNs (Kipf and Welling, 2016) are applied to augment the representations of the words in the input sentence with the contextual representations for EFP.

let H 0 be the the matrix with (h 1 , h 2 , .


We devise a method for embedding timexes that more explicitly reflects their temporal status.

we sample pairs of time expressions from synthetic data, train character LSTM models to encode these time expressions and classify their temporal ordering.


To tackle this problem, we propose a novel end-to-end deep network model for reading comprehension, which we refer to as Episodic Memory Reader (EMR) that sequentially reads the input contexts into an external memory, while replacing memories that are less important for answering unseen questions.

we train an RL agent to replace a memory entry when the memory is full, in order to maximize its QA accuracy at a future timepoint, while encoding the external memory using either the GRU or the Transformer architecture to learn representations that considers relative importance between the memory entries.


We pose this memory management problem as a learning problem and train both the memory representation and the scheduling agent using reinforcement learning.

we propose to train the memory module itself using reinforcement learning to replace the most uninformative memory entry in order to maximize its reward on a given task.


However, when t > N , when the memory becomes full, it selects an existing memory entry to delete.

it outputs an action based on π(i|S (t) ), which denotes the selection of i th memory entry to delete.


EMR-biGRU A major drawback of EMR-Independent is that the evaluation of each memory depends only on the input x (t) .

the importance is computed between each memory entry and the new data instance regardless of other entries in the memory.


A policy that replaces the newest data instance.

it first fills in the memory and then discards all following data instances.


However, we find that the graph-based features (Leakage and Advanced) make the problem feasible on a wide range of datasets.

on the datasets like QuoraQP and ByteDance, the leakage features are even more effective than the unlexicalized features.


In this paper, we introduce query-agnostic indexable representations of document phrases that can drastically speed up open-domain QA.

our dense-sparse phrase encoding effectively captures syntactic, semantic, and lexical information of the phrases and eliminates the pipeline filtering of context documents.


The dense vector is effective for encoding syntactic and semantic cues, while the sparse vector is good at encoding precise lexical information.

the embedding of each span (i, j) in the document x k is represented with This is also illustrated in Figure 1 right.


The inner product allows more coherent phrases to have more similar start and end encodings.

where • indicates inner product operation and We use term-frequency-based encoding to obtain the sparse embedding s k i:j for each phrase.


That is, where • indicates inner product operation and We use term-frequency-based encoding to obtain the sparse embedding s k i:j for each phrase.

we largely follow DrQA  to construct 2-gram-based tf-idf, resulting in a highly sparse representation (d d ≈16M) for each document.


This allows us to model the dense query embedding differently from the dense embedding in the phrase index while sharing all parameters of the BERT encoder.

given the contextualized word representations of the question, we obtain the the dense query vector by where h 1 1 is the encoding corresponding to the (first) special token and we obtain the others in a similar way.


Hence (1) training the models, (2) indexing the embeddings, and (3) performing inner product search at inference time are non-trivial for both (a) computational time and (b) memory efficiency.

we carry out this section assuming that we have a constrained hardware environment of 4 P40 GPUs, 128 GB RAM, 16 cores and 2 TB of PCIe SSD storage, to promote reproducibility of our experiments under academic setting.


One straightforward way to define the loss is to define it as the negative log probability of the correct answer where Pr(i, j) ∝ exp(l i,j ).


We reduce the size of each vector by quantization.

we convert each float32 value to int8 with appropriate offset and scaling.


Motivated by that, we propose neural variational language model (NVLM).

our framework consists of two probabilistic components: a constituency parser and a joint generative model of sentence and parse tree.


The parser takes as input a sentence x and predicts the corresponding parse tree y.

we use an encoder-decoder structure to parameterize the parser.


11in Appendix E. In order to stabilize the learning process, we use standard variance reduction techniques to reduce the variance of gradient (Greensmith et al., 2004;Mnih and Gregor, 2014;.

we first standardize the signal (rescaling it to zero mean and unit variance) and then subtract a baseline function b(x).


(2016), we use importance sampling technique to estimate P (x).

we use our parser P ✓ 1 (y|x) as the proposal distribution.


Combined Effect The impact of our features and application logic filtering is much more substantial for GMDP models, both on average and for the four domains where GMDP outperforms ADAGRAD (four rightmost domain columns of the table).

while in these four setups the combination of GMDP training with the features and filtering yields a substantial improvement over AdaGrad training with these additions, excluding either the features or the filtering often results in an advantage for AdaGrad training.


Intermediate Training Given the robust success of LM pretraining, we explore methods of further improving on such sentence encoders.


The underline idea of our HSP model is decomposition and integration.

as shown in Figure 1, our HSP model first decomposes a complex question into sub-question sequence with a question decomposer (QD), and then extracts key semantic information based on the generated sub-questions and the original complex question with an information extractor (IE).


We apply global max pooling to construct the encoding for each sentence.

we apply an element-wise max operation over the temporal dimension of the hidden states.


DisSent and InferSent do well on different sets of tasks.

disSent outperforms In-ferSent on TREC (question-type classification).


Although DisSent is trained on a dataset derived from the same corpus as SkipThought, DisSent almost entirely dominates SkipThought's performance across all tasks.

on the SICK  dataset, DisSent and SkipThought perform similarly on the relatedness task (SICK-R), but Dis-Sent strongly outperforms SkipThought on the entailment task (SICK-E).


We show in Figure 3 that we tend to have low distances overall for the successfully aligned pairs.

whenever our extraction pipeline yields a match, the dependency parsing patterns do extract high quality training pairs.


These residuals account for 64% of the variance in the balanced confusions (R 2 = 0.6431, F (1, 223) = 401.8, p < .001).

we come close to predicting the balanced confusions from the unbalanced ones.


In this work, we experiment with spectral methods of signal representation and summarization as mechanisms for constructing such word-sequence embeddings in an unsupervised fashion.

we explore an algorithm rooted in fluid-dynamics, known as higher-order Dynamic Mode Decomposition, which is designed to capture the eigenfrequencies, and hence the fundamental transition dynamics, of periodic and quasi-periodic systems.


2010, annotators decide relative orders instead of a complete order over all systems.

4 system outputs are randomly grouped into 2 pairs to make 2 comparisons.


Table 4 shows the SEMBLEU and SMATCH scores several recent models.

we asked for the outputs of Lyu (Lyu and Titov, 2018), Gros (Groschwitz et al., 2018), van Nood (van Noord and Bos, 2017) and Guo (Guo and Lu, 2018) to evaluate on our SEMBLEU.


2017), while our reranker applies domain-general quality-measuring features compatible with both domain-specific (e.g., λcalculus) and general-purpose (e.g., Python) MRs (more in § 3).

our reranker mainly uses two features, whose scores are given by two external models: a reconstruction model and a matching model.


Similar models have also been applied for pragmatic inference in instruction-following agents for modeling the likelihood of causing the speaker to produce the utterance given an inferred action (Fried et al., 2018), while we use p(z →x) as one qualitymeasuring feature in our reranker.

we implement p(z → x) using an attentional sequence-to-sequence network (Luong et al., 2015), which takes as input a tokenized MR z.


(2018), we allow some source names to be changed or swapped in an as-graph constant after their original assignments.

after we build a constant according to the edge grouping and source assignments described above, we generate multiple variants of the constant that have different source names.


We complement the state with an attention over the input sentence (Bahdanau et al., 2014).

we use general attention (Luong et al., 2015).


This method allows the use of an external evaluation measure as reward (Paulus et al., 2017).

we want to maximize the expected Smatch reward, where p θ is the policy specified by the parser parameters θ and g s is a graph sampled from p θ .


We observe a consistent trend across both of our metrics, with the tasks encoded in a natural progression: POS tags processed earliest, followed by constituents, dependencies, semantic roles, and coreference.

it appears that basic syntactic information appears earlier in the network, while high-level semantic information appears at higher layers.


We observe the same general ordering on the 12-layer BERT-base model ( Figure A.2).

there appears to be a "stretching effect", where the representations for a given task tend to concentrate at the same layers relative to the top of the model; this is illustrated side-by-side in Figure A.3.


Because many annotations are uninteresting -for example, 89% of part-of-speech tags are correct at layer 0 -we use a heuristic to identify ambiguous sentences to visualize.

6 Figure 3 shows two 6 we look for target edges (s1, s2, label) where the highest scoring label has an average score τ (label|s1, s2) ≤ 0.7, and look at sentences with more than one such edge.


We obtain videos from seasons 1-8 and segment episodes using laughter cues from its audience.

we use open-source software for laughter detection (Ryokai et al., 2018) to obtain initial segmentation boundaries and fine-tune them using the subtitles' timestamps.


lowed to extract each of them is described below: Text Features: We represent the textual utterances in the dataset using BERT (Devlin et al., 2018), which provides a sentence representation u t ∈ R dt for every utterance u.

we average the last four transformer layers of the first token ([CLS]) in the utterance -using the BERT-Base model -to get a unique utterance representation of size d t = 768.


In this paper, we tackle these tasks in the context of complex arguments on a diverse set of topics.

our dataset consists of manually curated argument trees for 741 controversial topics covering 95,312 unique claims; lines of argument are generally of depth 2 to 6.


To this end, we consider a neural network grammar with latent variables.

we employ a grammar as the backbone of our approach in which nonterminals represent sentiment signals and grammar rules specify sentiment compositions.


Depending on how a sentiment label is represented, we develop three increasingly complex models.

the first model, which is introduced in Section 4.1, uses a weighted grammar to model the first-order correlation between sentiment labels in a tree.


Sentiment grammars provides a principled way for explicitly modeling sentiment composition, and through parameterizing the emission rules with neural encoders, it can take the advantage of deep learning.

by adding a weighted grammar on top of a tree-LSTM, our model is reminiscent of LSTM-CRF in the sequence structure.


We optimize the objective with gradient-based methods.

gradients are first calculated over the sentiment grammar layer, before being back-propagated to the tree-LSTM layer.


The performance improvement of ConTree+WG over ConTree reflects the benefit of handling sentiment composition explicitly.

the phrase level binary classification task, Con-Tree+WG improves the accuracy by 2 points.


The obvious promotion of BCN+LVG4 and BCN+LVeG shows that explicitly modeling sentiment composition with fine-grained sentiment subtypes is useful.

bCN+LVeG improves the sentence level classification accurracies by 1.4 points (fine-grained) and 0.7 points (binary) compared to bCN (our implementation), respectively.


In this work we focus on Japanese and show the potential use of transfer learning techniques in text classification.

we perform binary and multi-class sentiment classification on the Rakuten product review and Yahoo movie review datasets.


These models should be evaluated on our adversarial dataset.

it will be interesting if Botschen et al.


Reason Encoder: Then we extract reasons from the utterances, which is the main contribution of this work.

we focus on the major reasons that most people are concerned with, which possess two properties: 1) they are focal points mentioned to support a specific stance; 2) they recur in multiple utterances.


In this section, we study what has been learned in the reason encoder of RCN.

we show the attentive activations in the reason en- Reason 1: @HillaryClinton is a liar & corrupt .


Interestingly, adding out-of-domain data improved F1 of all classifiers except SVM-BoNG tfidf .

the precision of the MLP classifiers increased by transfer learning.


To alleviate these problems above, we propose a novel embedding refinement method to obtain context-aware embedding for TABSA.

we use a sparse coefficient vector to select highly correlated words from the sentence, and then adjust the representations of target and aspect to make them more valuable.


This work has studied span representations for ASP.

we have investigated (i) an LSTM-minus-based span representation originally developed for other NLP tasks and (ii) a task-specific extended representation capturing the AM/AC distinction for ASP.


Our model borrows the idea of a translational scoring function from , which learns embeddings such that for a given valid triple t k ij = (e i , r k , e j ), the condition h i + g k ≈ h j holds, i.e., e j is the nearest neighbor of e i connected via relation r k .

we try to learn entity and relation embeddings to minimize the L1-norm dissimilarity measure given by d We train our model using hinge-loss which is given by the following expression where γ > 0 is a margin hyper-parameter, S is the set of valid triples, and S denotes the set of invalid triples, given formally as replace head entity ∪ {t k ij | e j ∈ E \ e j } replace tail entity Our model uses ConvKB (Nguyen et al., 2018) as a decoder.


We assume the Distributional Inclusion Hypothesis (DIH) which states that a word (relation) r entails another word (relation) q if and only if in any context that r can be used, v can be used in its place (Dagan et al., 1999;Geffet and Dagan, 2005;Kartsaklis and Sadrzadeh, 2016).

in a correct and complete knowledge graph, we have: X r,e 1 ,e 2 = 1 → X q,e 1 ,e 2 = 1 =⇒ X r,e 1 ,e 2 ≤ X q,e 1 ,e 2 .


Note that the amount of the data that is used for all the methods is the same.

we have only used the triples from the NewsSpike corpus for both link prediction and entailment detection tasks and the gain in performance of the both tasks is merely because the two tasks learn complementary information.


For example, the well-known TransE model (Bordes et al., 2013) captures relational similarity between entity pairs by considering a translation vector for the relations connecting them.

it learns embeddings for entities and relations such that e 2 − e 1 ≈ r for any correct triple (r, e 1 , e 2 ).


The most simple approach to modeling text pairs using the VAE framework is to consider two text sequences separately (Zhao et al., 2018;Shen et al., 2018a).

a generator is trained to reconstruct a single input sequence rather than integrating both sequences, and the two latent representations encoded from a variational posterior are given to a classifier network.


Due to this particular choice of q φ , z s depends only on x s and is independent of the label information possibly permeated in x t .

this design induces q φ to extract the features needed for controlling the semantics only from x s , while preventing q φ from encoding other biases.


This constraint enforces the sequence x t generated by conditioning onỹ and z s to actually have the relationshipỹ with x s .

we also introduce a constraint on z that keeps the distributions of z t (the latent content variable obtained by encoding the generated sequence x t ) and z s close: it pushes the generated sequence x t to be in a similar semantic space with the ground-truth target sequence x t .


To assess the effect of each element, we experiment with model variants where some of the components are removed.

we conduct an ablation study for the following variants: (i) without cross-sentence generation (i.e.


(2019), contains 877K tuples covering a variety of social commonsense knowledge around specific event prompts (e.g., "X goes to the store").

aTOMIC distills its commonsense in nine dimensions, covering the event's causes (e.g., "X needs to drive there"), its effects on the agent (e.g., "to get food") and its effect on other direct (or implied) participants (e.g., "Others will be fed").


This feature is narrative in the sense that it takes into account whether an entity is central to the story in the text.

we observed that many sentences which shared an event hierarchy also share some coreferring mentions beside events argument.


We compromise and use two-sentence generations.

in the two-sentence case, we find ourselves in a Goldilocks zone wherein generations are challenging for deep models, yet as we shall soon see, easy for humans.


However, despite its success, it remains unclear whether the representations produced by BERT can be utilized for tasks such as commonsense reasoning.

it is not clear whether BERT shed light on solving tasks such as the Pronoun Disambiguation Problem (PDP) and Winograd Schema Challenge (WSC).


In all our experiments, we used the out-of-thebox BERT models without any task-specific finetuning.

we use the PyTorch implementation of pre-trained bert − base − uncased models supplied by Google 1 .


1 (bottom), our method outperforms the best system in the 2016 competition (58.3%) by a large margin.

we achieve 68.3% accuracy, better than the more recently reported results from (Liu et al., 2017) (66.7%), who makes use of three KBs and a supervised deep network.


However, all these approaches require the answer to explicitly exist in the text.

they are unable to resolve pronouns in light of abstract/implicit referrals that require background knowledge -see (Saba, 2018) for more detail.


Attracted by the success of recently proposed language representation model BERT, in this paper, we introduce a simple yet effective reimplementation of BERT for commonsense reasoning.

we propose a method which exploits the attentions produced by BERT for the challenging tasks of PDP and WSC.


The authors employ Amazon Mechanical Turk to collect 8k conversations with 127k QA pairs.

they pair two crowd-workers: a questioner and an answerer to chat about a passage.


(2018), we use denoising autoencoders along with back-translation (described in Section 3.1.1) for pretraining the language models in both the primary and secondary languages.

we first train the model to reconstruct their inputs, which will expose the model to the grammar and vocabulary specific to each language while enforcing a shared latent-space with the help of the shared encoder and decoder layers.


We initialize the encoder and decoder weights using unsupervised pretraining and finetune these weights further during the supervised QG model training.

in each step of training, we update the weights W Esec , W E shared , W D shared and W Dsec using QG data in the secondary language and W E pri , W E shared , W D shared and W D pri using QG data in the primary language.


3) Controllable trade-off : the trade-off between content preservation and style polarity could be tuned in our method.

we tune it by biasing the number of operation steps.


To stabilize the high-level policy learning, we introduce auxiliary supervision to the pointer.

we extend the pointer to an attentionbased classifier, i.e., for j = 1, 2.


Low-Level Policy Gradients All the extrinsic and intrinsic rewards are used for low-level policy learning.

the rewards for φ 1,2,3 are For φ = φ 1 , φ 2 , φ 3 , the policy gradient is Overall Objectives The overall objectives for θ are the classification loss in Eq.


As discussed in §4.3, we adopt a heuristic inference policy for operator selection.

we enumerate each operator and select the operated sentencex 2 which maximizes the criterion: where LM 2 (x 2 ) denotes the probability ofx 2 computed by the language model LM 2 , p(s j |•) is the classifier defined in §4.4.2, and η is a balancing hyper-parameter.


However, the extraction model presented in that work is limited to the closed-domain setting of basketball game tables and summaries.

they assume that each table has exactly the same set of attributes for each entity, and that the entities can be identified in the text via string matching.


In this section we compare several automatic evaluation metrics by checking their correlation with the scores assigned by humans to table-to-text models.


We check how different components in the computation of PARENT contribute to its correlation to human judgments.

we remove the probability w(g) of an ngram g being entailed by the table from Eqs.


Precision Recall F-score 0.428 0.310 0.351 employ the Pointer-Generator Network (PG-Net) (See et al., 2017) for this purpose.

we use a sequence-to-sequence model, whose encoder and decoder are both single-layer bi-directional LSTMs.


We overcome this issue by leveraging recent progress in unsupervised machine translation Lample and Conneau, 2019;Artetxe et al., 2018).

we collect a large corpus of natural questions and an unaligned corpus of cloze questions, and train a seq2seq model to map between natural and cloze question domains using a combination of online back-translation and de-noising auto-encoding.


For this, we need large corpora of cloze questions C and natural questions Q. Cloze Corpus We create the cloze corpus C by applying the procedure outlined in Section 2.2.2.

we consider Noun Phrase (NP) and Named Entity mention (NE) answer spans, and cloze question boundaries set either by the sentence or sub-clause that contains the answer.


we apply attention, LIME and input perturbation (IP).


The preference for the difficult to say option can be explained by the fact that in many cases, both models were explained with the same or very similar fact lists.

we found that IP provided identical top five fact lists in 120 out of 200 tasks.


To measure how well word embeddings are mixed across languages, we draw on concepts from network science.

some cross-  (b) high modularity Figure 1: An example of a low modularity (languages mixed) and high modularity cross-lingual word embedding lexical graph using k-nearest neighbors of "eat" (left) and "firefox" (right) in English and Japanese.


In our setting, the language l of each word defines its group, and high modularity indicates embeddings are more similar within languages than across languages (Newman, 2003;Newman and Girvan, 2004).

good embeddings should have low modularity.


As a third downstream task, we turn to an important task for low-resource languages: lexicon expansion (Gupta and Manning, 2015;Hamilton et al., 2016) for document retrieval.

we start with a set of EN seed words relevant to a particular concept, then find related words in a target language for which a comprehensive bilingual lexicon does not exist.


Methodologically, we introduce a new paired-sample multiplicative mixed-effects model to obtain language difficulty coefficients from at-least-pairwise parallel corpora.

the model is aware of inter-sentence variation and can handle missing data.


5 If we were to assume that our language models were perfect in the sense that they captured the true probability distribution of a language, we could make the former claim; but we suspect that much of the difference can be explained by our imperfect LMs rather than inherent differences in the expressed information (see the discussion in footnote 3).

the crude tools we use are recurrent neural network language models (RNNLMs) over different types of subword units.


Our mixedeffects model assumes that this multiplier is lognormally distributed over the sentences : that is, log( / ) ∼ N ( , 2 ), where mean is the difficulty of language .

/ = exp( + ) where ∼ N (0, 2 ) is residual noise, yielding equations (1)-(2).


10 We jointly fit the intent sizes and the language difficulties .

9 we deal with data missing completely at random (MCAR), a strong assumption on the data generation process.


We did experiment with priors but found them so quickly overruled by the data that it did not make much sense to spend time on them.

for full inference, we implemented all models in STAN (Carpenter et al., 2017), a toolkit for fast, state-of-the-art inference using Hamiltonian Monte Carlo (HMC) estimation.


We still see German as the hardest language, but almost all other languages switch places.

we can see that the variance of the char-RNNLM is much higher.


The results are again not quite as striking as we would hope.

in Mood's median null hypothesis significance test neither 26A ( > .3 / .7 for BPE/char-RNNLM) nor 81A ( > .6 / .2 for BPE/char-RNNLM) show any significant differences between categories (detailed results in Appendix F.1).


We have to find a tradeoff: take as many Bibles as possible that share as many verses as possible.

we cast this selection process as an optimization problem: select Bibles such that the number of verses overall (i.e., the number of verses shared times the number of Bibles) is maximal, breaking ties in favor of including more Bibles and ensuring that we have at least 20000 verses overall to ensure applicability of neural language models.


For that reason, we argue that, while recent research on cross-lingual word embeddings has almost exclusively focused on mapping methods, future work should consider alternative approaches to try to overcome the limitations of this paradigm.

we believe that an interesting direction would be to adapt joint methods so they can work with monolingual corpora.


This analysis calls for further research on alternatives to current mapping methods, which have been very successful on unsupervised settings.

we would like to explore new methods to jointly learn cross-lingual embeddings on monolingual corpora.


(2019) recently showed that BLI results do not always correlate well with downstream performance.

they observe that some mapping methods that are specifically designed for BLI perform poorly in other tasks.


Furthermore, we show the distinctive nature of TWEETQA by comparing the collected data with traditional QA datasets collected primarily from formal domains.

we demonstrate empirically that three strong neural models which achieve good performance on formal data do not generalize well to social media data, bringing out challenges to developing QA systems that work well on social media domains.


On the other hand, the TWEETQA also has its unique properties.

a significant amount of questions require certain reasoning skills that are specific to social media data: • Understanding authorship: Since tweets are highly personal, it is critical to understand how questions/tweets related to the authors.


The proposed dataset informs us of the distinctiveness of social media from formal domains in the context of QA.

we find that QA on social media requires systems to comprehend social media specific linguistic patterns like informality, hashtags, usernames, and authorship.


As pointed out by a number of prior researches, there are four dominated variables: topic, author, time and language use.

we should control the first three variables to study the effect of language use.


An NQ-pair was preserved only if it was appropriate for a human to raise the question given the piece of news.

the question should be related to the given news while not mentioning extra information.


The original decoder employs two multi-head attention layers to include both the context in encoder and decoder, while we only need one to learn the decoder context, since the context in encoder is a vector (i.e., d k ).

after applying the word and positional embeddings to (w k 0 , .


In this section, summarization is modeled as a sequence labeling problem.

a document is viewed as a sequence of sentences and a summarization model is expected to assign a True or False label for each sentence, where True means this sentence should be included in the summary.


The proposed approach further extends the said architectures with semantic-based concept generalization, in an effort to improve the overall system performance.

semantic-based approaches utilizing (semantic) graphs produce the desired summaries through the extraction of ontological and syntactical relations in text, mainly by reducing the graph or by locating its key concepts (Khan et al., 2018;Joshi et al., 2018;Moawad and Aref, 2012).


The approach that achieved the best result was that of the combination of cosine similarity of averaged word2vec vectors and cosine similarity based on bag of words.

the best performance was achieved when the windows around the candidate and the generalized concepts were set to 10 and 6, respectively.


However, our approach can be applied to any dataset.

for future work, this study could be replicated for related fields like Machine Translation or Natural Language Generation.


To manage this issue we apply kernel smoothing over the language model to adapt it from the full vocab V down to the candidate word list C. Our smoothing process focuses on the output embeddings from the pretrained language model.

first we form the Voronoi partition (Aurenham-mer, 1991) over all the embeddings using the candidate set C. each word type w in the full vocabulary V is exactly assigned to one region represented by a word type w in the candidate set C, such that the distance from w to w is not greater than its distance to any other word types in C. As above, we use cosine similarity between corresponding word embeddings to define the regions.


(2017) to obtain a sequence of states A hierarchical convolutional decoder generates the target sentences (based on the encoder outputs).

a document-level decoder first generates sentence vectors (LSTM Document Decoder in Figure 1), representing the content specification for each sentence that the model plans to decode.


(2017) combines word embeddings with positional embeddings.

the word representation w ti of each target word y ti is combined with vector e i indicating where this word is in the sentence, w ti = emb(y ti ) + e i .


Table 9 shows examples of system output on the development set.

we show summaries generated by CONVS2S and CONVS2D+Topic, and also include the reference Gold standard.


the metric is more strongly positive when a form is less predictable given other forms in the language and more strongly negative when a form is more strongly predictable.


We will use this syncretism-collapsed representation for all simulations below.

this assumption will allow us to simply count the surface forms of each word in Wikipedia without dealing with the tricky issue of assigning individual words to the correct combination of morphosyntactic features (see, Cotterell et al., 2018b, for detailed discussion).


All word forms, paradigms, and morphosyntactic features are taken from the UniMorph project (Kirov et al., 2018).

we examine the following 28 languages: Albanian, Arabic, Armenian, Basque, Bulgarian, Czech, Danish, Dutch, English, Estonian, French, German, Hebrew, Hindi, Irish, Italian, Latvian, Persian, Polish, Portuguese, Romanian, Russian, Spanish, Swedish, Turkish, Ukrainian, Urdu, and Welsh.


However, there is wide variability in the average irregularity score between languages.

in the most regular language, Portuguese, correct forms are about 25,000 times more likely on average than alternative forms.


Training with perceptual context improves multimodal neural models compared to training on language alone.

augmenting a predictive language model with images that illustrate the sentences being learned enhances its next-word or masked-word prediction ability.


For each area, we tested the left and right hemispheres independently and compared these performances with the bilateral frontal lobe as well as the activity across all regions.

we examined the primary visual areas (left and right occipital lobe), speech and language processing areas (left temporal) and verbal memory (right temporal), sensory perception (left parietal) and integration (right parietal), language related movements (left frontal) and nonverbal functioning (right frontal).


We evaluate effectiveness of this synthetically generated brain data for data augmentation in the stimulus prediction task (Mitchell et al., 2008).

we train a decoding model (X2) to predict brain activity during a stimulus reading based on GloVe vectors for nouns.


Representations learned by BERT are the most effective in predicting brain activity.

most models are able to predict activity in the left temporal region of the brain with high accuracy.


Still, some patterns are considerably different from those encountered in human language.

lSTMs generally have a preference for the reverse version of an iconic language, and only show a weak tendency towards avoidance of redundant coding.


Second, we look at the cultural evolution of a language by means of the iterated language learning paradigm (see Kirby et al., 2014, for a survey).

we investigate the changes that modern Seq2Seq networks exposed to a language through multiple generations introduce, checking which biases they expose.


Seq2Seq agents might show similar preferences when tested as Speakers.

they might show a learning and preservation preference for either fixed nomarking languages or free marking languages.


This is in line with the typological observation that even nonconfigurational languages favor (at least statisti- 9 We run more simulations in this case as we noticed that the final language depends on the initial seed, and hence there is high variance with only 5 runs.

we start with 5 different parents and simulate 10 generations, repeating each experiment with 5 different seeds cally) certain orders (Hale, 1992;Mithun, 1992) and thus an equiprobable distribution of orders, as it is the case in our free word-order+markers language, is unlikely.


Ifû * matches one of the true samples, the agent succeeds, otherwise it fails (in iterated learning, P (u|t) corresponds to the parent's distribution).

we are not evaluating the model on a perfect fit of the ground-truth (parent's, in case of iterated learning) distribution, but we score a hit for it as long as it outputs a combination in P (u|t).


In this paper, we propose to resolve this problem by modeling and leveraging the head-driven phrase structures of entity mentions, i.e., although a mention can nest other mentions, they will not share the same head word.

we propose Anchor-Region Networks (ARNs), a sequence-to-nuggets architecture for nested mention detection.


To address this issue, we propose Bag Loss, an objective function which can be used to train ARNs in an end-to-end manner without any anchor word annotation.

our Bag Loss is based on at-least-one assumption, i.e., each mention should have at least one anchor word, and the anchor word should strongly indicate its entity type.


Inspired by the recent success of pointer networks (Vinyals et al., 2015;Wang and Jiang, 2016), this paper designs a pointer-based architecture to recognize the mention boundaries centering at an anchor word.

our region recognizer will detect the mention nugget "the department of education" for anchor word "department" by recognizing "the" to be the left boundary and "education" to be the right boundary.


To address the second problem, we design Bag Loss based on the at-least-one assumption, i.e., for each mention at least one word should be regarded as its anchor word.

we refer to all words belonging to the same innermost mention as a bag.


Bag Loss selects anchor words according to their associations with the bag type.

only words highly related to the bag type (e.g., "department" in "the department of education") will be trained towards the bag type, and other irrelevant words (e.g., "the" and "of" in the above example) will be trained towards NIL.


This result further verifies the necessity of Bag Loss.

because not all words in a mention are related to its type, it will introduce remarkable noise by regarding all words in mentions as anchor words.


We can see that there is still a significant performance gap from the anchor detector to entire ARNs.

there exist a number of mentions whose anchor words are correctly detected by the anchor detector but their boundaries are mistakenly recognized by the region recognizer.


This paper proposes Anchor-Region networks, a sequence-to-nuggets architecture which can naturally detect nested entity mentions by modeling and exploiting head-driven phrase structures of entity mentions.

an anchor detector is first used to detect the anchor words of entity mentions and then a region recognizer is designed to recognize the mention boundaries centering at each anchor word.


For textual networks, when the embedding objective is network topology reconstruction, a natural choice is to treat the text as context (Tu et al., 2017).

when modeling the edge w uv , S v and S u are respectively treated as the context for context-aware embeddings z u|c and z v|c , which are then used in the prediction of edge likelihood.


A more flexible solution is to employ an attention mechanism, which only attends to content that is relevant to a specific query (Vaswani et al., 2017).

attention models leverage a gating mechanism to de-emphasize irrelevant parts in the input; this method pools information only from the useful text, which is also a fixed length vector but that only encodes information with respect to one specific content (Santos et al., 2016).


We call π ∈ R n×m + a valid transport plan if it properly redistributes mass from µ to ν, i.e., i π ij = ν j and j π ij = µ i .

π breaks mass at {x i } into smaller parts and transports π ij units of x i to y j .


To alleviate this issue, we consider other more computationally efficient surrogate objectives.

we adopt the negative sampling approach (Mikolov et al., 2013), which replaces the bottleneck Softmax with a more tractable approximation given by is the sigmoid function, and p n (v) is a noise distribution over the nodes.


We adopt an approach similar to that used for some natural language inference (NLI) tasks (Bowman et al., 2015;Poliak et al., 2018).

given a scientific paper in PDF format, our system first extracts the key contents from the abstract and experimental sections, as well as from the tables.


To predict correct TDM triples and associate the appropriate scores, we adopt a natural language inference approach (NLI) (Poliak et al., 2018) and learn a binary classifier for pairs of document contexts and TDM label hypotheses.

we split the problem into two tasks: (1) given a document representation DocTAET, we would like to predict whether a specific TDM triple can be inferred (e.g., give a document we infer <Summarization, Gigaword, ROUGE-2>); (2) we predict whether a <Dataset, Metric> tuple (DM) can be inferred given a score context SC.


Finally we link these candidates to the valid TDM triples in a taxonomy 12 based on Jaccard similarity.

we predict a TDM triple for a paper if the similarity score between the triple and a candidate is greater than α (α is estimated in the  training set).


Inspired by Machine Reading Comprehension (MRC), we regard the product title and product attribute as 'context' and 'query' respectively, then the 'answer' extracted from 'context' equals to the attribute value wanted.

we model the contexts of title and attribute respectively, capture the semantic interaction between them by attention mechanism, and then use Conditional Random Fields (CRF) (Lafferty et al., 2001) as output layer to identify the corresponding attribute value.


In this paper, we propose the parallel Seq2Seq network with the coverage attention to alleviate the overlapping phrase problem.

we integrate the linguistic constraints of keyphrases into the basic Seq2Seq network on the source side, and employ the multi-task learning framework on the target side.


Seriousness of the problem (top-k, k = 10)  In this paper, we propose a parallel Seq2Seq network (ParaNet) with the coverage attention to alleviate the overlapping phrase problem.

we exploit two standalone encoders to encode separately the source text and syntactic constraints into network on the source side, and then applies multi-task learning framework to generate the keyphrases and part-of-speech (POS) tags for words in keyphrases on the target side.


In this study, we propose the parallel Seq2Seq network with the coverage attention to alleviate the overlapping problem (including sub-phrase and super-phrase problems) in existing keyphrase generation methods.

we incorporate the linguistic constraints of keyphrases into the basic Seq2Seq network, and employ multi-task learning framework to enhance generation performance.


(2017) developed semi-supervised deep learning model on the Twitter corpus.

they used the Convolution Neural Network (CNN) for classification.


Set decoder D: Due to its powerful ability of LSTM to model sequence dependency, we also implement D as a LSTM model to capture highorder correlations between labels.

the hidden state s t of the set decoder D at timestep t is computed as: where [e(y t−1 ); c t ] denotes the concatenation of vectors e(y t−1 ) and c t , e(y t−1 ) is the embedding of the label y t−1 generated at the last time-step, and c t is the context vector obtained by the attention mechanism.


Once we updated the slot representation v k in the current iteration, the logit b kt becomes larger when the dot product p k|t • v k is large.

when a prediction vector p k|t is more similar to a slot representation v k , the dot product is larger, indicating that it is more likely to route this word to the k-th slot type (Line 7).


Since a rule pattern can only match one word in the aspect term, the function TermFrom in Algorithm 2 tries to obtain the whole term based on this matched seed word.

it simply returns the word w s when it is a verb.


We also compare with two simplified versions of RINANTE: directly using the mined rules to extract terms; only using human annotated data to train the corresponding neural model.

the second simplified version uses a BiLSTM-CRF structured model with the embeddings of each word in a sentence as input.


To address this label confusion problem, this paper proposes cost-sensitive regularization, which can force the training procedure to concentrate more on optimizing confusing type pairs.

we introduce a costweighted term into the training loss, which penalizes more on mislabeling between confusing label pairs.


Commonly, these methods regard event detection as a wordwise classification task with one NIL class for tokens do not trigger any event.

a neural network automatically extracts high-level features and then feed them into a classifier to categorize words into their corresponding event sub- CT CR MT NIL CC BC 41.3 14.4 2.1 1.6 39.0 1.7 CT 8.5 42.7 4.7 2.6 40.6 0.9 CR 5.7 7.3 50.0 1.1 32.3 2.9 MT 3.0 7.7 6.1 28.7 51.3 3.2 types (or NIL).


We refer to this phenomenon as label confusion.

there are mainly two types of label confusion in event detection: 1) trigger/NIL confusion; 2) sibling sub-types confusion.


In this paper, inspired by cost-sensitive learning (Ling and Sheng, 2011), we introduce costsensitive regularization to model and exploit the label confusion during model optimization, which can make the training procedure more sensitive to confusing type pairs.

the proposed regularizer reshapes the loss function of model training by penalizing the likelihood of making wrong predictions with a cost-weighted term.


Therefore, we turn to pre-trained language models, attempting to leverage their knowledge learned from the large-scale corpus for event generation.

this paper proposes a framework based on pre-trained language models, which includes an event extraction model as our baseline and a labeled event generation method.


We hypothesize this is due to the issue of a disconnect between training and test-time objectives.

the system is trained solely to raise likelihood of gold-standard extractions, and during training the model is not aware of its test-time behavior of ranking a set of system-generated assertions across sentences that potentially include incorrect extractions.


Our approach can be regarded as a new type of autoencoder model to unsupervised dependency parsing that combines the benefits of both generative and discriminative techniques.

our approach breaks the context-free independence assumption in previous generative approaches and therefore becomes more expressive.


By conditioning dependency prediction on the whole input sentence, discriminative methods are capable of utilizing not only local information, but also global and contextual information of a dependency in determining its strength.

both Grave and Elhadad (2015) and Cai et al.


Unlike neural DMV, however, when computing the probability of a dependency, we rely on not only local information as in DMV, but also global and contextual information from a compressed representation of the input sentence produced by neural networks.

instead of modeling the joint probability of the input sentence and its dependency parse as in a generative model, we model the conditional probability of the sentence and parse given global information of the sentence.


We extend the neural DMV such that when predicting the probability of a grammar rule in parsing a sentence, the model incorporates not only local information of the rule but also global information of the sentence.

we model each grammar rule probability conditioned on a continuous vector.


Learning Algorithm: We optimize our objective function using the expectation-maximization (EM) algorithm.

the EM algorithm alternates between E-steps and M-steps to maximize a lower-bound of the objective function.


PFAs have some conditions on their structure.

finally, each state must be accessible and co-accessible.


Computing α n 0,n+1 only requires knowledge of the terms of the form α k i,j , where i, j ≥ k + 1, or of the form a k 0,k+1 .

only the term α k 0,k+1 and the terms in the bottom right k × k sub-table of α k need to be considered at step k + 1. .


In the KMP DFA, state q i has exactly one transition to state q i+1 and |Σ| − 1 transitions to lower (or equal) states.

there is no path from a state of label i to a state with label at least i + 2 that does not go through state i + 1.


In this paper we express the external syntactic information as vectors of discrete features, because this enables us to explore different ways of injecting the syntactic information into the neural SRL model.

we propose three different syntax encoding methods: a) a full constituency tree representation (Full-C); b) an SRLspecific span representation (SRL-C); and c) a dependency tree representation (Dep).


We conclude that the extra complexity of the autoencoder model is not justified.

dep with auto-encoder hurts SRL accuracy (0.6% behind the model with the constituency features).


MTL enables learning many tasks jointly, encapsulating them in a single model and leveraging their shared representation (Caruana, 1997;Ruder, 2017).

we will use a hard-sharing architecture: the sentence is first processed by stacked BILSTMs shared across all tasks, with a task-dependent feed-forward network on the top of it, to compute each task's outputs.


In particular, we will use a hard-sharing architecture: the sentence is first processed by stacked BILSTMs shared across all tasks, with a task-dependent feed-forward network on the top of it, to compute each task's outputs.

to benefit from a specific parsing abstraction we will be using the concept of auxiliary tasks (Plank et al., 2016;Bingel and Søgaard, 2017;Coavoux and Crabbé, 2017), where tasks are learned together with the main task in the MTL setup even if they are not of actual interest by themselves, as they might help to find out hidden patterns in the data and lead to better generalization of the model.


Different from attention mechanism, the module reduces noise in semantic aspect level rather than word level.

we selectively replace some words in the sentence with abstract aspects.


Sequence-labeling is particularly convenient for nearest-neighbor based approaches, since a predictionŷ can be formed by simply concatenating labels extracted from the label-sequences associated with neighbors.

we will assume we have access to a database m) and their corresponding true label-sequences y (m) .


In particular, we will assume we have access to a database m) and their corresponding true label-sequences y (m) .

we will predict a labelingŷ for x by considering each token x t , selecting a labeled token x  we consider a very simple token-level model for this label-agnostic copying, where the probability that x's t'th label y t is equal to y , and is independent of the surrounding labels, conditioned on x and D. 2 we define where the above probability is normalized over all label tokens of all label-sequences in D. Above, (both in R D ) represent the contextual word embeddings of the t'th token in x and the k'th token in x (m) , respectively, as obtained by running a deep sequence-model over x and over x (m) .


At test time, we predictŷ t to be the label type with maximal marginal probability.

we setŷ t to be where z ranges over the label types (e.g., POS or named entity tags) present in D. As noted in the introduction, predicting labels in this way allows for the prediction of any label type present in the database D used at test time, and so we can easily predict label types unseen at training time without any additional retraining.


We evaluate zero-shot transfer performance by training on one dataset and evaluating on another, without any retraining.

we consider three zero-shot transfer scenarios: training with Universal POS Tags on the Penn Treebank and then predicting the standard, fine-grained POS tags, training on the CoNLL 2003 NER data and predicting on the fine-grained OntoNotes NER data (Hovy et al., 2006) using the setup of Strubell et al.


We see that in all cases the label-agnostic model outperforms standard fine- tuned BERT, often significantly.

we note that when going from universal POS tags to standard POS tags, the fine-tuned label-agnostic model manages to outperform the standard mostfrequent-tag-per-word baseline, which itself obtains slightly less than 92% accuracy.


I've been trying to get it for a while now!

listener: quite an accomplishment and you should be proud!


As discussed above, one of the ultimate goals in human conversation is that information can be exchanged effectively through interaction.

we argue that successful multi-turn dialogues are determined by the joint experience of both participants in the conversation, i.e., both participants need to get aware of their counterparts and express themselves effectively.


From the toy example, it can be observed that the coverage vector v t increases with the amount of expressed knowledge.

a higher mean value of v t indicates that the participants have expressed more background knowledge, which gives a better chance for them to know more about each other.


These results lead to the following crucial conclusion: while in-domain data are insufficient to train response selection models from scratch for many target domains, such data are invaluable for adapting a pretrained general-domain model to the target domain.

the results indicate that the synergy between the abundant response selection Reddit data and scarce in-domain data is effectively achieved through the proposed training regime, and both components are crucial for the final improved performance in each target domain.


Blocks World in Minecraft Minecraft provides an ideal setting to simulate Blocks World, although there are two key differences to physical toy blocks: Minecraft blocks can only be placed on a discrete 3D grid, and they do not need to obey gravity.

they do not need to be placed on the ground or on top of another block, but can be put anywhere as long as one of their sides touches another block.


GTMN sets meta-word expression as a goal of generation and dynamically monitors expression of each variable in the meta-word during the decoding process.

gTMN consists of a state memory panel and a state controller where the former records status of meta-word expression and the latter manages information exchange between the state memory panel and the decoder.


Technically, rather than comparing M i .v t−1 with M i .g and adopting operations accordingly, we propose a soft way to update the state memory panel with SUB and ADD, since (1) it is difficult to identify over-expression or sub-expression by comparing two distributed representations; and (2) the hard way will break differentiablility of the model.

we define g t ∈ R d×l as a gate to control the use of SUB or ADD where g be the changes from the SUB operation and the ADD operation respectively, then M i .v t−1 is updated aŝ where • means element-wise multiplication, and g t (i), ∆ SU B t (i), and ∆ ADD t (i) can be defined as and where ⊕ is a concatenation operator.


Our approach integrates conversation generation with on-demand MRC.

we use an MRC model to effectively encode the conversation history by treating it as a question in a typical QA task (e.g., SQuAD (Rajpurkar et al., 2016)), and encode the web document as the context.


For inference, we found that top-k random sample decoding (Fan et al., 2018) provides the best results for all the systems.

at each decoding step, a token was drawn from the k most likely candidates according to the distribution over the vocabulary.


DLI can be trained with contextual SLU jointly if utterances are sorted one by one: selecting the right utterance from remaining candidates based on previously sorted context.

given a response and its context, the DLI task requires our model to infer whether the response is the right one that matches the dialogue context, similar to the next sentence prediction task (Logeswaran and Lee, 2018).


Persona-agnostic meta-learning (PAML) We define the persona meta-dataset as D =   Compute adapted parameters with gradient descent: end for 9: where α is learning of the inner optimization, and the training loss.

crossentropy loss is used for training the response generation: The meta-learning model is then trained to maximize the performance of the adapted model f θ p i to the unseen dialogues in D valid p i .


In this work, we introduce a hierarchical neu- ral attention architecture, integrating turn-level attention with word-level attention for multi-turn dialogue comprehension in a question-answering manner, where we evaluate performance on a corpus preserving linguistic features from real-world spoken conversation scenarios.

we examine how our approach is able to address challenges from limited training data scenarios and from lengthy and out-of-distribution test samples.


NLU is to extract core semantic concepts from the given utterances, while the goal of NLG is to construct corresponding sentences based on given semantics.

understanding and generating sentences are a dual problem pair shown in Figure 1.


Assuming that we have two spaces, the semantics space X and the natural language space Y, given n data pairs {(x i , y i )} n i=1 , the goal of NLG is to generate corresponding utterances based on given semantics.

the task is to learn a mapping function f (x; θ x→y ) to transform semantic representations into natural language.


Note that two aforementioned models BERT+RNN and BERT+RNN+Ontology use the linear layer to transform a hidden vector to an output vector, which depends on a candidate slot-value list.

the models require re-training if the ontology is changed, which implies that these models have lack of scalability.


The availability of task descriptions has made zero-shot learning (Norouzi et al., 2013;Socher et al., 2013) popular.

work on zeroshot utterance intent detection has relied on varied resources such as click logs (Dauphin et al., 2013) and manually defined domain ontologies (Kumar et al., 2017), as well as models such as deep structured semantic models  and capsule networks (Xia et al., 2018).


CSRR employs hierarchical latent variables based on VAEs to represent the meaning of utterances and meanwhile learns the relationship between query and response.

cSRR draws the background of the conversation with a discourse-level latent variable and then models the consistent semantics between query and response, e.g.


In these approaches, no syntactic or semantic annotation is needed for training: representation is induced from scratch in an end-to-end fashion, in such a way as to benefit a given downstream task.

these approaches provide an inductive bias specifying that (hierarchical) structures are appropriate for representing a natural language, but do not make any further assumptions regarding what the structures represent.


To this end, we rely on the Perturb-and-MAP framework (Papandreou and Yuille, 2011).

we perturb the potentials (arc weights) with samples from the Gumbel distribution and compute the most probable structure with the perturbed potentials: Each element of the matrix G ∈ R n×n contains random samples from the Gumbel distribution 4 which is independent from the network parameters θ, hence there is no need to backpropagate through this path in the computation graph.


When KL(q φ (z|x)||p(z)) = 0, it indicates that z and x are independent (i.e., q φ (z|x) = q φ (z) = p(z)).


We investigate this by providing additional information about the premise to a hypothesis-only model.

we provide the last noun of the premise, i.e., we form a hypothesis-plus-one-word model.


While it is tempting to hypothesize other ways a model can cheat, it is infeasible to enumerate over all of them.

if we could write down all the necessary tests for test-driven development (Beck, 2002) of a machine learning model, we would already have a rule-based system that can solve our task.


However, since these classi-fiers can be easily fooled if they rely on superficial patterns, the resulting dataset may still contain artifacts.

a much stronger model (BERT) that sees the full-input easily solves the dataset.


Certifiable Robustness Finally, an alternative approach for creating models that are free of artifacts is to alter the training process.

model robustness research in computer vision has begun to transition from an empirical arms race between attackers and defenders to more theoretically sound robustness methods.


The distribution vector P (w) of a word w can be calculated in multiple ways.

in this work, we leverage a pretrained language model to compute P (w) and condition on all the words preceding w. for the t-th word x t in a sentence, we have where LM (w j |x <t ) denotes the probability of the j-th word in the vocabulary appearing after the sequence Note that the language model is pretrained using the same training corpus of the NMT model.


In this paper we are focused only on semi-supervised domain adaptation -when knowledge from a large labeled dataset in a source domain can be somewhat transferred to help improve the same task on a target domain, which typically has a significantly smaller number of labels.

this paper focuses on domain adaptation for the detection of question duplicates in community question answering forums (Shah et al., 2018;Hoogeveen et al., 2015), as well as for RTE tasks (Dagan et al., 2005;Zhao et al., 2017).


Both T2 and T3 outperform T1, hence providing strong evidence that GRL significantly improves overall feature learning if compared to maximizing the domain classification loss.

the comparison between T3 and T1, shows that learning exactly the same feature set using GRL for adversarial domain adaptation is more effective than maximizing the loss.


Although we have shown that the approach works in principle, by achieving state-of-the-art results on four standard datasets, it needs to be further explored in order to understand why it works.

we need to understand why words extracted with χ 2 yield better results compared to ANOVA, how many words should be extracted given a specific task, if other statistical tests might even improve the outcomes, etc.


Empirical studies show that our approach can significantly improve in translation quality with an increased model depth.

we achieve 1.0 and 0.6 BLEU score improvement over the strong Transformer baseline in English→German and English→French translations.


With our method, the performance of a well-optimized 6block model can be further boosted by adding two additional blocks, while simply using Transformer (8B) will lead to a performance drop.

we achieve a 29.92 BLEU score on En→De translation with 1.0 BLEU improvement over the strong baselines, and achieve a 0.6 BLEU improvement for En→Fr.


(2) Ensemble learning (Ensemble): we present the two-model ensemble results for fair comparison with our approach that involves a two-pass deepshallow decoding.

we present the ensemble performances of two independently trained 6-block models (Ensemble 6B/6B), and ensemble of one 6-block and one 8-block model independently trained from scratch (Ensemble 6B/8B).


We propose two variants of MHA, namely a black-box MHA (b-MHA) and a white-box MHA (w-MHA).

in contrast to previous language generation models using M-H, b-MHA's stationary distribution is equipped with a language model term and an adversarial attacking term.


Given the stationary distribution (π(x)) and transition proposal, M-H is able to generate desirable examples from π(x).

at each iteration, a proposal to jump from x to x is made based on the proposal distribution (g(x |x)).


The novel idea behind our hybrid generative-discriminative method is to explicitly capture the fine-grained information inferred from raw texts, utilizing the information to help interpret the predicted classification results and improve the overall performance.

we introduce the notion of an explainable factor and a minimum risk training method that learn to generate reasonable explanations for the overall predict results.


However, there is a significant shortcoming of this generating process: it fails to build strong connections between the generative explanations and the predicted overall results.

the generative explanations seem to be independent of the predicted overall results.


We report the results in Table 8, and we can see that over half of the judges think that our GEF can generate explanations more related to the classification results.

for 57.62% of the tested items, our GEF can generate better or equal explanations comparing with the basic model.


In this paper, we study modeling behavioral codes used to asses a psychotherapy treatment style called Motivational Interviewing (MI), which is effective for addressing substance abuse and related problems.

we address the problem of providing real-time guidance to therapists with a dialogue observer that (1) categorizes therapist and client MI behavioral codes and, (2) forecasts codes for upcoming utterances to help guide the conversation and potentially alert the therapist.


We used the audio and visual feature extractors pre-trained on YouTube videos and the Kinetics dataset (Kay et al., 2017) (Refer to  for the detail video features).

we used the 2048-dimensional I3D flow features from the "Mixed 5c" layer of the I3D network (Carreira and Zisserman, 2017) for visual features and 128dimensional Audio Set VGGish  for audio features.


We study the problem of imposing conversational goals on open-domain chat agents.

we want a conversational system to chat naturally with human and proactively guide the conversation to a designated target subject.


This paper makes a step towards open-domain dialogue agents with conversational goals.

we want the system to chat naturally with humans on open domain topics and proactively guide the conversation to a designated target subject.


We decouple the whole system into separate modules and address the challenges at different granularity.

we explicitly model and control the intended content of each system response by introducing coarse-grained utterance keywords.


The agent then produces an utterance x n+1 as a response, aiming to satisfy (1) transition smoothness by making the response natural and appropriate in the current conversation context, and (2) target achievement by driving the conversation to reach the designated target.

we consider a target is achieved when either the human or the agent mentions the target or similar word in an utterancesuch a definition is simple and allows easy measurement of the success rate.


We explore a solution that addresses the two desiderata separately.

we maintain smooth conversation transition by turn-level supervised learning on open-domain chat data, and we inject target-guiding behavior with a rulebased guiding strategy.


Hybrid Kernel-based Method We further study a hybrid approach that combines neural feature extraction with pairwise closeness measuring.

given a pair of a current keyword and a candidate next keyword, we follow (Xiong et al., 2017) by first measuring the cosine similarity of their normalized word embeddings, and feeding the quantity to a kernel layer consisting of K RBF kernels.


Same as the turn-level transition module, the conditional response retrieval module can also be learned with open-ended conversation data in a supervised manner.

we maximize the likelihood of observed response given its history and predicted keyword, while minimizing the likelihood of randomly sampled negative responses.


We then augment the data by automatically extracting keywords of each utterance.

we apply a rule-based keyword extractor which combines TF-IDF and Part-Of-Speech features for scoring word salience.


Retrieval-Stgy augments the above base retrieval system with the proposed target-guided strategy (section 4.2).

it first extracts the keywords of current utterance with the extractor used in section 5, and applies the target-guided rule to obtain a set of candidate keywords.


We first evaluate the performance of each conversation turn, in terms of both turn-level keyword prediction and response selection.

we disable the discourse-level target constraint, and focus on measuring how accurate the systems can predict the next keyword and retrieve the correct response on the test set of the conversation data.


Following the experimental settings in prior work (Lewis et al., 2017;Li et al., 2016b), we developed a task simulator to automatically produce target-guided conversations.

we use the base Retrieval agent to play the role of human which retrieves a response without knowing the end target.


To avoid infinite conversation without ever reaching the target, we set a maximum allowed number of turns, which is 8 in our experiment.

an agent that does not achieve the target after producing 8 responses is considered to fail in the case.


We finally perform human evaluation for a more thorough system comparison in terms of different aspects.

we use the DialCrowd toolkit (Lee et al., 2018) to setup human evaluation interfaces, and undertook two types of human studies as below.


The second evaluation compares our bestperforming Kernel agents with other agents sideby-side.

we ask a human turker to converse with the Kernel agent and a randomly selected comparison agent in the same test case.


Annotations for persuaders' utterances included diverse argument strategies and task-related non-persuasive dialogue acts.

we identified 10 persuasion strategy categories that can be divided into two types, 1) persuasive appeal and 2) persuasive inquiry.


Emotion appeal refers to the elicitation of specific emotions to influence others.

we identified four emotional appeals: 1) telling stories to involve participants, 2) eliciting empathy, 3) eliciting anger, and 4) eliciting the feeling of guilt.


After identifying and categorizing the persuasion strategies, the next step is to analyze the factors that contribute to the final donation decision.

understanding the effects of the persuader's strategies, the persuadee's personal backgrounds, and their interactions on donation can greatly enhance the conversational agent's capability to engage in personalized persuasion.


We use the KL divergence between model and ground truth (GT) response sets to measure how well a model can approximate the GT distribution of words.

we define distributions p gt and p m based on each set of responses and calculate the KL divergence pm(w) for each GT response.


We demonstrate that the above construction of a word's representation disambiguated in a context is useful by comparing with state-of-the-art unsupervised methods for polysemy disambiguation on two tasks: Word Sense Induction and contextual similarity.

we compare with MSSG, the K-Grassmeans model of (Mu et al., 2017), and the sparse coding method of (Arora et al., 2018).


(4) to obtain p. In this section, we further integrate combination rules into our sememe-incorporated SC models.

we can use totally different composition matrices for MwEs with different combination rules: where w r c ∈ R d×2d and R s refers to combination rule set containing syntax rules of MwEs, e.g., adjective-noun and noun-noun.


These become obvious when looking at examples from the aforementioned datasets (see Table 2, which includes an example from VUAMC).

many metaphors annotated in the binary datasets differ widely in their metaphoricityi.e., their degree of being a metaphor.


We want to create a resource that allows us to probe embeddings for S-classes.

we have the following desiderata: (i) We need a corpus that is S-class-annotated at the token level, so that we can train sense embeddings as well as conventional word embeddings.


We investigate embeddings by probing: Is the information we care about available in a word w's embedding?

we probe for S-classes: Can the information whether w belongs to a specific S-class be obtained from its embedding?


Intuitively, if a neural pathway was approximating a function similar to one of the phenomena examined by the linear probes, then the loadings of each neuron in the pathway would be similar in relative shape to the weights of the relevant linear probe.

if the pathway and the probe are viewing the same phenomenon, the neurons with stronger weights in the probe should have higher loadings in the pathway and vice versa.


However, they are compositionally less rigid than some other types of multiword expressions such as, e.g., idioms (as, e.g., [to] kick the bucket) or multiword lexical units (as, e.g., President of the United States or chief inspector).

they are formed by a freely chosen word (the base), which restricts the selection of its collocate (e.g., rain restricts us to use heavy in English to express intensity).


Below we explore the potential of such relation vectors for semantic relation classification.

we trained them for all word pairs from DiffVec and LexFunc using two different variants of the SeVeN model (Espinosa-Anke and Schockaert, 2018).


As for LMC, how to set a threshold probability for judging whether a given sentence is correct remain explored, and there exists great trade-off between the precision and the recall as reported in (Jia et al., 2013).

utility of M by comparing the experimental results of Ours − and Ours + , we can observe that the latter achieves better performance, which validates the effectiveness of utilizing Confusionsets that can help improve the probability of generating correct target characters.


In this paper, we propose a general framework for data augmentation in low-resource machine translation that not only uses target-side monolingual data, but also pivots through a related highresource language (HRL).

we experiment with a two-step pivoting method to convert high-resource data to the LRL, making use of available resources to better approximate the true data distribution of the LRL.


To avoid such case, we force the model to choose actions such that it will always satisfy the latency constraints.

if the model reaches the aggressive bound, it must choose a target word other than ε with highest score, even if ε has higher score; if the model reaches the conservative bound, it can only choose ε at that step.


Note that for each individual application of the Structure Layer, we are building multiple levels (L) of nested entities.

within each Structure Layer there is a loop of length L. By building the structure before the Update Layer, the updates to the token embeddings can utilize information about which entities each token is in, as well as neighbouring entities, as opposed to just using information about neighbouring tokens.


In addition, on higher levels, the directions from other words to each of these three tokens will also be identical.

the use of "directions" 4 allows the network to represent entities as a single embedding in a fully differentiable fashion, whilst keeping the sequence length constant.


The use of directional vectors D in the Update Layer can be thought of as an alternative to the positional encodings in Transformer (Vaswani et al., 2017).

instead of updating each token embedding using neighbouring tokens embeddings with a positional encoding, we update using neighbouring token embeddings, and the directions to those tokens.


We evaluate these cases in the experiments.

we make the following contributions: • We propose a DL architecture for ER that learns attribute agnostic and transferable representations from multiple source datasets using dataset (domain) adaptation.


The resultant attribute representations are then used to compare attributes of each entity record pair.

we compute the element-wise absolute difference between the two attribute vectors for each attribute and construct attribute similarity vectors (sim 1 and sim 2 in Fig.


To this end, we apply the technique of dataset (domain) adaptation developed in image recognition (Ganin and Lempitsky, 2015).

we build a dataset classifier with the same architecture as the matching classifier ("Dataset Classifier" in Fig.


First, deep transfer active learning (DTAL) achieves the best performance in the low-resource setting of each dataset.

dTAL outperforms the others to the greatest degree in Cora (97.68 F1 points) probably because Cora is the most complex dataset with 8 attributes in the schema.


Class-specific Token-level Loss The first modification we make is to pick a target entity class and modify ∆(y i , y) to have word-wise loss of tgt for false positives on the target class and loss of t gt for false positives on other classes.

let y j i be j-th element of sequence y i , we define ∆(y i , y) = j w j , where if y j i = y j tgt , if y j i = y j and y j = target class t gt , if y j i = y j and y j = target class Note that the target class in the above equation contains all the labels related to the target entity type; that is, if the target class is ORG, we consider B-ORG and I-ORG to be the related labels.


Moreover, even when such pretrained models are available, they inevitably come with potentially undesirable biases baked in, based on the data on which they were trained (Recasens et al., 2013;Bolukbasi et al., 2016;Zhao et al., 2019).

for social science applications, it may be preferable to exclude such confounders by only working with in-domain or curated data.


A powerful way of encoding and decoding text is to use sequence models.

f µ (x) and f σ (x) would map from a sequence of tokens to a pair of vectors, µ and σ, and f d (z) would similarly decode from z to a sequence of tokens, using recurrent, convolutional, or attention-based networks.


Here, for an n-layer MLP encoder on word counts c i , we build on that idea, and use as representations a weighted sum over θ i and the internal states of the MLP, h i , with weights to be learned by the downstream classifier.

5 for any sequence-to-vector encoder, f s2v (x), we propose to augment the vector representations for each document by concatenating them with a weighted combination of the internal states of our variational encoder (Peters et al., 2018a).


For all experiments we make use of the Deep Averaging Network (DAN) architecture (Iyyer et al., 2015) as our baseline sequence-to-vector encoder, f s2v (x).

embeddings corresponding to each token are summed and passed through a multi-layer perceptron.


8 http://nlp.stanford.edu/projects/ glove/ High resource In the high-resource setting, we assume access to plentiful computational resources and massive amounts of out-of-domain data, which may be indirectly accessed through pretrained models.

we evaluate the performance of a Transformer-based ELMO (Peters et al., 2018b) and BERT, both (a) off-theshelf with frozen embeddings and (b) after semisupervised fine-tuning to both unlabeled and labeled in-domain data.


We also experiment with ELMO trained only on in-domain data as an example of high-resource LM pretraining methods, such as Dai and Le (2015), when there is no out-of-domain data available.

we generate contextual word representations with a Transformer-based ELMO.


PBLM is designed to feed structure-aware task classifiers.

in the PBLM-CNN architecture that we consider here (Figure 1b), 4 the PBLM's softmax layer (that computes the probabilities of each pivot to be the next unigram/bigram) is cut and a matrix whose columns are the PBLM's h t vectors is fed to the CNN.


The only difference is that in cases where the next unigram or bigram is a pivot, instead of predicting the actual pivot identity, PBLM should predict PosPiv or NegPiv according to the pivot's class.

the representation learned by the first PBLM model is only sensitive to whether a pivot is positive or negative and not to the actual pivot identity.


At each iteration the PBLM is exposed to additional # P /K pivots, where # P is the total number of pivot features.

at the first iteration the first # P /K pivots are exposed, at the second iteration the next # P /K are also exposed and so on till the last (Kth) iteration in which all pivots are exposed.


Table 2 and Table 1 further reveals that model selection by development data has a more negative impact on NoTRL, compared to RF2 and BasicTRL.

for NoTRL there are only two cases where the model that performs best on the test set (max column of Table 2) was selected by the development data (the numbers reported in Table 1): B-D (84.2%) and K-A (86.1%).


To capture these sequential dependencies, STANCE encodes each mention using a bidirectional long short-term memory network (LSTM) (Hochreiter and Schmidhuber, 1997;Graves and Schmidhuber, 2005).

each character c i in a mention m is represented by a d-dimensional vector, h i , where h i is the concatenation of the hidden states corresponding to c i produced by running the LSTM in both directions.


Aligning the mentions is posed as a transport problem, where the goal is to convert one mention into another while minimizing cost.

we solve the Kantorovich formulation of optimal transport (OT).


In the first experiment, we compare STANCE with both classic and learned similarity models in alias detection.

we compare STANCE to following approaches: • Deep Conflation Model (DCM) -state of the art model that encodes each string using a 1-dimensional CNN applied to character ngrams and computes cosine similarity (Gan et al., 2017).


To do so, we compare variants of STANCE with components removed and/or modified.

we compare the following variants: • WITHOUT-OT (-OT) -STANCE with LSTM encodings and CNN scoring but without optimal transport-based alignment.


We can therefore train the entire network end-toend from a cross-entropy objective, where a loss is incurred for incorrect decisions on the level of token pairs.

we set y i,j = 1 when w i and w j corefer (coreferential links), and also when both w i and w j are part of the same mention span (self links).


To address this, we propose teacher annealing, which mixes the teacher prediction with the gold label during training.

the term in the summation becomes where λ is linearly increased from 0 to 1 throughout training.


A crucial difference distinguishing our work from the STILTs, Snorkel MeTaL, and MT-DNN KD methods in Table 2 is that we do not single-task fine-tune our model.

we do not continue training the model on individual tasks after multi-task training.


We examine the quality of the MR extraction with a qualitative study evaluating YELPNLG MR to NL 1 2 3 4 5 6 7 8 9 1 0 1 1 pairs on various dimensions.

we evaluate content preservation (how much of the MR content appears in the NL, specifically, nouns and their corresponding adjectives from our parses), fluency (how "natural sounding" the NL is, aiming for both grammatical errors and general fluency), and sentiment (what the perceived sentiment of the NL is).


For the +ADJ, +SENT, and +STYLE MRs, each MR is a longer relational tuple, with additional style feature information to encode, such that an input sequence x 1:S = (f attr , f val , f 1:N ), and where each f n is an additional feature, such as adjective or mention order.

in the case of +STYLE MRs, the additional features may be sentence-level features, such as sentiment, length, or exclamation.


Then, we find the total number of unique templates each model produces, finding that each "more informed" model produces more unique templates: BASE produces 18k, +ADJ produces 22k, +SENT produces 23k, and +STYLE produces 26k unique templates.

given the test set of 30k, +STYLE produces a novel templated output for over 86% of the input MRs.


This is problematic for comparing the performance of the KGLM to traditional language models on Linked WikiText-2 since there are a large number of rare entities whose alias tokens are outof-vocabulary.

even if the KGLM identifies the correct entity and copies the correct alias token with high probability, other models can attain better perplexity by assigning a higher probability to UNK.


By relying on memorization, existing language models are unable to generate factually correct text about real-world entities.

they are unable to capture the long tail of rare entities and word types like numbers and dates.


This makes our setting closer to style transfer in computer vision, in which an image is generated that combines the content from one image and the style from another (Gatys et al., 2016).

in our setting, we seek to generate a sentence that combines the semantics from one sentence with the syntax from another, and so we only require a pair of (unparsed) sentences.


To solve this problem, we follow  and take an approach based on latentvariable probabilistic modeling, neural variational inference, and multi-task learning.

we assume a generative model that has two latent variables: y for semantics and z for syntax (as depicted in Figure 2).


At test time, we want to have different combinations of semantic and syntactic inputs, which naturally suggests separate parameterizations for q φ (y|x) and q φ (z|x).

q φ (y|x) is parameterized by a word averaging encoder followed by a three-layer feedforward neural network since it has been observed that word averaging encoders perform surprisingly well for semantic tasks (Wieting et al., 2016).


However, since our purpose is not to reduce model size (unlike Chen and Gimpel, 2018), we marginalize out the latent code to get the embeddings during both training and testing.

where c w is the latent code for word w, v cw is the vector for latent code c w , and e w is the resulting word embedding for word w. In our models, we use 10 binary codes produced by 10 feedforward neural networks based on a shared word embedding, and then we concatenate these 10 individual cluster vectors to get the final word embeddings.


Based on this, we introduce a word position loss (WPL).

as shown in  where softmax(•) t indicates the probability at position t. Empirically, we observe that adding WPL to both the syntactic encoder and decoder improves performance, so we always use it in our experiments unless otherwise indicated.


While the SCPN yields very strong metric scores, there are several differences that make the SCPN results difficult to compare to those of our models.

the SCPN uses a supervised parser both during training and at test time, while our strongest results merely require a POS tagger and only use it at training time.


After serializing all the words in the 'Value' columns, for the i-th word in the table x a k i whose attribute is a k (the k-th attribute), we use the attribute name a k and the word's position in that tuple to locate the word (Lebret et al., 2016).

we utilize a triple z a k i = {a k , p a k i+ , p a k i− } to represent the structure information for word x a k i , in which p a k i+ and p a k i− are the positions of x a k i counted from the beginning and end of a k , respectively.


• Experimental results show that our proposed approach generally outperforms the other approaches on two style transfer datasets.

to the content preservation, Style Transformer achieves the best performance with a significant improvement.


they are all the positive reviews for a specific product), and we refer this shared characteristic as the style of these sentences.

a style is defined by the distribution of a dataset.


DSS-VAE explicitly models syntax in the continuous latent space of VAE, while retaining the sampling and manipulation benefits.

we introduce two continuous latent variables to capture semantics and syntax, respectively.


(1) by adopting two separate latent variables z sem and z syn to capture semantic and syntactic information, respectively.

we assume that the probability of a sentence x in DSS-VAE could be computed as: where p(z sem ) and p(z syn ) are the priors; both are set to be independent multivariate Gaussian N (0, I).


We are inspired by the research in text style transfer and apply auxiliary losses to regularize the latent space (Hu et al., 2017;Fu et al., 2018).

we adopt the multi-task and adversarial losses in John et al.


It is seen that the line of DSS-VAE is located to the upper-left of the competing methods.

the plain VAE and CGMH are "inadmissible," meaning that DSS-VAE simultaneously outperforms them in both BLEU-ori and BLEU-ref, indicating that DSS-VAE outperforms previous state-of-the-art methods in unsupervised paraphrase generation.