Recall that we restricted the choice of source domains to the 12 CQADupStack forums.

the second is unsupervised DQD via representation learning (Charlet and Damnati, 2017;Lau and Baldwin, 2016), which requires only unlabeled questions.


We are grateful to the anonymous reviewers for their helpful feedback.

(1) Compression   We train this model using the Adam optimizer (Kingma and Ba, 2014) and binary cross-entropy loss, with accuracy as the training metric.


This is because our approaches are centered around word usage.

to verify whether human evaluators are in agreement with our characterization model, we conducted a user study using Mturk (Amazon, 2005).


By choosing perceived offensiveness and discrepancies in perception as proxy measures for MAS, we hope to shift studies of MAS from a judgment call on an individual's intent to a description of how it affects its targets, which can be described as an objective measure of human behavior that can be validated through study.

this moderate agreement is on-par with other difficult annotation tasks, such as those for connotation frames (Rashkin et al., 2016), which had 0.52 percentage agreement for rating the polarity of the sentence towards a target, dimensions of social relationships (Rashid and Blanco, 2017), which had κ val- ues as low as 0.59, and Word Sense Disambiguation (Passonneau et al., 2012), which reported α values for some words below 0.30 at determining meaning.


On the other hand, methods similar to the proposed model could be developed for such purposes and not shared with the broader community.


For the purpose of training the embedding f θ we compose it with a discriminative classifier g φ : R D → R Y with parameters φ predicting the author of an episode, where Y is the number of authors in the training set.

prior work in this area has primarily focused on classifying an author as a member of a closed and typically small set of authors (Stamatatos, 2009;Schwartz et al., 2013;Shrestha et al., 2017).


To amortize both costs and annotators' effort at answering questions, each HIT requires the participants to annotate three stories after answering demographic questions.

my favorite set was all the art that was made out of stylish trash .


if annotators are asked to write their own text given a reference sentence, they may simply produce arbitrarily paraphrased output which does not exhibit a stylistic diversity.


The TA shows that the annotators rarely indicate the same scenes, even if they are asked to annotate an event in the screenplay that is described by a specific synopsis sentence.

it still remains competitive against the baselines, indicating that tracking TPs in screenplays fully automatically is feasible.


Traditional suicide risk assessment like Suicide Probability Scale (Bagge and Osman, 1998), Adult Suicide Ideation Questionnaire (Fu et al., 2007), Suicidal Affect-Behavior-Cognition Scale (Harris et al., 2015), etc.

the experimental result showed that by combining all of three emotion traits together, the proposed model could generate more discriminative suicidal prediction performance.


Although highly beneficial, those models' use may not be sufficient for the case of dialogue as response generation for goal-oriented dialogue from extremely limited data requires specialized tools.

the base optimization objective is as follows: where x usr is user's query, x sys is the system's response, c is the dialogue context, and F e and F d are respectively hierarchical encoder and decoder.


The model (as well as its variants listed above) is implemented in PyTorch (Paszke et al., 2017), and the code is openly available 1 .

we use it as-is (ZSDG) as well its variation as follows.


Given the target domain, we first train LAED model(s) on the MetaLWOz data -here we exclude from training every domain that might overlap with the target one.

for the NLU ZSDG setup, we annotated all available SMD data and randomly selected a subset of 1000 utterances from each source domain, and 200 utterances from the target domain.


We first describe the task we are addressing in this paper, and the corresponding base model.

we perform an exhaustive ablation study of DiKTNet by comparing it to all its variations mentioned above: HRED, HRED+ELMo, and HRED+LAED.


In addition, with the recently adopted technique of training dialogue systems end-to-end data-efficiency of such systems becomes the key question in their adoption in practical applications.

we have a set of dialogues in source domains and just a few seed dialogues in the target domain.


All bold-face results are statistically significant to p < 0.01. that MGT results in more general representations of language, thereby facilitating better transfer.

across both base architectures, MGT outperforms ensembling.


In such case, the personal information about "School" may be fake.

if n crt < n wrg , the worker's decision will be "Fraud" and the manager's decision will be "Fraud" too.


Then, based on the KG, we present structured dialogue management (Section 3) to explore the optimal dialogue strategy with reinforcement learning.

we find that, compared with Full-S and HP-S, MP-S is unable to learn any useful dialogue policy during training.


It means that using the hierarchical policy to unfold dialogues is necessary for our task.

the system actions in our task are selecting nodes in the KG to generate questions.


Given the same or similar dialog context, there may exist many valid expressions for the responses with the same dialog act a, each corresponding to a certain configuration of z.

this is because that taking average metrics may cause that the safety responses get higher scores than meaningful and diverse responses if most of these valid responses are not related to the ground-truth.


In this Corpus, the dialog act categories are {Inform, Question, Directive, Commissive}.

in our experiment, we also find that HAE-CT still faces serious safe response problems.


To tackle the challenges, we propose to take advantage of the hierarchical nature of response generation.

cNN models have been shown to be efficient for NLP and have achieved excellent results in sentence modeling and classification.


This method needs the preorganized topic/theme annotations for each conversation, which are prohibitively expensive to obtain.

note that if annotators rate different options, this triplet will be counted as "tie".


For example, when "The Shining" is mentioned, some of the top biased words are "creepy", "gory" and "scary", which are consistent with the style of the horror movie, and "stephen", who is the original creator of the movie.

then it combines the information with its hidden representation to form its updated representation at the next layer.


By further introducing the recommender system's knowledge of the items that have appeared in dialog, we guide the dialog system to generate responses that are more consistent with the user's interests.

kBRD (D) stands for only incorporating the dialog contents.


These actions can be regarded as users' feedbacks that reflect users' interest.

an ideal recommender dialog system is an endto-end framework that can effectively integrate the two systems so that they can bring mutual benefits to one another.


The word embedding is also 1000 dimension, trained from random initialization.

while moving towards z AE of an arXiv-style sentence "This would be an interesting viewpoint", the responses gradu-context Do you want to play a game?


Table 6 presents the generated examples of our models and baselines, our model can extract the keywords from the context which is helpful to generate an informative response, but the HRED model often generates safe responses like "Metoo" or "Y es".

we first rewrite the contexts in the test set with CRN, and then retrieve 10 candidates with the rewritten context from the index 6 .


Numbers in parentheses indicate the gains or losses after adding the persona conditions.

the gain achieved by the IMN ctx model is limited.


For the relevance score, the retrievalindependent Seq2Seq-MMI establishes a strong baseline.

relevance Fluency ours 2.69 3.11 3.20 Seq2Seq-MMI 2.64 3.02 3.14 Table 4: Ablation study on the ranker.


First, their skeleton extractor is pre-trained by the lexical overlap between the retrieved response and the golden response.

for a fair comparison, the lengths of the skeletons (the number of preserving words) generated by PMI and Keywords are kept as the same with the one generated by our skeleton extractor.


DSTRead (Gao et al., 2019) formulate the dialogue state tracking task as a reading comprehension problem by asking slot specified questions to the BERT model and find the answer span in the dialogue history for each of the pre-defined combined slot.

then for each s in S, the third CMRD generates the value sequence V d,s based on the corresponding h s,d .


When the size of paired data is small, the basic Seq2Seq model tends to generate more generic responses.

different from existing work, we assume that n is small (e.g., a few hundred thousands) and further assume that there is another set with T i a piece of plain text sharing the same characteristics with {Y i } n i=1 (e.g., both are questions) and N > n. Our goal is to learn a generation probability P (Y |X) with both d P and d U .


Indeed, existing big conversation data mix various intentions, styles, emotions, personas, and so on.

suppose that we have a dataset D is a pair of message-response, and n represents the number of pairs in D P .


We conduct our experiments on a short-text conversation benchmark dataset (Shang et al., 2015) which contains about 4 million post-response pairs from the Sina Weibo 2 , a Chinese social platforms.


This is consistent with our motivation that open-domain short-text conversation covers a wide range of topics and areas, and the top frequent words are not enough to capture the content of most training pairs.

many enhanced encoder-decoder approaches have been proposed to improve the quality of generated responses.


(1) The model tends to select more gain from a lower Kullback-Leibler (KL) divergence during training, which encourages the approximate posterior close to Gaussian prior, rendering the latent space of the former unused.

our work is supported by National Natural Science Foundation of China (61976154), Tianjin Natural Science Foundation (18JCY-BJC15500), National Natural Science Foundation of China (61771333), Tianjin Municipal Science and Technology Project (18ZXZNGX00330), and the Foundation of State Key Laboratory of Cognitive Intelligence, iFLYTEK(CIoS-20190001).


(2) The speakers in a dyadic conversation have different linguistic characteristics, sentiments and personalities.

(3) Compared with the Gau-Generators in Generator KL, the vMF-Generators (i.e., SVN, SSVN Gau−E and SSVN) have the much higher KL values, indicating that the vMF is a better selection than Gaussian to solve the KL-vanishing problem.


We compare our models with Information Retrieval (IR) based models and recommendation-only models.

additionally, the recommendations are not grounded in real observed movie preferences, which may make trained models less consistent with actual users.


First, our response generation task only includes turns where the system's dialogue act is CONFORM SQL.

after annotation and reviewing, 3,007 of them were finished and kept in the final dataset.


The difficulty of constructing dialogue-based NLIDB systems stems from these requirements.

the majority of previous work focus on converting a single, complex question into its corresponding SQL query.


In task-oriented dialog systems, the user simulator's task is to complete a pre-defined goal by interacting with the system.

we used policy gradient method to train dialog systems (williams, 1992).


When the dialog policy is bad, even if the NLG module can generate natural system responses, the users would still think the system is unnatural.

this is because SL-template's dialog manager is learned with supervised learning, less rigid than the agenda-based dialog policy, which further leads to a less rigid behaviour of the trained dialog system.


The performance of the simulator directly impacts the RL policy.

another common automatic metrics used to assess the language model is BLEU, but since this is a user simulator study and we don't have ground truth, BLEU score is not available.


One of the main drawbacks of HOCA is the generation of high-order tensor, the size of the high-order tensor will increases exponentially with the number of modalities as n i=1 t i , resulting in a lot of computation.

space (G) Training Time (s) HOCA-U 4.1 16418 HOCA-UBT 9.7 37040 L-HOCA-UBT 5.6 24615 The theoretical complexity of different attention mechanisms is illustrated in section 4.3.


We simply compare the descriptions generated by HOCA-U and L-HOCA-UBT, respectively.

we implement the multimodal correlation between the different modalities in a more efficient way with lowrank approximation which has been widely used in the community of vision and language (Lei et al., 2015;Liu et al., 2018;.


The goal of our work is to deal with unpaired image-caption data for image captioning.

baseline (10%) : a group of sheep standing next to each other.


To compute the cross entropy loss, we use the pseudo-label assignment for the Unlabeled-COCO images.

these pseudo-labels are not noisefree, thus treating them equally with the paired ones is detrimental.


After learning unimodal (H) and multimodal (Ĥ) representations of context, we use a Memory Fusion Network (MFN) (Zadeh et al., 2018a) to model the punchline ( , -∈/,$ , -∈/,0 , -∈/,1 2 .

a full understanding of humor requires analyzing the context of the punchline.


Execution of the command then requires mapping the command into the physical visual space, after which the appropriate action can be taken.

talk2Car contains images in realistic settings accompanied by free language in contrast to curated datasets such as MS-COCO.


We use only the articles from trustworthy media sources, according to our MBFC labels.


Table 1: Accuracy and Average Precision for individual feature types, calculated using 10-fold cross-validation using the Snopes dataset (S), and the Snopes+Reuters dataset (S+R).

there are cases when an image is completely legitimate, but it is published alongside some text that does not reflect its content accurately.


Our new model generalizes the planning space of VPN to reason about intermediate goals and obstacles, and includes recurrent action generation for trajectories with multiple goals.


During fine-turning, we continuously generate additional examples using model failures.

without the implicit reasoning discriminator (-Implicit discriminator), the additional examples make learning too difficult, and do not help.


E (I, B, R, T ) is a noncontextualized representation for each token and of its position in text, but also of the content and position of the bounding boxes.

vCR has much longer questions and answers compared to other popular visual Question Answering (vQA) datasets, such as vQA v1 (Antol et al., 2015), vQA v2 (Goyal et al., 2017) and GQA (Hudson and Manning, 2019), requiring more modeling capacity for language understanding.


GQA (Hudson and Manning, 2019) uses real scenes from Visual Genome, but the language is artificially generated.

we found this to be surprisingly competitive on VCR compared to published baselines, perhaps due to our choice of powerful pretrained models.


input sig-   nals that can help to differentiate domains apart have has no impact on the task itself.

they are prone to fail when there is a severe change in the marginal distribution that is task irrelevant.


The overall distribution of α values at convergence can be seen in Figure 3.

the decoder self and context attention only learn to distribute these parameters in a single mode.


Encoder Architecture: Another way to encourage this robustness is through inductive bias in the encoder architecture.

in this work we present a probabilistic latent variable model capable of disentangling stylistic features of fonts from the underlying structure of each character.


Since our model attempts to learn a smooth manifold over the latent style, we can also perform interpolation between the inferred font representations, something which is not directly possible using either of the baselines.

we propose a deep probabilistic approach that combines aspects of both these lines of past work.


These are broken down into training, dev, and test splits of 7649, 1473, and 1560 fonts respectively.

we use a convolutional neural network which takes in a batch of characters from a single font, concatenated with their respective character type embedding.


MacCartney and Manning (2009); Angeli and Manning (2014); and others incorporated knowledge of verb signatures within a natural logic framework (MacCart-ney, 2009;Sánchez Valencia, 1991) in order to perform natural language inference.

in this paper, we revisit the question of whether neural models of natural language inference-which are not explicitly endowed with knowledge of verbs' lexical semantic categorieslearn to make inferences about veridicality consistent with those made by humans.


None of the traditional criteria for basic color terms hold up robustly.

the sequence gamma can have larger magnitude if, despite poor sifting, the order of the basic terms is correct.


The results indicate that the BiLSTM-CRF framework is a better fit for encoding order information and long-range context dependency for such sequence labeling task.

in sentence S1, the negative focus is the propositional clause until the market close, yielding the interpretation that mutual fund trades take effect, but not until the market close.


Although methods like (Elsner et al., 2007;Li and Jurafsky, 2017) attempt to combine these aspects of coherence, to our knowledge, no methods consider all the three aspects together in a single framework.

our bi-LSTM sentence encoder gives a representation h i ∈ R 2p for each sentence s i in the document.


We observed that the naive pairwise ranking loss that uses a fixed margin unfairly penalizes the locally positive sentences during training.

the main differences between our implementation and the implementation referred in their paper are that we used a bi-LStM (as opposed to simple RNN) for sentence encoding and trained the network with the Adam optimizer (as opposed to AdaGrad).


As shown in Table 6, the addition of the global model and LM loss to the local model improves performance on the standard discrimination task by 1.34%.

this section presents details of our experiment procedures and results.


Subsequent studies extended the basic entity grid model.

we show that most of these models often fail on harder tasks with more realistic application scenarios.


Here, except the random model, all of the baselines are based on neural networks, which are typically more competitive than traditional approaches (e.g., utilizing handcraft features).

the primary goal of sentence set ordering is to put an unordered set of sentences into a coherent paragraph in the correct order.


This is related to the global dependencies of an entire paragraph.

we discuss the reason for our highest performance among previous models by classifying the factors affecting performance into four categories: sentence set modeling, topic latent vectors, permutation invariance, and attention at decoder.


In this section, we present TGCM as a novel topic-guided coherence modeling for SSO.

in Table 2-3, the several values of some models are directly taken from (Cui et al., 2018), while we implemented the rest of models using their public code with the same experimental setup.


Although TGCM-S do not utilize topical context features at encoder and decoder (such as t p and t s ), TGCM-S also employs the transformer-based attentive pointer decoder.

the pairwise model only learns the relative order from sentence-pair interactions.


The topic latent vector t s i captures how topical a sentence s i is in itself (local context), whereas the topic latent vector t p representes the overall theme of a paragraph p (global context).

other parameters were initialized randomly based on He et al.


Since our decoder directly receives the set of sentence vectors regardless of their input order via attention, our encoder is free from the constraint that all sentences must be compressed into a single fixedlength vector as a paragraph representation.

these models can capture global dependencies regardless of the order of input sentences.


The attention layer repeatedly decides the next sentence from sentence-pair interactions between the current predicted sentence and each paragraph-independent candidate {s i } 5 1 .

with the multiple attention modules, TGCM draws global dependencies among sentences by attending over the topic-sensitive sentence vectors repeatedly.


Its results were only slightly less good than those of stand alone GEN. To further investigate comparisons between different architectures for solving the attachment problem, we compared various local models extended with the MS-DAG decoding algorithm discussed in Section 5, giving the global results shown in the righthand columns of Tables 3 and  4.

we include LFs are expert-composed functions that make an attachment prediction for a given candidate: each LF returns a 1, a 0 or a -1 ("attached"/"do not know"/"not attached") for each candidate.


To generate a large number of discourse structures via distant supervision from sentiment, we propose a four-step approach, shown in Figures 2 and 3.

although in principle new corpora could be created, the annotation process is expensive and time consuming.


We study this question on the ACL and EMNLP paper collections and present an analysis on how well deep learning techniques can infer the authors of a paper.

a separate convolutional layer is dedicated to learning from part-of-speech sequences.


For instance, in Figure 1, with identified key elements, one can easily categorize the incident in dimensions of "age of harasser" (adult), "single/multiple harasser(s)" (single), "type of harasser" (unspecified), "type of location" (park) , "time of day" (day time).

karlekar and Bansal (2018) were the first group to our knowledge that applied NLP to analyze large amount ( ∼10,000) of sexual harassment stories.


This could explain why the regular BiLSTM model got lower performance than the CNN model.

the missing information is hence marked as "unspecified".


In this study, we manually annotated those stories with labels in the dimensions of location, time, and harassers' characteristics, and marked the key elements related to these dimensions.

when training with key element extractions, it put almost all attention on the harasser "young man" (S2), which helped the model make correct prediction of "young harasser".


The volume of social media data is huge, and the more we can extract from these data, the more powerful we can be as part of the efforts to build a safer and more inclusive communities.

the position of each word in this context sequence is crucial information for the CNN model to make the correct predictions (Nguyen and Grishman, 2015).


The results show that our approach in general performs better in the Twitter dataset.

to the best of our knowledge, our work is the first that investigate VAEs as a tool for training data expansion, so as to enhance machine learning performance with limited amount of labeled data.


We design a hierarchical seq2tree model that can better capture information of the AST of the equation.

these studies lost sight of important information of the math equation ASTs (e.g., parent and siblings of each node), despite of their promising results.


One point to note is that our training procedure is based on finding shortest paths in a complete KB, and we use this same procedure on the incomplete KB setting.

the classifiers used for retrieval in the different iterations are identical.


Our implementation shows comparable performance on the WikiMovies (MetaQA 1-hop) dataset, as shown in Table 3.

both models are limited by the number of facts and text that can fit into memory.


As we discussed, different from the traditional RC task, the answer string will appear multiple times in different paragraphs in OpenQA.

besides, in the distantly supervised setup (Mintz et al., 2009), it is postulated that the paragraphs that contain the answer string are ground truths, while even some of the positive paragraphs are wrong-labeled as they do not concern the question, e.g.


These paragraphs are inherently relevant to each other and provide enhanced evidence for the correct answer.

this indicates that SearchQA contains more positive paragraphs than other datasets.


The scoring function of ComplEx is given by where e s , e o , w r ∈ C n are the embeddings of s, o, and r, respectively.

a matrix is block circulant if it can be written in the form 1b) .


in this paper, we strive toward the above two issues via incorporating diversified contexts and answer-aware loss.


Comparison with NAQANet The major difference between our model and NAQANet is that NAQANet does not have the reasoning module, i.e., M 0 is simply set as M P .

our NumNet model outperforms strong baselines with a large margin on the DRoP dataset.


Hence, the learned model behaves more like a question language model with some loose context constraint, while it is unaware of the strong requirements that it should be closely grounded by the context and should be answered by the given answer.

these works either reduced the ground-truth data size or simplified the span-prediction QA task to answer sentence selection.


In terms of syntactic correctness, KEAG and MHPGM both perform well thanks to their architectures of composing answer text and integrating knowledge.

an overview of the neural architecture of KEaG is depicted in Figure 1 3.2 Sequence-to-sequence model KEaG is built upon an extension of the sequenceto-sequence attentional model (Bahdanau et al., 2015;Nallapati et al., 2016;See et al., 2017).


Each tuple (question, answer, and document) is fed into the BERT model as: " [CLS]  As a remark, we want to highlight that both the DRD and ARD discriminators have been trained on different datasets than the final multiple choice QA model, which is trained and evaluated on the ARC Easy and Challenge datasets.

as shown in Figure 2, each supporting document is associated with a list of scores computed by the discriminators.


This decision step is accomplished by a simple feed-forward network (Figure 3).

replacing GrU with Long-Short Term Memory (LSTM) cells (Hochreiter and Schmidhuber, 1997) gives similar performance but the training procedure is computationally more expensive.


Humans capture that information by learning, by experiencing and from commonsense knowledge.

the weights in dictate to what extent the network should attend each document.


In this work we use linguistic annotations as a basis for a Discourse-Aware Semantic Self-Attention encoder that we employ for reading comprehension on narrative texts.


to be specific, in the system without the paragraph-level neural retrieval module, we re-trained the sentence-level retrieval module with negative sentences directly sampled from the term-based retrieval set and then also re-trained the downstream QA or verification module.


Progress on MRS has been made by improving individual IR or comprehension sub-modules with recent advancements on representative learning (Peters et al., 2018;Radford et al., 2018;Devlin et al., 2018).

this gives the final term-based retrieval set for HOtPOtQA.


The task has gained popularity with its natural combination of information retrieval (IR) and machine comprehension (MC).

to validate the system design decisions in Sec 3 and reveal the importance of semantic retrieval towards downstream, we conducted a series of ablation and analysis experiments on all the modules.


We add a "maybe" choice in PubMedQA to cover uncertain instances.

under reasoning-free setting where the annotator can see the conclusions, a single human achieves 90.4% accuracy and 84.2% macro-F1.


To which organ system do the esophagus, liver, pancreas, small intestine, and colon belong?

although various neural Qa methods have achieved high performance on some of these datasets Trivedi et al., 2019;Tymoshenko et al., 2017;Seo et al., 2016;Wang and Jiang, 2016;De Cao et al., 2018;Back et al., 2018), we argue that more effort must be dedicated to explaining their inference process.


Unsurprisingly, training and testing in the same domain (e.g., Fiction) leads to the best performance on sentence selection.

in MultiRC, AutoROCC outperforms our baseline of BERT + entire passage (row 10 vs 22) by 8.3% EM0, indicating that AutoROCC can filter out irrelevant content.


Note that some of the works discussed here transfer knowledge from external datasets into the QA task they address (Chung et al., 2017;Pan et al., 2019;Min et al., 2017;Qiu et al., 2018;Chen et al., 2017).

the improvements are not large, e.g., the maximum improvement is 1.6% (when k = 4), which indicates that ROCC is robust to a certain extent to lexical variation.


Open-domain question answering (QA) Inspired by the series of TREC QA competitions, 2 Chen et al.

centered around entity names, this model risks missing purely descriptive clues in the question.


Because off-the-shelf IR systems generally optimize for shallow lexical similarity between query and candidate documents in favor of efficiency, a good proxy for this overlap is locating spans of text that have high lexical overlap with the intended supporting documents.

it is computationally inefficient, and has high variance especially for the second reasoning step and forward, because the context depends greatly on what queries have been chosen previously and their search results.


The input to the first autoencoder is the word embedding of the natural language.

we collect all the answers where they have a single code block and their corresponding questions are labeled with an SQL tag from 'Posts.xml'.


An example question and four query substructures with highest prediction prob-abilities are shown in the top of Figure 4.

figure 3 depicts the framework of SubQG, which contains an offline training process and an online query generation process.


Path Ranking Algorithm (PRA) is the first path-based reasoning method.

when the agent selects a valid action, Line 12 is executed.


For each (h, r) pair, there is a ground truth t and about 10 generated false t. It is divided into a training set and a test set.

the pretrained embedding dimension is set to 100.


Many links could miss among the entities in a KG.

valid action means that there is an output relation connected with the current entity, while invalid action denotes that there is no relation.


Given a knowledge graph G = (E, R, T), where E, R and T are the entity set, relation set and KG triple set, respectively, and a news text snippet for which entity linking has been performed to build the mentioned entity set L ⊂ E, the text-based knowledge graph updating task is to read the news snippet S and update T accordingly to get the final triple set T and the updated graph G = (E, R, T ).

this requires far more complicated updating rules.


Finally, to accommodate the fixed input dimension of the discriminator D, we can simply select a subset P e ⊆ Ω E with the top N occurrence frequency, where N is normally much smaller than |Ω E |.

as for the longer paths, despite their potential utility values, they are more likely to contain worthless or even noisy inference steps, thus we learn them only in the training phase.


Nevertheless, the evidential paths they used are gathered by random walks, which might be inferior and noisy.

automated reasoning on KGs has been a longstanding task for natural language processing.


Finally, through such a trialand-error process, the well-trained policy-based agent can be used to find evidential paths for predictions.

recently, DeepPath (Xiong et al., 2017) and MINErVA (Das et al., 2018) were proposed to address the problem above by using reinforcement learning, where they are committed to learn a policy which guides the agent to find more superior evidential paths to maximize the expected reward.


The classical approach is to find the optimal reward function by inverse reinforcement learning (IRL) (Russell, 1998;Ng et al., 2000) to explain expert behaviors.

to encourage the agent to find more diverse evidential paths, it is desirable to train the agent by imitating each trajectory instead of each of its state-action pairs.


Considering the model complexity, we also add l 2 -norm of all learnable parameters to the final loss function.

only using BiLSTM fusion without deep fusion, the accuracy drops by 2.1% on the test set of SciTail dataset.


When one sentence learns attention to another sentence, the attention is performed between two parallel layers and oriented to the intermediate representations from the preceding layer of another one.

given the representations of P and Q: , each cross attention P(t)→Q will use the original semantics H 0 Q of Q for interaction, where t = {1, • • • , T} and t = 1 represents P using the original representation H 0 P .


The first five models in Table 3 are all implemented in (Khot et al., 2018).

semantics to be paid attention are uncertain and unstable for matching because semantics are changed at different layers.


Moreover, the methods stated above assume information from different paths between an entity pair only contributes to the relation inference linearly.

its simple structure has flaws in dealing with complicated relations like 1to-N, N-to-1 and N-to-N.


In addition, to shown the effect of collaboration, we train a model variation with the parameters of the extractor frozen.

cPL denotes our proposed final model (with all the components).


A straightforward solution for the OKGR problem is to directly add extracted facts (by a pre-trained relation extraction model) to the graph.

similarly, we train the extractor on the corpus labeled by distant supervision.


It first uses PCNN-ATT to extract relational triples from the corpora, and augments KG with the triples whose prediction confidences are greater than a threshold.

the value of γ determines how the parameters are updated and to what extent the the internal states will be influenced by the future.


Inspired by the coattention mechanism , we propose an attention-based inferer (ABI) to estimate the mean and standard deviation of a distribution belongs to p θ (•) or q φ (•) by capturing semantic interactions of input sequences.

vari-ational Autoencoder (vAE) based models have shown great potential in modeling the one-tomany problem and generate diversified inferences (Bowman et al., 2015;Zhao et al., 2017).


To simulate human reasoning process, ADIN is stacked with multiple asynchronous inference sub-layers, and each sub-layer consists of two inferential modules in an asymmetrical manner.

furthermore, we study the effect of two inferential modules in one asynchronous inference sub-layer.


In this paper, we propose an asynchronous deep interaction network (ADIN) for natural language inference.

different from a simple semantic matching task, reasoning should be asynchronous and fully interpretable (Yi et al., 2018).


This may be due to many reasons, such as more typos and emojis within the original reviews, and so forth.


This is useful for reasons such as marketing, overall enjoyment of interaction, and mental health therapy.


(1 -very negative, 3 -neutral, 5 -very positive) Each participant evaluates every piece of text.

nouns typically carry more, and phrases the most (since they consist of multiple words).


because English and French share many similarities, both KECG and baselines get satisfactory results on DbP15K FR−EN .


We find probes on ELMo2 to be strikingly more selective than those on ELMo1, consistent across all probes, both for part-of-speech tagging and dependency head prediction.

they take this as evidence that gains in part-of-speech probing accuracy on the trained representations over the untrained representations are due to linguistic properties, not memorization.


Constraining the number of training examples is effective for part-of-speech, suggesting that learning each linguistic task requires fewer samples than our control task.

(2017) defined completely random tasks related to Rademacher complexity (Bartlett and Mendelson, 2001) to understand the capacity of neural networks to overfit, showing that they are expressive enough to fit random noise, but still function as effective models.


The IB method introduces the new random variable T, the tag sequence that compresses X, by defining the conditional distribution p ✓ (t | x).

the Maryland Advanced Research Computing Center provided computing facilities.


, x n , it is possible that x i contains not only information about word i but also information describing word i + 1, say, or the syntactic constructions in the vicinity of word i.

language  Table 1: Statistics of the datasets used in this paper.


We encode each goldsegmented sentence in our treebank via the ELMo model for that language, which yields a tensor S ELMo = R N ×L×D , where N is the number of words in the sentence, L = 3 is the number of ELMo layers, and D = 1024 is the ELMo vector dimensionality.

a closer look at the breakdown per language reveals that this picture is slightly distorted by different sentence length distributions in different languages.


The scoring is based on the top three words on the stack and the first word of the buffer, and the input to the MLP includes the BiLSTM vectors for these words as well as their leftmost and rightmost dependents (up to 12 words in total).

one of the most striking results, however, is that both parsers improve their accuracy on longer sentences, with some models for some languages in fact being more accurate on medium-length sentences than on shorter sentences.


Semantic parsing is the task of mapping natural language to machine interpretable meaning representations, which in turn can be expressed in many different formalisms, including lambda calculus (Montague, 1973), dependency-based compositional semantics (Liang et al., 2011), frame semantics (Baker et al., 1998), abstract meaning representations (AMR; Banarescu et al.

we also conducted various ablation studies to examine the contribution of individual features to crosslinguistic semantic parsing.


By leveraging a multilingual BERT self-attention model pretrained on 104 languages, we found that fine-tuning it on all datasets concatenated together with simple softmax classifiers for each UD task can meet or exceed state-of-the-art UPOS, UFeats, Lemmas, (and especially) UAS, and LAS scores, without requiring any recurrent or language-specific components.

we performed a high-level visual analysis of the BERT attention weights to see if they have changed on any discernible level.


We display an average of scores over all (89) treebanks with a training set.

an interesting observation is that for Lemmas and UFeats, the classifier prefers to also incorporate the information of the first 3 layers.


The OpenBookQA dataset provides the core science fact used to create the question.

collecting knowledge gaps for such questions and common-sense knowledge to capture these gaps are interesting directions of future research.


It is worth noting that recent large-scale language models (LMs) (Devlin et al., 2019;Radford et al., 2018) have now been applied on this task, leading to improved state-of-the-art results (Sun et al., 2018;Banerjee et al., 2019;Pan et al., 2019).

the baseline models were developed for this dataset using hyper-parameter tun-ing; we do not perform any additional tuning.


We present the test accuracies on the two ques-  Table 4: Test accuracy on the the OBQA-Short subset and OBQA-Full dataset assuming core fact is given.

we take a different approach of using this background knowledge in an explicit inference step (i.e.


No Relation Score: We ignore the entire relation-based score (score r ) in the model and only rely on the fact-relevance score.

these tables are expensive to create and these questions often need multiple hops (sometimes up to 16 (Jansen et al., 2018)), making reasoning much more complex.


The grounded schema graphs are usually much more complicated and noisier, unlike the ideal case shown in the figure.

for each question concept c i ∈ C q and answer concept c j ∈ C a , we can efficiently find paths between them that are shorter than k concepts 4 .


Due to this property of SQuAD, translating the triple (p, q, a) is non-trivial: when p and a are translated into target language, p t and a t , a t ⊆ p t may no longer hold.

due to the commonality of BidAF and BERT, taking question and passage as input and generating passage representations for answer span prediction, we can apply BERT to our approach, as shown in Figure 2 (Center), by modifying Refinery module of modeling confidence score.


Therefore, we need to find the answer span a t in the target language, by matching with a s in the source language s. We overviewed a highprecision baseline  where a s and p s are independently translated, and a t is found only when the translation of a s is exactly and consecutively found in that of p s .

our method is extensible to any language with an existing open-source NMT, leveraging only its attention scores (Bahdanau et al., 2014).


This material is based upon work supported by the National Science Foundation under Grant No.

two sentences within paradigm form a minimal pair if they differ in one feature (licensor, NPI, or scope), but have different acceptability.


For example, a popular view states that NPIs are licensed if and only if they occur in downward entailing (DE) environments (Fauconnier, 1975;Ladusaw, 1979), i.e.

a result showing generally good performance for a model should be regarded as possibly hiding actual differences in performance that a different task would reveal.


(2017) with the aid of a bilingual lexicon.


However, even F still has d 2 entries, which is far too many to be practical.

let t ∈ [0, 1] d be a vector of typological features for language ∈ D t ∪ D e .


We finetune a BERT model on the IMHO+context dataset described in Section 3.2 using both the masked language modeling and next sentence prediction objectives.

we assume we have ADUs and predict components and relations at that level.


On inspection, this model is mostly predicting SUPPORTED for most adversarial instances.

it has been observed in related NLP tasks that as models become more complex, it is difficult to fully understand and characterize their behaviour (Samek et al., 2017).


Miculicich Werlen and Popescu-Belis (2017) collect such a list for English-French for the pronouns it and they.

generated noisy example 2: He was creative, generous, funny, loving and talented, and we will miss him dearly.


We use zero-padding (shown as shaded boxes) to make P r and P s fixed-length.

it is available for each source language for which English translations are generated in the WMT tasks, e.g., Czech, French, German, etc.


P s ) as query vectors, the rows of K r (resp.

the focus was on cross-lingual pronoun prediction, which required choosing the correct pronouns in the context of an existing translation, i.e., this was not a realistic translation task.


We also confirmed that the multilingual redundancy of Wikipedia was effective: using more languages led to significantly better performances.

such causalities satisfy the above three desiderata.


Figure 5 (a) shows the effect of travel status on review summarization.

model Testing At test time, We use beam search with a beam size of 4.


To perform the aspect-level evaluation, we first define seven aspects: location, service, room, value, facility, food, and hotel, where hotel describes the overall attitude.

extensive experiments on TripAtt show that ASN achieves state-of-the-art performance on review summarization.


Therefore, We take gender, age and travel status as our attribute information, and collect near 3 million attribute-reviewsummary triplets.

1 , and 2 show strategies based on attribute embedding, and represent Attribute Selection strategy, and Attribute Prediction strategy, respectively.


Second, users also explicitly label their travel status when booking hotels, such as traveled solo or traveled on business.

the best per-   formance is obtained when the attribute-specific vocabulary size is 100, that's the reason why we set our attribute-vocabulary size to 100.


Furthermore, they find that the Average Word Embedding sentence encoder works at least as well as encoders based on CNN and RNN.

adding a representation of the whole document (i.e.


(ii) We test our method on the Pubmed and arXiv datasets and results appear to support our goal of effectively summarizing long documents.

more recently, inspired by the success of LSTm-minus in both dependency and constituency parsing, Liu and Lapata (2017) extend the technique to discourse parsing.


Specifically, a gate p gen ∈ [0, 1] is introduced to switch between copy mode and generation mode.

besides relying on parallel data and text as in previous work, our model attends to relevant external knowledge, encoded as a temporary memory, and combines this knowledge with the context representation of data before generating words.


This result demonstrates the relevance of external knowledge and sparseness of original data are the main factors affecting system 1 Available at https://github.com/hitercs/ WikiInfo2Text performance.

in the references, nationality of a person is frequently mentioned.


Note that the rewards usually reflect the quality of extracted summary and measured by standard evaluation protocol.

then an earlier and more detailed sentence about "fans rioting" is appended to the summary by performing re-reading.


Inspired by the reading cognition of human beings, we propose HER, a two-stage method, to mimic how people extract summaries.

they still sequentially process text and tend to extract earlier sentences over later ones due to the sequential nature of selection (Dong et al., 2018).


In the dynamically masking method, we order the weights from big to small at first, then go on masking two neighbors until the ratio between them is over a threshold.

the training objective is to let the relevant part attended by attention c contribute more to the summarization, while let the irrelevant or less relevant parts attended by attention o contribute less.


inspired by the phrase-based translation models, Yao et al.


The loss can be calculated as follows: where y (1) and y (2) are the outputs of two tasks.

in addition, both MS and MT can significantly help to produce better summaries.


2We use the LDC MT dataset, which belongs to the news domain similar to our En2ZhSum, during the training of CLS+MT.

then we employ LexRank (Erkan and Radev, 2004), a strong and widely used unsupervised summarization method, to summarize the translated document.


Although end-to-end deep learning has made great progress in natural language processing, no one has yet applied it to CLS due to the lack of large-scale supervised dataset.

in this work, we introduce a novel approach to directly address the lack of data.


We also visualize in Figure 3 a comparison between Pointer-Gen+ARL-SEN and Pointer-Gen+RL-SEN according to how sensational the test set headlines are.

in this paper, we propose a model that generates sensational headlines without labeled data.


How to be a county party secretary like JIAO Yulu?

the results on the test set show that the baseline classifier gets 60% accuracy, which is worse than the proposed classifier (which achieves 65%).


And the color of orange/black further indicates the better model and its score.

the following shows an example of a RL model maximizing the sensationalism score by generating a very unnatural sentence, while its sensationalism scorer gave a very high score of 0.99996: 十个可穿戴产品的设计原 则这消息消息可惜说明 Ten design principles for wearable devices, this message message pity introduction.


A headline generation model can be trained with MLE, RL or a combination of MLE and RL.

none of these work considered the sensationalism of generated outputs.


The training model is also optimized in a way that adapts to different data, which is based on a novel method of distantly-supervised learning guided by reference summaries and testing set.

first, this graph provides a huge concept space with multi-word terms that cover concepts of worldly facts as concepts, instances, relationships, and values 2 .


The morphological realisation (MR) module consists in producing inflected word forms based on lemmas coupled with morphological features.

since for the system predictions we do not have a parse tree, we additionally record the distance between head and dependent (in the reference data) and we compare it with the distance between the same two items in the system output.


For the exact match, most languages score above average (from 0.51 to 0.71).

the third line (WO+MR (S −c )) shows the BLEU scores for our WO+MR model, i.e., when the MR model is applied to the output of the WO model.


although our model needs the matched pseudoparallel corpus as a starting point, it has high tolerance to recover from occasional low-quality matches.


This indicates that subsequent iterative refinement indeed allows for 30−50% low-quality pairs in the initial pseudo-parallel corpus.

the average difference gap between the calculated score, 52.63, and a perfect BLEU score, 100, shows that the five human rewritten sentences contain significant lexical differences.


Additionally, both rewards require reference summaries.

we leave the learning of a generic summarisation evaluation metric for future work.


The extractor, on the other hand, applies an actor-critic RL algorithm on top of a pointer network.

bERT+MLP+Pref significantly outperforms (p < 0.05) all the other models that do not use bERT+MLP, as well as the metrics that rely on reference summaries (see Table 1).


Generating accurate and diverse paraphrases automatically is still very challenging.

to the best of our knowledge, this is the first work on using GAN with VAEs for paraphrase generation task.


Backpropagation of gradients from the discriminative model to the generative model is difficult for sequence generative models.

our proposed adversarial training and two-path loss take an apparently positive effect on alleviating the exposure bias problem and generating diverse but realistic predictions.


(2018) utilized results from pre-executed operations to improve the fidelity of generated texts.

since the record in the row is not sequential, we use a self-attention network which is similar to Liu and Lapata (2018) to model records in the context of other records in the same row.


2019, we conducted the following human evaluation experiments at the same scale.

unlike the unordered nature of rows and columns, the history information is sequential.


Following notations in Section 2.3, β t,i ∝ exp(score(d t , row i )) obtains the attention weight with respect to each row.

we keep most recent history information sequence within history window and denote them as hist(r i,j ).


The resultant summary consists of salient words or phrases that are selected by integer linear programming (ILP).

sTDs and sTDs/NC take both elements into consideration and perform much better than these baselines.


In fact, abstractive approaches for MDS are more like words or phrases recombination.

many studies attempt to construct well connected graphs, e.g., group sentences into clusters (Wan and Yang, 2008;Banerjee et al., 2015) and construct dense features with distributed embeddings (Yasunaga et al., 2017).


Recently, neural generative models have shown promising results in paraphrase generation.

all the data are manually labeled whether two sentences convey the identical meaning.


We should lower the reward of that good output for other generators, but reducing the reward of Y * to zero will affect the stability of other generators.

rewards will be increased as the generators increase the generation probability of better sequences while decreasing the chances of worse sequence generation.


Therefore, the word embeddings used as additional input play an important role in helping the model to deal with language information.

the complete model has slightly more parameters than the model without graph encoders (57.6M vs 61.7M).


Abstractive summarization is a task to generate a short summary that captures the core meaning of the original text.

perfect consistency between attention weights occasionally disturbs the model to generate proper outputs.


Table 1 shows the evaluation results with their consistency.

our proposed HCL with non-normalized attention weights can accurately compute this inconsistency, contrary to the HCL with normalized attention weights.


Feedback comment generation in general is the task of generating feedback comments given a text (referred to as essay, hereafter).

then, each sentence is tokenized and put into lowercase.


We find that our model can capture distant answer-relevant dependencies such as "mean temperature" while the proximity-based baseline model wrongly takes neighboring words of the answer like "coldest" in the generated question.

our  Table 5: Performance for the average relative distance between the answer fragment and other non-stop sentence words that also appear in the ground truth question (BLEU is the average over BLEU-1 to BLEU-4).


We find that the performance drops at most 36% when the relative distance increases from "0 ∼ 10" to "> 10".

different from Heilman and Smith (2010) which directly use the simplified sentence for generation and Cao et al.


In the first case, there are two subsequences in the input and the answer has no relation with the second subsequence 4 .

to address this issue, we extract the structured answer-relevant relations from sentences and pro-pose a method to jointly model such structured relation and the unstructured sentence for question generation.


Then they adopt an overgenerate-and-rank approach to select the best candidate considering several features.

they explicitly encode the relative distance between sentence words and the answer via position embedding and positionaware attention.


we believe that especially in 5 http://www.statmt.org/wmt17/ translation-task.html 6 http://opus.nlpl.eu/EMEA.php 7 Our baseline scores differ from those provided by the task organisers because of differences in truecasing strategies.


Besides, the results indicate that our generated reviews have no statistical difference in terms of the helpfulness scores from those written by consumers towards certain products, with average helpfulness scores 3.10 and 3.03 for machinegenerated and real-world reviews respectively.

besides the statistical and semantical metrics, we also design an empirical study to test the personalized performance of our generated reviews.


The proposed RevGAN model follows a three-staged process: In Stage 1, we propose to use Self-Attentive Recursive Autoencoder for mapping the discrete user reviews and product descriptions into continuous embeddings for the advantage of capturing the ''essence" of textual information and the convenience for subsequent optimization processes.

we condition the sentiment labels on the discriminator to artificially change the rules that the discriminator works, and force it backpropagate loss functions that update generator policy correspondingly.


It is defined as the fraction of unique n-grams in the summary that are novel, normalized by the length ratio of the generated and reference summaries.

this shows the possible benefits that can be obtained by exposing the model to the evaluation data in unsupervised setups.


(Paulus et al., 2017) combined supervised and reinforcement learning, demonstrating improvements over competing approaches both in terms of ROUGE and on human evaluation.

our experiments show that the best performing set of metrics consists of RoUGE-L in conjunction with QA conf and QA f score , both computed at an article-level, and hence unsupervised.


We adopted the following automatic metrics.

results The annotation results in Table 4 show that our model significantly outperforms baselines in both metrics.


Among the 1,000 generated texts, 79.0% of texts have scores above 4.

at time step t, the plan decoder makes a binary prediction for each input item by estimating P (d i ∈ g t |g <t , x, z p ): where σ denotes the sigmoid function, h i is the vector of input item d i , and h p t is the hidden state of the plan decoder.


Similarly, to measure conciseness, we report how often summaries contain at least one non-restrictive relative clause (RelCl.)

for instance, a human writer places the underlined sentence in the input article next to the first sentence in the summary to improve topical coherence as they are about the same topic ("elections").


NeuSum ) uses a seq2seq model to predict a sequence of sentences indices to be picked up from the document.

our model consists of a sentence extraction model joined with a compression classifier that decides whether or not to delete syntax-derived compression options for each sentence.


Our high-level approach to summarization is shown in Figure 1.

these summaries frequently contained short sentences to fill up the budget, and the collection of summaries returned tended to be less diverse than those found by beam search.


Typically, an encoder encodes the input x i to a semantic representation c i , while a decoder controls or modifies the stylistic property and decodes the sentence x i based on c i and the pre-specific style l i .

the overall style transfer performance is still nonoptimal.


The classifier objective L T style thus dominates Eqn.

explicitly learning precise stylized information within each domain is crucial to generate domain-specific styles.


For example, consider a passage from the SQuAD dataset in Table 7, where except the question word who, the model sequentially copies everything from the passage and achieves a QBLEU score of 92.4.

one way to approach this is by re-visiting the passage and answer with the aim to refine the initial draft by generating a better question in the second pass and then improving it with respect to a certain aspect.


• Reddit (Ouyang et al., 2017): is a collection of personal posts from reddit.com.

mmR is extremely biased to the importance aspect on XSum and Reddit.


We looked into the individual word pairs marked as proper nouns in the DE and DA data, as these languages are related and RCSLS performs comparably on them otherwise, and did not find any patterns that could explain the large differences.

by more canonical we mean, for example, indefinite instead of definite forms of nouns and adjectives (see Ex.


Alternative approaches for enrichment exists, of course, but we wonder how worthwhile further efforts would be.

the uncovered issues of high noise levels (proper nouns) and limited coverage (missing gold standard targets) clearly have a crucial impact on BDI results obtained on the MUSE dataset, and need to be addressed.


Recently, morphological inflection has frequently been cast as a sequenceto-sequence task, mapping the characters of the input word together with the morphological features specifying the target to the characters of the corresponding inflected form (Cotterell et al., 2018).

closely is early stopping: a separate development or validation set is used to end training as soon as the loss on the development set L D (θ) starts increasing or model performance on the development set D starts decreasing.


Development sets are impractical to obtain for real low-resource languages, since using all available data for training is often more effective.

hyperparameters are taken from Sharma et al.


We obtain a different picture for TRANSL: results are equal for 3 languages, and better for DevLang for the remaining 2.

while longer training seems better for MORPH, not all performance loss can be explained by aborting training too early.


Constraining search that way increases the run time as the γ-bounds are lower.

greedy and beam search both achieve reasonable BLEU scores but rely on a high number of search errors 5 to not be affected by a serious NMT model error: For 51.8% of the sentences, NMT assigns the global best model score to the empty translation, i.e.


Among these, some works are implicitly on temporal commonsense, such as event durations (Williams, 2012;Vempala et al., 2018), typical temporal ordering (Chklovski and Pantel, 2004;Ning et al., 2018a,b), and script learning (i.e., what happens next after certain events) (Granroth-Wilding and Clark, 2016;Li et al., 2018).

for instance, given two events "going on a vacation" and "going for a walk," most humans would know that a vacation is typically longer and occurs less often than a walk, but it is still challenging for computers to understand and reason about temporal commonsense.


1 for the five phenomena studied here and Table 1 for basic statistics of it.

existing works have not studied all five types of temporal commonsense in a unified framework as we do here, nor have they developed datasets for it.


Jia and Liang (2017) addressed this problem and proposed an adversarial version of the SQuAD dataset, which was created by adding a distractor sentence to each paragraph.

as shown in Section 2, the maximization of MI needs positive samples and negative samples drawn from joint distribution and the product of marginal distribution respectively.


In this paper, we propose a meta-based algorithm for multi-hop reasoning (Meta-KGR) to address the above problems, which is explainable and effective for few-shot relations.

to the best of our knowledge, this work is the first research on fewshot learning for multi-hop reasoning.


(2018) create an NLI test set specifically to show the deficiencies of state-of-the-art models in inferences that require lexical and world knowledge.

2 Taking special account of both the switchable and associative instances suggests the following evaluation protocol for a given model.


We do so by systematically examining threats to the validity of experiments involving recent CSR models.

when evaluating on SwAG, it is important to determine whether the prediction relies on an understanding of the context or on shallow patterns in the LM-generated counterfactuals.


One is that these type of sentences can get a high generation probability since the generator is actually a language model.

considering the simplicity of the model and the ease of training, we adopt the neural constrained language model of  as the generator.


is the first endeavor to apply neural network to this task, which adopts a constrained neural language model (Mou et al., 2015) to guarantee that a pre-given word sense to appear in the generated sequence.

1 Generating creative and interesting text is a key step towards building an intelligent natural language generation system.


For generator, we first tag each word in the English Wikipedia corpus with one word sense using an unsupervised WSD tool 2 .

the generator can be any model that is able to generate a pun sentence containing a given word with two specific senses.


In our joint model, grid search is used to determine β and results are shown in Figure 2.

further, Table 4 gives two examples of the generated questions on SQuAD dataset, by the base-line model and our joint model respectively.


In this case, the symmetry of T is broken by larger unary potentials.

setup Unconditional generation is still considered a challenging task for both, GANs and latent stochastic models,  and standard RNNs form a very competitive baseline (semeniuta et al., 2018).


as ψ(w i , w j ) = A ij for some parameter matrix A ∈ R V ×V .

this confirms that our model can learn beyond pairwise interactions through the latent chain.


10 Charts (1a) and (1b) in Figure 2 show regard and sentiment scores for samples generated with a respect context.

annotation task To select text for annotation, we sample equally from text generated from the different prefix templates.


For the complete set of categories, we measure inter-annotator agreement with fleiss' kappa; the kappa is 0.5 for sentiment and 0.49 for regard.

we define the regard towards different demographics as a measure for bias.


(4) The evaluation of the proposed masking strategy on the two fact verification datasets indicates that, while the in-domain performance remains on par with that of the model trained on the original, lexicalized data, it improves considerably when tested in the out-of-domain dataset.

to mitigate this dependence on lexicalized information, we experiment with two strategies of masking.


Also the distribution of labels was made similar to that of FNC.

with organization-c1, the misc-c1 entered commercial service.


Idiosyncrasies Distorting Performance We investigate the correlation between phrases in the claims and the labels.

our analysis of the data demonstrates that this unexpectedly high performance is due to idiosyncrasies of the dataset construction.


The aspect embeddings are initialized randomly and learned as parameters.

we find that the expressions of conflict opinions usually are lengthy and implicit.


The key problems were that many of the regions were far too granular (e.g.

more complex tasks like Named-Entity Recognition often rely on contiguous, cleanly segmented text for successful processing.


In addition, we include the results using neural models such as CNN and RNN in Table 4.

intuitively, words with higher topic coherence and lower degree of overlapping among different topics should be assigned higher reward in the next iteration of learning.


In this work, a novel method for determining a stopping criterion is proposed that models the rate at which relevant documents occur using a Poisson process.

a range of studies have demonstrated that ranking the results of the Boolean query can support the identification of relevant studies by placing them higher in the ranking, e.g.


Second, relevance judgments in nearly all newswire test collections are annotations on documents, not on individual sentences or passages.

we begin with BERT Large (uncased, 340m parameters) from Devlin et al.


It performed worse (highest RBO .091), likely due to being more sensitive to unusual subword unit combinations generated by the MT system.

(2017) rather than weakly supervised BM25 scores (Dehghani et al., 2017), which would produce a smoothed tf.idf neural BM25 model that behaves differently than a standard neural IR model.


Transformers (Vaswani et al., 2017) already make extensive use of linear maps in multi-headed attention and appear to be justified in doing so.

there is no evidence that models exclusively represent relationships in this manner.


While the meanings of defining words are important in dictionary definitions, it is crucial to capture the lexical semantic relations between defined words and defining words.

neural pattern-based word-pair embedding models (Washio and Kato, 2018a,b;Joshi et al., 2019) unsupervisedly learn two neural networks: a word-pair encoder and pattern encoder, both of which encode the word-pair and lexico-syntactic pattern respectively into the same embedding space.


To provide the definition decoder with information regarding lexical semantic relations, we use an additional loss function with word-pair embeddings as follows: where S is a set of stopwords.

lexical semantic relations in definitions are not explicit.


In countries that speak multiple main languages, mixing up different languages within a conversation is commonly called codeswitching.

from the left image of figure 2, in word-level, the model mostly chooses the correct language embedding for each word, but also combines with different languages.


unlike GPA-based approaches, MPPA does not require a multi-way dictionary, but only bilingual dictionaries which are much easier to obtain even in an unsupervised manner.


Since a parallel corpus, albeit small, is available, formality style transfer usually takes a seq2seq-like approach (Rao and Tetreault, 2018;Niu et al., 2018a;Xu et al., 2019b).

this work is supported in part by the National Natural Science Foundation of China (Grand Nos.


In the past few years, style-transfer generation has attracted increasing attention in NLP research.

the preprocessed sentence serves as a Markov blanket, i.e., the system is unaware of the original sentence, provided that the preprocessed one is given.


(2018) adopt an eraseand-replace approach and design their methods to erase the style-related words first and then fill in words of different style attributes.

previous works often suffer from content leaking problem.


We follow the assumption that g(•, •) is a DAG consisting of N nodes and edges among them (Liu et al., 2019;Xie et al., 2019b;Pham et al., 2018).

we find that the model failed to optimize when n = 2.


Figure 1 and 2 shows that JoBi not only consistently performs the best over the entire range of parameters, but also delivers a performance improvement that is especially large when the batch size or the negative ratio is small.

it should be noted that on FB15K-237, all JoBi models outperform all the baseline models, regardless of the base model used.


Bilinear models such as DistMult and ComplEx are effective methods for knowledge graph (KG) completion.

in our preliminary experiments on baselines, we found that the choice of loss function had a large effect on performance, with negative log-likelihood (NLL) of softmax consistently outperforming both max-margin and logistic losses.


Other often used dimensionalities include 200 (Tang et al., 2015;Ling et al., 2016;Nickel and Kiela, 2017) and 500 (Tang and Liu, 2009;Perozzi et al., 2014).

the exact definition of 'low' dimensionality is rarely explored.


We further demonstrate our model's increased capabilities on humor identification problems, such as the previously created datasets for short jokes and puns.

this previous research has gone into many settings where humor takes place.


Nodes will diverge because local gradients differ.

(2017) with eight layers of bidirectional LSTM consisting of 225M parameters.


While the rule of thumb is to scale learning rate linearly with batch size (Goyal et al., 2017), the Transformer model is also sensitive to high learning rates (Aji and Heafield, 2019).


To train a multilingual BERT model for our sequence prediction tasks, we add a softmax layer on top of the the first wordpiece (Schuster and Nakajima, 2012) of each token 3 and finetune on data 3 We experimented with wordpiece-pooling (Lee et al., 2017) which we found to marginally improve accuracy but at a cost of increasing implementation complexity to maintain.

mMiniBERT performs well and outperforms the state-of-the-art Meta-LSTM on the POS tagging task and on four out of size languages of the Morphology task.


We wish to estimate p(x|y), the conditional probability of utterance x given dialogue act y.

let x and y denote an input sentence and the corresponding semantic representation respectively.


On PTB, we also compare to two models using structural information in language modeling: parsing-reading-predict networks (PRPN; Shen et al., 2018a) predicts syntactic distance as structural features for language modeling; orderedneuron LSTM (ON-LSTM; Shen et al., 2018b) posits a novel ordering on LSTM gates, simulating the covering of phrases at different levels in a constituency parse.

a language model does not have access to future words, and hence running a backward RNN from right to left is less straightforward: one will have to start an RNN running at each token, which is computationally daunting (Kong et al., 2016).


The first uses logic for semantic representation, including ATIS (Price, 1990;Dahl et al., 1994) and GeroQuery (Zelle and Mooney, 1996).

to this end, the tencent multilingual embeddings are chosen, which contain both Chinese and English words in a multi-lingual embedding matrix.


Intuitively, the same semantic parsing task can be applied cross-lingual, since SQL is a universal semantic representation and database interface.

as discussed above, column names are selected by attention over column embeddings using sentence representation as a key.


Thus, the parameters of the gating GCN are trained from the relevance loss and the usual decoding loss, a ML objective over the gold sequence of decisions that output the query y. Discriminative re-ranking Global gating provides a more accurate model for softly predicting the correct subset of DB constants.

finally, we compute two oracle scores to estimate future headroom.


In transductive learning, because an unlabeled test set can be used for training, it is possible to adapt LMs directly to the word distributions of the test set.

we fine-tuned an LM on an unlabeled test set.


Experiments using the SentEval suite showed that DCT embeddings outperform the commonlyused vector averaging on most tasks, particularly tasks that correlate with sentence structure and word order.

both EigenSent⊕Avg and ELMo performed better than all other models on SST-5.


The BERT model architecture consists of multiple layers of Transformers and uses a specific input representation, with two special tokens, [CLS] and [SEP], added at the beginning of the input sentence pair and between the sentences (or bag of sentences) respectively.

the semantics of a sentence are usually more dependent on local context, rather than all sentences in a long docu- ment.


(a) We observe significantly different levels of complexity dependent on the population ratio.

the cost function is then: whereV (h) refers to using the predicted value but not updating the value sub-network according to this loss function.


We next examine what happens when we expose different linguistic communities to each other.

become more tightly coupled, as evident from the higher success rate among those in Fig.


The first row is based on the balanced dataset, and the rest are based on the imbalanced dataset with different oversampling ratios.

we extracted instances using the same methods as described above, but we filtered out COMMENT/REPLY pairs in which a condescension-related word appeared.


This model, called HEURISTIC LABELS, serves to demonstrate how well the abstractive model would perform with a perfect extractive step.

we believe this reflects the difficulty of evaluating this task and summarization in general.


An example instance of the summary cloze task is presented in Figure 1.

being able to predict the next sentence of a summary is an important component of a full topic-focused MDS system and a worthwhile task on its own.


The second recurrent layer consists of regular Gated Recurrent Units (GRUs), which are used to update the polished fact from q k to q k+1 using h k Tm .

hefei Prison proposed a commutation sentence since PERS had repentance and received two awards during his sentence.


On one hand, the sections in the prototype summary that are not highly related to the prototype document are the universal patternized words and should be emphasized when generating the new summary.

we investigate the influence of the iteration number when facts are extracted.


While the space of programs is exponential, we observed that abstract programs which are instantiated into correct programs are not very complex in terms of the number of production rules used to generate them.

each column is encoded by averaging the embeddings of words under its column name.


The structural constraint discussed above now corresponds to assuming that each span in a question can be aligned to a unique row or column slot.

the representation of a slot is contextually aware of the entire abstract program (Dong and Lapata, 2018).


In this work, we focus on only the DM formalism.

then, the next source node index d u i+1 is the same as the target node the module points to.


In the case of UCCA, we sort the children based on their UCCA node ID.

in Table 2, we also conduct ablation study on beam search to investigate contributions from the model architecture itself and the beam search algorithm.


Scalars p gen , p enc and p dec act as a soft switch to control the production of target node label from different sources.

mST algorithms have to be used to search for a valid prediction.


As suggested by Dozat and Manning (2016), we project v i and h t to a lower dimension for reducing the computation cost and avoiding the overfitting of the model.

we experiment with the following traversal variants: (1) random, which sorts the sibling nodes in completely random order.


LN is the layer normalization (Ba et al., 2016) and h 0 t is always initialized with s 0 .

the top node, in our case p2, only received errors from the top node's softmax.


6 The reason is that the random order potentially produces a larger set of training pairs since each random order strategy can be considered as a different training pair.

if we adopt the Smatch-weighted metric, our method achieves a better score i.e.


We define x i to be the vector at node i (in the example trigram, the We skip the standard derivative for Ws.

the combined strategy benefits from both worlds.


Empirically, we find that computation time is manageable when limiting the application of s t (•) to candidates that share the same entities as the unlabeled utterance.

the starting point is a user who needs a semantic parser for some domain, but has no data.


It assumes that if two entities have a relation in KGs, then all sentences mentioning the two entities express this relation.


NER is an essential pre-processing step for many natural language processing (NLP) applications, such as relation extraction (Bunescu and Mooney, 2005), event extraction (Chen et al., 2015), question answering (Mollá et al., 2006) etc.

we thank the anonymous reviewers for their insightful comments.


Specifically, on OntoNotes and MSRA, 'w/o T-graph' obtains worse performance than others, showing that Tgraph is important.

• We achieve the state-of-the-art results in various popular Chinese NER datasets, and our model achieves a 6-15x speedup over the existing SOTA model.


We can clearly see that removing any graph causes obvious performance degradation, but the importance of different graphs varies from dataset to dataset.

for instance, " ¬: : ::" (Beijing Airport) and ": : ::" (Airport) are the self-matched words of the character ": : :" (airplane).


Compared with lattice LSTM (Zhang and Yang, 2018), Our model gains a 0.91% improvement in F1 score.

word information is very useful in Chinese NER, because word boundaries are usually the same as named entity boundaries.


This work has been supported in part by National Science Foundation SMA 18-29268, DARPA MCS and GAILA, IARPA BETTER, Schmidt Family Foundation, Amazon Faculty Award, Google Research Award, Snapchat Gift and JP Morgan AI Research Award.

we found that, by explicitly adapting the model along label distribution shift, consistent improvements can be achieved on distant supervision but not on human annotations.


We choose MLP as the interaction function in our DGLSTM-CRF according to performance on the development set.

how to make good use of the rich relational information as well as complex long-distance interactions among words as conveyed by the complete dependency structures for improved NER remains a research question to be answered.


(2017)) yields the following estimator: Examining the expression above, we can see that for a fixed value of a, the numerator of the of the ratio grows with the frequency of w i in either language L 0 or L 1 .

these models did not achieve any substantial performance gain to justify their additional complexity.


Evaluations on multiple text classification tasks show that ProSeqo significantly improved accuracy compared to state-of-the-art on-device neural network SGNN (Ravi and Kozareva, 2018) on short text with +3.4% for MRDA, +4.5% on SNIPS, +5.3% on SWDA and +8.9% on ATIS; and long documents with +23% on Amazon product reviews, +33.9% on AG news, +35.9% on Y!A.

for ATIS and SNIPs, the most recent state-of-art approaches use joint intent and slot prediction model (Hakkani-Tur et al., 2016;Liu and Lane, 2016;Goo et al., 2018), where the slot model recognizes the enti- AG Y!A AMZN ProSeqo (our on-device model) 91.5 72.4 62.3 SGNN (Ravi and Kozareva, 2018)(on-device) 57.6 36.5 39.3 fastText-full (Joulin et al., 2016) 92.5 72.3 60.2 CharCNNLargeWithThesau.


(Liu and Lane, 2016) showed that joint intent and slot model improves upon the individual ones.

we noticed that the test set of the Y!A dataset in (Yang et al., 2016) was smaller compared to the data used in (Zhang et al., 2015;Joulin et al., 2016) and our experiments.


However, this sample-wise comparison may be severely disturbed by the various expressions in the same class.

we choose it as an interaction function in this paper.


In the distance metric learning models (Matching Networks, Prototypical Networks, Graph Network and Relation Network), all the learning occurs in representing features and measuring distances at the sample-wise level.

we aim to perform meta-learning on the training set, and extract transferable knowledge that will allow us to deliver better few-shot learning on the support set and thus classify the test set more accurately.


1 1 https://cogcomp.seas.upenn.edu/page/ publication_view/883 The plague in Mongolia, occurring last week, has caused more than a thousand isolation Supervised text classification has achieved great success in the past decades due to the availability of rich training data and deep learning techniques.

unfortunately, some classes are still challenging, such as "evacuation", "infrastructure", and "regime change".


Then those explanations are parsed into formal constraints which are further combined with unlabeled data to yield new label oriented classifiers through posterior regularization.

in this work, we only tried to make use of the class names and their definitheir probabilities after softmax, then do softmax to get new probabilities.


We expect that the constrained models should have accuracies at least on par with the baseline (though one of the key points of this paper is that accuracy by itself is not a comprehensive metric).

our goal is to build models that minimize inconsistency with domain knowledge stated in logic.


The example in §1 is an allowed label assignment.

with t-norms to relax logic, we can systematically convert rules as in (1) into differentiable functions, which in turn serve as learning objectives to minimize constraint violations.


It produces undesirable outcomes: the encoder yields meaningless posteriors that are very close to the prior, while the decoder tends to ignore the latent codes in generation (Bowman et al., 2015).

the joint distribution is typically assumed as a standard multivariate Gaussian.


Recently, researchers have also extended this method to unsupervised learn-ing (Hsu et al., 2019) and online learning scenarios .

we propose to learn lowrank embeddings of sentences by tensor decomposition to capture their contextual semantic similarity, which works well regardless of the size of documents (Hosseinimotlagh and Papalexakis, 2018).


On CPU, InferSent is about 65% faster than SBERT.

we compute average GloVe embeddings using a simple for-loop with python dictionary lookups and NumPy.


These two options are also provided by the popular bert-as-a-service-repository 3 .

sentEval fits a logistic regression classifier to the sentence embeddings.


Let us consider the AUC risk, i.e., bipartite ranking risk (Narasimhan and Agarwal, 2013): Figure 1: An overview of the framework.

we also provide the results with varying thresholds and heuristic thresholds in Appendix C, where the trends of performance for each method do not differ much from Table 4.


Unlike supervisedlearning, where we have positive and negative data, the threshold information is provided from labels and we can draw the decision boundary accordingly.

let sym be a symmetric loss such that sym (z) + sym (−z) = K, where K is a positive constant.


We demonstrate how to optimize the area under the receiver operating characteristic curve (AUC) effectively and also discuss how to adjust it to optimize other well-known evaluation metrics such as the accuracy and F 1measure.

one can learn a bipartite ranking function effectively by AUC optimization in our setting, which suggests the idea to learn a reliable ranking function first, and then adjust the threshold.


multi-view spectral clustering (MVSC) (Kanaan-Izquierdo et al., 2018) is a competitive standard multi-view clustering approach.

here, the target utterances are sampled randomly from the corpus, and the context utterances are sampled from within each pair of adjacent utterances.


To this end, we adopt prototypical networks (Snell et al., 2017), a metric learning approach, to solely rely on the encoders to form the classifiers instead of introducing additional classification layers.

multi-view spectral clustering can work with representations that are separately learned for the individual views and the multi-view information is aggregated using the common eigenvectors of the data similarity Laplacian matrices.


ENTY"What was the first satellite to go into space?"

0.97 "What is the name of David Letterman's dog?"


Hence, a mapping from D to a l-dimensional embedding, with l n, is naturally provided by the projection hidden function˜ x = c U S − 1 2 .

moreover, the investigated approach is is computationally affordable, as it roughly corresponds to a forward pass across the network.


"What was the last year that the Chicago 0.73 "The film Jaws was made in what year?"

lRP allows to identify fragments of the input playing key roles in the decision, by propagating relevance backwards.


For VAE-0.5 (Bowman et al., 2016), we implement KL annealing by increasing the KL weight linearly from 0.1 to 1.0 in the first 10 epochs and adopt word dropout rate of 0.5 to alleviate the posterior collapse.

the estimation and maximization of the MI in the high-dimensional space are difficult.


We now expand on the brief literature review in Section 1 to better contextualize our work.

we also outperform K-center greedy Coreset at all sampling percentages without utilizing additional diversity-based augmentation.


Thus, it seems to have a inductive bias for class boundaries, similar to the above works.

ensembles versus Single Models: A similar experiment was conducted to investigate the overlap between a single FTZ model and a probabilistic committee of models (5-model ensemble with FTZ (Lakshminarayanan et al., 2017)) to identify comparative advantages of using ensemble methods.


At each iteration, we train the model on the current train set and use a model-dependent query strategy to acquire new samples from the pool, get them labeled by an oracle and add them to the train set.

our findings are as follows: (i) We find that utilizing the uncertainty query strategy using a deep model like FastText.zip (FTZ) 1 to actively construct a representative sample provides query and train sets with remarkably good sampling properties.


(ii) We finds that a single deep model (FTZ) used for querying provides a sample set similar to more expensive approaches using ensemble of models.

we establish a new stateof-the-art baseline for further research in deep active text classification.


In this paper, we show that if we have prior knowledge of such biases, we can train a model to be more robust to domain shift.


This reduces to bias product when g(x i ) = 1.

it is the features we will use in our bias-only model.


Transforming it into a three-player game helps to improve the performance of the evaluation set while lower the accuracy of the complement predictor.


Performance The clues in text classification tasks are typically short phrases (Zaidan et al., 2007).


Hence, a possible third direction is to instead approximate the objectives to the un-normalized model distributions.

1 In the following, we will derive an objective function from a divergence D with the data distribution as first argument, and the second being the distribution of the parametric model: A large number of divergence measures has been introduced for a variety of applications (Basseville, 2013).


They show gains of up to 1 point in perplexity, and a slight improvement over the MLE baseline for WT2, confirming the results obtained on a single model.

it is interesting to note that the power transformations will here be applied on the posterior classification probabilities p C θ instead of categorical probabilities p θ .


In this work, we explore the possibility of affecting how words are learned depending on their frequency by using alternative loss functions.

their training cost grows linearly with the number of words in the vocabulary, often making it prohibitively slow.


We display these values for our models trained with exact objectives in Table 4.

the three divergences presentend in Section 3 are defined on positive measures: in theory, we can simply use the exp function on the scores s θ and do not need to normalize them: neither the α and β divergences are scale invariant (see right column of table 1 and Cichocki and Amari 2010).


For an input x, we consider perturbationsx, in which every word x i can be replaced with any similar word from the set S(x, i), without changing the original sentiment.

data augmentation is unable to cover the exponentially large space of perturbations that involve many words, so it does not prevent errors caused by changing many words.


In addition, we would like to apply the proposed methods for other pretrained models.

the training loss of fine-tuning BERT tends to monotonously decrease along the optimization direction, which eases optimization and accelerates training convergence.


Our contributions are: • A reinforcement learning formulation to guide question-asking strategies for learning from language.

4 While setting up the optimization problem, the value of µ can be adapted to reflect this intuition.


In the E-step of the Posterior Regularization training (Ganchev et al., 2010), the computation of the posterior regularizer remains unchanged.

this follows a Wittgensteinian view of language as a cooperative game (Wittgenstein, 1953) between agents (here, the teacher and a learner) with a shared goal (here, building an effective classifier).


At each step t, the learner's action a t consists of choosing a question to ask the teacher.

in scenarios where there is a lot of labeled data available enabling robust inductive inference, we would like to primarily rely on it rather than explanations.


However, by compressing the web search results into a knowledge graph, we significantly reduce the number of tokens by an order of magnitude and make it possible for a model to access the entirety of the search information.

current approaches extractively select portions of web text as input to Sequence-to-Sequence models using methods such as TF-IDF ranking.


To facilitate question answering, the dataset provides the top 100 web search hits from querying the question, which results in 200K words on average.

these approaches have been applied to extractive question answering tasks that require span identification, rather than abstractive text generation in an information synthesis setting.


we also list the wMT18 top-2 systems for De!En translation in Table 4: RwTH (Graça et al., 2018) and UCAM (Stahlberg et al., 2018) systems, which are both ensemble models.


Neural machine translation (briefly, NMT) (Bahdanau et al., 2014;Gehring et al., 2017;Vaswani et al., 2017) is well-known for its outstanding performance (Hassan et al., 2018), which usually relies on large-scale bitext for training.

when turning to clean tuning, we can obtain another 1.2 and 2.3 points improvement.


We want to express gratitude to the anonymous reviewers for their hard work and kind comments, which will further improve our work in the future.

knowledge graph embedding models map relations and entities into continuous vector space.


They use a score function to measure the truth value of each triple (h, r, t).

is the score for negative sample (h i , t ′ i ) corresponding to current positive entity pair should be small for task T r which represents the model can properly encode truth values of triples.


Thus gradients of parameters indicate how should the parameters be updated.

furthermore, using loss gradient as one kind of meta information is inspired by MetaNet (Munkhdalai and Yu, 2017) and MAML (finn et al., 2017) which explore methods for few-shot learning by metalearning.


Our MetaR is tested on different settings of datasets introduced in Table 3.

gradients of parameters indicate how should the parameters be updated.


These errors on in-vocabulary terms can be explained by inconsistencies in annotation across the two domains: • In the PPCEME, to may be tagged as either infinitival (to, e.g., I am going to study) or as a preposition (p, e.g., I am going to Italy).

as a secondary evaluation, we measure performance on the full PTB tagset in Table 3, thereby enabling direct comparison with prior work (Yang and Eisenstein, 2016).


These pronouns are significant sources of errors for baseline models: for example, a BERT-based tagger makes 216 errors on 272 occurrences of the pronoun thee.


On the other hand, the last example is very difficult for humans (row 4), possibly due to the relatively neutral text.


The goal here is not to build an ensemble of DNNs to surpass current classification state of the art results, but instead to test our hypothesis to determine if machine RPs can fit IRT models that can benefit NLP tasks.

as this posterior is usually intractable, VI approximates it by the variational distribution: Where π θ j () and π b i () denotes different Gaussian densities for different parameters whose means and variances are determined by minimizing the KL-Divergence between q(θ, b) and π(θ, b|Y ).


generate target sequences iteratively but require the target sequence length to be predicted at start.

we propose a method to exploit token embeddings of the pre-trained language model to warm-start the training of edits like appends and replaces which are associated with a token argument.


For each pair of responses (one from ARAML and the other from a baseline, given the same input post), five annotators were hired to label which response is better (i.e.

aRaML a man is wearing a hat and holding a toothbrush as he stands on the grass of a field.


We evaluate this model on three neural machine translation (NMT) benchmark datasets, achieving comparable performance with state-of-the-art nonautoregressive NMT models and almost constant decoding time w.r.t the sequence length.

for FlowSeq to predict the entire sequence in parallel, it needs to know its length in advance to generate the latent sequence z.


To better understand why the approach achieves these gains, we designed the experiments on SCAN domain to address the following questions: (1) Does the proposed model work in the expected way that humans do (i.e., visualization)?

we thank Kenneth Church, Mohamed Elhoseiny, Ka Yee Lun and others for helpful suggestions.


In training, We expect this is possible when we have prior knowledge that Y 1 depends only on X 1 and Y 2 depends only on X 2 .

both input words and Algorithm 1 Proposed approach.


For TurnLeft task, the accuracy drops only in D, but not in B or C, indicating at least one of entropy regularization is necessary.

it appears with other combinations of words at test time.


10, 000 examples are held out to serve as the validation set.

if we replace the [MASK] token with a pronoun instead of the correct candidate, the resulting sentence sometimes sounds unnatural and would not occur in a human-written text.


While DBMs are able to correctly identify a small set of triples with relative features such as (skyscraper, apartment, tall) as discriminative, interpretation of relative relations requires two types of features which are not targeted by the models, namely: (i) the extraction of precise numerical reference points at scale (dealing with variations of dimensional units) and (ii) the ability to extrapolate the relations for unobserved lexemes by an explicit mechanism of comparative/transitive reasoning.

while incidental attributes can be captured as a second-order distributional phenomena, these corpora do not reflect explicit commonsense knowledge, in particular with regard to extra-linguistic phenomena.


Definition Based Models: False negatives comprise the majority (83%) of all model errors.

discriminative attributes can also occur as incidental, sensorial or relative instances.


The higher compression ratio renders it more challenging for the student model to absorb important weights.

with the number of epochs increasing, the student model learned with this vanilla KD framework quickly reaches saturation on the test set (see Figure 2 in Section 4).


Different from previous knowledge distillation methods (Hinton et al., 2015;Sau and Balasubramanian, 2016;Lu et al., 2017), we adopt a patient learning mechanism: instead of learning parameters from only the last layer of the teacher, we encourage the student model to extract knowledge also from previous layers of the teacher network.

once the network has been trained, there will be a high degree of parameter redundancy.


This further confirms what we showed in Figure 2.

this phenomenon is called Posterior Collapse, where the Kullback-Leibler (KL) divergence between the posterior and the prior (often assumed to be a standard Gaussian) vanishes.


This is equivalent to maximizing the following evidence lower bound ELBO, In this case, Mean-field (Kingma and Welling, 2014) assumption is often used for simplicity.

uNK said the company 's uNK group is considering a uNK standstill agreement with the company traders said that the stock market plunge is a uNK of the market 's rebound in the dow jones industrial average one trader of uNK said the market is skeptical that the market is n't uNK by the end of the session the company said it expects to be fully operational by the company 's latest recapitalization i was excited to try this place out for the first time and i was disappointed .


ization (Chen et al., 2008) to get the Cholesky factor L. The covariance matrix Σ = w • I + aa T formed in this way is guaranteed to be positive definite.

we assume that the members of variational family Q are dimensional-wise independent, meaning that the posterior q can be writ- The simplicity of this form makes the estimation of ELBO very easy.