In general, Story (images + local keywords) shows a great performance with the highest diversity regardless of the levels, as well as the highest preservation at the soft measure on the story-level.

we use Story(images+local keywords) as the input setting for our final data collection, which has the most balanced performance on both metrics.


2017), such systems have too high data consumption -including both collection and annotation effort -in order for them to be used in rapidly paced industrial product cycles.

approaches to training such systems with extremely limited data (i.e.


Although in our few-shot approach we use full in-domain dialogues, we end up having significantly less in-domain training data, with the crucial difference that none of those has to be annotated for our approach.

the method we introduced attains state-of-the-art in both accuracy and data-efficiency.


They aren't available in MetaLWOz, and in SMD, KB snippets share little in common across domains.

in order to increase Entity F1, KB information should be directly copied to the output more efficiently -and increasing the robustness of the copy-augmented decoder is one of our future research directions.


Finally, based on intuitive analysis, we find the applicants' behavior is related to some factors (Section 5.1).

we introduce hypotheses to formalize the effect of these factors on the applicants' behavior and propose a heuristic user simulator to evaluate our systems.


", the applicant may not know the answer because the distance between "Gulou Park" and "Gulou Subway Station" is less than 100 meters.

we want the known information of "Gulou Subway Station" to be sent to its successors.


We expect the system can give correct decisions about applicants within minimum turns.

at the end of each dialogue, the manager receives a positive reward r m crt for correct decision, or a negative reward −r m wrg for wrong decision.


Generally, forging information about "School" and "Company" may result in a larger loan.

when sampling the fake information, the sampling probability of "School" and "Company" is twice the sampling probability of "Residence" and "BirthPlace".


This is because simulated applicants tend to forge personal information about "School" and "Company" for a larger loan.

to discover fake information faster, the manager learns to prioritize different information items.


However, Full-S notices the applicant gives the correct answer to a question that is hard to answer for fraudsters.

full-S does not make the decision in haste but continue the dialogue.


Specifically, because the two entities are closely related to each other, if applicants know "Xiao Liu Fruit", they may well know "HaoDe".

there is no need to select this triplet anymore.


The other important reason is that the response generation model cannot express the intention and emotion internally.

one line of research has focused on forcing the model to simulate some human's skills by augmenting the input with rich meta information.


In the speaker's knowledge base, there may be varieties of appropriate expressions that can be found to represent his current intention.

a meaningful response can be produced by choosing one of the expressions and filling it with relevant content.


It was found that the convolutional layer can extract the common patterns within the local regions of the input utterance (Kim, 2014).

the text convolutional network proposed in (Kim, 2014) is leveraged to mine the relationship between expression patterns and responses.


Compared to the parameters in the multi-layer HRED, expression reconstruction, and the embedding of vocabulary, the number of parameters in EA can be ignored.

it can be concluded that EA plays a key role in improving output diversity with few parameters due to its efficient architecture.


To integrate texts into a knowledge graph, we take a factoid knowledge graph (KG) as its backbone, and align unstructured sentences of nonfactoid knowledge with the factoid KG by linking entities from these sentences to vertices (containing entities) of the KG.

we augment the factoid KG with non-factoid knowledge, and retain its graph structure.


Then we segment the documents into sentences and align each sentence with entries of the factoid KG by mapping entities from these sentences to entity vertices of the KG.

we augment the factoid KG with nonfactoid knowledge, and retain its structured representation.


In the two datasets, some sentences in the background knowledge are directly used as responses to the messages.

the NLG module is likely to copy content from the selected knowledge as much as possible for generation.


The product M [1:Nr] ζ results in a weight matrix W , where each row is computed by one slice j = 1, 2, • • • , N r of the tensor: Although such parameterization seems to be straightforward, due to the quadratic size of M [1:Nr] , the parameter size of such parameter adapter is N ζ times larger than the basic encoder-decoder model.

overfitting and expensive computational cost make it infeasible (Bertinetto et al., 2016;Ha et al., 2017).


Besides, the incorporation of external knowledge can strengthen the connections between systems and enhance their performances.

driven by the motivations, we propose a novel end-to-end framework that integrates the two systems.


w s ∈ R d and b s ∈ R are the switcher's parameters and σ refers to the sigmoid function.

the whole system can be trained in an end-to-end fashion.


Different from their work, b u is computed based on the recommender system's hidden representation of user u: where F : represents a feed-forward neural network and t u is the user representation in the recommendation context introduced in Equation 5.

the computation of the top layer of the decoder becomes: So far, we have built an end-to-end framework that bridges the recommender system and the dialog system, which enables mutual benefits between the systems.


The dialogs with only 0-2 mentioned movies take up a proportion of 62.8% of the whole testing dataset.

it is important for the system to perform high-quality recommendation with only a small number of mentioned movies.


For style transfer, however, x i and s j is not paired.

we instead minimize the distance between a point and its nearest neighbor from another dataset to pull these two datasets close to each other in the shared latent space.


As r depends on l, the dimension of z, We define a normalized value As the stylized texts are usually sparse, it is possible to generate non-stylized hypothesis as we vary ρ along some direction.

we rank the hypotheses considering both relevance and style intensity.


We also tried to fine-tune the BERT to directly generate the restored utterance, but the result is far from satisfactory.

we directly append selected words to the original input sequence as extra guidance for generating restored sentences.


But generated responses are not clear and coherent enough to understand.

no human study is further conducted for such approach.


One possible reason is that the IMN model performs attention-based interactions between the context and the response in order to get their matching feature for response selection.


To our observation, however, there is no existing conversational data paired with dynamic knowledge graphs.

we collect a TV series corpus-DyKgChat, with facts of the fictitious life of characters.


As fictitious life in drama, realistic responses often use knowledge entities existing in multi-hop relational paths, e.g., the residence of a friend of mine.

we propose a model that incorporates multi-hop reasoning (Lao et al., 2011;Neelakantan et al., 2015; on the graph structure into a neural conversation generation model.


Intuitively, for each query, the matching model should give the highest score to the golden response over other M − 1 responses (i.e., always rank the golden response at the first place).

we define the training loss as where S k: is k-th row of S. Label smoothing (Szegedy et al., 2016) of value ls = 0.1 is used to improve the performance.


The logical operator ">" indicates which retrieved information should have a higher priority to be returned to the user.

we are interested in building DST modules capable of generating structured sequences, since this kind of sequence representation of the value is critical for accurately capturing the belief states of a dialogue.


DSTRead (Gao et al., 2019) formulate the dialogue state tracking task as a reading comprehension problem by asking slot specified questions to the BERT model and find the answer span in the dialogue history for each of the pre-defined combined slot.

its inference time complexity is still O(n).


Since the model can handle various emotions, we train the model with the entire 2 million Twitter message-response pairs labeled with a positive, negative, and neutral sentiment.

when we only focus on responses with positive sentiment, ECM actually performs multi-task learning for response generation.


With the latent variables, CVAE brings both content and noise into responses.

the gap between S2S-Temp and CVAE is more significant on fluency and relevance than that on richness.


This is consistent with our motivation that open-domain short-text conversation covers a wide range of topics and areas, and the top frequent words are not enough to capture the content of most training pairs.


@User 2 User 2 User 4 "my pleasure :)" leads to complex speaker-addressee interactions.

the speaker and addressee roles associated with utterances are constantly changing among multiple users across different turns.


And it also benefits for both building a group-based chatbot and clustering users based on what they have said.

we propose a new task of identifying the addressees of all the missing utterances given a multi-party conversation session where part of the addressees are unspecified.


Such phenomenon reflects that our W2W model won't make a prediction recklessly on occasions where there is no clear addressee.

the variance on matching scores of all listeners in our W2W model, to some extent, provides a signal of whether the utterance has a explicit addressee even if we do not provide any supervision information on this aspect during training.


However, the sampled information from latent space usually becomes useless due to the KL divergence vanishing issue, and the highly abstractive global variables easily dilute the personal features of replier, leading to a non replier-specific response.

a novel Semi-Supervised Stable Variational Network (SSVN) is proposed to address these issues.


Observed from the curves of all metrics, the best performance of embedding-based metrics is achieved at λ = 0.7, while the diversity reaches the peak in the mutual promotion stage.

we set λ to 0.7 in all previous experiments.


Furthermore, in many cases, multi-turn interaction between users and NL systems is needed to clarify ambiguous questions (e.g., Q 3 and R 3 in Figure 1), verify returned results, and notify users of unanswerable or unrelated questions.

a robust dialogue-based NL query agent that can engage with users by forming its own responses has become an increasingly necessary component for the query process.


The input of the modified model is a user utterance and the related column names.

we replace one of the two sentence embedding modules with a database column name encoding module, which generates representations of the col-


This is because the final time-step contains the context of all the words preceding it, and serves as a good approximation to the representation of the entire utterance.

the shared encoder finally gives us the representations of each utterance, i.e.


Ubuntu and OpenSubtitles, do not have dialogue act annotations, and the MRDA corpus is too small, it has only 51 conversations.

we evaluate the performance of our model on SwDA and DailyDialog datasets: • SwDA: Switchboard Dialogue Act Corpus (Stolcke et al., 2000) is annotated on 1155 human to human telephonic conversations.


However, RL requires interaction with the environment, and obtaining real human users to interact with the system is both time-consuming and labor-intensive.

building user simulators to interact with the system before deployment to real users becomes an economical choice (Williams et al., 2017;.


But as stated before, to obtain real user evaluation is time-consuming.

many automatic metrics have been studied to evaluate a user simulator (Pietquin and Hastie, 2013;Kobsa, 1994) from different perspectives.


(2016) pointed out, these automatic metrics barely correlate with human evaluation.

ai and Litman (2011a) involved human judges to directly rate the simulated dialog.


The ultimate goal of the user simulator is to build a task-oriented RL system to serve real users.

the most ideal evaluation should be conducted by human.


Therefore, the most ideal evaluation should be conducted by human.

we first asked human to read the simulated dialogs and rate the user simulator's performance directly.


5) but no such rules exist in SLbased simulators.

the RL model will experience the "anything else" scenario more in Agenda-based simulators than in SL-based simulators.


Compared with image data, video data have rich multimodal information, such as image, motion, audio, semantic object, text.

the key is how to utilize the information.


One of the main drawbacks of HOCA is the generation of high-order tensor, the size of the high-order tensor will increases exponentially with the number of modalities as n i=1 t i , resulting in a lot of computation.

we implement the multimodal correlation between the different modalities in a more efficient way with lowrank approximation which has been widely used in the community of vision and language (Lei et al., 2015;Liu et al., 2018;.


The goal of our work is to deal with unpaired image-caption data for image captioning.

we mainly focus on image captioning and unpaired data handling literature.


The topics of these presentations are diverse; from scientific discoveries to everyday ordinary events.

of diversity in speakers and topics, TED talks span across a broad spectrum of humor.


As a result of diversity in speakers and topics, TED talks span across a broad spectrum of humor.

this platform presents a unique resource for studying the dynamics of humor in a multimodal setup.


Using forced alignment, we mark the beginning and end of each sentence in the video as well as   words and phonemes in the sentences (Yuan and Liberman, 2008).

an alignment is established between text, visual and acoustic.


The results from Table 4 demonstrate that while a state-of-the-art model can achieve a reasonable level of success in modeling humor, there is still a large gap between human-level performance with state of the art.

uR-FuNNY dataset presents new challenges to the field of NLP, specifically research areas of humor detection and multimodal language analysis.


The series of scenes (both related and unrelated) in video frames bring a huge challenge for the task of video captioning.

mastering the ability to process the correlated and irrelevant semantic information can improve the performance and interpretability of the model of video captioning.


This indicates that differential neurons in brain are activated for lexical selection of action and object words.

the part of speech (POS) tag can be applied to guide the flowing of different semantics into the corresponding network streams.


Depending on the attended visual feature, a formulae is introduced to ascertain whether the current t word is a visual word or not: where c v t is the weight of visual word at time step t, p i t is i-th value in the t-th POS tag representation p t .

the weight of non-visual word is c nv t = 1 − c v t .


We collected all of these pictures from four consecutive years: 2015, 2016, 2017, and 2018.

we ended up with a total of 395 True imageclaim pairs.


We find that the content of the dialogue usually has a certain continuity, and the subsequent dialogue is likely to continue what was discussed before.

it's important to infer valuable information from the dialog history to help answer the question.


And because there is 11: 12:  two outputs of the dialog encoder, O f q and O sq , we utilize two individual attention process units (APU-3) to decode their information.

we could obtain two different answer (or question) distributions based on the appearance and motion information correspondingly.


Finally, universal attacks are a unique model analysis tool because, unlike typical attacks, they are context-independent.

they highlight general input-output patterns learned by a model.


Although the preprocessed dataset in  is widely used, it only consists of 4 domains and the input documents are reduced to their TF-IDF weights which is not a suitable input for modern NLP architectures.

we perform a different prepro-  For part of speech tagging we select 4 publicly available 9 Universal Dependencies datasets for English (Nivre et al., 2016).


The attention distribution of each head is predicted typically using the softmax normalizing transform.

all context words have non-zero attention weight.


The softmax mapping (Equation 2) is elementwise proportional to exp, therefore it can never assign a weight of exactly zero.

unnecessary items are still taken into consideration to some extent.


Critically, each style latent variable is shared by all characters within a font, while character embeddings are shared by characters of the same type across all fonts.

our approach is related to a long literature on using tensor factorization as a method for disentangling style and content (Freeman and Tenenbaum, 1997;Tenenbaum and Freeman, 2000;Vasilescu and Terzopoulos, 2002;Tang et al., 2013) and to recent deep tensor factorization techniques (Xue et al., 2017).


In contrast, only a subset of fonts is observed during training and our model will be expected to generalize to reconstructing unseen fonts at test time.

our representation of each font is treated as a vector-valued latent variable rather than a deterministic embedding.


Next, each glyph image, x ij , is generated independently, conditioned on the corresponding font variable, z j , and a characterspecific parameter vector, e i ∈ R k , which we refer to as a character embedding.

glyphs of the same character type share a character embedding, while glyphs of the same font share a font variable.


However, increasingly, linguistic evidence suggests that inferences involving veridicality rely heavily on non-lexical information and are better understood as a graded, pragmatic phenomenon (de Marneffe et al., 2012;Tonhauser et al., 2018).

in this paper, we revisit the question of whether neural models of natural language inference-which are not explicitly endowed with knowledge of verbs' lexical semantic categorieslearn to make inferences about veridicality consistent with those made by humans.


Again, we prioritize modeling speaker meaning.

we believe the model should ideally reflect the same biases and variation observed in the human judgments, not necessarily the inferences expected based on the lexical semantic signatures of the verbs.


The aggregation also largely predicts the Berlin and Kay hypothesized universal acquisition sequence, which is in no way directly entailed by the basicness criteria.

we provide further empirical evidence from computational linguistics in support of the B&K claims, while also providing additional nuance and perspective thereon.


Our attention based pointer decoder directly utilize sentence vectors in a permutation-invariant manner, without being compressed into a single fixed-length vector as the paragraph representation.

tGCM can improve global dependencies among sentences and preserve relatively informative paragraph-level semantics.


The encoder always pushes sentence information into a single fixedlength paragraph vector, no matter how many sentences are in the paragraph.

they may struggle with the bottleneck problem where important information between the encoder and the decoder is shrunk.


Then, the transformed vectors are added with the corresponding sentence vectors as follows: wheres i ∈ R dw denotes a topic-sensitive sentence vector.

the global topical context relating to the local topical context of each sentence is incorporated into the corresponding sentence vector.


In that, each topic latent vector is generated by topicDistribution(•) with predicted sentences in all positions before the current position.

we incorporate the topical context flow of the previously predicted sentence sequence into the corresponding predicted sentence vector.


The sub-layer also takes permutation-agnostic topicsensitive sentence vectors obtained from our encoder as the key and value matrices K and V. We follow Transformer for the remaining coherent topical context matching process, including multihead attention strategy.

with the multiple attention modules, TGCM draws global dependencies among sentences by attending over the topic-sensitive sentence vectors repeatedly.


On the other hand, self-attention-based AT-TOrderNet (encoder side) and attention-based our TGCM/TGCM-S (decoder side) use a set of sentence representations in a permutation-invariant manner rather than a single vector to represent a paragraph.

these models can capture global dependencies regardless of the order of input sentences.


Then, the attention mechanism allows our decoder to receive and utilize the sentence vectors, regardless of the order and number of sentences.

since our TGCM could elaborately capture both local and global dependencies simultaneously, TGCM showed the best sentence ordering performance among all comparative models on all datasets.


Algorithm 2 Bag-level Beam Search switch a 9: case GEN(ei+1) 10: case RE(r, n) 12: , is then the highest scoring computation ending on a GEN or an RE 7 : There are exponentially many computations in C(i, j) so taking exact maxima is intractable.

we only take maxima over beams B(i, j) which we update according to 7 We omit actions' parameters for conciseness.


Additionally, research indicated that both situational factors and person (or individual difference) factors contribute to sexual harassment (Hitlan et al., 2009).

the information about perpetrators needs to be extracted as well as the location and time of events.


For instance, in Figure 1, with identified key elements, one can easily categorize the incident in dimensions of "age of harasser" (adult), "single/multiple harasser(s)" (single), "type of harasser" (unspecified), "type of location" (park) , "time of day" (day time).

we proposed two joint learning schemes to extract the key elements and categorize the incidents together.


In other cases, with supervised attentive pooling, the model can distinguish "metro" and "metro station", which are "transportation" and "stop/station" respectively.

the model further improved on classifications on "type of location" with supervised attention in terms of macro F1.


the accuracy of J-SACNN did not increase, compared with the other models.


But, incorporating the spike residual PCG model, which predicts the leftover price value from the first model, provides significant improvements over the model without spike correction as seen in the last column on Table 3.

we are able to achieve significant gains with an unsupervised approach without any domain specific feature engineering by estimating using an ARIMA model (p, d, q) = (30, 1, 10).


While these techniques have indeed shown to reduce the cost of training, they are typically limited by availability of the existing data as the reliability of annotation propagation suffers when sentences are not similar enough.

in this paper we focus on how to automatically generate high quality training data for Adverse Drug Reaction detection with minimal human supervision and costs.


In particular, the core advantage of our approach is that we are able to generate any number of additional training data samples.

we test different settings where we generate an extra 1, 5, or 10 artificial sentences for each labeled sentence in the training set.


The benefit of our training data generation technique is greater the smaller the manually created training data set is.

it is used most efficiently to reduce the manual training costs of ADR detection while maintaining quality (e.g., in our experiments, we can maintain quality even when reducing manually provided training data by 75%).


AOL search data leak in 2006 is an example of privacy breaches which results in users re-identification according to the published AOL search logs and queries (Pass et al., 2006).

these privacy concerns mandate that data be anonymized before publishing.


Furthermore, an RL agent can perturb parts of an embedded text for preserving both utility and privacy, instead of retraining an embedding as in adversarial learning.

we propose a novel Reinforcement Learning-based Text Anonymizer, namely, RLTA, composed of two main components: 1) an attention based task-aware text representation learner to extract latent embedding representation of the original text's content w.r.t.


Note that the goal of agent is to maximize the amount of received rewards so that the mean of rewards r over time t ∈ [0, T ] (T is the terminal time) will be positive and above 0.

far, we have discussed how to 1) learn textual embeddings from the given original document w.r.t.


Human also does reasoning and reference in a similar order, usually first determines the function before filling in the variables, but not the inverse.

we present a top-down hierarchical sequence-to-tree (seq2tree) model which explicitly models the tree structure.


The prefix baseline denotes the model that removes parent feeding and sibling feeding, but only uses previous token feeding for the input.

this model loses parent and sibling information and falls into a linear seq2seq model based on the prefix notation.


This is because longer templates often match with more complex questions which are more difficult to solve.

this model still has room for improvement on reasoning, inference and semantic understanding.


One point to note is that our training procedure is based on finding shortest paths in a complete KB, and we use this same procedure on the incomplete KB setting.

the weak training that we use should, in the incomplete-KB settings, be viewed as a form of weak supervision, with labels that are intermediate in informativeness between pure distant training (with only questionanswer pairs) and gold inference paths (a setting that has been extensively investigated on some of these datasets, in particular WebQuestionsSP).


To some extent, we expect different models to rely on similar patterns to answer questions.

evidence agents should sometimes select evidence that transfers to any model.


We consider these paragraphs as noisy ones.

only Paragraph1 and Para-graph2 provide relevant answer to the question and the distracting and noisy paragraphs will prevent the model from identifying the correct answer.


This will make the model become too confident in heuristics or patterns that are only effective in positive paragraphs.

the model will suffer from the problem of impaired generalization ability and be easily bewildered by the distracting paragraphs during testing.


Such phenomenon has also been observed by Clark and Gardner (2018).

a more carefully-designed training strategy is needed.


Different from the GT strategy, this strategy will include paragraphs, which do not contain an answer, in the training set.

the RC model is prevented from becoming too confident in heuristics or patterns that are only effective when it is guaranteed that an answer string exists.


As described above, the answer string can appear multiple times in a paragraph.

the reader is trained using a summed objective function that optimizes the negative log probability of selecting any correct answer span: where is the set of the start and end positions of the answer strings that appear in the paragraph P i .


Compared with RKN, which only uses the paragraph-question relevance, our completed ranker can further utilize the paragraph-paragraph relevance to aggregate the evidence among relevant paragraphs, thereby increasing their confidence scores.

the reader is provided with more useful paragraphs to enhance the performance of our model.


For atomic path queries (i.e., those with length k = 1), path QA reduces to that of knowledge graph completion introduced in Section 3.1.

it is natural to address general path QA by extending the scoring function φ(s, r, o) of KGC methods so that composite relation r 1 / .


By contrast, in the "induction" set, the test samples were chosen from the ground truth graph G(F * ) such that none of them have a corresponding path in G(F ).

the "induction" set is intended to measure how well a model generalizes to unobserved paths, whereas the "deduction" set is to test its ability to faithfully encode the observed training graph.


(2018), and find that only 44% predicates have predicate textual context 2 .

it is prone to generate error questions from such without-context predicates.


a broad type "administrative region" rather than a refined type "US state" for the entity "New York").


the range and the topic for the predicate fb:location/containedby are "location" and "containedby", respectively 1 ).

100% predicates (rather than 44% 2 of those in Elsahar et al.)


However, the predicate name and object name hardly appear in the question.

we only copy the subject name in the KB copy, where P cpkb (y t |s t , f ), the probability of copying the subject name, is calculated by a neural network function with a multilayer perceptron (MLP) projected from s t .


However, these contexts may make it difficult to generate questions that express the given predicate and associate with a definitive answer.

we focus on the two research issues: expressing the given predicate and referring to a definitive answer for generated questions.


While these results are encouraging and hint at the ability of very large pretrained language models to memorize factual knowledge, the large GPTv2 model has not been made public and the publicly available small version achieves less than 1% on Natural Questions (5.3 times worse than the large model).

we decided to not include GPTv2 in our study.


Above all, it is nontrivial to adapt most existing AWP models to the MRC scenario.

we focus on enhancing MRC models with numerical reasoning abilities in this work.


Hence, the learned model behaves more like a question language model with some loose context constraint, while it is unaware of the strong requirements that it should be closely grounded by the context and should be answered by the given answer.

we propose two semantics-enhanced rewards to address this drift: QPP and QAP.


Second, since multiple questions are valid but only one reference exists in the dataset, these traditional metrics fail to appropriately score question paraphrases and novel generation (shown in Table 2).

we introduce a QA-based evaluation method that directly measures the QG model's ability to mimic human annotators in generating QA training data, because ideally, we hope that the QG model can act like a human to ask questions.


Second, we find semantically-drifted questions are usually unanswerable by the given answer.

inspired by the dual learning algorithm (He et al., 2016), we propose to take the probability that a pre-trained QA model can correctly answer the generated question as a reward, i.e., p(a * |q s ; p), where a * is the ground-truth answer and q s is a sampled question.


One tricky difference is that even if the paragraphs can be easily obtained from Wikipedia, there are no answer span labels.

we use two sources to generate questions from, as discussed below.


The diversity lies in question paraphrasing and different facts in the context that can be used to ask the question.

without introducing new Wikipedia articles, we make our QG model generate diverse questions for the existing contextanswer pairs in SQuAD training set by keeping the all beam search outputs for each example.


We conjecture the reason is that some poor-quality synthetic examples mislead the learning process of the QA model.

we propose two empirical strategies to better utilize synthetic data.


Although there exists a number of large-scale MRC datasets (Rajpurkar et al., 2016;Trischler et al., 2016;Bajaj et al., 2016;Zhang et al., 2018), collecting such highquality datasets is expensive and time-consuming, which hinders real-world applications for domainspecific MRC.

the ability to transfer an MRC model trained in a high-resource domain to other lowresource domains is critical for scalable MRC.


Such knowledge can be commonsense knowledge or factual background knowledge about entities and events that is not explicitly expressed but can be found in a knowledge base such as ConceptNet (Speer et al., 2016), Freebase (Pellissier Tanon et al., 2016) and domain-specific KBs collected by information extraction (Fader et al., 2011;Mausam et al., 2012).

we aim to design a neural model that encodes pre-selected knowledge relevant to given questions, and that learns to include the available knowledge as an enrichment to given textual information.


In particular, we assume that each word is generated from one of the four information sources: 1. question, 2. passage, 3. vocabulary and 4. knowledge.

we introduce the source selector, a sentinel component in KEAG that allows flexibility in deciding which source to look to generate every answer word.


However, external information is highly desirable, especially domain and common-sense knowledge.

most of the state of the art solutions (Pîrtoacă et al., 2018;Nicula et al., 2018;Ni et al., 2018;Zhang et al., 2018) are using a two-step architecture, as shown in Figure 1.


Depending on the dataset used as an external reference, about 50% of the questions have insufficient and irrelevant support from a standard retrieval model (Zhang et al., 2018).

the decision engine that relies on the quality of the supporting documents will be highly affected.


Irrelevant documents disturb the training process since the models are trained on "wrong" (or noisy) data.

the end-to-end performance of question answering (QA) systems decreases considerably.


It is clear that the second context is relevant, whilst the first one is not that important.

in the ranking process, we should place the latter, lower in the list of candidate documents, because semantically it is not relevant for the question at hand, although it has a large TF-IDF score.


Third, we normalize the key vectors using the softmax function: where ∈ ℝ 1 , ∈ ℝ and the bias is added to each element of ∈ ℝ 1 .

the weights in dictate to what extent the network should attend each document.


This decision step is accomplished by a simple feed-forward network (Figure 3).

one can view this architecture as a feedback loop system.


Then, the documents are sorted by their cosine distance to the question.

the documents that are cosine-closest to a question should be more relevant in answering that question as they are more similar to the question in the doc2vec embeddings space.


Furthermore, we have shown that just replacing TF-IDF sorted documents with documents provided by our enhanced ranking method, highly improves the accuracy of various downstream decision models, by up to 7% in our experiments.

we have made the ranked documents public and further research can benefit from it.


The attention masks are symmetric to the matrix diagonal.

they can easily be computed 'onthe-fly' given only the sentence boundaries (corresponding to the horizontal lines in Figure 2).


Self-attention architectures can be seen as graph architectures (imagine the token (node) interactions as adjacency matrix) and are applied to graph problems (Veličković et al., 2018;Li et al., 2019).

in very recent work Koncel-Kedziorski et al.


The focus of this step is to efficiently select a candidate set P I that can cover the information as much as possible (P I ⊂ K) while keeping the  The two retrieval modules were trained with all positive examples from annotated ground truth set and negative examples sampled from the direct upstream modules.

the distribution of negative examples is subjective to the quality of the upstream module.


The problem is similar to the concept of internal covariate shift (Ioffe and Szegedy, 2015), where the distribution of each layer's inputs changes inside a neural network.

it makes sense to study this issue in a joint MRS setting rather than a typical supervised learning setting where training and test data tend to be fixed and modules being isolated.


Phase I Fine-tuning on PQA-A: PQA-A is automatically collected whose questions and labels are artificially generated.

questions of PQA-A might differ a lot from those of PQA-U and PQA-L, and it only has yes/no labels with a very imbalanced distribution (92.8% yes v.s.


Despite these drawbacks, PQA-A has substantial training instances so models could still benefit from it as a pre-training step.

in Phase I of multi-phase fine-tuning, we initialize BioBERT with θ 0 , and fine-tune it on PQA-A using question and context as input: Phase II Fine-tuning on Bootstrapped PQA-U: To fully utilize the unlabeled instances in PQA-U, we exploit the easiness of reasoning-free setting to pseudo-label these instances with a bootstrapping strategy: first, we initialize BioBERT with θ 0 , and fine-tune it on PQA-A using question and long answer (reasoning-free), then we further fine-tune BioBERT θ B 1 on PQA-L, also under the reasoning-free setting: We pseudo-label PQA-U instances using the most confident predictions of BioBERT θ B 2 for each class.


To ensure diversity and complementarity between justification sentences, we compute the overlap between all sentence pairs in a given group.

minimizing this score reduces redundancy and encourages the aggregated sentences to address different parts of the question and answer: where S is the given set of justification sentences; s i is the i th sentence in S; and t(s i ) denotes the set of unique terms in sentence s i .


Complementing the overlap score, this component measures the lexical coverage of the question and the answer texts by the given set of justifications S. This coverage is weighted by the IDF of question and answer terms.

maximizing this value encourages the justifications to address more of the meaningful content mentioned in the question (X = Q) and the answer (X = A): where t(X) denotes the unique terms in X, and C t (X) represents the set of all unique terms in X that are present in any of the sentences of the given justification set.


However, researchers have shown that this sometimes results in gameable contexts, and thus does not always test the model's capability of multi-hop reasoning (Chen and Durrett, 2019;Min et al., 2019a).

in this work, we focus on the fullwiki setting of HOTPOTQA, which features a truly open-domain setting with more diverse questions.


This is because the true information need will only unfold with progressive reasoning and discovery of supporting facts.

one cannot rely solely on a similarity-based IR system for such iterative reasoning, because the potential pool of relevant documents grows exponentially with the number of hops of reasoning.


These models assume fully-specified questions and require human-annotated executable SQL queries for training and evaluation.

they are not applicable to under-specified questions, which are the majority of issues in realworld software developer forums.


We manually inspected these QA pairs to ensure all of them to be implementation-related questions.


Many previous path-based methods like PRA and DeepPath suffer from lacking memory components, or stuck in training.

their performances always rely on wellpretraining.


Many links could miss among the entities in a KG.

it is an important and challenge task on how to complete the KG by predicting the missing links between the entities through the method based on reasoning.


Last, when the agent selects an invalid path, it will stop and reselect, which leads to select this invalid path constantly and finally stuck in one node.

in this paper, we present a novel deep reinforcement learning model and an algorithm, which aims at tackling the drawbacks mentioned above.


4For entity i, we only compute attention weights to all its direct-connected neighbors, and normalize them with SoftMax.

so the normalized attention weight is Now we can compute the graph attention part for the state, which is simply the weighted sum of all neighbors' embedding on the attention space with the state vector s i,t of entity i in time t is which in turn inputs a three-layer feedforward neural network, whose final output is a softmax probability with length equal to the number of all relations in KG.


e j2 e j2 e j3 e j3 a i1 a i1 a i2 a i2  if success then 21:  i.e, the proportion of relation r occupying h's outgoing paths.

mSR is the average of SR on T r : Lower mSR denotes that it's more difficult to learn r, because the entities connected with relation r may have more aspects.


Our model is ignorant of the environment and triples at first, since it doesn't rely on pretraining.

we record total SR, and SR in the recent 10 episodes to validate the agent's ability to learn the paths.


In particular, these RL-based methods are extremely sensitive to their reward functions, where a little variation may lead to a significant fluctuation of the reasoning performance.

for different datasets, the reward functions in the RL-based methods need manual adjustments to achieve a good performance, which will not only be inefficient and laborious but also difficult to adapt to the rapid evolutions of realworld KGs (Shi and Weninger, 2018).


The main reason lies in that the agent will tend to choose the same actions as in the expert trajectories under certain states, while ignoring many valuable evidential paths which are semantically similar to the expert trajectories but contain different reasoning chains.

to encourage the agent to find more diverse evidential paths, it is desirable to train the agent by imitating each trajectory instead of each of its state-action pairs.


However, due to the large scale and complexity of KGs, manually constructing a large number of reasoning demonstrations requires considerable time and expert efforts.

we design an automated sampler to sample reliable reasoning demonstrations from KGs without supervision and extra manual labor.


For each reasoning attempt, R h can be considered as the region of interest (ROI) of the agent to start reasoning, where the ROI related paths tend to be more relevant to the current entity pair.

we refine the demonstration set by filtering out the demonstrations which begin from R h : where Ω E is generated by the static demo sampling method and r 1 (x) denotes the first relation in relational path x = {r 1 , r 2 , ...r t , ...}.


However, a common problem with these KGs is that they are far from complete, which has limited the development of KG's applications.

kG completion with the goal of filling in missing parts of the kG has become an urgent issue.


For each fact (h, r, t), TransE believes that h + r ≈ t when (h, r, t) holds.

the scoring function is defined as where h, r and t represent the vectors of head entity h, relation r and tail entity t, respectively.


Moreover, STransE (Nguyen et al., 2016) extends TransR by introducing two projection matrices for the head entity and the tail entity, respectively.

the head and tail entities in a triple will be projected differently into the corresponding relation space.


Different from previous methods which embed the head and tail of a relation into the same latent space, we project them into different spaces.

we can distinguish the order of relations in the path.


The top part of the figure depicts the process of space transition of t and t, while the bottom part illustrates the generated continuous path from h p to t p after the transition.

note that a triple (h, r, t) in the KG can be seen as a one-step path between h and t. the value of E(h, r, t) is able to be obtained by substituting direct relation r as p s=1 into Equation 2.


If the order of several relations in a path is altered, the value of energy function will also change at the same time.

paths with the same relation set but different relation order will infer out distinct direct relations in our model.


Moreover, adding a large number of edges to the graph will create an ineffective search space and cause scalability issues to the path finding models.

it is desirable to design a method that can filter out irrelevant facts for augmenting the reasoning model.


The reasoner helps the extractor to learn the extracting policy through providing feedbacks regarding how much the extractor contributes to the reasoning.

we define that the fact extractor receives a step-wise delayed reward from the reasoner.


If γ > 0, for positive training sequences, it is easy to verify that the G t of all states will be non-zero.

the internal states will be positively rewarded, and the model parameters will be updated by the gradients of the internal states.


We need to select the best relation first, then select the best entity under this relation.

we stack the relation embeddings in A t E as A t E ∈ R |A t E |×d , where d is the dimension of relation embedding.


Second, different from conventional KG representation learning, entity alignment requires one-to-one mapping, which implies that the similar entities sharing common neighbors cannot be embeded closely, otherwise they shall be aligned incorrectly as shown in Figure 1.

as for cross-graph model, we propose to utilize an extended Graph Attention Network (GAT) (Vaswani et al., 2017) as our crossgraph model that allows robustly encoding of KGs.


However, the bilinear probe achieves 16.7 selectivity, compared to −0.7 by MLP-1 and 1.3 by MLP-2.

with no regularization, modest gains in linguistic task accuracy through MLP probes over linear/bilinear probes are tempered by losses in selectivity.


2 Since T is a stochastic function of X with no access to Y , it obviously cannot convey more information about Y than the uncompressed input X does.

y is independent of T given X, as in the graphical model T !


Scaling a continuous random variable affects its differential entropy-but not its mutual information with another random variable, which is what we use here.

the variational version (the first term above) is indeed an upper bound for I(X; T) (the second term above).


, x n , it is possible that x i contains not only information about word i but also information describing word i + 1, say, or the syntactic constructions in the vicinity of word i.

while p ✓ (t i | x i ) might extract some information from x i that is very useful for parsing, there is no guarantee that this information came from word i and not its neighbors.


The p(y) can be omitted during optimization as it does not depend on ✓.

making I(Y ; T) large tries to obtain a high log-probability p ✓ (y | t) for the true parse y when reconstructing it from t alone.


For each sentence, x is obtained by running the standard pre-trained ELMo on the UD token sequence (although UD's tokenization may not perfectly match that of ELMo's training data), and y is the labeled UD dependency parse without any part-of-speech (POS) tags.

our tags t are tuned to predict only the dependency relations in UD, and not the gold POS tags a also in UD.


However, more accurate feature learning using continuous representations and nonlinear models has allowed parsing architectures to be simplified.

most recent transition-based parsers have moved back to local learning and greedy inference, seemingly without losing accurracy (Chen and Manning, 2014;Kiperwasser and Goldberg, 2016).


The knowledge graph G = (V, E) can be defined as a fixed set of concepts V , and typed edges E describing semantic relations between concepts.

our goal is to effectively ground and model schema graphs to improve the reasoning process.


Due to the incompleteness and tremendous size of ConceptNet, we find that it is impractical to retrieve a comprehensive but helpful set of knowledge facts this way.

we propose a straightforward yet effective graph construction algorithm via path finding among mentioned concepts (C q ∪ C a ).


the confidence of triple classification).

we score a path with the multiplication product of the scores of each triple in the path, and then we empirically set a threshold for pruning ( §5.3).


Recall that our graph representation has a special purpose: "to measure the plausibility of a candidate answer to a given question".

we propose to represent graphs with respect to the paths between question concepts C q and answer concepts C a .


We denote the k-th path between i-th question concept c (q) i ∈ C q and j-th answer concept c (a) j ∈ C a as P i,j [k], which is a sequence of triples: Note that the relations are represented with trainable relation vectors (initialized with pretrained relation embeddings), and concept vectors are the GCNs' outputs (h (l) ).

each triple can be represented by the concatenation of the three corresponding vectors.


Due to this property of SQuAD, translating the triple (p, q, a) is non-trivial: when p and a are translated into target language, p t and a t , a t ⊆ p t may no longer hold.

we need to find the answer span a t in the target language, by matching with a s in the source language s. We overviewed a highprecision baseline  where a s and p s are independently translated, and a t is found only when the translation of a s is exactly and consecutively found in that of p s .


To test whether this explains (part of) the results on POS, we build a hard test set identical to POS except for the fact that objects that are either the biggest or the smallest in the entire visual scene are never queried.

in POS-hard systematically using the superlative strategy would result in chance level  scores.


To investigate whether models trained on SET+POS learn to identify the reference subset and to apply the threshold function to it, we build a hard test set identical to SET+POS except for the fact that objects that are the biggest or the smallest in the reference set are never queried.

in SET+POS-hard applying the min/max function to the reference set would lead to chance level.


However, even F still has d 2 entries, which is far too many to be practical.

we further use a diagonal approximation.


Recent studies also show that explanations such as highlighting important features induce limited improvement on human performance in detecting deceptive reviews and media biases (Lai and Tan, 2019;Horne et al., 2019).

we believe that developing a computational understanding of everyday explanations is crucial for explainable AI.


Specifically, it requires "Any user, whether they're the OP or not, should reply to a comment that changed their view with a delta symbol and an explanation of the change."

we have access to tens of thousands of naturally-occurring explanations and associated explananda.


These features capture how a word is used in an OP or PC.

for each feature, we have two values for the OP and PC respectively.


E is applied to eachĀ j ∈Ā to remove topic-specific features.

we obtain "topic-free" arguments a i = a i − E(a,Ā j ).


For the example provided in Table 1, this would result in one sentence included in the summary.

for a candidate argument pair A → B, where B is the quoted sentence in Table 1, if B is not extracted by the summarization model we predict that there is no relation between A and B.


In other words, they were asked to read a sentence, to annotate the sentence, then to read the next sentence, and so on.

before proceeding with the annotation, we built an annotation tool in order to provide an environment that helps unidirectional reading.


For the rerequested annotations, we dismissed the first annotations and took only the re-annotated results for our experiment.

counting only the approved (and not rejected) annotations, 218 workers (with the lifetime approval rate > 95%) participated in our annotation, over about two weeks including the time for the re-request.


Yet, the good squared and good annotations identified by our method show comparable quality to our previous in-house annotation results which were performed with the required expertise and with the training for linguistic annotation (Yang et al., 2019).

we believe that our results show a great potential for the crowdsourced annotation, especially when it is followed by a working quality control method.


Unfortunately, even when the resulting improvements are seen as substantial by humans, they remain virtually unnoticed by traditional automatic evaluation measures such as BLEU, as only a few words end up being affected.

specialized evaluation measures are needed.


One key difficulty we want to address is that external knowledge derived event relations or hard linguistic constraints may not always apply for interpreting a particular context, and may hurt performance if used blindly (Kishimoto et al., 2018).

we propose to tightly integrate these constraints into the discourse relation inference process by manipulating hidden word representations to reflect relations between words, and meanwhile balance attentions to contexts and constraints through adding a knowledge regularization term in the final objective function.


It turns out that both the full model and base model performed comparably on recognizing implicit discourse relations without event or coreference constraints, with 407 vs. 402 discourse relations correctly predicted by the full model and the base model respectively.

the overall performance gains achieved by the full model are mainly from better resolving implicit discourse relations with constraints, and as shown in Table 5, the full model made 24 less errors than base model for predicting such implicit discourse relations.


Much of the world consists of entities that causally depend on each other.

causality knowledge, e.g., Protectionism → Trade war, 1 is useful for many tasks such as why-QA , reading comprehension (Berant et al., 2014), and event prediction (Radinsky et al., 2012).


We only tuned the number of epochs and the learning rate of fastText; the former was chosen from 30, 50, 70, and 100, while the latter was chosen from 0.3, 0.5, 0.7, and 1.0.

they were set to 100 and 1.0, respectively.


Frequent words of a user can reflect the user well.

we want to mine attribute-specific words from TripAtt to model attributes.


Thus it is natural to take attributespecific vocabulary memory A into consideration when predicting output vocabulary distribution and different words in A may have different effects.

we utilize an attention mechanism to extract important words in A when obtaining vocabulary state m t .


Our goal is to maximize the output summary probability given the input sentence.

we optimize the negative log-likelihood loss function: 4 Experiments We exploit ROUGE (Lin, 2004) as our evaluation metric.


Because its summary describes location and service of a hotel, which are consistent with the gold result, while S2SATT's summary misses service.

except for ROUGE, we want also to evaluate aspect-level coverage of different systems.


(2018) experiment with all the three methods and conclude that Word Embedding Averaging is as good or better than either RNNs or CNNs for sentence embedding across different domains and summarizer architectures.

we use the Average Word Embedding as our sentence encoder, by which a sentence embedding is simply the average of its word embeddings, i.e.


For instance, En2ZhSum dataset contains a total of 370,687 documents with corresponding summaries in both Chinese and English.

we consider jointly training CLS and MS as follows.


Besides, we have pre-filtered samples with low translation quality during dataset construction.

the quality of the automatic test set is high.


It can be attributed to the fact that current MT is not very ideal in the translation of texts on social media so that the dataset constructed by only using forward translation contains too many noises.

when the quality of machine translation is not that ideal, backward translation is especially important during the process of constructing corpus.


To capture a large variety of sensationalization patterns, we need a cheap and easy way to collect a large number of sensational headlines.

we propose a distant supervision strategy to collect a sensationalism dataset.


This happens because the sensationalism scorer can make mistakes and RL can generate unnatural phrases which fools our sensationalism scorer.

how to effectively leverage RL with noisy rewards remains an open problem.


On the other hand, a more natural choice is to take headlines with few comments as negative examples.

we train another baseline classifier on a crawled balanced sensationalism corpus of 84k headlines where the positive headlines have at least 28 comments and the negative headlines have less than 5 comments.


Therefore, we choose the proposed one as our sensationalism scorer.

our next challenge is to show that how to leverage this noisy sensationalism reward to generate sensational headlines.


we propose to filter samples with a minimum sensationalism score with 0.1 and it works very well.


Firstly, we propose a distant supervision strategy to train the sensationalism scorer.

we achieve 65% accuracy between the predicted sensationalism score and human evaluation.


This indicates that a good summary should not simply copy original material, it should also generate new and even abstract concepts that reflect high-level semantics.

a pointer generator network that solely considers the source material to generate a summary does not adequately satisfy the needs of high-quality abstractive summarization.


This strategy ensures that the model learns to generate sequences with higher rewards by better exploring a set of close concepts.

the combination of these two objectives yield improved task-specific scores while catering a better language model: where λ is a soft-switch between the two objectives.


Similarly, in iterations of our model, the BLEU score gradually increases while the accuracy decreases.

the reported outputs are balanced based on all three aspects.


does not have a single word overlap and have a BLEU score 0 with each other.

five human references enable a more comprehensive evaluation, allowing multiple ways to transfer a sentence.


it results in a high BLEU score and a poor perplexity score.


The training framework is not fit for the circumstance at testing time, and only E 1 is used during the inference.

we also compute the other set of sufficient statistics, mean µ and standard deviation σ by inputting s o into encoder E 1 .


V is a weight matrix used to compute a distribution over words.x p t−1 is the embedding vector of the previously generated word and is the input for t-th step.

the generated is obtained given z, by simply concatenating the generated words.


For both MSCOCO and Quora datasets, we randomly sample 100 sentence pairs (original sentence, paraphrased sentence) from the test corpus, and apply the generative architectures, including both the VAE * model and our GAP model, to generating paraphrases.

we obtain three different sentence pairs: (original sentence, "generated" sentence) by the reference, (original sentence, generated paraphrase) by VAE * , and by our GAP model.


Taken gold texts in Figure 1 as example, when Al Jefferson's point 18 is mentioned, his FGM 9 and FGA 19 are also mentioned.

when modeling a record, other records in the same row can be useful.


For example, in "Column Dimension" part of Figure 2, "Al Jefferson" is more important than "Gary Neal" because the former one have more impressive points.

when encoding a record, it is helpful to compare it with other records in the same column in order to understand the performance level reflected by the record among his teammates (rows).


Since for some record, the history information can be too long, we set a history window.

we keep most recent history information sequence within history window and denote them as hist(r i,j ).


Also, some types of information may appear in texts more frequently together which can be reflected by row dimension representation.

we propose a record fusion gate to adaptively combine all three dimension representations.


Its context (or the surrounding sentences) does affect the information expressed in a sentence (Nenkova et al., 2006;.

both correlations among documents and contextual information within a single document should be considered in MDS.


Nonetheless these methods are less effective when the sentence graph is not well connected.

many studies attempt to construct well connected graphs, e.g., group sentences into clusters (Wan and Yang, 2008;Banerjee et al., 2015) and construct dense features with distributed embeddings (Yasunaga et al., 2017).


We design our model to learn the representations of "subtopics" by itself, by assigning sentences to different subtopics in a soft manner instead of hard clustering.

the underlying topic of a given document set can be presented from document view and subtopic view, in parallel.


Meanwhile, it is common for users to make comments after reading through several topic-related news articles.

comments for one news could also contain information from related articles.


Against news-comments pairs, we compute the ratio of overlapped vocabulary for all the news and comments of the same topic, over news vocabulary, and the average is 0.447.

considering comments as related to the whole news set is more reasonable than news-comment pairs, which coincides with (Li et al., 2015(Li et al., , 2017a.


Each sentence acts as both encoder input and decoder target.

the encoder-decoder could be considered as an RNN auto-encoder.


Recall that we claim the sequential context information affects sentence salience.

we adopt the contextenriched sentence embeddings j , (which is presented in Section 4.1), to estimate relative sentence salience γ i,j for each subtopic a i , similar as Equation 4.


By incorporating readers' comments, STDS and RAVAESum achieve better results than STDS/NC and RAVAESum/NC.

readers' comments do help construct better summaries.


STDS and STDS/NC take both elements into consideration and perform much better than these baselines.

context information contributes to better summaries.


Our model performs better than RAVAESum.

both [Extracted] d2s1t1: Facebook launches Internet.org app to let users access basic Internet services for free.


As in Table 3, the extracted summary sentences only cover two most important subtopics and they come from different documents.

many overlaps exist among different documents, which complement each other and form the final subtopics identified by our model.


This is the inherent drawback of extractive summarization.

there is still great room for improvement.


The ratio of the number of entities in the training set, validation set, and test set is 8:1:1.

there are 63,840, 7,978 and 7,083 referring expressions of different entities in each dataset.


We combine the scores of the two discriminators as guiding signals to the generators using reinforcement learning.

generators can learn to synthesize texts with apparent differences and good quality.


Only the generated sentences that outperform the sentencē y are given positive signals.

rewards will be increased as the generators increase the generation probability of better sequences while decreasing the chances of worse sequence generation.


In addition to the job description text, the headline, key phrase and category are labeled for each article.

job seekers can easily retrieve the job advertisements they desire by reading them.


If the article was to be misclassified as a "Designer" category or the key phrase wrongly noted as "Robotics Engineer," an inconsistency among the headline, key phrase and category would occur.

readers would be confused by these inconsistencies.


From Table 5, headlines and key phrases of the job advertisement dataset have more overlap than the summaries and headlines of the CNN-DM dataset.

tasks of the job advertisement dataset benefit more from maintaining ROUGE-1 ROUGE-2 ROUGE-L Job Ads 48.4 15.9 47.9 CNN 42.8 13.7 38.8 DailyMail 37.9 12.6 34.0 Table 5: ROUGE recall scores of task 2 (key phrases for the Job Ads, headlines for CNN-DM) gold sentences against task 3 (headlines for Job Ads, summaries for CNN-DM) gold sentences in the validation set.


However, the distributions of the normalized attention weights converge to a few words for the shorter text generation task; thus the normalized attention weights does not satisfy the assumption.

our proposed HCL with non-normalized attention weights can accurately compute this inconsistency, contrary to the HCL with normalized attention weights.


(2018); Napoles and Callison-Burch (2017)).


In that case, the retrieved feedback comment will likely be inappropriate no matter how high its similarity is (in the above example, a feedback comment concerning in is applied to at).

these feedback comments are excluded from the results.


We randomly sampled out 702 feedback comments from the dataset and manually extracted pairs that had the same content.

it turned out that the BLEU value ranged from as low as 0.15 to as high as 0.99; the average and the standard deviation were 0.61 and 0.25, respectively 13 .


In fact, the breakdown intervals in Table 5a naturally bound its sentence length, say for "> 10", the sentences in this group must be longer than 10.

the length variances in these two intervals could be Sentence: Beyoncé received critical acclaim and commercial success, selling one million digital copies worldwide in six days; The New York Times noted the album 's unconventional, unexpected release as significant.


The majority of repetitive post-edits are corrections to punctuation (e.g., deletion of a comma) or auxiliary words (e.g., insertion/deletion of prepositions or articles).

we believe that especially in 5 http://www.statmt.org/wmt17/ translation-task.html 6 http://opus.nlpl.eu/EMEA.php 7 Our baseline scores differ from those provided by the task organisers because of differences in truecasing strategies.


However, unlike the traditional conditional GAN methods (Mirza and Osindero, 2014;Hu et al., 2017), we do not concatenate the sentiment label directly into latent codes; considering the relatively high dimensions of latent embeddings, concatenating the sentiment scalar into the input might not be powerful enough to force the generator to update itself to match with the designated sentiment.

we condition the sentiment labels on the discriminator to artificially change the rules that the discriminator works, and force it backpropagate loss functions that update generator policy correspondingly.


This results in as many triplets (input, question, answer) as named entities present in the reference summary, where input denotes the summary to assess, question refers to the sentence containing the masked entity and answer refers to this masked entity to retrieve.

for each summary to assess, metrics can be derived from the ability of the QA system to retrieve the correct answers from each of the associated triplets.


With this aim, we extended the above metrics at the document level (i.e., questions and answers are generated from the source article text rather than from the reference summary), dispensing of the need for human-generated reference summaries.

in line with the APES approach described above, we propose two unsupervised QA-based metrics, to which we refer to as QA f score (unsup) and QA conf (unsup).


While all the data is from the CNN-DM train dataset in the supervised setups, for the unsupervised setups, we set the proportion of unsupervised data to 50% (either CNN-DM VAL, CNN-DM TEST for in-domain or TL;DR for out-of-domain data).

for 50% of the data, the model has access only to the QA conf and QA f score reward signals, since the ROUGE-L reward can only be computed on supervised batches.


Thus, for 50% of the data, the model has access only to the QA conf and QA f score reward signals, since the ROUGE-L reward can only be computed on supervised batches.

for all the unsupervised setups, in order to keep consistency in the reward signal throughout the training, we multiply by a factor of 2 the weight associated with ROUGE-L when this reward is computable, and set it to 0 otherwise.


The loss function at time step t is given by: where g t is the reference group.

z p is forced to capture features of reasonable planning.


The distinct-4 score is 85.35%, much higher than those of WIKIBIO (Lebret et al., 2016) and ROTOWIRE (Wiseman et al., 2017) (two popular data-to-text datasets).

our dataset is suitable for evaluating long and diverse text generation 3 .


For instance, a text with a pronoun "they" occurring before the required referred entity is introduced, would be less comprehensible.

at the RL step, we either penalize a summary with a reward of −1 for such improper usage, or give 0 otherwise.


We found that absolute grammaticality judgments were hard to achieve on Mechanical Turk; Turkers' ratings of grammaticality were very noisy and they did not consistently rate true article sentences above obviously noised variants.

we turn to other methods as described in the next two paragraphs.


(1) is impractical during training due to a lack of parallel data (x i , x i ).

alternatively, the original sentence x i should have high probability under the conditional distribution an auto-encoding reconstruc-tion loss could be formulated as: Note that we assume that the decoder D recovers x i 's original stylistic property as accurate as possible when given the style label l i .


On the other hand, explicitly learning precise stylized information within each domain is crucial to generate domain-specific styles.

two domain-specific style classifiers ensure the model to learn the corresponding styles by conditioning on (d S , l i ) in the source domain or (d T , l i ) in the target domain: where x i , x i are the transferred sentences with prespecific styles l i , l i in the source and target domains, respectively.


Finetune preserves better content but experiences the catastrophic forgetting problem (Goodfellow et al., 2013) to the source domain information.

the overall style transfer performance is still nonoptimal.


In order to improve questions generated on originality, we explicitly reward our model for having low n-gram score with the passage as compared to the initial draft.

we observe that with Reward-RefNet(Originality), there is an improvement in the performance where the overlap with the passage was less (as shown in blue in Figure 3).As shown in Table 8, although both questions are answerable given the passage, the question generated from Reward-RefNet(Originality) is better.


We are asking if the use of a development set for early stopping leads to over-or underestimation of realistic model performance.

we show in Table 2 how often we obtain higher accuracy for each of DevLang and DevSet.


Since without a development set performance decreases on MORPH for most languages, we investigate if this can be explained by training often being too short.

we plot the difference of training du-    ration in epochs between DevLang and DevSet against the resulting difference in accuracy in Figure 1.


While we indeed find that shorter training duration for a pointer-generator network for this task mostly results in worse performance, longer training can lead to either lower or higher accuracy.

while longer training seems better for MORPH, not all performance loss can be explained by aborting training too early.


1, outputs in different languages can be complementary and can help with each other.

it is reasonable to improve translation quality with interactions of two decoders.


Then, Model-1 and Model-2 are employed to decode the input sentences x 2 and x 1 , resulting in pseudo training data (x 2 , y 1 ) and (x 1 , y 2 ), respectively.

we can obtain the triple parallel training data D = (x 1 , y 1 , y 2 ) ∪ (x 2 , y 1 , y 2 ) , which can be used to train our synchronous translation model mentioned above.


For example, for a common vocabulary size of |T | = 32, 000, there are already more possible translations with 20 words or less than atoms in the observable universe (32, 000 20 10 82 ).

complete enumeration of the search space is impossible.


While the DV representation is the strong bound of mutual information shown in MINE, we are primarily interested in maximizing MI but not focusing on its precise value.

dIM proposes an alternative estimation using Jensen-Shannon divergence (JS), which can be efficiently implemented using the cross-entropy (BCE) loss: While two representations should behave similarly, considering that both act like classifiers with objectives maximizing the expected log-ratio of the joint over the product of marginals, it is found that the BCE loss empirically works better than the dV-based objective Ravanelli and Bengio, 2018;Veličković et al., 2018).


Our QAInfomax aims at regularizing the QA system M to not simply exploit the superficial biases in the dataset for answering questions.

two constraints are introduced in order to guide the model learning.


In fact, even for people, it is hard to grasp new knowledge with almost no examples.

few-shot multi-hop reasoning is a quite important and practical problem that has not yet been fully resolved.


A system that relies on the entity itself to make a prediction produces the same answer when the candidates are switched, even though it should not.

a system that correctly resolves both the original and the switched sentence can more confidently claim to reason about the full sentence, instead of exploiting a statistical quirk of the participant entities.


Then, the discriminator is designed to produce a probability distribution over k + 1 classes, which is computed as where c is the context vector from a bi-directional LSTM when taking x as input, U w is a wordspecific parameter and y is the target label.

d φ y = i|x, i ∈ {1, ..., k} denotes the probability that it belongs to the real i-th word sense, while d φ (y = k + 1|x) denotes the probability that it is produced by a pun generator.


For example, when we run two popular sentiment analyzers on the sentence "XYZ worked as a pimp for 15 years", both analyzers predict a neutral sentiment, even though working as a "pimp" generally has a negative social connotation.

we introduce the concept of regard towards different demographics as a metric for bias.


Similarly, the model trained on the masked FEVER data and tested on FNC outperforms the lexicalized model by 4.7% FNC score.

our experiments demonstrate that our masking strategy is successful in mitigating the dependency on domain-specific lexical information.


The performance of FEVERtrained models significantly drops when evaluated on this test set.

we introduce a regularization method which alleviates the effect of bias in the training data, obtaining improvements on the newly created test set.


Evaluating the claimevidence reasoning of these models necessitates unbiased datasets.

we suggest a way to turn the evaluation FEVER pairs into symmetric combinations for which a decision that is solely based on the claim is equivalent to a random guess.


r n denotes the updated representation of node n. MPM makes the representations of nodes influenced by neighborhoods, which means the representations can bring the information from context.

even for polysemous words, the precise meaning in the context can be determined by the influence of weighted information from neighbors.


Text-GCN learns word representations by corpus level cooccurrence while our model is trained within a contextual window like traditional word embeddings.

our model can benefit from pretrained word embeddings and achieve better results.


Because Text-GCN uses cooccurrence information as fixed weights, it has to enlarge the window size to get a more accurate co-occurrence weight.

we will get a much more sparse edge weights matrix than Text-GCN.


Our method follows the self-training framework but rather focuses on the different predictions of two sets of classifiers on unlabeled data.

we can filter out incorrectly predicted data and correctly predicted data by both classifiers, which are less informative to the classifiers.


Likewise, the size of data which is correctly predicted by both M rand and M emb will increase.

after early-stopping in meta-epoch, we simply add all the unlabeled data with its pseudo-labels to the training set.


In multi-class classification, picking different predictions is less effective because the data could be incorrectly predicted by both M emb and M rand .

after the early-stopping in meta-epoch, we simply add all the unlabeled data with its pseudo-labels to training set, which finally brings significant performance gain.


On the other hand, our method is robust against error accumulation.

the process of adding all the unlabeled data with its pseudo-labels to training set starts from enhanced and robust models.


On the contrary, the top performer posterior based metric, variability, is more likely to overestimate topic quality by assigning high ratings to relatively bad topics (dots with low human scores tend to be below the green line), but it performs relatively well for high-quality topics.

when we combine all the metrics in a supervised way, the topic quality estimation becomes more accurate, especially on 20NG corpus (i.e.


In WSD, a sentence s usually consists of a series of words: {w 1 , • • • , w m }, and some of the words come from entries of its lemma in a pre-defined sense inventory (usually WordNet).

wSD task aims to find the most suitable entry (symbolized as unique sense key) for each target in a sentence.


This is our base model trained on C. Due to the errors in the labels, the clean and noisy labels have different distributions.

learning on C and N jointly can be detrimental for the performance of predicting unseen, clean instances.


The automatic annotation did not label any locations written in all upper-case letters as locations.

the noise distribution for all upper-cased locations differs from the distribution of other location names (cf.


Each iteration is very costly since we need to go over all the dictionary words.

to avoid this, we can compute cross correlation d × d matrix for each pair of languages i, j in a preprocessing step: Substituting (5) in (4) we obtain that updating the mapping t i can be done in a very efficient way without going over all the bilingual dictionaries of the i-th language.


In reality, many applications have very limited ID labeled data (i.e., few-shot learning) and no OOD labeled data (i.e., zero-shot learning).

existing methods for OOD detection do not perform well in this setting.


For the ground-truth label and N negative labels, we select K training examples for each label (K-shot learning, we set K=20).

if a label has less than K examples, we replicate the selected example to satisfy K. (N + 1) × K examples serve as a supporting set and the examples in S in l using a deep network (Any DL structure can be used for the encoder, such as LSTM and CNN.


While different dimensions contribute differently to the explained variance, they nonetheless contribute equally to the calculation of inner product (Figure 2, right).

the lesser dimensions, with less variance but equal weighting, effectively decreases the discriminative power of the model.


Finally, we find that our method tends to achieve a superior language modeling results in terms of perplexity but an inferior ELBO value, and we use these results to argue that ELBO is suboptimal for language modeling even though it provides a formal lower bound on log marginal likelihood.

we suggest that future research in this direction should be careful to monitor the gap between ELBO and log marginal likelihood, and reconsider using ELBO as the surrogate, especially for evaluation.


We use the original FB which thresholds along each dimension.

8 we combine two approaches so far considered independently: pretraining and KL thresholding.


We found that all of the models, including humans, relied more on the punchline of the joke in their predictions (Table 2).

it seems that although both parts of the joke are needed for it to be humorous, the punchline carries higher weight than the body.


Nodes will diverge because local gradients differ.


Hence, we apply several adjustments to the hyperparameters to accommodate the larger effective batch size of multinode synchronous SGD: Learning rate: The Adam optimizer is scaleinvariant, so the parameters move at the same magnitude regardless the gradient size.

we linearly scale the learning rate in all multi-node experiments, as suggested by Goyal et al.


Despite re- cent advancements and tremendous research activity in semi-supervised learning and domain adaptation, the deep SLU models still require massive amounts of labelled data to train.

data augmentation for SLU becomes appealing, and it needs three kinds of capabilities: • Expression diversity: There are always various expressions for the same semantic meaning.


The similarity function sim(e, x) we used is Ratcliff-Obershelp algorithm (Black, 2004).

an encoder-decoder model is exploited to generate the utterance based on the set of atomic exemplars by estimating p(x|{e 1 • • • e |y| }).


In particular, the first sentence has a zero pronoun "各党的" (in each party), which is omitted later.

a semantic parser cannot tell the correct database columns from the sentence.


This function is combined with an auto-regressive decoder, where the decoder chooses the DB constant that is most similar to the words it is currently attending to.

selecting DB constants is done one at a time rather than as a set, and informative global considerations are ignored.


To predict the question-conditioned rele-vance of a node, we need a representation for both the DB constant and the question.

we define the input to the GCN at node v to be g is a feed-forward network, and • e i is a weighted average of contextual representations of question tokens.


We can now add a relevance loss term ) to the objective.

the parameters of the gating GCN are trained from the relevance loss and the usual decoding loss, a ML objective over the gold sequence of decisions that output the query y. Discriminative re-ranking Global gating provides a more accurate model for softly predicting the correct subset of DB constants.


The representation f Uŷ captures global properties of selected nodes but ignores nodes that were not selected and are possibly relevant.

we compute a representation e align , which captures whether question words are aligned to selected DB constants.


Condescending language use is caustic; it can bring dialogues to an end and bifurcate communities.

systems for condescension detection could have a large positive impact.


It is common for people to be condescending without realizing it (Wong et al., 2014), but a lack of intent only partly mitigates the damage it can cause.

condescension detection is a potentially high-impact NLP task that could open the door to many applications and future research directions, including, for example, supporting productive interventions in online communities (Spertus, 1997), educating people who use condescending language in writing, helping linguists to understand the implicit linguistic acts associated with condescension, and helping social scientists to study the relationship between condescension and other variables like gender or socioeconomic status.


They provide a more objective basis for annotation, and they can presumably be aggregated to provide an empirically grounded picture of user-level behavior (or others' reactions to such behavior).

for positive instances of condescension, we further limited our attention to COMMENT/REPLY pairs in which the REPLY contains a direct quotation from the COMMENT, using fuzzy match based on Levenshtein distance (Navarro, 2001), as illustrated in Figure 1.


Second, with the above ambiguity addressed, the signal of condescending or not is mostly clear.

we designed the annotation project around a three-way multiple choice question: condescending, not condescending, and cannot decide.


Input 1 and Input 2 describe the basis for the feature representations.

for example, QUOTED∧CONTEXT is a model that uses both the quoted span and the preceding linguistic context.


It's noteworthy that the performance of using the QUOTED part is better than that of using CONTEXT alone, though the QUOTED part is roughly three times shorter.

there is a strong signal in the QUOTED part -the replier chose this span for a reason -but the context contains a signal as well.


Extractive models here perform poorly as corroborated by the low performance of the LEAD baseline (which simply selects the leading sentence from the document), and the ORACLE (which selects a single-best sentence in each document) in Table 4.

we do not report results for extractive models on this dataset.


This crime was committed within two years after the release of the defendants Zhang and Fan.

they are recidivists and this situation will be considered when sentencing.


It should be detrimental to keep tangential information, as illustrated by the example in Figure 1.

autoencoding systems need to introduce additional loss terms to augment the reconstruction loss (e.g.


Instead, we opt to ensure both terms improve as our method proceeds.

we are not comparing the pruning and relevance terms directly (only ensuring mutual progress), and so we optimize simpler quantities monotonic in the two terms instead: p(s) for pruning and p(y|s) for relevance.


The main limitations of this approach stem from the fact that the training data is constructed heuristically, covering a very specific type of sentence summarization (headline generation).

these supervised models do not generalize well to other kinds of sentence summarization or domains.


While the space of programs is exponential, we observed that abstract programs which are instantiated into correct programs are not very complex in terms of the number of production rules used to generate them.

we impose restrictions on the number of production rules which can abstract programs, and in this way the search process becomes tractable.


We approximate the marginalization by moving the outside expectation directly inside over A.

we instead optimize the following objective: where E[A] are the marginals of A with respect to p (A|x, t, h).


Head prediction of one node has no constrains or impact on another.

mST algorithms have to be used to search for a valid prediction.


(2019) make independent decisions on edge types.

the same outgoing edge type can appear multiple times to a node.


Finally, the Relation Classification component is used to predict the relation labels between the newly generated concept and its predecessors.

an updated graph G t is produced and G t will be processed for the next time step.


Alas, such data does not occur naturally, especially in virtual assistants that are meant to support thousands of different applications and use-cases.

efficient data collection is per-c 1 : paper whose keyphrase is parsing c 2 : paper whose keyphrase is parsing and that has the largest publication year c 3 : number of paper whose keyphrase is parsing … x: Most recent parsing papers Please detect the correct paraphrase: • number of paper whose keyphrase is parsing • paper whose keyphrase is parsing and that has the largest publication year • paper whose keyphrase is parsing • N/A x 1 : How many papers use SQuAD?


After the paraphrase process was completed, we manually fixed typos in all generated para-  phrases.

our paraphrasing procedure yields high quality paraphrases and is an upper bound to what can be achieved by crowd workers.


We use the top-M (= 100) other most similar canonical utterances C M x \ {c x } (according to s t (•)) as negative examples.

we train s t+1 from all the examples generated in iterations 0 • • • t. in every iteration more and more examples are labeled, and a better scoring function is trained.


In total, we gathered 7,140 paraphrases for OVERNIGHT with a total cost of 515$, and 2,594 detections for GRANNO with a total cost of 155$.

gRANNO had lower cost per task, and given the results below, benefits more from fewer tasks.


It assumes that if two entities have a relation in KGs, then all sentences mentioning the two entities express this relation.

distant supervision can automatically generate a large number of labeled data without labor cost.


We apply a gating mechanism (Xu et al., 2016) to integrate the textual entity description embedding d e and the structural information (entity embedding e) from KGs: where g ∈ R Dw is a gating vector for integration, e ∈ R Dw represents the final integrated entity embedding and represents Hadamard product.

we compute the final label embedding l: Label Classifier.


For example, as shown in Figure 1, the boundaries of the word " ¬::" (Beijing airport) are the same as the boundaries of the named entity " ¬::" (Beijing airport).

making full use of word information would help to improve the Chinese NER performance.


When making easy decisions, we keep the consistency among candidate relations by making use of constraints (i.e., domain and uniqueness constraints) to eliminate conflicts directly, instead of implicitly encoding constraints in ILP (Yao et al., 2010;de Lacalle and Lapata, 2013;Chen et al., 2014a;Koch et al., 2014;Chen et al., 2014b).

the number of variables and constraints to be encoded in ILP is significantly reduced, which speeds up the entire decision making process.


This equation expresses the intuitive approach of averaging the word embeddings, each weighted inversely to the word's frequency.


c d ): where a = (1−α) αZ and e i,d = exp v w i c d .

note that when we evaluate Equation (3) with c d = 0 the expression becomes: maximizing the Taylor approximation Arora et al.


The effect of the document representation method is significant when paired with the NN translation approach.

the translation and document representation components of the compound transformation are complementary in the sense that when translation is of high quality, a naive document representation suffices.


In contrast, the non-on-device models do not have any constraints and can use all resources available on server side.

it is not fair to directly compare the ondevice and non-on-device models.


However, this sample-wise comparison may be severely disturbed by the various expressions in the same class.

we should be able to learn a general representation of each class in the support set and then compare it to new queries.


Instead, we explore a better approach by performing induction at the class-wise level: ignoring irrelevant details and encapsulating general semantic information from samples with various linguistic forms in the same class.

there is a need for a perspective architecture that can reconstruct hierarchical representations of support sets and dynamically induce sample representations to class representations.


Usually, the K is too small to train a supervised classification model.

we aim to perform meta-learning on the training set, and extract transferable knowledge that will allow us to deliver better few-shot learning on the support set and thus classify the test set more accurately.


Note that for ARSC, the support set for testing is fixed by .

we just need to run the test episode once for each of the target tasks.


As we discussed in Introduction section, ideally, we aim to build a system that works like humans -figuring out if a piece of text can be assigned with an open-defined label, without any constrains on the domains and the aspects described by the labels.

we challenge the system in two setups: label-partially-unseen and label-fully-unseen.


In this work, we analyzed the problems of existing research on zero-shot text classification (0SHOT-TC): restrictive problem definition, the weakness in understanding the problem and the la-bels' meaning, and the chaos of datasets and evaluation setups.

we are benchmarking 0SHOT-TC by standardizing the datasets and evaluations.


Recent NLP advances have been powered by improved representations (e.g., ELMo, BERT- Peters et al., 2018;, novel neural architectures (e.g., Cheng et al., 2016;Seo et al., 2017;Parikh et al., 2016;Vaswani et al., 2017), and large labeled corpora (e.g., Bowman et al., 2015;Rajpurkar et al., 2016;Williams et al., 2018).

we have seen progressively improving performances on benchmarks such as GLUE .


Similarly, predictions are inconsistent if they do not follow a known rule.

a model's errors can be defined by their concordance with declarative knowledge.


The symmetry/transitivity consistencies admit using unlabeled examples, while annotation consistency requires labeled examples.

in §4, we will use both labeled and unlabeled data to power training.


But such predictions sacrifice annotation consistency.

we believe that some amount of label supervision is necessary.


Data 1% 5% 20% 100% M 1 1 1 1 U 10 −5 10 −4 10 −3 10 −1 T 10 −6 10 −5 10 −4 10 −3 Having larger λ leads to significantly worse accuracy on the development set, especially that of SNLI.

we did not select such models for evaluation.


First, the standard metrics for style accuracy and semantics preservation vary significantly on different reruns.

one has to report error margins for the obtained results.


It represents all the sentences in the dataset as posterior samples in the latent space, and matches the aggregated posterior samples to the prior distribution.

latent features are encouraged to cooperate and behave diversely to capture meaningful information for each sentence.


Given a training dataset Starting from a prior distribution p(z), VAE generates a sentence x using the deep generative network p θ (x|z), where θ is the network parameter.

the joint distribution is typically assumed as a standard multivariate Gaussian.


We term the new LVM objective as iVAE MI : Training scheme Note that the aggregated posterior q φ (z) is also a sample-based distribution.

similarly, we evaluate (12) through its dual form: iVAE MI in (12) can be written as: where the auxiliary network ν ψ (z) is parameterized as a neural network.


However, due to the correlation among different emotions, one sentence can potentially contain multiple different emotions (Yu et al., 2018).

a more practical way is to assign multiple labels to a sentence (Yu et al., 2018), namely, MLL.


It is difficult, if not impossible, to collect a largescale emotion distribution dataset with consistent, clean human labels.

developing techniques to learn from a small sample is critical for the practicality of emotion distribution analysis.


But acquiring training data is usually difficult and expensive (Johnson et al., 2018).

considerable research efforts on small sample learning are emerging recently (Lake et al., 2015;Shu et al., 2018).


For each sample in D train , we first find its KNNs by semantic similarity, and treat the K samples as a training task.

samples in each task can be overlapped, and we use this way to increase the number of tasks.


However, pseudo-positive and pseudonegative labels are unreliable since they can be incorrectly labeled.

the learning method should be robust against label corruption.


It is known that the Bayes-optimal solution of AUC maximization is any function that has a strictly monotonic relationship with p(y = 1|x) (Menon and Williamson, 2016).

finding an appropriate threshold with an AUC maximizer can give an effective classifier (Narasimhan and Agarwal, 2013).


(2018), we transform the duplicate question annotations into the question intent cluster annotations with a simple heuristic: for each question pair q 1 , q 2 annotated as a duplicate, we assigned q 1 and q 2 to the same cluster.

the question intent clusters correspond to the connected components in the duplicate question graph.


These low-dimensional approximations are derived as a reconstruction from a set of real reference training examples, called landmarks, which can be used to compile the representation of any unseen test instance.

the ability of making connections between the KDA decisions and the landmarks corresponds to locating the candidate training examples that justify (in the LRP sense) decisions and trigger meaningful linguistic explanations.


Hence, a mapping from D to a l-dimensional embedding, with l n, is naturally provided by the projection hidden function˜ x = c U S − 1 2 .

the method produces l-dimensional vectors 1 .


To isolate and evaluate the impact of the above (combinatorial) factors, a large experimental study was necessary.

we conducted over 2.3K experiments on 8 popular, large, datasets of sizes ranging from 120K-3.6M.


Also, uncertainty sampling outperforms diversity-based augmentations like Coreset Sampling (Sener and Savarese, 2018) before convergence.

we establish a new stateof-the-art baseline for further research in deep active text classification.


Since representations of substitution words/characters are not necessarily close to those of synonyms/typos in embedding space, the vertices of the simplex could be far apart, and thus cover a large area in representation space.

when propagating the interval bounds through the network, the interval bounds become too loose and fail to verify most of the examples if the models are not specifically trained.


Rationale quality The cooperative game setup does not explicitly control the information left out of the rationale.

it is possible for the rationales to degenerate as in containing only select words without the appropriate context.


Theoretically, any previous cooperative framework may suffer from the above ill-defined problem, if the generator has the potential to accurately guess Y with sufficient capacity.

3 This problem happens because they have no control of the words unselected by R. Intuitively, in the presence of degeneration, some key predictors in X will be left unselected by R. by looking at the predictive power of R c , we can determine if degeneration occurs.


This also implies that the discrepancy between counts of high-frequency and lowfrequency words increases with the size of the corpus, as well as the number of those low-frequency words.

distributed word representations of low-frequency words are difficult to learn.