To get the multi-perspective information, it is necessary to distinguish the role of each context word and model their relationship especially when one context word could take on multiple roles (Zhang et al., 2018b).

this is difficult to realize for the HAN model as its final representation for the context is produced with an isolated relevance with the query word which ignores relations with other context words.


We notice that for a reasonably small adapter size, adapter performance gradually converges to its peak and stays steady, with almost no over-fitting for a long enough period, easing final model selection.

with full fine-tuning, optimal model selection becomes challenging due to rapid over-fitting on the adaptation corpus.


Although it might be possible to reduce this gap further by increasing the adapter size beyond b = 4096, there might be more efficient ways to approach this problem, including more expressive network architectures for adapters, joint fine-tuning of adapters and global model parameters, etc.


Multilingual Neural Machine Translation (NMT) models have yielded large empirical success in transfer learning settings.

these black-box representations are poorly understood, and their mode of transfer remains elusive.


One may also use advanced and domain-specific neural machine translation system to achieve better translation performance, while we leave it for individuals, and this is beyond the scope of this paper.

for span-extraction reading comprehension task, a major drawback of this approach is that the translated answer may not be the exact span in the target passage.


Rapid progress has been made in the field of reading comprehension and question answering, where several systems have achieved human parity in some simplified settings.

the performance of these models degrades significantly when they are applied to more realistic scenarios, where answers are involved with various types, multiple text strings are correct answers, or discrete reasoning abilities are required.


(2019) extend previous one-type answer prediction (Seo et al., 2017) to multi-type prediction that supports span extraction, counting, and addition/subtraction.

they have not fully considered all potential types.


Existing methods use unsupervised training with encoder-decoder architectures (Lei et al., 2016;Zhang and Wu, 2018), and adversarial domain transfer where the model is trained on a source domain and adversarially adapted to a target domain (Shah et al., 2018).

such approaches typically fall short of the performances that are being achieved with in-domain supervised training.


Therefore, we can add more instances to the training splits with these methods.

if not otherwise noted, we use the same number of training instances as in the original training splits with duplicates.


Automatic DQD is often approached as a supervised problem with community-generated training labels.

smaller CQA forums may suffer from label sparsity: On Stack Exchange, 50% of forums have fewer than 160 user-labeled duplicates, and 25% have fewer than 50 (see Figure 1).


These include undersampling (Kotsiantis and Pintelas, 2003), oversampling, and feature selection (Zheng et al., 2004) at the data level.

due to random undersampling, potentially useful samples can be discarded, while random oversampling poses the risk of overfitting.


These themes were developed qualitatively from transcripts with therapists.

while they were comprehensive in the clinical domain, here, we aim to create a more general framework for classifying MAS in online interactions.


Further, our focus on gender also allows us to reliably recruit crowdworkers across the gender spectrum, whereas other social categories such as race or religion are more difficult to recruit in a balanced proportion through traditional mechanisms.

despite this current focus, both the typology and annotation approach are designed for  of perceived offensiveness between annotator gender.


Hence, we prefer browsing the reviews online for the sake of finding our desirable products.

it is quite time-consuming for potential buyers to sift through millions of online reviews with uneven qualities to make purchase decisions.


One approach to define f θ might be to manually define features of interest, such as stylometric or surface features (Solorio et al., 2014;Sari et al., 2018).

when large amounts of data are available, it is preferable to use a data-driven approach to representation learning.


7 The most closely related work is author identification on social media.

previous work in this area has largely focused on distinguishing among small, closed sets of authors rather than the open-world setting of this paper (Mikros and Perifanos, 2013;Ge et al., 2016).


To computationally model a meaning-preserved variance of text across styles, many recent works have developed systems that transfer styles (Reddy and Knight, 2016;Hu et al., 2017;Prabhumoye et al., 2018) or profiles authorships from text (Verhoeven and Daelemans, 2014;Stamatatos et al., 2018) without parallel corpus of stylistic text.

the absence of such a parallel dataset makes it difficult both to systematically learn the textual variation of multiple styles as well as properly evaluate the models.


A series of works by (Koppel et al., 2011Koppel and Winter, 2014) and their shared tasks (Stamatatos et al., 2018) show huge progress on author profiling and attribute classification tasks.

none of the prior works have collected a stylistic language dataset to have multiple styles in conjunction, parallely annotated by a human.


With a single image, on the other hand, the outputs produced by annotators tend to be diverse.

the image can be explained with a variety of contents, so the output meaning can drift away from the reference sentence.


Top five uni/bigram keywords are chosen at each story, which are called global keywords.

another top three uni/bigram keywords are chosen at each image/sentence in a story, which are called local keywords.


Context-Aware Model (CAM) A simple baseline model would compute the semantic representation of each sentence in the synopsis using a pretrained sentence encoder.

classifying segments in isolation without considering the context in which they appear, might yield inferior semantic representations.


Specifically, we want to track   the five sentences with the highest posterior probability of being TPs and sequentially assign them TP labels based on their position.

it is possible to have a cluster of neighboring sentences with high probability, even though they all belong to the same TP.


The performance of the end-to-end TAM model drops slightly compared to the same model using goldstandard TP annotations.

it still remains competitive against the baselines, indicating that tracking TPs in screenplays fully automatically is feasible.


We would also like to point out that, as can be seen in the table, our results have quite high variance -the main source of it is the nature of our training/evaluation setup where we average over 10 runs with 10 different sets of seed dialogues.

in the majority of cases with comparable means, DiKTNet has a lower variance than    The comparison of the setups with different latent representations also gives us some insight: while the VAE-powered HRED model improves on the baseline in multiple cases, it lacks generalization potential compared to the LAED setup.


Negative candidates which are lexically similar to the ground truth response should result in models that carefully consider each word in order to produce fine-grained representations and identify minute differences between candidate responses.

very semantically distant candidate responses should result in very broad and abstract representations of language.


Identity fraud detection is of great importance in many real-world scenarios such as the financial industry.

few studies addressed this problem before.


Based on the above finding, we aim to design a dialogue system to detect identity fraud by asking derived questions.

there are three major challenges in achieving this goal.


To generate derived questions, we link all personal information entities to nodes in an existing Chinese KG 3 and crawl triplets that are directly related to them.

owing to the fact that the KG is largely sparse, nearly a half of entities 4 cannot be linked.


Besides, existing work requires labeled data, which is often hard to get.

we focus on detecting identity fraud through multi-turn interactions and use reinforcement learning to explore the anti-fraud policy without any labeled data.


In such tasks, a set of system actions can be pre-defined based on the business logic and slots.

the system actions in our task are selecting nodes in the KG to generate questions.


The neural encoder-decoder models have shown great promise in neural conversation generation.

they cannot perceive and express the intention effectively, and hence often generate dull and generic responses.


Existing works introduce dialog acts to label a cluster of responses and a latent variable is learned to select a dialog act for response generation (Zhao et al., 2017;Serban et al., 2017a).

it is not effective to capture the output diversity since the natural correlation between the expression patterns and dialog acts is not learnt.


A simple implementation is to directly embed the act information into the Seq2Seq model.

as shown in our experiments, it still suffers from safe response problem.


Formally, the first way updates its state only according to [w, 0 • e], while the second according to [w, 1 • e].

these methods may cause that the content (expression) is so powerful that the responses are without any effective expression (meaningful content).


But, only a little works (Zhao et al., 2017;Serban et al., 2017a) on open domain endto-end modeling take dialog acts into account.

many attempts have also been made to improve the architecture of Seq2Seq models by changing the training methods.


Because of its great potential in understanding and modeling conversations, SEQ2SEQ has been widely applied in different kinds of conversation scenarios including technical supporting, movie discussions, and social entertainment, etc.

when confronting conversations with diverse topics or themes, SEQ2SEQ is usually prone to make generic meaningless responses due to its oversimplified parameterization.


This architecture shows great success in neural dialogue generation (Shang et al., 2015;Serban et al., , 2017Shen et al., 2017;Clark and Cao, 2017;Xing et al., 2017;Choudhary et al., 2017;.

with a single set of model parameters and the oversimplified model architecture, the flexibility of the model is rather limited, especially when confronting conversations with diverse topics or themes.


We observe that the decoding speeds of CVAE and LAED are relatively comparable with our model.

when comparing with TA-SEQ2SEQ and DOM-SEQ2SEQ that also elaborately and explicitly model conversations with diverse topics or themes, ADAND shows a clear superiority in decoding speed.


These actions can be regarded as users' feedbacks that reflect users' interest.

due to its implicitness, such feedback can only reflect a part of users' interest, causing inaccuracy in recommendation.


From the figure, it can be found that while there is no mentioned item in the dialog, the baseline and the one only with knowledge perform the worst.

the two models with dialog incorporation perform significantly better.


2018, we align latent spaces for different styles.

we also align latent spaces encoded by different models (S2S and AE).


(2019b) maximized the average of capped distance between points from the same latent space.

we found that the results are sensitive to the cap value.


As in the automatic evaluation results, STYLEFU-SION and MTask show the highest appropriate-   ness (not statistically different) apart from the Human system.

sTYLEFUsION outputs are much more stylized.


Dialogue systems have attracted increasing attention due to the promising potentials on applications like virtual assistants or customer support systems (Hauswald et al., 2015;Poulami Debnath, 2018).

studies (Carbonell, 1983) show that users of dialogue systems tend to use succinct language which often omits entities or concepts made in previous utterances.


First, the context is treated as a whole for calculating its attention towards profile sentences.

each context is composed of multiple utterances and these utterances may play different roles when matching different profile sentences.


In general, all these methods adopted a contextlevel persona fusion strategy, which first obtained the embedding vector of a context and then computed the similarities between the whole context and each profile sentence to acquire the persona representation.

such persona fusion is relatively too coarse.


It is reasonable that the contextpersona matching is more important because contexts provide the fundamental semantic descriptions for response selection.

the single persona-response matching can also achieve a hits@1 of 48.8% and an MRR of 60.9%, which shows the usefulness of utilizing persona information to select the best-matched response.


The lengths of shortest paths from sources to targets are shown in Figure 3.

most probabilities lie on two and three hops rather than zero and one hop, so key-value extraction based text generative models (Eric et al., 2017;Levy et al., 2017;Qian et al., 2018) are not suitable for this task.multi-hop reasoning might be useful for better retrieving correct knowledge graph entities.


Among the baselines, KAware has better performance on Last1.

qadpt outperforms all baselines when the knowledge graphs slightly change (Last1 and Last2) in terms of accurate change rate.


For the scenario of zero-shot adaptation, Mem-Net and TAware show their ability to update responses when the knowledge graphs are largely changed.

qadpt is better to capture minor dynamic changes ( Last1 and Last2) and updates the responses according to the new knowledge graphs.


The goal of the interpretable matching model is to reveal token-level matching information between a query-response pair thus a matching skeleton can be derived from the response.

the training of the matching model does not rely on such fine-grained annotations.


In addition, it might be a little bit surprising to see that PMI and keywords give almost the same performance on all three metrics, telling a given query is not that necessary.

we found lots of skeletons proposed by PMI are identical to those of keywords.


In the first case, the retrieved utterance is very specific with elaborated details.

it is not a reasonable response due to the sudden topic drift.


use dialogue acts to capture the discourse variations in multi-round dialogues as guided knowledge.

such discourse information can hardly be extracted for short-text conversation.


During testing, a decoder sequentially generates a response using search strategies such as beam search.

these models frequently generate bland and generic responses.


Recently, Seq2Seq neural networks (Sutskever et al., 2014) have demonstrated excellent results on open-domain conversation (Shang et al., 2015;Sordoni et al., 2015;Vinyals and V. Le, 2015;Yao et al., 2015).

due to lacking of the global and relevant information guidance, they inherently tend to generate trivial and uninformative responses (e.g., "I don't know"), rather than meaningful and replier-specific ones Li et al., 2016).


It is observed that when k ≤ 3 , model performance improves with the increasing of k, which suggests that more distributions in GMD are helpful for modeling user personalization.

for k > 3, model performance slightly drops with the increasing of k, The potential reason is GMD with three distributions is effective enough for modeling personalization, and sophisticated GMD might suffer scarce datasets for training.


(2018) explores this approach in detail.

the dataset introduced in that work does not capture higher-level strategic behaviors that can impact the quality of the recommendation made (for example, it may be better to elicit user preferences first, before making a recommendation).


(2013) automatically produces template-based questions from user reviews.

no conversational recommender systems have been built based on these works due to the lack of a large publicly available corpus of human recommendation behaviors.


Second, as the model becomes better at the game, we observe an increase in the length of dialogue.

it remains shorter than the average length of human dialogues, possibly because our reward function is designed to minimize it, which worked better in experiments.


The goal of these systems is to help users accomplish a specific task, such as flight or hotel booking or transportation planning.

to achieve these goals, task-oriented dialogue systems rely on pre-defined slots and values for request processing (which can be represented using simple SQL queries consisting of SELECT and WHERE clauses).


Comparing to existing context-dependent text-to-SQL datasets, CoSQL contains significantly more turns, out of which 11,039 user utterances are convertible to SQL.

all NL utterances in ATIS and SParC can be mapped to SQL.


For example, SQL AGG components occur most frequently in the beginning of dialogues, as a result of users familiarizing themselves with the amount of data in the DB or other statistical measures.

the frequencies of almost all SQL components in SParC increase as the question turn increases.


Reinforcement Learning has gained more and more attention in dialog system training because it treats the dialog planning as a sequential decision problem and focuses on long-term rewards .

rL requires interaction with the environment, and obtaining real human users to interact with the system is both time-consuming and labor-intensive.


For instance, RL systems built with more complicated user simulators will have lower scores on the automatic metrics, compared to those built using simpler user simulators.

the good performance may not necessarily transfer when the system is tested by real users.


In this work, we build a similar agenda-based user simulator in the restaurant domain, and focus more on analyzing the effects of using different user simulators.

it's not feasible to build agenda-based user simulators for more complex tasks without an explicit agenda.


(2018) introduced the Neural User Simulator (NUS) which learned user behaviour from a corpus and generates natural language directly instead of semantic output such as dialog acts.

unlike in ABUS, how to infuse the agenda into the dialog planning and assure consistency in data-driven user simulators has been an enduring challenge.


Dialog action space can also be on the word-level.

previous study shows degenerate behavior when using word-level action space , as it is difficult to design a reward.


"Auto-Success" has been used to reflect the solved ratio previously.

it's not necessarily correlated with the user-rated solved ratio.


pare the success rate of S 1 , ..., S n on U .

by looking at the fifth row for the SL-Retrieval simulator, it will prefer Sys-SLT (0.975) over Sys-AgenG (0.965), but actually the average performance of Sys-AgenG (0.882) is better than Sys-SLT (0.616) from the last row.


Bahdanau Attention (Bahdanau et al., 2014) is widely used to calculate the attention weights according to the features of individual modalities, since Bahdanau Attention is originally proposed for machine translation which can be considered as a unimodal task.

video captioning is a multimodal task and different modalities are able to provide complementary cues to each other when calculating the attention weights.


Given an image x ∈ D x u , we retrieve a caption in the unpaired dataset, i.e.,ỹ ∈ D y u , that has the highest score obtained by the discriminator, i.e., the most likely caption to be paired with the given image as vice versa for unpaired captions: By this retrieval process over all the unpaired dataset, we have image-caption pairs {(x i , y i )} from the paired data and the pairs with pseudolabels {(x j ,ỹ j )} and {(x k , y k )} from the unpaired data.

these pseudo-labels are not noisefree, thus treating them equally with the paired ones is detrimental.


Over the last few years, vision-language tasks such as image captioning (Xu et al., 2015) and visual question answering (VQA) (Antol et al., 2015;Anderson et al., 2018) have provided a testbed for developing a cognitive agent.

the agent performing these tasks still has a long way to go to be used in real-world applications (e.g., aiding visually impaired users, interacting with humanoid robots) in that it does not consider the continuous interaction over time.


In general, the algorithms we introduce again outperform the NoStruct baseline.

to the crowdlabeled experiments, AP (slightly) outperformed the other algorithms.


We also constructed a dataset from English sentence-tokenized Wikipedia articles (not including captions) and their associated images from Im-ageCLEF2010 (Popescu et al., 2010).

to RQA and DIY, there are no explicit connections between individual images and individual sentences, so we cannot compute AUC or precision, but this corpus represents an important organicallymultimodal setting.


Ultimately, the choice to split the annotations to three streams and use more than one attention heads, each focusing on different types of mentions, leads to better performance.

even the multi-head supervised attentive model, does not score as well as the multiview (non-attentive) model.


(2018), where an attention head is replaced by a model trained to predict syntac-tic dependencies (Dozat and Manning, 2017).

our model uses explicit supervision for all self-attention heads and is trained to predict the correct attention scores in a multi-task fashion.


(Lu et al., 2016) proposed a "visual sentinel", which applies the hidden state in LSTM as supervision to adaptively decide if it is necessary to input visual feature to the language model when generating the next word.

the supervision information from the hidden state isn't credible which cannot make sure it contains the corresponding visual decision signals.


The features of the POS tag in the linguistic computer vision.

they ignore the important property of POS tag, which relates to different visual semantics with different types of POS tag.


Considering this strong correlation between the two tasks, some joint models are proposed based on the multi-task learning framework (Zhang and Wang, 2016;Liu and Lane, 2016) and all these models outperform the pipeline models via mutual enhancement between two tasks.

their work just modeled the relationship between intent and slots by sharing parameters.


All these models outperform the pipeline models via mutual enhance- 5 We only consider the first subword label if a word is broken into multiple subwords ment between two tasks.

these joint models did not model the intent information for slots explicitly and just considered their correlation between the two tasks by sharing parameters.


This is due to the fact that these RPN-systems have to align every proposed region with the command.

the non-RPN systems only have to encode the full image once and then reason over this embedding.


In future versions, Talk2Car will be expanded to include the above annotations and dialogues.

this first version already offers a challenging dataset to improve current methods for the joint processing of language and visual data and for the development of suitable machine learning architectures.


In most cases, this involves some kind of digital manipulation, e.g., cropping, splicing, etc.

there are cases when an image is completely legitimate, but it is published alongside some text that does not reflect its content accurately.


The tool extracts metadata in the form of about 100 features such as size, resolution, GPS location.

most of this metadata turns out to be missing from our images: only five features could be extracted for more than half of the images from the Snopes dataset.


If there is virtually no change, then the cell has reached its local minima for error at that quality level.

if there is a large change, then the pixels are not at their local minima and are effectively original.


conventional frameworks like reinforcement learning (RL) or imitation learning (IL) are poorly suitable.


Unlike instruction-level evaluation, cascaded evaluation executes the instructions in sequence.

instead of starting of starting only from the start state of the first instruction, we create separate examples for starting from the starting state of each instruction in the interaction and continuing until the end of the interaction.


We see a similar trend for the implicit discriminator when looking at full game points, an interaction-level metric that does not account for performance on over 80% of the data because of error propagation.

the proportion of points scored computed using cascaded evaluation shows the benefit of both mechanisms.


Recent neural architectures including Transformer (Vaswani et al., 2017) and BERT (Devlin et al., 2018) have dramatically increased our ability to include a broad window of potential lexical hints.

the same capacity allows for multimodal context, which may help model the meaning of words in general, and also sharpen its understanding of instances of words in context (e.g., Bruni et al.


Similarly, SPICE also has a lower capability to identify action inconsistencies, which shows the limitation of scene graphs to capture this feature.

the n-gram-based metrics prefer to identify movement changes.


This has been the basis of overcoming domain-shift using many self-training techniques which select the most confident examples as new training examples, together with the predicted class as pseudo-label (McClosky et al., 2006).

modern neural networks are known to give wrongly calibrated confidence scores (Guo et al., 2017) meaning that the associated probability scores to the predicted class label does not reflect its correctness likelihood.


We find that none of the papers reported all of the items in our checklist.

every paper reported at least one item in the checklist, and each item is reported by at least one paper.


( , 2020 -as the goal of our analysis is to learn a smooth manifold of font styles that allows for stylistic inference given a small sample of glyphs.

many other style transfer tasks in the language domain (Shen et al., 2017) suffer from ambiguity surrounding the underlying division between style and semantic content.


Broadly speaking, our architecture represents the underlying shape that defines the specific character type (but not the font) as coarse-grained information that therefore enters the transpose convolutional process early on.

the stylistic content that specifies attributes of the specific font (such as serifs, drop shadow, texture) is represented as finer-grained information that enters into the decoder at a later stage, by parameterizing filters, as shown in the right half of Figure 2.


The conventional approach for computing loss on image observations is to use an independent output distribution, typically a Gaussian, on each pixel's intensity.

deliberate analysis of the statistics of natural images has shown that images are not well-described in terms of statistically independent pixels, but are instead better modeled in terms of edges (Field, 1987;Huang and Mumford, 1999).


Nearest neighbors provides a strong baseline on the full test set, even outperforming GlyphNet.

it performs much worse on the hard subset.


As shown in previous work Ponti et al., 2018), retrofitting (CLSRI-AR) and the cross-lingual post-specialization transfer (X-PS) are substantially better in the LS task than the original distributional space.

our full CLSRI-PS model results in substantial boosts in the  LS task (13-17%) over the previous best reported scores of X-PS as well as over CLSRI-AR.


Interestingly, for STS both X-PS and CLSRI-AR damage the performance of the distributional baseline.

the full CLSRI-PS model still shows a substantial improvement over all baselines.


Briefly, the sentence meaning approach to NLI takes the position that NLP systems should strive to model the aspects of a sentence's semantics which are closely derivable from the lexicon and which hold independently of context (Zaenen et al., 2005).

the speaker meaning approach to NLI takes the position that NLP systems should prioritize representation of the goaldirected meaning of a sentence within the context in which it was generated (Manning, 2006).


That is, both "He knows that the answer is 5" and "He does not know that the answer is 5" imply that "The answer is 5".

a verb like "manage to" has the signature +/− since, in a positive environment, the complement projects ("I managed to pass"→"I passed") but, in a negative environment, the negation of the complement projects ("I did not manage to pass"→ ¬"I passed").


As ordering the sequence is more nuanced than sifting basic from non-basic, one might expect consistently lower correlations.

the sequence gamma can have larger magnitude if, despite poor sifting, the order of the basic terms is correct.


During training, we maximize the logprobability of the correct tag sequence: While during decoding, we predict the output sequence that obtains the maximum score given by As discussed in Section 1, contextual information is vital for negative focus detection.

the BiLSTM-CRF model might fail to identify the important information related to a negative focus in contexts.


10 Consider the previous sentence S2, the author emphasize that the Nucor plant will shut down, thus the most likely negative focus should be anything in S3, whereas the prepositional phrase on some days only provides a temporal statement in our view.

different annotators might have different understandings and interpretations for this case.


Then, the pointer network based decoders predict the correct sentence sequence.

the paragraph vector depends on the permutation of input sentences.


To address the issue, ATTOrderNet (Cui et al., 2018) employs self-attention at the encoder to capture global dependencies regardless of an input sentence order.

similar to traditional models, they also compress sentence vectors into a single fixed-length vector via average pooling.


Since they are dependent on the permutation of the input sentences, their paragraph-level representations are not reliable.

self-attention-based AT-TOrderNet (encoder side) and attention-based our TGCM/TGCM-S (decoder side) use a set of sentence representations in a permutation-invariant manner rather than a single vector to represent a paragraph.


For dialogues with multiple interlocutors, extraction of their discourse structures provides useful semantic infor-mation to the "downstream" models used, for example, in the production of intelligent meeting managers or the analysis of user interactions in online fora.

despite considerable efforts to retrieve discourse structures automatically (Fisher and Roark, 2007;Duverle and Prendinger, 2009;Li et al., 2014;Joty et al., 2013;Ji and Eisenstein, 2014;Yoshida et al., 2014;Li et al., 2014;Surdeanu et al., 2015), we are still a long way from usable discourse models, especially for dialogue.


Once our MILNet model is trained, we can use it to obtain the attention score A E i and the sentiment score S E i for each EDU E i in a document (see (c) in Figure 3).

while each A E i is already a scalar, S E i is a vector with |C| elements, one for each sentiment class C i .


The first set of results in Table 3 shows that the hierarchical right/left branching baselines dominate the completely right/left branching ones.

their performance is still significantly worse than any discourse parser (intra-and interdomain).


It was a single perpetrator that engaged in touching or groping more often, rather than groups of perpetrators.

commenting happened more frequently when harassers were in groups.


Many common English nouns, adjectives and verbs, whose contribution to semantics is minimal (Forman, 2003) were also removed from the vocabulary.

namedentities were retained for their newsworthiness and a set of "trigger" words were retained that depict events (e.g.


Since PCG is also capable of representing other stock time series as potential influencers in the network, we can use this to model the propagation of shocks in the market as shown in Figure 2.

these links were not used for prediction performance to maintain parity with our baseline.


We build our method upon variational autoencoders (VAE), a deep probabilistic neural model which learns latent text features and their distributions effectively.

to other approaches using variational autoencoders for text generation, we modify the mechanism for generating new artificial samples such that we obtain samples structurally and semantically similar to a specific subset of the original data.


Embedding based methods, however are limited by the existing sentences available in a given corpus.

our approach is capable of generating new sentences not existing in the corpus, thus largely expanding the training data.


Here, data is embedded into a latent space which is modelled by conditional distributions, and samples from this distributions can be decoded into new artificial data instances.

to shallow models such as Skip-Gram (Mikolov et al., 2013) which also embeds into latent spaces, deep generative models have been shown to capture the implicit semantics and structure of the underlying data more effectively.


This is demonstrated by the performance of CRF+Doc2Vec (Table 5).

even using CRF+VAE (1 sample) shows higher F-score than CRF without notable loss of precision.


For instance in the Tweet "Ive had no appetite since I started on prozac", the annotators did not annotate no appetite as an ADR.

our model was able to predict it correctly as an ADR, but due to this mistake in test data is considered a false positive.


Online vendors use this data to understand users preferences and further predict their future needs.

user-generated data is rich in content and malicious attackers can infer users' sensitive information.


The goal of our model is to manipulated learned embedded representation such that any potential adversary cannot infer users' privateattribute information.

a challenge is that the text anonymizer does not know the adversary's attack model.


In some applications where the privacy of users are important and critical, we can set the α parameter above 0.5.

if the users privacy is not top priority, this parameter can be set to a lower value than 0.5 which although it does not protect users' private attribute as good as when α >= 0.5, but it does protect users' private attribute at a reasonable level.


Equation x = 660 / ( 32 + AST of the equations.

these approaches are based on traditional methods and require feature engineering.


We choose Key-Value Memory Networks (Miller et al., 2016) and GRAFT-Net (Sun et al., 2018) as our baseline models: to the best of our knowledge, these are the only ones that can use both text and KBs for question answering.

both models are limited by the number of facts and text that can fit into memory.


Our setup is not directly comparable to standard QA setups, as we aim to evaluate evidence rather than raw QA accuracy.

each judge model's accuracy is useful to know for analysis purposes.


Since our model is better at filtering out the irrelevant data, the improvement in SearchQA is not as significant as in other datasets.

our (RS→WS, RK, SUM) model still achieves the SoTA result by adopting the SUM method and HAS-QA, which is a strong baseline, also adopts the similar method.


As the number of paragraphs increases, the chances are better that the answer is included in these paragraphs.

the difficulty and running time for the reader also increase.


(2019) also ranked the paragraphs and took into account both the ranking results and the scores produced by the reader to predict the final answer.

they mainly utilize the paragraph-question relevance to rank the paragraphs and train the reader mainly on the positive paragraphs.


Commutativity of relation matrices was not recognized as an issue in the past research because the main focus was on predicting the truth value of atomic triplets.

when path queries are concerned, commutativity poses a problem.


Previous work (Elsahar et al., 2018) usually obtained predicate textual contexts through distant supervision.

the distant supervision is noisy or even wrong (e.g.


Previous study found that most questions contain the subject name or its aligns in SimpleQuestion (Petrochuk and Zettlemoyer, 2018).

the predicate name and object name hardly appear in the question.


(2018) demonstrated the effectiveness of POS copy for the context.

such a copy mechanism heavily relies on POS tagging.


Following (Serban et al., 2016;Elsahar et al., 2018), we adopt some word-overlap based metrics (WBMs) for natural language generation including BLEU-4 (Papineni et al., 2002), ROUGE L (Lin, 2004) and METEOR (Denkowski and Lavie, 2014).

such metrics still suffer from some limitations (Novikova et al., 2017).


One might think that if a lot of words in a question overlap with those in a passage, the answer may be easily detected from the matched sentence in the passage.


Monolingual vs. Multilingual: For the DrQA model, we observe that the multilingual training significantly improves the performance on English comparing with the monolingual training with only the English dataset.


Existing work focuses on understanding linguistic and semantic properties of word representations or how well pretrained sentence representations and language models transfer linguistic knowledge to downstream tasks.

our investigation seeks to answer to what extent pretrained language models store factual and commonsense knowledge by comparing them with symbolic knowledge bases populated by traditional relation extraction approaches.


2018leveraged deep Q-network to solve the AWP problems, achieving a good balance between effectiveness and efficiency.

all the existing AWP systems are only trained and validated on small benchmark datasets.


Theoretically, − → E and ← − E are complement to each other .

as a number may occur several times and represent different facts in a document, we add a distinct node for each occurrence in the graph to prevent potential ambiguity.


By doing this, it learns the mappings among different languages and performs good on the XNLI dataset.

xLM only uses a single cross-lingual task during pre-training.


Similar to translation language model in XLM, this task is based on the bilingual sentence pairs as well.

as it doesn't use the original words as input, we can train this task by recovering all words at the same time.


1 In contrast to the rapid progress shown in Question Answering (QA) tasks (Rajpurkar et al., 2016;Yang et al., 2018), the task of Question Generation (QG) remains understudied and challenging.

as an important dual Context: ...during the age of enlightenment, philosophers such as john locke advocated the principle in their writings, whereas others, such as thomas hobbes, strongly opposed it.


Sachan and Xing (2018) proposed a self-training cycle between QA and QG.

these works either reduced the ground-truth data size or simplified the span-prediction QA task to answer sentence selection.


Generate from Existing Articles In SQuAD (Rajpurkar et al., 2016), each context-answer pair only has one ground-truth question.

usually, multiple questions can be asked.


Previous works (Fried et al., 2018;Dhingra et al., 2018) proposed to first pre-train the model with synthetic data and then fine-tune it with ground-truth data.

we find when the synthetic data size is small (e.g., similar size as the ground-truth data), catastrophic forgetting  will happen during fine-tuning, leading to similar results as using ground-truth data only.


On the one hand, the copy mechanism provides detailed background information for generating a question.

if not copying correctly, the question could be syntactically incorrect.


For instance, in the first example, the passage suggests that people fled from Sudan to Chad, while the generated question describes the wrong direction.

overall we think that the current question generator provides reasonable synthesized questions, yet there is still large room to improve.


(2018) took a generative approach where they added a decoder on top of their extractive model to leverage the extracted evidence for answer synthesis.

this model still relies heavily on the extraction to perform the generation and thus needs to have start and end labels (a span) for every QA pair.


A relevant example of such a question is the following: An important characteristic of all solutions developed for this task is that they are not given explicitly any external information in the form of documents supporting the correct answer or semistructured information.

external information is highly desirable, especially domain and common-sense knowledge.


When a valid document is provided -guaranteed to contain the correct answer -the exact match (EM) score obtained by DrQA is 69.5.

when the supporting document has to be retrieved from Wikipedia, by an information retrieval engine, the EM score drops to 27.1.


ing has focused on integrating external knowledge (linguistic and/or knowledge-based) into recurrent neural network models using Graph Neural Networks (Song et al., 2018), Graph Convolutional Networks (Sun et al., 2018;De Cao et al., 2019), attention (Das et al., 2017;Mihaylov and Frank, 2018;Bauer et al., 2018) or pointers to coreferent mentions (Dhingra et al., 2017).

in this work we examine the impact of discourse-semantic annotations ( Figure 1) in a self-attention architecture.


Therefore our method is most related to LISA (Strubell et al., 2018), which uses joint multi-task learning of POS and Dependency Parsing to inject syntactic information for Semantic Role Labeling.

we do not use multi-task learning, but directly encode semantic information extracted by pre-processing with existing tools.


the annotator 2 cannot use the long answer, so reasoning over the context is required for 3 Both are qualified M.D.


Strictly speaking, most yes/no/maybe research questions can be answered by "maybe" since there will always be some conditions where one statement is true and vice versa.


This resembles the transfer learning discussed by Howard and Ruder (2018), where the source domain would be the BM25 sentences, and the target domain the ROCC justifications.

one important distinction is that, in our case, all this knowledge comes solely from the resources provided within each dataset, and is retrieved using unsupervised method (BM25).


We conjecture this happens because the BERT language model was trained on a large text corpus that comes from these two do-  mains.

importantly, AutoROCC is more robust across domains that are different from these two, since it is an unsupervised approach that is not tuned for any specific domain.


Designed to be more challenging than SQuAD-like datasets, they feature questions that require context of more than one document to answer, testing QA systems' abilities to infer the answer in the presence of multiple pieces of evidence and to efficiently find the evidence in a large pool of candidate documents.

since these datasets are still relatively new, most of the existing research focuses on the few-document setting where a relatively small set of context documents is given, which is guaranteed to contain the "gold" context documents, all those from which the answer comes (De Cao et al., 2019;Zhong et al., 2019).


Recent work on open-domain question answering largely follow this retrieve-and-read approach, and focus on improving the information retrieval component with question answering performance in consideration (Nishida et al., 2018;Kratzwald and Feuerriegel, 2018;Nogueira et al., 2019).

these one-step retrieve-and-read approaches are fundamentally ill-equipped to address questions that require multi-hop reasoning, especially when necessary evidence is not readily retrievable with the question.


Both datasets feature a few-document setting where the gold supporting facts are provided along with a small set of distractors to ease the computational burden.

researchers have shown that this sometimes results in gameable contexts, and thus does not always test the model's capability of multi-hop reasoning (Chen and Durrett, 2019;Min et al., 2019a).


Further inspection reveals that despite Elasticsearch improving overall recall of gold documents, it is only able to retrieve both gold documents for 36.91% of the dev set questions, in comparison to 28.21% from the IR engine in (Yang et al., 2018).

gOLDEN Retriever improves this percentage to 61.01%, almost doubling the recall over the single-hop baseline, providing the QA component a much better set of context documents to predict answers from.


Since the prediction result for each query substructure is independent, the score for query structure S i is measured by joint probability, which is Assume that should be 1 in the ideal condition.


Alike FP, our model also attains better results in LP, and the SOTA results on FB15K-237 dataset 2 .

we also notice that AttnPath doesn't attain the best result under a small part of query relations, even lower than TransE / R. By analyzing triples related to these relations, we found that: 1) they have more outgoing edges of the other relations pointing to the entities which are not the Figure 2: Total SR and SR-10 for two relations of NELL-995.


IE-gold + R-GCN combines the advantages of IE-gold and R-GCN, and performs best among the baselines.

in the 1hop-subgraphs, the Added Acc and Deleted Acc are only 0.4681 and 0.2448, which are still quite low.


However, the targets may be very far and hard to reach, it is the shortcuts that make the successful information delivery possible.

if there are too many shortcuts, the messages are easy to get to wrong targets.


On the one hand, for existing RL-based methods, their results on FB15K-237 are generally lower than those on NELL-995 since FB15K-237 is more complex and arguably more difficult to design proper reward functions manually.

our framework reliefs this problem to some extent by dynamically learning superior reward functions, thus we make greater improvements on challenging FB15K-237.


As a result, semantics to be paid attention are uncertain and unstable for matching because semantics are changed at different layers.

the intermediate representations tend to be affected by error propagation in multi-layered attentions, in which if the first attention aligns the wrong position, the second attention will now have the incorrect information as input for alignment.


These methods try to incorporate information of relation paths to get better performance.

they pay less attention to the order of relations in a path when learning representations of the path.


In recent years, there has been a surge of interests in interpretable graph reasoning methods.

these models often suffer from limited performance when working on sparse and incomplete graphs, due to the lack of evidential paths that can reach target entities.


Given a daily-life event, human can easily understand it and reason about its causes, effects, and so on.

it still remains a challenging task for NLP systems.


This is useful for reasons such as marketing, overall enjoyment of interaction, and mental health therapy.

due to limited data with emotional content in specific semantic contexts, the generated text may contain incorrect semantic content.


Adjectives typically carry less semantic meaning and SMERTI likely has more trouble figuring out how best to infill the text.

nouns typically carry more, and phrases the most (since they consist of multiple words).


Constraining the number of training examples is effective for part-of-speech, suggesting that learning each linguistic task requires fewer samples than our control task.

for dependency edge prediction, this leads to significantly reduced linguistic task accuracy.


We could estimate these by sampling t from the token encoder p ✓ (t | x) and then evaluating all q , p ✓ , r , and s ⇠ probabilities.

in fact we use the sampled t only to estimate the first expectation (by computing the decoder probability q (y | t) of the gold tree y); we can compute the KL terms exactly by exploiting the structure of our distributions.


In all experiments so far, we did not model this constraint explicitly to investigate whether the model is able by default to predict rank correctly.

in exploring model configurations we also report on whether adding this constrant leads to better performance .


The authors of BERT recommend not to mask words randomly with [MASK] when fine-tuning the network.

we discovered that masking often reduces the tendency of the classifiers to overfit to BERT by forcing the network to rely on the context of surrounding words.


We surmise that this is due to using mixed batches on an unbalanced training set, which skews the model towards predicting larger treebanks more accurately.

we find that fine-tuning on the treebank individually with BERT weights saved from UDify eliminates most of these gaps in performance.


To speed up training, we employ bucketed batching, sorting all sentences by their length and grouping similar length sentences into each batch.

to ensure that most sentences do not get grouped within the same batch, we fuzz the lengths of each sentence by a maximum of 10% of its true length when grouping sentences together.


We primarily focus on the OpenBookQA dataset since it is the only dataset currently available that provides partial context.

we believe such an approach is also applicable to the broader setting of multi-hop RC datasets, where the system could start reasoning with one sentence and fill remaining gap(s) using sentences from other passages.


This allows using an attention-based approach of indirectly combining information (Dhingra et al., 2018;Cao et al., 2019;Song et al., 2018).

open domain question answering datasets  come with no context, and require first retrieving relevant knowledge before reasoning with it.


Some RC systems (Mihaylov and Frank, 2018;Kadlec et al., 2016) and Textual Entailment (TE) models (Weissenborn et al., 2017;Inkpen et al., 2018) incorporate external KBs to provide additional context to the model for better language understanding.

we take a different approach of using this background knowledge in an explicit inference step (i.e.


The OpenBookQA dataset provides the core science fact used to create the question.

in 20% of the cases, while the core science fact inspired the question, it is not needed to answer the question .


It has been shown that simply fine-tuning large, pre-trained language models such as GPT (Radford et al., 2018) and BERT (Devlin et al., 2019) can be a very strong baseline method.

there still exists a large gap between performance of said baselines and human performance.


(2018) follow this line of works to incorporate the embeddings of related knowledge triples at the word-level and improve the performance of natural language understanding tasks.

to our work, they do not explicitly impose graph-structured knowledge into models , but limit its potential within transforming word embeddings to concept embeddings.


For RCQA, a well-known resource is SQuAD (Rajpurkar et al., 2016) with 100K QA data created by human, followed by NarrativeQA (Kočiskỳ et al., 2018), SQuAD 2.0 (Rajpurkar et al., 2018), and CoQA (Reddy et al., 2018).

as these datasets support only English, supporting other languages requires either annotation efforts in a comparable scale (Lim et al., 2018), or modeling efforts to overcome the limitation of training resources in terms of quantity or quality.


The objective of original QA model minimizes the sum of the negative log probabilities: where N is the number of examples in training set, s i and e i are the start and end indices of the i-th example, respectively.

for down-weighting noise, we combine this QA loss function with weighted confidence score, based on function f (CS(p, q)): in initial steps of training, the confidence score is not reliable, and may cause unstable training.


Not only for question answering, such method has been successful in sentiment classication (Zhou et al., 2016), relation extraction (Faruqui and Kumar, 2015), and causal commonsense (Yeo et al., 2018).

these approaches are dependent on quality of translation.


model avg seen ± sd avg unseen ± sd CNN+LSTM 0.608 ± 0.01 0.4036 ± 0.003 CNN+LSTM+SA 0.7813 ± 0.009 0.235 ± 0.006 FiLM 0.8489 ± 0.014 0.153 ± 0.02 chance 0.5 0.5  We tackle the modeling of size GAs as a relational problem in the domain of visual reasoning, and show (in contrast with Kuhnle et al., 2018) that FiLM is able to learn the function underlying the meaning of these expressions.

none of the models develop an abstract representation of GAs that can be applied compositionally, an ability that even 4-year-olds master (Barner and Snedeker, 2008).


Consistent with results from previous experiments, detecting the presence of a licensor is slightly more challenging for models fine-tuned with CoLA or NPI data.

the overall lower performances in scope detection compared with detecting the presence of the licensor is not found in the minimal-pair experiments.


Based on this method alone, we might conclude that BERT's knowledge of this domain is near perfect.

the other methods show a more nuanced picture.


By considering results from several evaluation methods, we demonstrate that BERT has systematic knowledge of NPI licensing.

this knowledge is unequal across the different features relevant to this phenomenon, and does not reflect the Boolean effect that these features have on acceptability.


Neural language models have achieved stateof-the-art performances on many NLP tasks, and recently have been shown to learn a number of hierarchically-sensitive syntactic dependencies between individual words.

equally important for language processing is the ability to combine words into phrasal constituents, and use constituent-level features to drive downstream expectations.


The LSTM (1B), LSTM (PTB), and RNNG models show zero or negative singular expectation for the pl or sg conditions, as expected.

the LSTM (enWiki) and ActionLSTM models show positive plural expectation in this condition, indi- cating that they have not learned the humanlike generalizations.


By comparing the BARE and OEST models' columns in Table 1, the non-conditional baseline BARE is superior for 71 / 77 languages (the exceptions being Chamorro, Croatian, Italian, Swazi, Swedish, and Tuareg).

the same columns in Table 3 and Table 2 reveal an opposite pattern: OEST outperforms the BARE baseline in 70 / 77 languages.


In particular, we find that identifying generic frames benefits from removing topic features, which are actually the hardest case.

removing topic features cannot help in identifying topic-specific frames.


agreement) given that a relationship exists.

we predict both argument components as well as the existence of a relation between them.


(2014) introduce a scheme to annotate inter-turn relations between two posts as "target-callout", and intraturn relations as "stance-rationale".

their empirical study is reduced to predicting the type of inter-turn relations as agree/disagree/other.


Predicting an argumentative relation is made more difficult by the fact that we need to consider all possible relation pairs.

some argumentative components may contain linguistic properties that allow us to predict when they are targets even without the full relation pair.


Various statistical methods have also been proposed to filter out the mistakes or (spamming) random responses of the † Corresponding author crowdsource workers (Liu et al., 2012;Hovy et al., 2013;Nguyen et al., 2017).

the way to filter out the mistakes or the random responses through statistical means is difficult to utilize for a fundamentally subjective annotation task.


They also showed that baseline classifiers could achieve moderate performance on the task.

we focus on local acceptability of arguments with restricted options for reason selection.


It focused on just two pronouns, it and they, and was applied to a single language pair.

we have a fully automated evaluation measure, we handle many English pronouns, and we cover multiple source languages.


This limits the evaluation measure both in terms of the language and also of the pronouns it is applicable to.

our framework requires only two candidate translations of the same text as input for comparison: this could be a reference vs. a system translation, or a comparison between two candidate translations (see Section 5.5).


First, we can easily verify such causalities, because Wikipedia articles tend to credibly attest them; for example, the Tobacco article states that inhaling its smoke can cause Lung cancer.

3 knowledge from other sources such as the web text tends to be difficult to verify, owing to a deluge of false information.


For example, in Wikidata, World War I (Q361) and the Paris Peace Conference (Q199820) show causality, as the former has the has effect relation with the latter.

world war I and the German invasion of Belgium (Q5551414) do not show causality, because they have a significant event relation between them.


Classical review encoder utilizes h i to represent review x and ends here.

we find that users with different backgrounds pay attention to different content of a review.


The classical summary decoder combines the context vector c t and the decoder state s t , and then feeds the merged vector into a linear layer to produce the vocabulary distribution.

when generating a summary, users with different attributes may have their own vocabulary.


Since the summary generated by S2SATT contains more overlapping words with the gold summary than the one generated by ASN contains, S2SATT obtains higher ROUGE scores than ASN.

from the view of aspects, ASN may be better.


Most recent works on neural extractive summarization have been rather successful in generating summaries of short news documents (around 650 words/document) (Nallapati et al., 2016) by applying neural Seq2Seq models (Cheng and Lapata, 2016).

when it comes to long documents, these models tend to struggle with longer sequences because at each decoding step, the decoder needs to learn to construct a context vector capturing relevant information from all the tokens in the source sequence (Shao et al., 2017).


However, the only information about sections fed into their sentence classifier is a categorical feature with values like Highlight, Abstract, Introduction, etc., depending on which section the sentence appears in.

in order to exploit section information, in this paper we propose to capture a distributed representation of both the global (the whole document) and the local context (e.g., the section/topic) when deciding if a sentence should be included in the summary Our main contributions are as follows: (i) In order to capture the local context, we are the first to apply LSTM-minus to text summarization.


They often assume that writing knowledge can be acquired from the training data alone.

when people are writing, they not only rely on the data but also consider related knowledge.


They often assume that all writing knowledge can be learned from the training data.

when people are writing, they will not only rely on the data contents themselves but also consider related knowledge, which is neglected by previous methods.


One may argue that neural models can learn such knowledge when enough parallel cooccurrence pairs such as (Guelma, Algerian) and (MC El Eulma, Algerian Ligue Professionnelle 1) are available.

even in such case, neural models still tend to make mistakes for sparse co-occurrence pairs as we will show in the experiments section.


Intuitively, incorporating an external KB should improve data-to-text generation performance.

to pinpoint the effect of the additional knowledge is not trivial since we know that (a) not all external knowledge is relevant and (b) neural models may memorize certain inference patterns when parallel data is big enough.


The reason why the absolute improvement is relative small is that the full WikiBio dataset consists of 728,321 parallel data-to-text pairs which are enough for neural models to memorize certain inference patterns for high frequency pairs.

as will be shown in the Section 4.5, the baseline fails on low co-occurrence frequency pairs, but the KBAtt avoids this problem with the contributions from the external knowledge.


In this case, KBAtt makes a plausible inference on nationality from birth place, so it generates "american".

this inference pattern doesn't hold for this case, because he is British.


Recently, Perez-Beltrachini and Lapata (2018) generalize this task to multi-sentence text generation, where they focus on bootstrapping generators from loosely aligned data.

most of the work mentioned above assume all the writing knowledge can be learned from massive parallel pairs of training data.


Note that the rewards usually reflect the quality of extracted summary and measured by standard evaluation protocol.

they still sequentially process text and tend to extract earlier sentences over later ones due to the sequential nature of selection (Dong et al., 2018).


It shows that all the three models can focus on different parts of the context to form summary at first and BANDITSUM performs the best after training 10k steps.

with training steps growing, BANDITSUM and HER w/o Local begin to prefer earlier sentences.


Traditional approaches to CLS are based on the pipeline paradigm, which either first translates the original document into target language and then summarizes the translated document (Leuski et al., 2003) or first summarizes the original document and then translates the summary into target language (Lim et al., 2004;Orasan and Chiorean, 2008;Wan et al., 2010).

the current machine translation (MT) is not perfect, which results in the error propagation problem.


In conclusion, TNCLS can generate more informative summaries, but it is difficult to improve the conciseness and fluency.

with the help of MT and MS tasks, conciseness and fluency scores can be significantly improved.


Generating headlines with many clicks is especially important in this digital age, because many of the revenues of journalism come from online advertisements and getting more user clicks means being more competitive in the market.

most existing websites 2 naively generate sensational headlines using only keywords or templates.


Thus, we train another baseline classifier on a crawled balanced sensationalism corpus of 84k headlines where the positive headlines have at least 28 comments and the negative headlines have less than 5 comments.

the results on the test set show that the baseline classifier gets 60% accuracy, which is worse than the proposed classifier (which achieves 65%).


In addition, many papers (Nallapati et al., 2017;Zhou et al., 2018b; use extractive methods to directly select sentences from articles.

none of these work considered the sensationalism of generated outputs.


Implicit methods (Shen et al., 2017b;Fu et al., 2018;Prabhumoye et al., 2018) transfer the styles by separating sentence representations into content and style, for example using backtranslation (Prabhumoye et al., 2018).

these methods cannot guarantee the content consistency between the original sentence and transferred output (Xu et al., 2018a).


Explicit methods (Zhang et al., 2018b;Xu et al., 2018a) transfer the style by directly identifying style related keywords and modifying them.

sensationalism is not always restricted to keywords, but the full sentence.


Pointer generator networks have also been extensively accepted by the ABS community due to their efficacy with long document summaries (Chen and Bansal, 2018;Hsu et al., 2018), title summarization , etc.

the current power of abstractive summarization falls short of their potential.


Further, pointer generator models can effectively adaptive to both extractor and abstractor networks (Chen and Bansal, 2018), and summaries can be generated by incorporating a pointer-generator and multiple relevant tasks (Guo et al., 2018), such as question or entailment generation, or multiple source texts .

work particularly targets the problem of the abstraction is rare.


Overall, the scores favour the delexicalised approach (negative delta in the all deprels column for all languages) supporting the results given by the automatic metric.

for some dependency relations, the lexicalised baseline shows usefulness of word information, for ex-  Table 4: Contraction Generation Results (BLEU scores).


Existing approaches try to explicitly disentangle content and attribute information, but this is difficult and often results in poor content-preservation and ungrammaticality.

we propose a simpler approach, Iterative Matching and Translation (IMaT), which: (1) constructs a pseudoparallel corpus by aligning a subset of semantically similar sentences from the source and the target corpora; (2) applies a standard sequence-to-sequence model to learn the attribute transfer; (3) iteratively improves the learned transfer function by refining imperfections in the alignment.


by aggregating system summaries' ROUGE scores across multiple input documents, we can reliably rank summarisation systems by their quality.

rOUGE performs poorly at summary level: given multiple summaries for the same input document, rOUGE can hardly distinguish the "good" summaries from the "mediocre" and "bad" ones (Novikova et al., 2017).


We believe this is the reason why ExtAbsRL has higher ROUGE scores.

extAbsRL extracts more redundant sentences: four out of 30 summaries by extAbsRL include redundant sentences, while Refresh and NeuralTD do not generate summaries with two identical sentences therein.


For instance, paraphrasing or machine translation exhibit a one-to-one relationship because the source and the target should carry the same meaning.

summarization or question generation exhibit one-to-many relationships because * Most work done during internship at Clova AI.


We model this kind of information in time dimension via selfattention.

unlike the unordered nature of rows and columns, the history information is sequential.


(2010) use the theme structure to define the representation for each sentence.

their solutions only consider statistical knowledge.


(2015) propose a sparse coding method to generate summaries that not only cover key content in news but also focuses highlighted by readers' comments.

they do not consider semantic information.


But they do not tackle sequential context information among sentences and treat them as separate instances.

we deal with sequential context information within each document and the relationships among documents.


In fact, one can also try to use a CNN to get t d .

our experiments suggest RNN performs better than CNN.


Cao and Clark (2019) factor the generation process leveraging syntactic information to improve the performance.

they linearize both AMR and constituency graphs, which implies that important parts of the graphs cannot well be represented (e.g., coreference).


On the same dataset, we have competitive results to Damonte and Cohen (2019).

we do not rely on preprocessing anonymisation not to lose semantic signals.


By penalizing such inconsistencies, the model enables the generation of more consistent outputs.

perfect consistency between attention weights occasionally disturbs the model to generate proper outputs.


It would, however, take the expertise of at least a teacher of English and also a computer engineer to achieve it; the former would have to think of what is typical in the given topic and then to make a set of rules; the latter then would have to turn them into computer-readable forms.

the neural retrieval-based method only requires a teacher of English to annotate a given corpus with feedback comments, without examining what is typical, which is much more effective and efficient.


On the one hand, it would require a successful parsing to recognize the sources of the errors.

it would require successful error detection to parse them correctly.


Compared with rule-based approaches, neural models (Yuan et al., 2017) can generate more fluent and grammatical questions.

question generation is a one-to-many sequence generation problem, i.e., several aspects can be asked given a sentence, which confuses the model during train and prevents concrete automatic evaluation.


From this correlation analysis against human judgments, we observe that, as expected, the Language Model metric captures readability better than ROUGE, while falling short on relevance.

the results obtained using the proposed QA-based metrics indicate their potential benefits especially under the unsupervised setting, with QA conf and QA f score capturing readability and relevance better than all the others reported metrics, including ROUGE.


To elaborate further, we notice that applying the learned coefficients for 1 to the results obtained by models reinforced on QA learned and QA equally , see Table 2, we obtain very similar scores (namely, 136.43 for QA equally and 136.4 for QA learned ).

the qualitative analysis reported in Tables 3 and 4 shows that while they perform sim-ilarly in terms of relevance, a significantly lower score for readability is obtained using QA equally .


Nema et al., 2018) using well-designed data encoder and attention mechanisms.


Improving the parser and deriving a more semantically-aware set of compression rules can help achieving better grammaticality and readability.

we note that such errors are largely orthogonal to the core of our approach; a more refined set of compression options could be dropped into our system and used without changing our fundamental model.


Without the manual deduplication mechanism, our model matches the ground truth around 80% of the time.

a low accuracy here may not actually cause a low final ROUGE score, as many compression choices only affect the final ROUGE score by a small amount.


A natural solution to the data-scarcity issue is to resort to massive data from other domains.

directly leveraging abundant data from other domains is problematic due to the discrepancies in data distribution on different domains.


This enables the model to learn generic style information from both domains.

explicitly learning precise stylized information within each domain is crucial to generate domain-specific styles.


Existing approaches focus on encoding the passage, the answer and the relationship between them using complex functions and then generate the question in one single pass.

by carefully analysing the generated questions, we observe that these approaches tend to miss one or more of the important aspects of the question.


Over 68.6%, 66.7% and 64.2% of the generated questions from RefNet were respectively more fluent, complete and answerable when compared to the EAD model.

there are some cases where EAD does better than RefNet.


It guarantees the maximum volume size of selected points with minimum number of points (Figure 2 (c)).

it does not reduce a redundancy between the points over the convex-hull, and usually choose larger number of sentences than k. Marcu (1999) shows an interesting study regarding an importance of sentences: given a document, if one deletes the least central sentence from the source text, then at some point the similarity with the reference text rapidly drops at sudden called the waterfall phenomena.


In most evaluations, ROUGE scores are linear to SO ratios as expected.

vO has high variance across algorithms and aspects.


One thing to note is that XSum and AMI have less new words in their target summaries.

paper datasets (i.e., PeerRead and PubMed) include a lot, indicating that abstract text in academic paper is indeed "abstract".


LexRank is highly biased toward the position aspect.

mmR is extremely biased to the importance aspect on XSum and Reddit.


(2018) investigate how to evaluate semi-supervised training algorithms in a realistic way; they differ from us in that they focus exclusively on semi-supervised learning (SSL) algorithms, and do not consider NLP explicitly.

in line with our conclusion, they report that recent practices for evaluating SSL techniques do not address the question of the algorithms' real-word applicability in a satisfying way.


As we will see in later sections, this is one of the main sources of search errors.

in many cases, the model score found by beam search is a reasonable approximation to the global best model score.


ELMo and BERT improve naive baselines by a large margin, indicating that a notable amount of commonsense knowledge has been acquired via pre-training.

even BERT still falls far behind human performance, indicating the need of further research.


Standard accuracy metrics indicate that modern reading comprehension systems have achieved strong performance in many question answering datasets.

the extent these systems truly understand language remains unknown, and existing systems are not good at distinguishing distractor sentences, which look related but do not actually answer the question.


Question answering tasks are widely used for training and testing machine comprehension and reasoning (Rajpurkar et al., 2016;Joshi et al., 2017).

high performance in standard automatic metrics has been achieved with only superficial understanding, as models exploit simple correlations in the data that happen to be predictive on most test examples.


Intuitively, the model is expected to choose the answer span after fully considering the entire question and paragraph.

traditional QA models suffered the overstability problem, and tended to be fooled by distractor answers, such as the one containing an unrelated human name.


and then search the target entity America from KGs as the answer.

as many KGs are constructed automatically and face serious incompleteness problems (Bordes et al., 2013), it is often hard to directly get target entities for queries.


These demonstrate that LM-based methods perform very well on the associative sentences, as expected.

their performance drops significantly on the non-associative subset, when information related to the candidates themselves does not give away the answer.


It shows that Pun-GAN can generate more vivid pun sentences compared with the previous best model CLM+JD.

there still exists a big gap between generated puns and human-written puns.


Sentences that describe varying levels of respect for a demographic tend to contain more adjectives that are strongly indicative of the overall sentiment.

sentences describing occupations are usually more neutrally worded, though some occupations are socially perceived to be more positive or negative than others.


expresses conflict sentiment towards ambience aspect.

most of existing studies ignore conflict opinions, for the reason that they are sparse in the datasets (Tang et al., 2016b;He et al., 2018).


Recently, researches have explored the graph neural network (GNN) techniques on text classification, since GNN does well in handling complex structures and preserving global information.

previous methods based on GNN are mainly faced with the practical problems of fixed corpus level graph structure which do not support online testing and high memory consumption.


cific points in the sequence when computing its output.

in this case, the attention attends the wrong context, as there are many words have no correlation or do not correspond to actual words.


Such approaches may achieve high-quality extraction and labeling.

they rely on extracted PDF source markup (not always available, e.g.


Aletras and Stevenson (2013) devised a new method by mapping the topic words into a semantic space and then computing the pairwise distributional similarity (DS) of words in that space.

the semantic space is still built on PMI or NPMI.


In recent years, Variational Autoencoder (VAE) has been proved more effective and efficient to approximating deep, complex and underestimated variance in integrals (Kingma and Welling, 2013;He et al., 2017).

the VAE-based topic models focus on the construction of deep neural networks to approximate the § The two authors contributed equally to this work.


When performing cross-language information retrieval (CLIR) for lower-resourced languages, a common approach is to retrieve over the output of machine translation (MT).

there is no established guidance on how to optimize the resulting MT-IR system.


The BM25 model was evaluated against both the Europarl and Wikipedia collections.

to avoid the performance degradation caused by crosscollection evaluation (Cohen et al., 2018), we only evaluate the Wikipedia-trained neural model on the Wikipedia evaluation collection.


We conclude that none of the n-gram precision components of BLEU or variations on it provide consistently better correlations with IR performance.

given a specific collection and model, it is likely one of the alternatives we explored here or other metrics (e.g.


(2017) built on this work, exploring several strategies for rotating embeddings to obtain more semantically meaningful dimensions.

to our knowledge, orthogonal transformations themselves have not been used to represent word relationships; our work is novel in this respect.


(2017a) convert WSD task to a sequence labeling task, thus building a unified model for all polysemous words.

neither of them can totally beat the best word expert supervised methods.


So far, we have obtained the adjective-noun pairs and value polarity relations for nouns.

it is still unclear whether the polarity is positive or negative.


While the meanings of defining words are important in dictionary definitions, it is crucial to capture the lexical semantic relations between defined words and defining words.

thus far, the utilization of such relations has not been explored for definition modeling.


Word analogy test results show that word representations trained with pre-trained Chinese n-grams perform better than those trained without (SISG(cjhr)), supporting our claim that our approach is able to transfer relevant knowledge from the Chinese language for detecting analogical relationships.

for the word similarity test, word vectors trained without Chinese embeddings perform better, suggesting that there are some trade-offs.


Previous works addressing this challenge mainly focused on word-level aspects such as word embeddings.

in many cases, languages share common subwords, especially for closely related languages, but also for languages that are seemingly irrelevant.


One popular approach is to estimate the relation between noisy and clean, gold-standard labels and use this noise model to improve the training procedure.

most of these approaches only assume a dependency between the labels and do not take the features into account when modeling the label noise.


Increasing the number of clusters introduces smaller clusters for which it is difficult to estimate the noise matrix, due to the limited training resources.

decreasing the number of clusters can generalize too much, resulting in loss of information on the noise distribution.


GPA was recently used to jointly transform multiple languages into a shared vector space (Kementchedjhieva et al., 2018).

gPA assumes that a multi-way word correspondence is available, which is often not the case.


Note that MGPA needs a multi-way dictionary constructed from the bilingual dictionaries.

mPPA uses directly the raw data (the bilingual dictionaries).


Adversarial training is a popular method to ensure the transferred sentences have the desired target styles.

previous works often suffer from content leaking problem.


(2018) also made use of a conditional discriminator for multiple style transfer.


The optimized network is inferred by choosing the edges with maximum weights in softmax.

dARTS is a "local" model because the softmax-based relaxation is imposed on each bundle of edges between two nodes.


Bilinear models such as DistMult and ComplEx are effective methods for knowledge graph (KG) completion.

they require large batch sizes, which becomes a performance bottleneck when training on large scale datasets due to memory constraints.


This technique is possible for standard benchmarks but not for large KGs, and we report results in Appendix D for all datasets small enough to allow for full contrastive training.

our main experiments use NLL of sampled softmax since our focus is on scalability.


As shown in Table 6, while AE achieves the best reconstruction when the noise is small (k = 1), its reconstruction deteriorates dramatically when k > 1, which suggests AE fails to learn a smooth latent space.

our method outperforms all the baselines by a large margin when k > 1.


BIOBERT is trained on PubMed abstracts and PMC full text articles, and CLIN-ICALBERT is trained on clinical text from the MIMIC-III database .

sCIBERT is trained on the full text of 1.14M biomedical and computer science papers from the semantic scholar corpus (Ammar et al., 2018).


We find that combining the sparse global gradient with the dense local gradient improves convergence.

adding local information means that nodes' parameters will diverge over time.


mMiniBERT Effectiveness The multilingual baseline mMeta-LSTM does not do well on lowresource languages.

mMiniBERT performs well and outperforms the state-of-the-art Meta-LSTM on the POS tagging task and on four out of size languages of the Morphology task.


Deep learning has achieved great success in the SLU field (Mesnil et al., 2015;Liu and Lane, 2016;Zhao et al., 2019).

it is notorious for requiring large labelled data, which limits the scalability of SLU models.


We observe no significant trend of favoring one branching direction over the other.

after training with the language modeling objective, PaLM-U shows a clear right-skewness more than it should: it produces much more right-branching structures than the gold annotation.


When we manually change the sentence pattern into "List the most common hometown of teachers", the parser gives the correct keyword.

the characterbased model is less sensitive to question sentences, which is likely because characters are less sparse compared with words.


In their work, they compare EigenSent with various sentence embedding models, including a different implementation of the Discrete Cosine Transform (DCT*).

to our implementation described in section 2.2, DCT* is applied at the word level along the word embedding dimension.


These models encode and contextualize sentences in two consecutive steps.

we propose an input representation which allows the Transformer layers in BERT to directly leverage contextualized representations of all words in all sentences, while still utilizing the pretrained weights from BERT.


(2017) is that they use the ROUGE scores to label the top (bottom) 20 sentences as positive (negative), and the rest are neutral.

we found it better to train our model to directly predict the ROUGE scores, and the loss function we used is Mean Square Error.


We observe that before finetuning, the attention patterns on [SEP] tokens and periods is almost identical between sentences.

after finetuning, the model attends to sentences differently, likely based on their different role in the sentence that requires different contextual information.


achieving 65.6% success rate between agents that never interacted with each other.

when the inter-group interaction occurred only half as frequently as the intra-group interaction, the agents from the two groups can play together with a much lower 52.4% success rate.


To the best of our knowledge, ours is the first work which builds end-to-end models on a large-scale dataset for topic-focused summarization.

generating Wikipedia to our work focusing on content selection for topic-focused summaries, there have been previous work interested in generating Wikipedia articles.


Another series of works focus on template-based methods such as (Oya et al., 2014).

template-based methods are too rigid for our patternized summary generation task.


On one hand, the sections in the prototype summary that are not highly related to the prototype document are the universal patternized words and should be emphasized when generating the new summary.

the sections in the prototype document that are highly related to the prototype summary are useful facts that can guide the process of extracting facts from input document.


With the IB objective (eq 3), there is no benefit to keeping any information from Z, which strictly makes the first term worse (more mutual information between source and summary) and does not affect the second (Z is unrelated to Y ).

1 In IB, this is a strict statistical relationship.


Thus, these supervised models do not generalize well to other kinds of sentence summarization or domains.

our method is applicable to any domain for which examples of the inputs to be summarized are available in context.


We also tried applying more complex transitions to − → x i like diagonal mapping (Trouillon et al., 2016), but did not observe improvements.

another option is to estimate this leads to poor alignment and performance drop because y t is not explicitly grounded on x i 3 .


Neural attention models (Bahdanau et al., 2015) with the seq2seq architecture (Sutskever et al., 2014) have achieved impressive results in text summarization tasks.

the attention vector comes from a weighted sum of source information and does not model the source-target alignment in a probabilistic sense.


Pointer generators are slightly better as it is trained to directly copy keyword from the source.

once it starts to enter the generation mode ("of british" in example 2 and "has been arrested" in example 3), the generation also loses control.


For this reason, our decomposition alone may not be very beneficial if coupled with standard attention.

our structured-attention model consistently performs much better than both baselines.


This kind of re-categorization has been shown to have considerable effects on the performance Guo and Lu, 2018).

one issue is that the precise set of re-categorization rules differs among different models, making it difficult to distinguish the performance improvement from model optimization or carefully designed rules.


In contrast, our method shows a strong capacity in capturing the main idea "the solution is about some patterns and a balance".

on the ordinary Smatch met- ric, their graph obtains a higher score (68% vs. 66%), which indicates that the ordinary Smatch is not a proper metric for evaluating the quality of capturing core semantics.


6 The reason is that the random order potentially produces a larger set of training pairs since each random order strategy can be considered as a different training pair.

the deterministic order stabilizes the maximum likelihood estimate training.


One prominent approach for data collection has been to automatically generate pseudo-language paired with logical forms, and paraphrase the pseudo-language to natural language through crowdsourcing (Wang et al., 2015).

this data collection procedure often leads to low performance on real data, due to a mismatch between the true distribution of examples and the distribution induced by the data collection procedure.


Knowledge Graphs (KGs) such as Freebase and DBpedia have shown their strong power in many natural language processing tasks including question answering  and dialog generation (Zhou et al., 2018).


MultiR (Hoffmann et al., 2011) and MIMLRE (Surdeanu et al., 2012) introduce multi-instance learning where the instances mentioning the same entity pair are processed at a bag level.

these methods rely heavily on handcrafted features.


Fortunately, the automatically constructed lexicon contains rich word boundaries information and word semantic information.

integrating lexical knowledge in Chinese NER tasks still faces challenges when it comes to self-matched lexical words as well as the nearest contextual lexical words.


Since without word boundaries information, it is intuitive to use character information 1 The code is available at https://github.com/ DianboWork/Graph4CNER only for Chinese NER (He and Wang, 2008;Liu et al., 2010;Li et al., 2014), although such methods could result in the disregard of word information.

word information is very useful in Chinese NER, because word boundaries are usually the same as named entity boundaries.


We observe significant improvements on distantly supervised datasets (i.e., KBP and NYT), with a up to 19% relative F1 improvement (Bi-GRU from 37.77% to 45.01% on KBP).

on the human-annotated corpus, the performance gain can be hardly noticed.


Many existing relation extraction (RE) models make decisions globally using integer linear programming (ILP).

it is nontrivial to make use of integer linear programming as a blackbox solver for RE.


Currently, research efforts have derived useful discrete features from dependency structures (Sasano and Kurohashi, 2008;Cucchiarelli and Velardi, 2001;Ling and Weld, 2012) or structural constraints (Jie et al., 2017) to help the NER task.

how to make good use of the rich relational information as well as complex long-distance interactions among words as conveyed by the complete dependency structures for improved NER remains a research question to be answered.


Empirically, we also found that those correctly retrieved entities of the DGLSTM-CRF (compared against the baseline) mostly correlate with the following dependency relations: "nn", "nsubj", "nummod".

dGLSTM-CRF achieves lower precisions against BiLSTM-CRF, which indicates that the dGLSTM-CRF model makes more false-positive predictions.


Note that, the bilingual data needed for this approach is coarsely taken from the same domain.

the texts need not be aligned beyond this coarse level.


We also experimented with the following neural network architectures for the classification: LSTM-RNN (Hochreiter and Schmidhuber, 1997), HAN (Yang et al., 2016), QRNN (Bradbury et al., 2017), and VDCNN (Conneau et al., 2017).

these models did not achieve any substantial performance gain to justify their additional complexity.


However, they are limited to short-text classification tasks.

our model effectively uses contextual information and combines recurrent and projection operations to achieve efficiency and enable learning more powerful neural networks that generalize well and can solve more complex language classification tasks.


Previous methods have proposed to overcome this by relying on character-level embeddings and other neural models like character-CNNs.

these methods are often complex and slow to compute for long text (e.g., convolution kernels on devices without significant computational capacity) and still require explicitly storing character or sub-word sequences.


In such challenging scenarios, recent studies have used meta-learning to simulate the few-shot task, in which new queries are compared to a small support set at the samplewise level.

this sample-wise comparison may be severely disturbed by the various expressions in the same class.


Such non-parametric models only need to learn the representation of the samples and the metric measure.

instances in the same class are interlinked and have their uniform fraction and their specific fractions.


We have a large labeled training set with a set of classes C train .

after training, our ultimate goal is to produce classifiers on the testing set with a disjoint set of new classes C test , for which only a small labeled support set will be available.


Style of a text is a very general notion that is hard to define in rigorous terms (Xu, 2017).

the style of a text can be characterized quantitatively (Hughes et al., 2012); stylized texts could be generated if a system is trained on a dataset of stylistically similar texts (Potash et al., 2015); and authorstyle could be learned end-to-end (Tikhonov and Yamshchikov, 2018b,c;.


Indeed, the output that copies input gives maximal BLEU yet clearly fails in terms of the style transfer.

a wholly rephrased sentence could provide a low BLEU between input and output but high accuracy.


Learning-based approaches alone, or combined with lexicon-based methods, usually produce state-of-the-art (SOTA) performance (Mudinas et al., 2012;Zhang et al., 2018), and thus are widely used nowadays.

learning-based methods usually demand a large amount of annotated data to train models, which has become one of the performance bottlenecks.


BERT uses a cross-encoder: Two sentences are passed to the transformer network and the target value is predicted.

this setup is unsuitable for various pair regression tasks due to too many possible combinations.


Training SBERT in the 10-fold cross-validation setup gives a performance that is nearly on-par with BERT.

in the cross-topic evaluation, we observe a performance drop of SBERT by about 7 points Spearman correlation.


(2018) for new tasks is the more suitable method, as it updates all layers of the BERT network.

sentEval can still give an impression on the quality of our sentence embeddings for various tasks.


Cosine-similarity treats all dimensions equally.

sentEval fits a logistic regression classifier to the sentence embeddings.


InferSent (Conneau et al., 2017) and Universal Sentence Encoder (Cer et al., 2018) both use (u, v, |u − v|, u * v) as input for a softmax classifier.

in our architecture, adding the element-wise u * v decreased the performance.


Theorem 1 shows that minimizing the corrupted risk R AUC-Corr (g) implies minimizing both the clean risk R AUC (g) and excessive terms.

since we only aim to minimize the clean risk, minimizing the corrupted risk may not effectively minimize the clean risk since excessive terms can be minimized instead and lead to overfitting.


Autoencoders have long been used for representation learning of images  and text (Li et al., 2015).

precisely reconstructing the clean input is probably too easy for high-capacity models.


A simple way to do so is putting a softmax classification layer on top of each encoder network.

we find that it is beneficial to directly perform classification in the k-means clustering space.


(2018) present a joint framework for deep multi-view clustering (DMJC) that is the closest work to ours.

dMJC only works with single-view inputs and the feature representations are learned using a multi-view fusion mechanism.


However, DMJC only works with single-view inputs and the feature representations are learned using a multi-view fusion mechanism.

aV-KMEaNS assumes that the inputs can be naturally partitioned into multiple views and carry out learning with the multi-view inputs directly.


In comparison to the semi-supervised neural neural models, CNN-RevGrad, Bi-LSTM-RevGrad, Adv-CNN, Adv-Bi-LSTM, SSL-VAE, which use the knowledge of a test set just like our model, we gain 8-11 pts in F1 score.

when evaluating on BioInfer dataset as a test set and AIMed as a training set, our model is in tie w.r.t.


Local explanation approaches focus on highlighting a handful of crucial features (Baehrens et al., 2010) or deriving simpler, more readable models from a complex one, e.g., a binary decision tree (Frosst and Hinton, 2017), or by local approximation with linear models (Ribeiro et al., 2016).

although they can explicitly show the representations learned in the specific hidden neurons (Frosst and Hinton, 2017), these approaches base their effectiveness on the user ability to study the quality of the reasoning and of the accountability as a side effect of the quality and the coherence of the features selection: this can be very hard in tasks where boundaries between classes are not well defined.


The MI not only measures how accurate the output agrees with its input, but also whether a meaningful latent variable can be learned in the latent space.

the estimation and maximization of the MI in the high-dimensional space are difficult.


It has been proposed in (Hoffman and Johnson, 2016) that decomposing the ELBO can produce an equivalent way to define the ELBO: where q φ (x, z) denotes the variational joint distribution induced by the posterior denotes the mutual information between x and z under q φ (x, z), and denotes the aggregated posterior (Makhzani et al., 2016).

the zero-forcing effect of the KL indicates if the latent variable z is a good representation of x, the mutual information between x and z should take a large value and the KL term in Equation (3) would be non-zero.


It is consistent with our motivation that the MI-regularized objective can alleviate the posterior collapse to improve VAEs' capability of generating fluent and diverse sentences with the effective use of the latent variable.

laggingVAE treats the MI as a stopping criterion for training the encoder RNN at the beginning of optimization.


Code and models are available at: https://github.com/drimpossible/Sampling-Bias-Active-Learning * indicates equal contribution † Work done at Verisk | AI Deep neural networks (DNNs) trained on large datasets provide state-of-the-art results on various NLP problems (Devlin et al., 2019) including text classification (Howard and Ruder, 2018).

the cost and time needed to get labeled data and to train models is a serious impediment to creating new and/or better models.


A recent empirical study (Siddhant and Lipton, 2018) investigating active learning in NLP suggests that Bayesian active learning outperforms classical uncertainty sampling across all settings.

the approaches have been limited to rel-atively small datasets.


Note that the current trend in deep learning is to train large models on very large datasets.

the aforementioned issues have not yet been investigated in the literature in such a setup.


We observe that across queries (∩Q), FTZ with entropy strategy has a balanced representation from all classes (high mean) with a high probability (low std) while Multinomial Naive Bayes (MNB) results in more biased queries (lower mean) with high probability (high std) as studied previously.

we did not find evidence of class bias in the resulting sample (∩S) in both models: Fast-Text and Naive Bayes (column 5 and 6 from Table  2).


A popular approach is to use an adversary to remove information about a target feature, often gender or ethnicity, from a model's internal representations (Edwards and Storkey, 2016;Kim et al., 2019).

the biases we consider are related to features that are essential to the overall task, so they cannot simply be ignored.