Our experiments indicate that adversarial attacks are not always the worst adversarial inputs, which can only be revealed via verification.

exhaustive verification is computationally very expensive.


Apparently this R is perfectly compact (only one word).

this rationale does not provide a valid explanation.


In this case, the two entities themselves suffice to serve as the rationales.

our model preserves the words like "working".


The corresponding objective function for MLE is derived from the Kullback-Leibler (KL) divergence between the empirical probability distribution representing the data and the parametric probability distribution output by the model.

the word frequency discrepancies in natural language make performance extremely uneven: while the perplexity is usually very low for frequent words, it is especially difficult to predict rare words.


These improvements have been obtained by explicitly incorporating in the model different ways of treating words according to their frequency.

learning is always made via (or approximating) Maximum Likelihood Estimation, which finds the distribution that maximizes entropy subject to the constraints given by training examples.


Applied to our prob-lem, the objective becomes: (2) In order to use the previously defined objectives, we need to compute the model probabilities p θ (y|x), which are usually obtained using a softmax function: applied on scores (or logits) s θ (x, y ) output by the model for every possible target token y ∈ Y.

as explained earlier, Y can be very large, hence computing all the scores and summing them is extremely slow.


It is interesting to note that the power transformations will here be applied on the posterior classification probabilities p C θ instead of categorical probabilities p θ .

the three divergences presentend in Section 3 are defined on positive measures: in theory, we can simply use the exp function on the scores s θ and do not need to normalize them: neither the α and β divergences are scale invariant (see right column of table 1 and Cichocki and Amari 2010).


This could be expected: intuitively, these values of α should make the model 'stretch' the probability mass.

as learning progresses, this phenomenon lessens, and the performance on rare words gets worse.


Besides, they are especially close across all values of γ: we can assume that this indicates that the corresponding Obj γ are 'closer' to the MLE objective.

tracking these values during training 13 shows that they all behave very similarly to perplexity.


word sequences alone (Kiperwasser and Goldberg, 2016;Dozat and Manning, 2016;Teng and Zhang, 2018), thereby allowing the output layer to make local predictions.

though explicitly capturing output label dependencies, CRF can be limited by its Markov assumptions, particularly when being used on top of neural encoders.


In NLP, label embeddings have been exploited for better text classification (Tang et al., 2015;Nam et al., 2016;.

relatively little work has been done investigating label embeddings for sequence labeling.


Capturing non-local dependencies between labels, these methods, however, are slower compared with CRF.

to these lines of work, our method is both asymptotically faster and empirically more accurate compared with neural CRF.


It gives an accuracy slightly higher than that of LSTM-softmax, which demonstrates the advantage of label embeddings.

it significantly underperforms BiLSTM-LAN (p-value<0.01), which shows the advantage of hierarchically-refined label distribution sequence encoding.


However, it predicts "with" incorrectly as "PP/NP" with the former supertag ending with "/PP".

biLSTM-LAN can capture potential long-term dependency and better determine the supertags based on global label information.


This suffices to compute interval bounds for feedforward networks and CNNs.

common NLP model components like LSTMs and attention also rely on softmax (for attention), element-wise multiplication (for LSTM gates), and dot product (for computing attention scores).


Across model architectures, we found that one epoch of certifiably robust training takes between 2× and 4× longer than one epoch of standard training.

iBP certificates are much faster to compute at test time than genetic attack accuracy.


Surprisingly, certifiably robust training nearly eliminated robustness errors in which the genetic attack had to change many words: the genetic attack either caused an error by changing a couple words, or was unable to trigger an error at all.

data augmentation is unable to cover the exponentially large space of perturbations that involve many words, so it does not prevent errors caused by changing many words.


Language model pre-training, such as BERT, has achieved remarkable results in many NLP tasks.

it is unclear why the pretraining-then-fine-tuning paradigm can improve performance and generalization capability across different tasks.


Recent work (Tenney et al., 2019b;Liu et al., 2019a;Goldberg, 2019;Tenney et al., 2019a) has shown that the pre-trained models can encode syntactic and semantic information of language.

it is unclear why pre-training * Contribution during internship at Microsoft Research.


In other words, the training loss of fine-tuning BERT tends to monotonously decrease along the optimization direction, which eases optimization and accelerates training convergence.

the path from random initial point to the end point is more rough, which requires a more carefully tweaked optimizer to obtain reasonable performance.


Second, existing approaches have focused on using language either as a standalone replacement for labeled data (Hancock et al., 2018), or to drive learning such as through specifying features for learning tasks (Eisenstein et al., 2009).

many realistic scenarios of learning from language would involve not learning from language alone, but learning from a mix of supervision, including both traditional labeled data, and natural language advice.


The advantage of using natural language as a medium is that it allows us to unify the different modes of supervision into a single, familiar user interface.

using natural language as a medium of supervision comes with its own set of challenges, as we discuss next.


In the E-step of the Posterior Regularization training (Ganchev et al., 2010), the computation of the posterior regularizer remains unchanged.

the M-step is modified so that the classifier parameters θ are learned using both the inferred labels for the unlabeled data, and provided labels for the labeled examples.


In learning scenarios where there is little labeled data, we would like to rely primarily on constraints specified from natural language explanations, and unlabeled data.

in scenarios where there is a lot of labeled data available enabling robust inductive inference, we would like to primarily rely on it rather than explanations.


Language models are traditionally evaluated with perplexity.

this measure suffers from several shortcomings, in particular strong dependence on vocabulary size and lack of ability to directly evaluate scores assigned to malformed sentences.


A better model is expected to give higher probability to sentences in the test set, that is, lower perplexity.

this measure is not always well aligned with the quality of a language model as it should be.


One method of obtaining sentence-sets is feeding acoustic waves into an ASR system and tracking the resulting lattices.

this requires access to the audio signals, as well as a trained system for the relevant languages.


Besides the source domain data, some methods utilize the target domain lexicons , unlabeled (Liu and Zhang, 2012) or partial-labeled target domain data  to boost the sequence labeling adaptation performance, which belong to unsupervised or semi-supervised domain adaptation.

we focus on supervised sequence labeling domain adaptation, where huge improvement can be achieved by utilizing only small-scale annotated data from the target domain.


There are also some methods (Guo et al., 2018;Kim et al., 2017;Zeng et al., 2018) explicitly weighting multiple source domain models for target samples in multi-source domain adaptation.

our work focuses on the supervised single source domain adaptation, which devote to implementing the knowledge fusion between the source domain and the target domain, not within multiple source domains.


From Figure 3(a), we can observe that model performance is improved when we add 20MB t data.

adding more clean back translated data starts to hurt the model performances.


Our goal is to learn a language model p θ based on sentences sampled from the training distribution  When x , classical statistical theory guarantees that a model trained via MLE performs well on the test distribution given sufficient data.

when p test x is not identical to p train x , MLE can perform poorly no matter how much data is observed.


incurs high losses) on rare sentences, we might initially try to define P as individual training examples to ensure low loss on all data points.

this is far too conservative, since the worst-case distribution would consist of exactly one data point.


In MLE, minimizing KL(p train is equivalent to minimizing the log loss because log p train x (x) can be treated as a constant.

in topic CVaR, the analogous baseline entropy term log p x|z (x | z) depends on z and thus is not a constant with respect to the outer supremum.


As we add data from ONEBWORD and α * decreases to 0.7, we find some positive transfer effects where the increased data from the ONEBWORD corpus improves the performance on Yelp.

as the fraction of nuisance data grows further and α * drops below 0.4 the MLE models suffer large increases in perplexity, incurring up to 10 additional points of perplexity.


Contextualized word embeddings such as ELMo and BERT provide a foundation for strong performance across a wide range of natural language processing tasks by pretraining on large corpora of unlabeled text.

the applicability of this approach is unknown when the target domain varies substantially from the pretraining corpus.


In either case, a primary benefit of contextualized word embeddings is that they seed the learner with distributional information from large unlabeled datasets.

the texts used to build pretrained contextualized word embedding models are drawn from a narrow set of domains: • Wikipedia in BERT (Devlin et al., 2019) and ULMFiT (Howard and Ruder, 2018); • Newstext (Chelba et al., 2013) in ELMo (Peters et al., 2018); • BooksCorpus (Zhu et al., 2015) in BERT (Devlin et al., 2019) and GPT (Radford et al., 2018).


This was shown to significantly reduce the transfer loss in Portuguese, and later in English (Yang and Eisenstein, 2016).

this approach relies on hand-crafted features, and does not benefit from contemporary neural pretraining architectures.


Upon inspection of the gathered labels, the high difficulty comes from the fact that there were many Turkers who labeled the data as neutral and also many who labeled it as contradiction.

an example that is easy for humans but difficult for the DNN models (Table  2, row 2) requires more abstract thinking than the earlier example.


On the other hand, the last example is very difficult for humans (row 4), possibly due to the relatively neutral text.

for the DNN models certain terms such as "stultifyingly contrived" may signal a more negative review and lead to the item being easier.


In particular, the internal uncertainty of the model is used as the basis for selecting how training examples are weighted.

model uncertainty depends upon the original training data the model was trained on, while here we use an external measure of uncertainty.


generate target sequences iteratively but require the target sequence length to be predicted at start.

our in-place edit model allows target sequence length to change with appends.


To tackle this problem, generative adversarial networks (GAN) with reinforcement learning (RL) training approaches have been introduced to text generation tasks Che et al., 2017;Lin et al., 2017;Shi et al., 2018;, where the discriminator is trained to distinguish real and generated text samples to provide reward signals for the generator, and the generator is optimized via policy gradient .

recent studies have shown that potential issues of training GANs on discrete data are more severe than exposure bias (Semeniuta1 et al., 2018;Caccia et al., 2018).


4a shows that for both FlowSeq and Transformer decoding is faster when using a larger batch size.

flowSeq has much larger gains in the decoding speed w.r.t.


Our observation is that conventional methods use a single representation for the input sentence, which makes it hard to apply prior knowledge of compositionality.

our approach leverages such knowledge with two representations, one generating attention maps, and the other mapping attended input words to output symbols.


On the other hand, "twice" contains only functional information, but no primitive information.

in general cases, each word may contain both primitive and functional information.


Due to its importance, pronoun resolution has seen a series of different approaches, such as rule-based systems (Lee et al., 2013) and end-to-end-trained neural models (Lee et al., 2017;Liu et al., 2019).

the recently released dataset GAP (Webster et al., 2018) shows that most of these solutions perform worse than naïve baselines when the answer cannot be deduced from the syntax.


In MASKEDWIKI, we looked for examples where masked nouns can be replaced with a pronoun, and only in 7 examples, we obtained a naturalsounding and grammatically correct sentence.

we estimated that 63% of the annotated examples in WIKICREM form a naturalsounding sentence when the appropriate pronoun is inserted, showing that WIKICREM consists of examples that are much closer to the target data.


These annotations can be found in Appendix A.

as shown in Section 6.2, training on WIKICREM alone does not match the performance of training on the data from the target distribution.


Additionally, the simplicity entailed by the vector space abstraction makes it an engineering-friendly representation, also explaining its widespread adoption and use (Freitas, 2015).

the latent features (dense vectors) at the center of most of the best-performing models have limited their application to two main uses: (i) computing semantic similarity and relat-edness measures and (ii) performing vocabulary generalization as an input layer on Machine Learning (ML) models.


For the triple (planet, moon, body) using immediate definitions of pivot and comparison incorrectly suggests that the triple (planet, moon, body) is discriminative.

after expanding using super-type definitions, the model correctly identifies body as a property of both planet and moon.


Pre-trained language models such as BERT have proven to be highly effective for natural language processing (NLP) tasks.

the high demand for computing resources in training such models hinders their application in practice.


Another direction is weight quantization (Gong et al., 2014;Polino et al., 2018), in which connection weights are constrained to a set of discrete values, allowing weights to be represented by fewer bits.

most of these pruning and quantization approaches perform on convolutional networks.


Using a weighted combination of ground-truth labels and soft predictions from the last layer of the teacher network, the student network can achieve comparable performance to the teacher model on the training set.

with the number of epochs increasing, the student model learned with this vanilla KD framework quickly reaches saturation on the test set (see Figure 2 in Section 4).


Ideally, we should pre-train BERT 6 [Large] and BERT 6 [Base] from scratch, and use the weights learned from the pretraining step for weight initialization in KD training.

due to computational limits of training BERT 6 from scratch, we only initialize the student model with the first six layers of BERT 12 or BERT 24 .


That is, we assume that the members of variational family Q are dimensional-wise independent, meaning that the posterior q can be writ- The simplicity of this form makes the estimation of ELBO very easy.

it also leads to a particular training difficulty called posterior collapse, where the KL divergence term becomes zero and the factorized variational posterior collapses to the prior.


The training PPL is monotonically decreasing in general.

when λ is small and the dependency relationships over latent codes are lost, the model quickly overfits, as the KL divergence quickly becomes zero and the validation loss starts to increase.


However, removing PE in the entire model greatly degrades the performance (from 34.71 to 14.47).

for SP, removing PE from our proposed attention variant dramatically degrades the performance (from 24.28 to 30.92).


The decoder self-attention is not an orderagnostic operation with respect to the order of inputs.

incorporating positional embedding into the attention mechanism may still improve performance.


Actually, the above process is based on the assumption that there is no dependency among the labels.

as shown in Figure 1, this assumption is hard to be satisfied in real-world datasets, and the complex label dependencies receive little attention in multi-label classification (Dembszynski et al., 2010;Dembczyński et al., 2012).


The training policies and prediction policies for all the labels can be viewed as a series of hyper-parameters.

to learn high-quality policies, one needs to specify both explicit and implicit label dependencies, which is not manually realistic.


After training, a single prediction policy (usually, a threshold of 0.5) is applied to all labels to generate the prediction.

as mentioned in Figure 1, these methods ignore the explicit and implicit label dependencies among the labels.


While disabling some heads improves the results, disabling the others hurts the results.

it is important to note that across all tasks and datasets, disabling some heads leads to an increase in performance.


Then we evaluate the word distributions between the generated documentd Y i and the ground-truth document d Y i , and use such signal to optimize the model.

straight-forward loss design, e.g., KL-divergence or Wasserstein distance between the word distributions ofd Y i and d Y i is not differentiable w.r.t.


However, the top 1 accuracy of the word translation is poor (< 20%), which makes it unable to select trustable sentence pairs using such a weak lexicon.

by leveraging our cross-lingual word embedding and unsupervised sentence representation, the selected sentences are much better (see more in Section 3.5).


(Fung and Cheung, 2004) use a small set of parallel corpora to initialize their EM lexical learning and further use this lexicon to iteratively mine sentence pairs.

in our unsupervised scenario, we have no bilingual sentence pairs to train such a model or lexicon to further select new sentence pairs.


For example, as you go from bottom to top layers, information about the past in left-to-right language models gets vanished and predictions about the future get formed.

for MLM, representations initially acquire information about the context around the token, partially forgetting the token identity and producing a more generalized token representation.


Second, both MT and MLM focus on a given token, as it either needs to be reconstructed or translated.

lM produces a representation needed for predicting the next token.


The languages were selected to represent different language families and morphological types, as we argue that fully unsupervised CLWEs have been designed to support exactly these setups.

we show that even the most robust unsupervised CLWE method (Artetxe et al., 2018b) still fails for a large number of language pairs: 87/210 BLI setups are unsuccessful, yielding (near-)zero BLI performance.


The superiority of weakly supervised methods (e.g., FULL+SL+SYM) over unsupervised methods is especially pronounced for distant and typologi-cally heterogeneous language pairs.

our study also indicates that even carefully engineered projection-based methods with some seed supervision yield lower absolute performance for such pairs.


A good transformation matrix W can make p G(vs) similar to p v t , so that D l can no longer distinguish between them.

this kind of similarity is at the distribution level.


Comparison with the state-of-the-art From the results shown in Table 3, we can see that in most cases, our method works better than previous supervised and unsupervised approaches.


our model are all related to "electronic device".

the translation candidates provided by MUSE are all related to "artillery", a different sense of the the word "battery" both in English and in Chinese, but not as common.


(2018) proposed a co-training algorithm to combine topological features and literal descriptions of entities.

combining these multi-aspect information of entities (i.e., topological connections, relations and attributes, as well as literal descriptions) remains under-explored.


Note that MAN propagates relation and attribute information through the graph structure.

for aligning a pair of entities, we observe that considering the relations and attributes of neighboring entities, besides their own ones, may introduce noise.


These figures prove that these two aspects of features are useful in making alignment decisions.

compared to MAN, HMAN shows more significant performance drops, which also demonstrates that employing the feedforward networks can better categorize relation and attribute features than GCNs in this scenario.


Since MAN propagates relation and attribute features via graph structures, it can still implicitly capture topological knowledge of entities even after we remove the topological features.

hMAN loses such structure knowledge when topological features are excluded, and thus its results are worse.


This suggests that our model can capture the characteristics of the source dataset via pretraining when using small supervision from language adaptation (i.e., small α).

pretraining introduces bias to the source space and the performance drops when larger weights are given to language adaptation; see the results with pretraining in Figure 3.


As Figure 5a shows, there is greater reduction in the classification loss for smaller values of α, i.e., when classification loss contributes more to the overall loss; see Equation (4).

figure 5b shows that the CSA loss decreases with larger values of α as the model pays more attention to the CSA loss; see the red and green lines in figure 5b.


Our method described so far does not rely on alignments from external statistical toolkits but performs self-training on alignments extracted from the layer average baseline.

gIZA++ provides a robust method to compute accurate alignments.


The improvement of the multi-task approach over the layer average baseline suggests that learning to translate helps produce better alignments as well.

still the multi-task approach falls short of the statistical and neural baselines, which have a strong advantage of having access to the full/partial target context.


Overall, given an input question, the layout policy learns what sub-tasks to perform, and the neural modules learn how to perform each individual sub-task.

since the the modular layout is sampled from the controller, the controller itself is not end-to-end differentiable and has to be optimized using Reinforcement Learning Algorithms like Reinforce (Williams, 1992).


The CompTreeNN and CompTreeNTN, while better, are not able to solve the task perfectly either.

it should be noted that the CompTreeNN outperforms our four standard neural models by ≈30% and the CompTreeNTN improves on this by another ≈10%.


If one could determine such rationales, new capabilities would become possible, including explanation, identifying alternative event orderings, and answering "what if..." questions.

this task is challenging as it requires knowledge of the preconditions and effects of actions, typically unstated in the text itself.


On first sight, this might seem like a simple solution to the dependency prediction task.

as we show later, this approach does not produce good results.


This includes datasets that test a system's ability to extract factual answers from text (Rajpurkar et al., 2016;Nguyen et al., 2016;Trischler et al., 2016;Mostafazadeh et al., 2016;Su et al., 2016), as well as datasets that emphasize commonsense inference, such as entailment between sentences (Bowman et al., 2015;Williams et al., 2018).

there are growing concerns regarding the ability of NLU systems-and neural networks more generally-to generalize in a systematic and robust way (Bahdanau et al., 2019;Lake and Baroni, 2018;Johnson et al., 2017).


Our work is also related to the domain of question answering and reasoning in knowledge graphs (Das et al., 2018;Xiong et al., 2018;Hamilton et al., 2018;Xiong et al., 2017;Welbl et al., 2018;Kartsaklis et al., 2018), where either the model is provided with a knowledge graph to perform inference over or where the model must infer a knowledge graph from the text itself.

unlike previous benchmarks in this domain-which are generally transductive and focus on leveraging and extracting knowledge graphs as a source of background knowledge about a fixed set of entities-CLUTRR requires inductive logical reasoning, where every example requires reasoning over a new set of previously unseen entities.


We found that time-constrained AMT annotators performed well (i.e., > 70%) accuracy for k ≤ 3 but struggled with examples involving longer stories, achieving 40-50% accuracy for k > 3.

trained annotators with unlimited time were able to solve 100% of the examples (Appendix 1.7), highlighting the fact that this task requires attention and involved reasoning, even for humans.


Even though this content is not strictly useful or needed for the reasoning task, it may provide some linguistic cues (e.g., about entity genders) that the models exploit.

the BERT-based models do not benefit from the inclusion of this extra content, which is perhaps due to the fact that they are already built upon a strong language model (e.g., that already adequately captures entity genders.)


The accuracy of machine learned speech recognizers (Hinton et al., 2012) and speech synthesizers (van den Oord et al., 2016) are good enough to be deployed in real-world products and this progress has been driven by publicly available labeled datasets.

conspicuously absent from this list is equal progress in machine learned conversational natural language understanding (NLU) and generation (NLG).


Since the ultimate goal is to be able to handle complex human language behaviors, it would seem that human-human conversational data is the better choice for spoken dialog system development (Budzianowski et al., 2018).

learning from purely human-human based corpora presents challenges of its own.


MultiWOZ is an entirely written corpus and uses crowdsourced workers for both assistant and user roles.

taskmaster-1 has roughly 13,000 dialogs spanning six domains and annotated with API arguments.


The need for high-quality, large-scale, goaloriented dialogue datasets continues to grow as virtual assistants become increasingly widespread.

publicly available datasets useful for this area are limited either in their size, linguistic diversity, domain coverage, or annotation granularity.


This creates a disparate conversation: agents are incentivized to operate within a set procedure and convey a patient and professional tone.


Commonly, datasets provide annotations at the turn level (Budzianowski et al., 2018;Asri et al., 2017;Mihail et al., 2017).

turn level annotations can introduce confusion for IC datasets, given multiple intents may be present in different sentences of a single turn.


(2018) builds a seq2seq neural network model for short texts to identify and recover ellipsis.

these methods are still limited to short texts or one-shot dialogues.


To handle this problem, memory networks are usually a great choice and a promising way.

existing memory networks do not perform well when leveraging heterogeneous information from different sources.


Their models mostly try to use the attention mechanism, including memory networks techniques, to fetch the most similar knowledge (Sukhbaatar et al., 2015), then incorporate grounding knowledge into a seq2seq neural model to generate a suitable re-sponse (Madotto et al., 2018).

existing memory networks equally treat information from multiple sources, e.g.


Comparing the generated sentences by humans, although entities and sentences are different with gold answer in example one, our approach is able to produce more fluent and accurate sentences.

the result on task weather forecasting neither HMNs and Mem2Seq can outperform SEQ2SEQ.


Due to their inherent capability in semantic alignment of aspects and their context words, attention mechanism and Convolutional Neural Networks (CNNs) are widely applied for aspect-based sentiment classification.

these models lack a mechanism to account for relevant syntactical constraints and long-range word dependencies, and hence may mistakenly recognize syntactically irrelevant contextual words as clues for judging aspect sentiment.


These words happen to exhibit similar patterns of how aspect words occur on sentence level (Lin and He, 2009).

the above methods, only exploiting global context, are arguably suboptimal for largely ignoring the rich information delivered by local context.


This is because the DMI can infer relational representations that capture some common latent relations between aspect and opinion words.

it still cannot completely capture the multi-word aspect terms (e.g.


Multi-task learning can learn better hidden states of the sentence, and better aspect embeddings.


Table 1 shows examples of aspects and five of their corresponding seed words from our experimental datasets (described later in more detail).

to a classification label, which is only relevant for a single segment, a seed word can implicitly provide aspect supervision to potentially many segments.


its effect on conversation recommendation has not been explored yet, and our work aims to fill the gap.


We can observe that Reddit dataset contains more conversations, with a higher average number of conversations per user.

twitter conversations are longer, with fewer participants.


Recently, neural networks based on multitask learning have achieved promising performance on fake news detection, which focus on learning shared features among tasks as complementary features to serve different tasks.

in most of the existing approaches, the shared features are completely assigned to different tasks without selection, which may lead to some useless and even adverse features integrated into specific tasks.


private model to alleviate the shared and private latent feature spaces from interfering with each other.

these models transmit all shared features in the shared layer to related tasks without distillation, which disturb specific tasks due to some useless and even harmful shared features.


Our assumption is that the valence of a concept should correlate with its perceived moral polarity, e.g., morally repulsive ideas should evoke an unpleasant feeling.

we do not expect this correspondence to be perfect; for example, the concept of dessert evokes a pleasant reaction without being morally relevant.


Predictions for democracy show a trend toward morally positive sentiment, consistent with the adoption of democratic regimes in Western societies.

predictions for slavery trend down and suggest a drop around the 1860s, coinciding with the American Civil War.


Garcia et al., 1997;Cofield et al., 2010;Lazarus et al., 2015;Chiu et al., 2017).

such manual approaches often require an exhaustive list of linguistic cues, which costs a significant amount of human effort and its generalizability may be limited by the small sample size.


association, associated with, predictor, at high risk of The statement shows that one variable directly changes the other.

the relation carries an element of doubt in it, which is normally via hedges or modalities.


Because some languages are used in multiple countries and some countries did not publish many papers, not all results in Table 6 and Table 7 are comparable.

we were able to identify four country-language pairs that correspond well to each other: Germany-German, Italy-Italian, France-French, and China-Chinese.


In a parallel development, researchers have recently started to view fact checking as a task that can be partially automated, using machine learning and NLP to automatically predict the veracity of claims.

existing efforts either use small datasets consisting of naturally occurring claims (e.g.


Contributions: The latter category above is the most related to our paper as we consider evidence documents.

existing models are not trained jointly for evidence identification, or for stance and veracity prediction, but rather employ a pipeline approach.


Further, Node2Vec (Grover and Leskovec, 2016) proposed a biased random walk to search a network and generate node sequences based on the depth-first search and width-first search.

those methods only embed the structure information into vector representations, while ignoring the informative textual contents associated with vertices.


Finally, we use N (v i ) to refer to v i 's neighbours, i.e., all nodes which are directly connected to v i .

1 to most existing models, where user representations are static, our model uses an encoder which takes as input a pre-computed user vector, performs a dynamic exploration of its neighbours in the social graph, and updates the user representation given the relevance of its connections for a target task.


Our work is motivated by the goal of assisting human moderators of online communities by preemptively signaling at-risk conversations that might deserve their attention.

we caution that any automated systems might encode or even amplify the biases existing in the training data (Park et al., 2018;Sap et al., 2019;Wiegand et al., 2019), so a public-facing implementation would need to be exhaustively scrutinized for such biases (Feldman et al., 2015).


As expected, the Seq2Seq model achieves higher scores with filtered conversation as input.


As shown in Table 3, applying Reinforcement Learning does not lead to higher scores on the three automatic metrics.

human evaluation (Table 4) shows that the RL model creates responses that are potentially better at mitigating hate speech and are more diverse, which is consistent with .


for all the testing Gab data can achieve 4.2 on BLEU, 20.4 on ROUGE, and 18.2 on METEOR.

this response is not considered as high-quality because it is almost a universal response to all the hate speech, regardless of the context and topic.


The sampled results in Figure 4 show that the Seq2Seq and the RL model can generate reasonable responses for intervention.

as is to be expected with machine-generated text, in the other human evaluation we conducted, where Mechanical Turk workers were also presented with sampled human-written responses alongside the machine generated responses, the human-written responses were chosen as the most effective and diverse option a majority of the time (70% or more) for both datasets.


(2018) defined this term when they found that the leaveone(token)-out (LOO) method might find a completely different set of most influential words if the most un-influential word is dropped from the sentence.

we do not observe this problem in our case: out of 419 predictions, 379 times the most influential word remains the most influential and 33 times it becomes the second most influential one after we drop the most un-influential words.


2) we cannot guarantee that the resulting text is still a natural input.

these challenges do not play much of a role in our domain.


This post fits the aggression label, by challenging authority and insulting, and the loss label, by evoking incarceration.

a loss label is more suitable, because: 1) though the anger seems to target specific players, the text as a whole reveals that the poster is highlighting institutional violence; and 2) the sad frown emoji, paired with the context that one of the poster's loved ones was recently arrested on a drug charge ("12z" refers to law enforcement's drug unit), cements that the user is grieving in this post.


It is convenient for computer scientists to regard the labels as ground truth, the dataset as static and try different models to improve the f-score.

as domain experts gained a deeper understanding about the twitter users, they marked 12% of the aggression labels as controversial (section 3); the annotation may evolve over time.


We are also grateful to our anonymous reviewers for their insightful and constructive feedback.

to many decades of research on oral code-switching, the study of written multilingual productions has only recently enjoyed a surge of interest.


Recent research from the sociolinguistic perspective has considered CS in an array of online genres: e.g., English-Jamaican Creole in email (Hinrichs, 2006), Malay-English language in online discussion forums (McLellan, 2005), English-Spanish bilingual blogging (Montes-Alcalá, 2007), and English-Chinese instant messaging (Lam, 2009).

these studies have analyzed productions of a small number of authors (even a single person), and each is restricted to a single language-pair.


A recently proposed zero-shot intent classification method, IntentCapsNet, has been shown to achieve state-of-the-art performance.

it has two unaddressed limitations: (1) it cannot deal with polysemy when extracting semantic capsules; (2) it hardly recognizes the utterances of unseen intents in the generalized zero-shot intent classification setting.


To improve business effectiveness and user satisfaction, accurately identifying the intents behind user ut-terances is indispensable.

it is extremely challenging not only because user queries are sometimes short and expressed diversely, but also because it may continuously encounter new or unacquainted intents popped up quickly from various domains.


(2018) extend capsule networks for zero-shot intent classification by transferring the prediction vectors from seen classes to unseen classes.

there are some key issues left to be resolved, including how to deal with polysemy in word embeddings and how to improve the model generalization ability to unseen intents in the generalized zero-shot intent classification setting.


A simple transfer strategy is to share the whole matrix in both target and source domains.

such a method will make it less flexible to capture domain-specific match information, since the local match information is likely to be varied across domains.


With the development of NLP technologies, news can be automatically categorized and labeled according to a variety of characteristics, at the same time be represented as low dimensional embeddings.

it lacks a systematic approach that effectively integrates the inherited features and inter-textual knowledge of news to represent the collective information with a dense vector.


Nonetheless, News2vec achieves state-of-art performance after sequence fine-tuning.

the result of Paragraph Vector cannot be further improved by fine-tuning.


Building on this line of work, Zhang and Lapata (2017) combine a novel sequence-to-sequence encoder-decoder model with a deep reinforcement learning framework that rewards the system for providing simple, fluent output, similar in meaning to the input.

one of the main challenges for such comprehensive end-to-end systems is the ability to address specific types of errors independently.


To further assess generalisability of the model, we test it on CEFR-LS, as well as BENCHLS for consistency (see Table 2).

as the words in BENCHLS dataset were selected at random rather than according to their complexity (Paetzold and Specia, 2016a), the results as expected are lower.


Previous work has framed this task as ranking words according to simplicity (Paetzold and Specia, 2016a).

a system that is only aimed at selecting the simplest candidate may return a substitution that does not fit the surrounding context, is ungrammatical or not semantically equivalent to the original.


Using the MedLit data in the RNN provides a significant improvement (65% vs 64%).

there is no differ-  ence in the BERT models, likely because of the a large amount of i2b2 training data.


We found that running the MedLit + TR i2b2 experiment with just 25% of the i2b2 dataset with the RNN model performs as well (no significant difference) as training on the full i2b2 dataset (Figure 4), and 12% better than the model using 25% of the i2b2 data alone.

when using the BERT model, MedLit + TR i2b2 only performs better than i2b2 alone if less than 25% of the training data is available.


These methods rely on the news click histories to model users' news reading interest.

these methods usually suffer from the data sparsity problem, since the news click behaviors of many users on news platforms are very limited, making it difficult for these methods to learn accurate representations of these users.


1, since the user posts the search query "toyota crown", and browses the webpage of "Pros and Cons of Toyota" for detailed information, she may have high interest in Toyota cars and is likely to read news articles related to Toyota cars.

this user may click only a few news articles, and her interest in toyota cars cannot be mined from the news click history.


(2014) proposed a Top-icMF method to represent users and items by incorporate the topics extracted from review texts to enhance the learning of latent user and item factors from the rating matrix via non-negative matrix factorization (NMF).

these methods only extract topics from reviews, and cannot effectively utilize the contexts and word orders in reviews, both of which are important for learning accurate user and item representations.


In this bipartite graph, users and items are represented as nodes, and the interactions of user-item pairs are regarded as edges.

different from typical graph neural networks (Veličković et al., 2017), the input of typical review-based recommender systems is only a user-item pair (two nodes with an edge) rather than the entire user-item graph.


The models capture multiplicative interactions between these elements and are thus able to make large shifts in event semantics with only small changes to the arguments.

previous work mainly focuses on the nature of the event and lose sight of external commonsense knowledge, such as the intent and sentiment of event participants.


Spelling correction (Mays et al., 1991;Islam and Inkpen, 2009) and grammar error correction (Sakaguchi et al., 2017) are useful tools which can block editorial adversarial attacks, such as swap and insertion.

they cannot handle cases where word-level attacks that do not cause spelling and grammar errors.


These models are node-based, i.e., they form pair representations based solely on the two target node representations.

entity relations can be better expressed through unique edge representations formed as paths between nodes.


They typically perform on the nodes by updating the representations during training.

a relation between two entities depends on different contexts.


Among three semi-supervised models, CPLS model achieves the greatest improvement, verifying the effectiveness of the projection functions.

the gain of CPLS model in the aspect of style accuracy is not that significant.


This example demonstrates that user comments closely relate to the intent of an edit, the actual edit operation, and the content and location in the document that underwent change.

during collaborative document authoring, the tracking and maintenance of comments becomes increasingly more challenging due to the large number of edits and comments that authors make.


(2010) consider both comment and document revision for lexical simplification.

they use comments as meta-data to identify trusted revisions, rather than directly modeling the relationship between comments and edits.


The aforementioned prior work mostly focuses on building the best neural network model independent of any model size or memory constrains.

recent work by Kozareva, 2018, 2019) show the importance of building ondevice text classification models that can preserve user privacy, provide consistent user experience and most importantly are compact in size, while yet achieving state-of-art results.


The BPE algorithm is deterministic because it segments greedily from left to right.

sR can sample multiple segmentations stochastically.


We can extract a set of top B(≥ N ) candidates using beam search with beam size B, namely in the descending order of likelihood.

in the case of deterministic segmentation, in the case of stochastic segmentation, same query q i , q j (i = j) with different token sequence t i , t j may exist.


By following (Kudo, 2018), we can find the most likely segmentation sequence t starting from all of the n-best segmentations t 1 , • • • ,t n of S(p) rather than from onlyt 1 .

we observe that this n-best decoding performs worse than one-best decoding.


On the other hand, BPE is poor when the query length (or prefix length) is short.

for a longer case, its MRR is almost close to that of the character baseline.


Because of controllability motivations, these axes and labels are generally known a priori.

counterfactual rewriting focuses on the causes and effects of a story, dimensions that can require more complex and diverse, yet potentially subtle, changes to accommodate the counterfactual event.


Neural question generation (NQG) has been extensively studied because of its broad application in question answering (QA) systems (Tang et al., 2018;Duan et al., 2017), reading comprehension (Kim et al., 2019;, and visual question answering (Fan et al., 2018).

the research of open-domain conversational question generation (CQG) Hu et al., 2018) is still in its infancy.


News titles are short and succinct, and thus only using news titles may lose quite a lot of useful information in comment generation.

a news article and a comment is not a pair of parallel text.


On the other hands, VAE-based models with the input of sequences usually adopt RNNs as encoders or decoders.

latent variable collapse (i.e., the latent variable collapses to obey the distribution of the prior) makes the sequential VAE to generate texts without information of latent code so that the latent code cannot have enough information of text features.


Taking TGVAE  as an example, it guides the generation of the VAE latent code by topic distribution.

it does not separate the semantic and structure latent codes explicitly.


Similar to existing topic modeling methods, we treat bag-of-words representations of texts as input.

since the encoder in the topic modeling component is expected to be as a discriminator and guarantee that texts generated by the sequence modeling decoder have specific topic information, the encoder cannot be with the discrete representations (e.g., one-hot representation) of texts as input.


This is a denoising preprocessing step that makes the model more reliable.

to make the execution of the discriminator applicable, our model takes all of the words in the document as input while outputting the words in a smaller vocabulary as a topic model does.


In this paper, we formulate phrase grounding as a sequence labeling task where we treat candidate regions as potential labels, and use neural chain Conditional Random Fields (CRFs) to model dependencies among regions for adjacent mentions.

to standard sequence labeling tasks, the phrase grounding task is defined such that there may be multiple correct candidate regions.


The success of these structured prediction methods shows the advantage of considering entity dependencies in learning and prediction.

these approaches capture certain relations in an ad hoc manner, and resort to approximate inference (Wang et al., 2016; or non-differentiable losses .


In recent years, video semantic comprehension has attracted much research attention, with a number of datasets and tasks being proposed such as activity recognition (Xu et al., 2017), dense video captioning (Krishna et al., 2017a), visual question answering (Lei et al., 2018(Lei et al., , 2019 etc.

most works are limited to only capturing coarse semantic information such as action recognition in broad categories.


All currently published models for addressing this problem can be categorized into two types: (i) top-down approach: it does classification and regression for a set of pre-cut video segment candidates; (ii) bottom-up approach: it directly predicts probabilities for each video frame as the temporal boundaries (i.e., start and end time point).

both two approaches suffer several limitations: the former is computationintensive for densely placed candidates, while the latter has trailed the performance of the top-down counterpart thus far.


These models suffer the same drawbacks as above mentioned in the top-down approach for NLVL.

with the advent of the first performance comparable bottom-up object detector: CornerNet (Law and Deng, 2018), the bottom-up approaches begin to gain unprecedented attention (Zhou et al., 2019b,a;Duan et al., 2019;, which inspires us to explore a decent bottom-up framework for NLVL.


In principle, a larger t usually gives us a more stable mistake estimation.

a larger t also requires more computation resources.


And (2) "Team A at Team B" is a way to say "Team A" as an away team playing with Team B as a home team.

in almost all cases (only 1 exception out of more than 100), "Team A" was labelled as ORG while "Team B" was labelled as LOC.


(2014) further applied previous detection models and manually corrected Icelandic Frequency Dictionary (Pind et al., 1991) POS tagging dataset.

these two methods are specifically developed for POS tagging and cannot be directly applied to NER.


Most state-of-the-art models for named entity recognition (NER) rely on the availability of large amounts of labeled data, making them challenging to extend to new, lowerresourced languages.

there are now several proposed approaches involving either cross-lingual transfer learning, which learns from other highly resourced languages, or active learning, which efficiently selects effective training data based on model predictions.


The variations on the second and third keyphrases are larger.

we found the variations are more about which keyphrases they choose to enter, not about whether a phrase is a keyphrase or not.


Knowledge graphs are structured representations of real world facts.

they typically contain only a small subset of all possible facts.


DistMult (Yang et al., 2015) is a special case of RESCAL with a diagonal matrix per relation, which reduces overfitting.

the linear transformation performed on entity embedding vectors in DistMult is limited to a stretch.


First, the chosen input texts must be classified into the same class by both models so the humans make decisions based only on the different explanations.

it is worth to consider both the cases where both models correctly classify and where they misclassify.


If they select the right model with high confidence, the explanation method will get a higher positive score.

a confident but incorrect answer results in a large negative score.


Each of the thresholds is set subject to sufficient purity of the classification results above it.

their method is applicable to CNNs with only one linear layer as F C, while our CNNs have an additional hidden layer (with ReLU activation).


Hence, they can help humans identify the poor model to some extent.

there was no clear winner when both CNNs predicted wrongly.


So, the evidence n-grams are more likely related to the predicted class than the other classes.

they may be less relevant than LIME's as the evidence is generated from a global explanation of the model (DTs).


First, LRP (N) generates good evidence for correct predictions, so it can gain high scores in the columns.

the evidence for incorrect predictions () is usually not convincing, so the counterevidence (which is likely to be the evidence of the correct class) can attract humans' attention.


(2012) proposed the BioNERDS, which is a NER system for detecting database and software names in scientific literature using extraction rules generated by examining 30 bioinformatics articles.

there has been no existing framework for modeling the resource role and function at such a fine-grained level in general domain scientific text.


Data-independent methods employ random projections to construct hash functions without any consideration on data characteristics, like the locality sensitive hashing (LSH) algorithm (Datar et al., 2004).

data dependent hashing seeks to learn a hash function from the given training data in a supervised or an unsupervised way.


To increase the modeling ability of (1), we may resort to more complex likelihood p θ (D|z), such as using deep neural networks to relate the latent z to the observation x i , instead of the simple softmax function in (2).

as indicated in , employing expressive nonlinear decoders likely destroy the distance-keeping property, which is essential to yield good hashing codes.


With reductions like the above one, any ANNS methods can be applied for MIPS.

it was shown that there are performance limitations for the reduction MIPS methods (Morozov and Babenko, 2018).


We found that the edge selection method is vital for the trade-off of effectiveness and efficiency in searching.

the existing edge selection techniques used in HNSW and ip-NSW are actually designed for metric distances, which are inapplicable for the non-metric measure, e.g., inner product.


On the other hand, the information captured by word embeddings can be seamlessly used in downstream tasks, which makes embedding a potential solution for the aforementioned problem.

conventional word embeddings using one unified embedding for each word are not able to distinguish different relations types (such as various syntactic relations, which is crucial for SP) among words.


To address this problem, the dependency-based embedding model (Levy and Goldberg, 2014) is proposed to treat a word through separate ones, e.g., 'food@dobj' and 'food@nsubj', under different syntactic relations, with the skip-gram (Mikolov et al., 2013) model being used to train the final embeddings.


In particular, quantization (Lin et al., 2016;Hubara et al., 2016) and groupwise low rank approximation (Chen et al., 2018a) achieve the state-of-the-art performance on tasks like language modeling and machine translation.

these methods are rather computationally inspired than being motivated by intuitions of the semantic composition and thus suffer from poor interpretability.


Notice that some previous methods compress model directly during training phase (Khrulkov et al., 2019;Wen et al., 2017).

our problem setup follows (Chen et al., 2018a;Shu and Nakayama, 2017;Chen et al., 2018b) that given a pre-trained model, we want to compress the model with limited fine-tuning.


This implies that local precision lost for low-rank basis in GroupReduce is more difficult to be recovered.

the collective information of MulCode due to the compositional property is more robust when imprecise local vectors present.


Recent studies have shown that word embeddings exhibit gender bias inherited from the training corpora.

most studies to date have focused on quantifying and mitigating such bias only in English.


This is due to morphological agreement and should not be considered as a stereotype.

gender bias in the embeddings of languages with grammatical gender indeed exists.


In gendered languages, all nouns are assigned a gender class.

inanimate objects (e.g., water and spoon) do not carry the meaning of male or female.


when analyzing languages like Spanish and French, considering the second is necessary.


To simplify the discussion, we focus on noun class systems where two major gender classes are feminine and masculine.

the proposed approach can be generalized to languages with multiple gender classes (e.g., German).


A few recent studies focus on measuring and reducing gender bias in contextualized word embeddings (Zhao et al., 2019;May et al., 2019;Basta et al., 2019).

they only focus on English embeddings in which the gender is mostly only expressed by pronouns (Stahlberg et al., 2007).


In English, these paths are extracted from sentences where x and y co-occur.

when x and y are in different languages, a new path definition is required.


Supervised English system Without crosslingual training samples, we cannot compare weakly supervised and fully supervised training for BILEXNET in a controlled fashion.

the supervised monolingual ENLEXNET model (Section 6) evaluated on the En-En test set offers a reference point: remarkably the F1 scores of BILEXNET are only 1 to 3 points lower than those obtained by the supervised English model (∼44 on the En-En test set).


This is similar to the the Co-  gALex shared task , where the first part of the task is to eliminate completely unrelated pairs, before predicting relations on the remaining pairs.

filtering out unrelated pairs is an easier task than filtering pairs in the Other category.


For example, given the monolingual example (drop, fall, Forward Entail) the model places the highest weight on the Hindi word , which captures the "moving downward" sense.

for the example (autumn, fall, Equivalence), the model correctly identifies Ú as the right translation.


Neural WSD systems (Kågebäck and Salomonsson, 2016;Raganato et al., 2017b) feed the continuous word representations into a neural network that captures the whole sentence and the word representation in the sentence.

in both approaches, the word representations are independent of the context.


(2017b) proposed a self-attention layer on top of the concatenated bidirectional LSTM hidden states for WSD and introduced multi-task learning with part-ofspeech tagging and semantic labeling as auxiliary tasks.

on average across the test sets, their approach did not outperform SVM with word embedding features.


This is surprising as the embedding directly provides a number's value, thus, the synthetic tasks should be easy to solve.

we had difficulty training models for large ranges, even when using numerous architecture variants (e.g., tiny networks with 10 hidden units and tanh activations) and hyperparameters.


Denoting the predicted restated query byz, simplifying E (x,y,z)∼D as E, the goal of the RL training is to maximize following objective: where Z is the space of all restated query candidates and r represents the reward defined by comparingz and the annotation z.

the overall candidate space Q×Z is vast, making it impossible to exactly maximize L rl .


Guided by the golden restated query z, in training, we find outz * by computing the reward of each candidate.

in inference, where there is no golden restate query, we can only obtainz * from F. Specially, for the v-th span in the followup query, we find u * = arg max u F u,v .


(2016) requires annotations on which phrases should be mapped to unknowns during the training phase.

such supervised knowledge is not required for our method.


As the interaction proceeds, the user question becomes more complicated as it requires longer SQL query to answer.

more query tokens overlap with the previous query, and thus the number of new tokens remains small at the third turn and beyond.


A series of pruning methods are then proposed to alleviate the imbalanced distribution, such as the korder pruning .

it does not extend well to other languages, and even hinders the syntax-agnostic SRL model as  has experimented with different k values on English.


Additionally, the use of CNN layers allows us to explicitly control the window size for phrase modeling, which has been shown to be critical for relevance matching (Dai et al., 2018;Rao et al., 2019).

the contextual encoder enables us to obtain long-distance contextual representations for each token.


Indeed, semantic matching methods alone are ineffective when queries are comprised of only a few keywords, without much semantic information to exploit.

the   context-aware representations learned from SM do contribute to RM, leading to the superior results of our complete HCAN model.


As is well known, the hard parameter sharing approach can provide representations for all the shared tasks and reduce the probability of overfitting on the main task.

this kind of sharing strategy somewhat weakens the representation framework maintains distinct model parameters for each task, due to the neutralization of knowledge introduced by the auxiliary task.


7 Note that these sentential and phrasal paraphrases are obtained by automatic methods.

dataset creation for downstream tasks generally requires expensive human annotation.


A recent task by  on the CONCODE dataset maps a single utterance to an entire method, conditioned on environment variables and methods.

we tackle the task of general purpose code generation in an interactive setting, using an entire sequence of prior NL and code blocks as context.


1 Natural language interfaces that allow users to query data and invoke services without programming have been identified as a key application of semantic parsing (Berant et al., 2013;Thomason et al., 2015;Dong and Lapata, 2016;Zhong et al., 2017;Campagna et al., 2017;Su et al., 2017).

existing semantic parsing technologies often fall short when deployed in practice, facing several challenges: (1) user utterances can be inherently ambiguous or vague, making it difficult to get the correct result in one shot, (2) the accuracy of state-of-the-art semantic parsers are still not high enough for real use, and (3) it is hard for users to validate the semantic parsing results, especially with mainstream neural network models that are known for the lack of interpretability.


Similarly, (Yao et al., 2019) relies on a pre-defined two-level hierarchy among components in an If-Then program and cannot generalize to formal languages with a deeper structure.

mISP aims for a general design principle by explicitly identifying and decoupling important components, such as error detector, question generator and world model.


For SQLNet, we also compare our system with the reported performance of DialSQL (Gur et al., 2018, Table 4).

since DialSQL is not open-sourced and it is not easy to reproduce it, we are unable to adapt it to SQLova for more comparisons.


We first observe that, via interactions with simulated users, MISP-SQL improves SyntaxSQL-Net by 10% accuracy with reasonably 3 questions per query.

we also realize that, unlike on WikiSQL, in this setting, the probabilitybased error detector requires more questions than the Bayesian uncertainty-based detector.


In all settings, MISP-SQL improves the base parser's performance, demonstrating the benefit of involving human interaction.

we also notice that the gain is not as large as in simulation, especially on SQLova.


Most human evaluation studies for (interactive) semantic parsers so far (Chaurasia and Mooney, 2017;Gur et al., 2018;Yao et al., 2019) use pre-existing test questions (e.g., from datasets like WikiSQL).

this introduces an undesired discrepancy, that is, human evaluators may not necessarily be able to understand the true intent of the given questions in an faithful way, especially when the question is ambiguous, vague, or containing unfamiliar entities.


In Example (1), though our baseline recovers a propositional phrase for the noun staff and another one for the noun funding, it fails to recognize the anaphora and antecedent relation between the two propositional phrases.

our approach successfully recognizes :prep-for c as a reentrancy node and generates one propositional phrase shared by both nouns staff and funding.


(2019) utilizes additional in-domain datasets to post-train BERT's weights and then fine-tune it on this task.

such a method requires a large corpus for post-training and the fine-tuning also takes a lot of computation resources and time.


The main challenge comes from multi-aspect sentences, which express multiple sentiment polarities towards different targets, resulting in overlapped feature representation.

most existing neural models tend to utilize static pooling operation or attention mechanism to identify sentimental words, which therefore insufficient for dealing with overlapped features.


Traditional approaches have designed rich features about content and syntactic structures to capture the sentiment polarity (Jiang et al., 2011;Pérez-Rosas et al., 2012).

these featurebased methods are labor-intensive and the performance highly depends on the quality of the features.


Very recently, CNN-based models have shown the strengths in efficiency to tackle the aspect-level sentiment classification (Xue and Li, 2018;Huang and Carley, 2018;.

all the previous methods utilize static pooling operation or attention mechanism to locate the sentimental words, which fails to handle the overlapped features.


Different from previous researches, we explore the dependence among relevant posts via the authors' backgrounds, since the authors with similar backgrounds, e.g., gender, location, tend to express similar emotions.

such personal attributes are not easy to obtain in most social media websites, and it is hard to capture attributesaware words to connect similar people.


On one hand, most websites may not contain useful personal information.

people are normally not willing to attach their personal information in social media.


Recently, with the development of artificial intelligence, neural network models have been successfully applied to various NLP tasks (Collobert et al., 2011;Goldberg, 2016).

few works use neural network models for emotion detection.


Most of previous studies consider each post individually in emotion detection, one of the most important tasks in sentiment analysis.

since the posts in social media are generated by users, it is natural that these posts can be connected through authors' personal background attributes.


For instance, all baselines miss proper noun "wendys" in the first example.

our approach increases the specificity of the output, leading to better content preservation.


With the development of deep learning, neural networks have obtained state-ofthe-art results on many sentiment classification datasets (Kim, 2014;Dong et al., 2014;Tang et al., 2015).

despite the promising results, recent work has shown that these models easily fail in adversarial examples 2 with little perturba- * Equal Contribution.


Second, we regard the examples that confuse the classifier as good attacking examples and give them high reward to train the generator.

not all confusing examples are useful for robustness improvements.


We posit that structural and semantic correspondence is both prevalent in opinionated text, especially when associated with attributes, and crucial in accurately revealing its latent aspect and sentiment structure.

it is not recognized by existing approaches.


Further, Solo and Friend contain reviews of business trips, although the authors did not select Business as the trip type.

this situation does not happen for Couple and Family.


In practice such models are notoriously hard to train and require the availability of very large datasets.

the injection of finegrained polarity information has been shown to be a key ingredient to build competitive sentiment predictors by Socher et al.


While these techniques are effective, they are not ideal in domains with limited data.

work such as ELMo (Peters et al., 2018) and BERT (Devlin et al., 2018) propose deeply connected layers to learn sentence embeddings by exploiting bi-directional contexts.


In their work (K Sarma et al., 2018), the authors demonstrate the effectiveness of KCCA based embeddings via classification experiments.

in order to verify that the word level adaptation performed by the KCCA step can capture relevant domain semantics, we perform the following experiment.


These words are used extensively when writing reviews about books.

words that shift the least are Nouns (69%) such as 'Higgins', 'Gardner' and 'Schaffer' that correspond to author and character names.


their inner working mechanisms and suffer from the lack of interpretability.

clearly understanding where and how such a model makes such a decision is rather important for developing real-world applications Marcus, 2018).


One possible way to alleviate this problem is to also leverage the soft-attention mechanism as proposed in .

this soft-attention mechanism may induce additional noise and lack interpretability because it tends to assign higher weights to some domain-specific words rather than real sentiment-relevant words (Mudinas et al., 2012;Zou et al., 2018).


(6)) to guide both clause and word selection.

when model training is finished, i.e., both high-level and low-level policy finish all their selections, the goal of sentiment rating predictor is to perform DASC.


This means that, interestingly, CHIM-based attribute representations have also learned information about the category of the product.

representations learned from the bias-attention method are not able to transfer well on this task, leading to worse results compared to the random and majority baseline.


The visualization results are shown in Fig 3, we can observe that the model without sentiment regularizer most focus on non-sentiment words such as XiaoMei, she and heard it which are inessential to provoke the emotion.

when we plus the SR. into model, we can see an obvious weights shift on attention distribution.


(2019) proposed a method based on learning to re-rank candidate emotion cause clauses with extracting a number of emotion-dependent and emotion-independent features.

these methods are heavily dependent on the expensive human-based features and are too difficult in a real-world application.


In addition, in UKPRank no individual labeling is provided, and individual quality scores are inferred from pairs labeling.

for IBMRank each argument is individually labeled for quality, and we explicitly demonstrate the consistency of these individual labeling with the provided pairwise labeling.


Table 6 shows that joint learning (BERT-Joint) hurts the performance compared to single-task BERT.

using additional information from the sentence-level for the token-level classification (BERT-Granularity) yields small improvements.


Thus, the weights in the encoder part will adjust according to the desired task-specific label as well.

in contrast, the decoder part does not have such information.


These approaches provided strong baselines for text classification (Wang and Manning, 2012).

sequential patterns and semantic structure between words play a crucial role in deciding the category of a text.


For example, a word like "killer" in the original embedding space is very close to negative sentiment words like, "bad" and "awful", especially if the word embeddings were produced on huge fact based datasets like Wikipedia or news datasets.

in the SST2 dataset, "killer" is often used to describe a movie very positively.


Incorporating the parent representation only along with the claim representation does not give significant improvement over representing the claim only.

incorporating the flat representation of the larger context along with the claim representation consistently achieves significantly better (p < 0.001) performance than the claim representation alone.


In such an operation, the GCN only considers the first-order neighborhood of a node when modeling its embeddings.

k successive GCN operations result in the propagation of information across the k-th order neighborhood.


(2018   CTC Table 1 shows the main BLEU results of different methods on the test set.

we cannot identify the best DA method because their rankings across the four translation tasks vary a bit.


Some of those measures have complex definition such as linear regions, others are very expensive to compute for models as large as Transformer such as Hessian and stiffness.

we compute weight norm with different forms proposed in Neyshabur et al.


To generate y given x with the channel model, we wish to compute arg max y log p(x|y) + log p(y).

naïve decoding in this way is computationally expensive because the channel model p(x|y) is conditional on each candidate target prefix.


For the direct model, it is sufficient to perform a single forward pass over the network parameterizing p(y|x) to obtain output word probabilities for the entire vocabulary.

the channel model requires separate forward passes for each vocabulary word.


These models enable zero-shot transfer, but achieve lower results than monolingual models.

we focus on making the training of monolingual language models more efficient in a multi-lingual context.


All of these studies, however, only create small datasets, which are inadequate for pretraining language models.

we are among the first to report the  3 Our method We propose Multi-lingual Fine-tuning (MultiFit).


Previous works mainly focus on adding different components into the NART model to improve the expressiveness of the network structure to overcome the loss of autoregressive dependency (Gu et al., 2017;Lee et al., 2018;.

the computational overhead of new components will hurt the inference speed, contradicting with the goal of the NART models: to parallelize and speed up neural machine translation models.


As we explain in §3, our investigation requires a definition of lexical semantics that is independent of grammatical gender.

in many gendered languages, word embeddings effectively encode grammatical gender because this information is trivially recoverable from distributional semantics.


Although our results provide evidence for the non-arbitrariness of noun-gender assignments, they must be contextualized.

to animate nouns, it is not clear that a single cross-linguistic category explains our results.


Similarly, we focus on perturbing over person names.

our method is readily extendable to other kinds of models as well as to other entity types.


Furthermore, we were unable to extract author gender in the professor dataset since the RMP reviews are anonymous.

in future work, we may explore the influence of author gender in the celebrity dataset.


For example, in Figure 1, it needs at least 4 hops (i.e., "fired"-"evidence"-"blood"-"soldiers") to figure out that the word "fired" means "shot" instead of "dismissed".

the above dependency tree based methods explicitly use only first-order syntactic relations, although they may also implicitly capture high-order syntactic relations by stacking more GCN layers.


Therefore, the trigger words and their related entities are unable to interact in multi-order graph attention network, which leads to a lower recall.

mOGANED still achieves the best performance in terms of precision, recall, F 1 -measure among all dependency based methods, which suggests the effective of multi-order representations.


Depending on the task at hand, IE often achieves high correctness (sometimes above 90%).

evaluating its coverage is inherently hard, as this would require exhaustively annotated corpora as gold standard.


(2018) fed external world knowledge (ConceptNet relations and coreferences) explicitly into MAGE-GRU (Dhingra et al., 2017) and achieved improvements compared to only using the relational arguments.

we here show that it works even better when we learn this knowledge implicit through next sentence prediction task.


The fine-tuning is then done as in ProposedRU.

to the above single-step pre-training, we also attempted to start from the pre-trained model for ProposedRU web , which was pre-trained using general web text, and then additionally pretrain it with causality-rich texts to make the model suitable for causality-event recognition 4 .


As the neural phrase structure parser is trained on increasingly larger corpora, the regression model of EEG amplitudes fits better and better.

this pattern only obtains with in-domain training data that is lexically-similar to the first chapter of Alice in Wonderland.


In contrast, the development of distributional modelling techniques and the availability of vast text corpora have allowed researchers to construct effective vector space models of word meaning over large lexicons.

this comes at the cost of interpretable, human-like information about word meaning.


A criticism of these real-valued embedding vectors is the opaqueness of their representational dimensions and their lack of cognitive plausibility and interpretability (Murphy et al., 2012;Ş enel et al., 2018).

human conceptual property knowledge is often modelled in terms of relatively sparse and interpretable vectors, based on verbalizable, human-elicited features collected in property knowledge surveys (McRae et al., 2005;.


Geographical SQA has posed great challenges to NLP and related research, ranging from scenario understanding to cross-modal knowledge integration and reasoning.

there is a lack of large datasets and benchmarking efforts for this task.


Existing SQA datasets for other domains include the TREC Precision Medicine track (Roberts et al., 2018) for the medical domain, and CAIL  for the legal domain.

sQA in the geography domain requires different forms of knowledge and different reasoning capabilities, and has posed different research challenges.


We chose this approach since synthetic data generation can be especially useful in novel and early-stage research efforts, as it provides a controlled environment and allows for detailed analyses of a model's ability to solve a task.

while methods such as Recurrent Entity Networks have shown promise for keeping track of the state-of-the-world in our experiments, this is still in scenarios where the complexity of natural language is relatively simple.


Second, previous work defines passages as articles, paragraphs, or sentences.

the question of proper granularity of passages is still underexplored.


Almost all previous state-of-the-art QA and RC models find answers by matching pas- DrQA (Chen et al., 2017) 37.7 44.5 41.9 48.7 32.3 38.3 29.8 -R 3 (Wang et al., 2018a) 35.3 41.7 49.0 55.3 47.3 53.7 29.1 37.5 OpenQA (Lin et al., 2018) 42.2 49.3 58.8 64.5 48.7 56.3 28.7 36.6 TraCRNet (Dehghani et al., 2019) 43.2 54.0 52.9 65.1 ----HAS-QA (Pang et al., 2019) 43.2 48.9 62.7 68.7 63.6 68.9 --BERT (Large) (Nogueira et al.,   sages with questions, aka inter-sentence matching (Wang and Jiang, 2017;Wang et al., 2016;Seo et al., 2017;Song et al., 2017).

bERT model simply concatenates a passage with a question, and differentiates them by separating them with a delimiter token [SEP], and assigning different segment ids for them.


This variance is especially high for local IDF, which serves as a strong signal as consistently observed in (Blair-Goldensohn et al., 2003).

in InsuranceQA, the variance 5894  of word signals are low.


Previous work usually utilizes knowledge graphs such as ConceptNet as external knowledge, and extracts triples from them to enhance the initial representation of the machine comprehension context.

such method cannot capture the structural information in the knowledge graph.


In semantic parsing, there is extensive work on using deep neural networks for training models over manually created logical forms in a supervised learning setup (Jia and Liang, 2016;Ling et al., 2016;Dong and Lapata, 2018).

creating labeled data for this task can be expensive and time-consuming.


Neural techniques question answering have improved (Devlin et al., 2018) machine reading comprehension (Rajpurkar et al., 2016, MRC): computers can take a single question and extract answers from datasets like Wikipedia.

qA models struggle to generalize when questions do not look like the standalone questions systems in training data: e.g., new genres, languages, or closely-related tasks (Yogatama et al., 2019).


A natural question is whether reinforcement learning could learn to retain the necessary context to rewrite questions in CQA.

our dataset could be used to pretrain a question rewriter that can further be refined using reinforcement learning.


You can consider any question it can answer to be too easy.

please note that the AI system incorrectly answering a question does not necessarily mean that it is good.


On standard answer selection datasets such as TrecQA (Wang et al., 2007) or WikiQA (Yang et al., 2015), Compare-Aggregate approaches achieve very competitive performance.


Previous studies on answer selection have focused mostly on small-scale datasets.

many community question answering (CQA) platforms such as Yahoo Answers and Stack Exchange have become an essential source of information for many people.


This is expected as BERT has been pretrained on a massive amount of unlabeled data.

our proposed techniques do add a significant amount of performance.


In other words, it cannot copy words with morphological changes.

morphological changes frequently happen when we transform a passage into a question as grammatical functions of some words (e.g., verbs, nouns, adjectives, etc) change.


The trained seq2seq model is prone to generating these "safe" questions, similar to the undiversified response generation in seq2seq-based dialogue model .

our model is able to generate a more relevant question including a rare word "amalgamated" as the word has a high overlap rate.


Stories generated with neural language models have shown promise in grammatical and stylistic consistency.

the generated stories are still lacking in common sense reasoning, e.g., they often contain sentences deprived of world knowledge.


7 The BERT multi-task versions perform better with highly correlated qualities like Q4 and Q5 (as illustrated in Figures 2 to 4 in the supplementary material).


This observation is consistent with many previous works of using RL/IL in NLP.

we also notice that the improvement of using the REINFORCE algorithm (line 6) is very small, only 0.18 on the average score.


In Table 4, we can see that the auxiliary loss model's improvements are even more amplified on D med and D late .


Further, at generation time, they heavily rely on rejection sampling to produce quatrains which satisfy any valid rhyming pattern.

we find that generators trained using our structured adversary produce samples that satisfy rhyming constraints with much higher frequency.


devised a type decoder which combines three type-specific generation distribution (including question type) with weighted sum.

the results displayed in their paper show that questions in dialogue are far different from questions for reading comprehension, which indicates a gap  In this paper, we propose a unified model to predict the question type and to generate questions simultaneously.


The first two examples represent DSR's generated tokens are more diverse.

it may suffer from problems as shown in example 3 and 4.


Recently, various deep learning approaches have been proposed based on the WikiSQL dataset (Zhong et al., 2017).

because WikiSQL contains only very simple queries over just a single table, these approaches (Xu et al., 2017;Huang et al., 2018;Yu et al., 2018a;Dong and Lapata, 2018) cannot be applied directly to generate complex queries containing elements such as JOIN, GROUP BY, and nested queries.


For text-to-SQL generation, several SQLspecific approaches have been proposed (Zhong et al., 2017;Xu et al., 2017;Huang et al., 2018;Yu et al., 2018a;Dong and Lapata, 2018;Yavuz et al., 2018) based on WikiSQL dataset (Zhong et al., 2017).

all of them are limited to the specific WikiSQL SQL sketch, which only supports very simple queries.


After their release, very large language models (LMs) were able to reach or surpass human-level performance on SNLI (Peters et al., 2018) and SWAG (Devlin et al., 2018).

researchers have found inadequacies in these datasets and the models trained on them.


Contextualized word embeddings have boosted many NLP tasks compared with traditional static word embeddings.

the word with a specific sense may have different contextualized embeddings due to its various contexts.


The LM updated with textbook data (BERT+Textbook), improves performance on the domains included in additional pre-training (Phy and Gov).

we suspect that the updated model becomes more specialized towards seen domains, which leads to performance degradation on the unseen domain of Psychology.


We limited the scope of this paper to the task of automatic short answer grading only.

our findings of the sensitivity of domain-specific BERT models appear generic.


Similarly, counterfactual reasoning has been studied in the logic community (Lewis, 2013), but again using formal frameworks.

wIQA treats the task as a mixture of reading comprehension and commonsense reasoning, creating a new NLP challenge.


It contains several phenomena in the "stress tests" (Naik et al., 2018) including word overlap, negation, and length mismatch.

these datasets are artificially constructed while CB data are naturally occurring.


In the past, there has been work on extracting high quality translations from crowd-sourced workers using automatic methods (Zaidan and Callison-Burch, 2011;Post et al., 2012).

crowd-sourced translations have generally lower quality than professional translations.


Overall, the domains and quantity of the existing parallel data are very limited.

both languages have a rather large amount of monolingual data publicly available (Buck et al., 2014), making them perfect candidates to track performance on unsupervised and semi-supervised tasks for Machine Translation.


Right: average sentence level BLEU against Wikipedia document id from which the source sentence was extracted; sentences have roughly the same degree of difficulty across documents since there is no extreme difference between shortest and tallest bar.

source sentences originating from Nepali Wikipedia (blue) are translated more poorly than those originating from English Wikipedia (red).


Nepali and Sinhala are languages with very different syntax and morphology than English; also, very little parallel data in these language pairs is publicly available.

a good amount of monolingual data, parallel data in related languages, and Paracrawl data exist in both languages, making these two language pairs a perfect candidate for research on low-resource MT.


On the one hand, if src is ignored, it is difficult to identify translation errors related to adequacy in mt, especially for fluent but inadequate translations (e.g., mt in Figure 1).

mt serves as a major source for generating pe since many words (e.g., "Ich" and "einen" in Figure 1) are copied from mt to pe.


occupations), we can rely on human annotations or external data.

for most words, evaluating the correctness of them is still an open problem.


(2017) use word embeddings as a quantitative lens to investigate historical trends of gender stereotypes.

due to the limitations in word embedding training, existing methods have constrained effects in this field.


One possible idea to tackle such problems is to employ a sarcasm scorer that can determine the sarcasm content in the generated output, and use the scores given by the sarcasm scorer for better training of the generator.

the sarcasm scorer may be external to the sequence-tosequence pipeline, and the scoring function may not be differentiable with respect to the model M θ .


In typical RL settings, the learner is typically initialized to a random policy distribution.

in our case, since some supervision is already available in the form of target sarcastic sentences, we pretrain the model with the loss minimization objective given in Eq.


Inspired by unsupervised machine translation (UMT) (Lample et al., 2018b), we treated our task as a translation problem, namely translating vernacular paragraphs to classical poems.

our work is not just a straight-forward application of UMT.


The UMT framework makes it possible to apply neural models to tasks where limited human labeled data is available.

in previous tasks that adopt the UMT framework, the abstraction levels of source and target language are the same.


For example, in  selected from famous modern prose.

compared with poem 4, poem 5 seems semantically more confusing.


We did not set any stopping criteria for the number of instructions to be generated for simplicity sake.

we generated as many number of instructions in the corresponding test record.


As mentioned above, the main difference is that SYPHON is two orders of magnitude faster than their system thanks to a novel decomposition and efficient SMT encoding.

we impose extra restrictions on the hypothesis space (i.e.


The resulting knowledge base, TEMPROB, contains observed frequencies of tuples (v1, v2, r) representing the probability of verb 1 and verb 2 having relation r and it was shown a useful resource for TempRel extraction.

tEMPROB is a simple counting model and fails (or is unreliable) for unseen (or rare) tuples.


The underlying assumptions behind most NER systems are that an entity should contain a contiguous sequence of words and should not overlap with each other.

such assumptions do Figure 1: Entities are highlighted with colored underlines.


Muis and Lu (2016a) proposed a hypergraphbased representation to compactly encode discontiguous entities.

this representation suffers from the ambiguity issue during decodingone particular hypergraph corresponds to multiple interpretations of entity combinations.


(2014);  extended the BIO tagging scheme to encode such complex structures so that traditional linear-chain CRF (Lafferty et al., 2001) can be employed.

the model suffers greatly from ambiguity during decoding due to the use of the extended tagset.


For r (k) or r (k,l) derived from a KB relation, we represent it by a trainable parameter vector.

for the one derived from a textual relation, we use the following encoders to compute its representations.


Since their method directly predicts a relation label for each surface pattern, it is more robust to the sparsity of surface patterns among a specific higher arity entity tuple.

due to their purely supervised training objective, its performance may degrade if the number of available training labels is small.


In the result, U+B performs significantly better (p < 0.005) than U and B, and this shows effectiveness of combining scores of both binary facts and unary facts.

there was no significant difference between U+B+N and N (p > 0.9).


From a humanities perspective, a VA constitutes an interesting phenomenon of enculturation (Holmqvist and Płuciennik, 2010) that deserves to be studied more in-depth, based on larger corpora.

we expect that the large-scale identification of VAs and other such stylistic devices is not only important in the humanities and for cultural reasons, but also in natural language processing to avoid mistakes in tasks such as machine translation and fact extraction.


Thus, we used a random sample of 105 articles (for each year 5).

we found only one VA: "If Nike is the Chicago Bulls of the athletic shoe market, retailers are holding their breath for a strong underdog player to emerge."


In other words, the more distinct these representations are, the more difficult identifying chemical compound entities becomes.

existing chemical NER methods do not deal with notation variants of chemical compounds, derived from the partial structures or the notation fluctuation peculiar to these chemical compounds.


These results contain two types of errors: idiosyncratic casing in the gold data and failures of the truecaser.

from the high scores in the Wikipedia experiment, we suppose that much of the score drop comes from idiosyncratic casing.


Naturally, in any crossdomain experiment, one will obtain higher scores by training on in-domain data.

our goal is to show that our methods produce a more robust model on out-of-domain data, not to maximize performance on this test set.


Its scorer makes some modifications to OIE2016.

it does not reward partial coverage of gold tuples, and forces one system prediction to match just one gold.


One-to-one match (OIE2016) is indifferent between the two which means that for OIE2016, adding more information in the same extraction has no value at all.

multi match (CaRB) assigns higher recall to system 1, since it contains strictly more information, and higher precision to system 2, since its prediction exactly matched a gold extraction.


Based on our empirical observation, the sentence-level sentiment classifiers without considering aspects can still achieve competitive results with many recent ABSA methods (see TextCNN and LSTM in Table 3).

even advanced ABSA methods trained on these datasets can hardly distinguish the sentiment polarities towards different aspects in the sentences that contain multiple aspects and multiple sentiments.


Researchers and practitioners typically have to resort to crowdsourcing.

as mentioned above, the crowdsourced annotations can be quite noisy.


This task is similar to the three-way aspect-level sentiment analysis that determines sentiment polarity towards aspect terms.

different from aspect-level sentiment analysis, the target in stance detection might not be explicitly mentioned in a given sentence.


Then the F1 average is calculated as: Note that the label "None" is not discarded during training.

the label "None" is not considered in the evaluation because we are only interested in labels "Favor" and "Against" in this task.


Based on massive amounts of data, recent pretrained contextual representation models have made significant strides in advancing a number of different English NLP tasks.

for other languages, relevant training data may be lacking, while state-of-the-art deep learning methods are known to be data-hungry.


Without L alignment , we observe a reduction in both accuracy and BLEU on Yelp.

this tendency is inconsistent on Amazon (i.e., -2.2 accuracy and +0.56 BLEU).


Hierarchical attention without normalization can make distinctions for the output vector of each class in the norm, so it works well.

using normalization is better in label attention.


The lower class label text for the upper class OTHER includes less verbose phrases such as "The user requesting," so label attention can give higher weight for important parts of the label text more precisely.

the F1 score of the proposed method for the upper class REPORT is worse than that of Non-hier.


The proposed method has a better F1 score than Non-hier for nine lower classes with a mean number of tokens in the lower class labels of 11.2.

the proposed method has a lower F1 score than Non-hier for seven lower classes with a mean number of tokens of 8.0.


Word embeddings have demonstrated strong performance on NLP tasks.

lack of interpretability and the unsupervised nature of word embeddings have limited their use within computational social science and digital humanities.


Relaxing the location assumptions with the Truncated Prior (γ = 1000) and placing informative priors on neutral words (ψ = 0.01) show varying degrees of improvements.

due to the limited number of hold-out test words, large improvements are required for significant differences -which only is observed for the sentiment dimension in the Senate corpus.


Such forms of bias typically do not depend on context outside of the sentence and can be alleviated while maintaining its semantics: polarized words can be removed or replaced, and clauses written in active voice can be rewritten in passive voice.

political science researchers find that news bias can also be characterized by decisions made regarding content selection and organization within articles (Gentzkow et al., 2015;Prat and Strömberg, 2013).


An alternative encoding method, Label Powerset (LP), captures dependencies explicitly: each label combination appearing in the training data is encoded as a new, unique label, reducing the task once again to a multiclass classification.

lP can introduce an exponentially large number of new labels, potentially with few training instances, thus exacerbating class imbalance.


Ancient History relies on disciplines such as Epigraphy, the study of ancient inscribed texts, for evidence of the recorded past.

these texts, "inscriptions", are often damaged over the centuries, and illegible parts of the text must be restored by specialists, known as epigraphists.


These repositories primarily consist in a researcher's mnemonic repertoire of such parallels, and in digital corpora for performing "string matching" searches (The Packard Humanities Institute, 2005;Clauss, 2012).

minor differences in the search query can exclude or obfuscate relevant results, making it hard to estimate the true probability distribution of possible restorations.


In addition to pairwise ranking, SemEval 2017 Task 6 also includes a global ranking subtask.

the majority of submissions build global rankings using a series of pairwise comparisons (Potash et al., 2017).


State-of-the-art humor recognition algorithms usually require a considerable amount of training data with labels to learn effective features (Yang et al., 2015).

such data are difficult to obtain -especially fine-grained humor annotations.


Like ours, their results are obtained without using the training set.

their system uses an n-gram language model trained on a 6.2GB subset of the News Commentary Corpus and the News Crawl Corpus.


Automatic data augmentation is commonly used in computer vision (Simard et al., 1998;Szegedy et al., 2014;Krizhevsky et al., 2017) and speech (Cui et al., 2015;Ko et al., 2015) and can help train more robust models, particularly when using smaller datasets.

because it is challenging to come up with generalized rules for language transformation, universal data augmentation techniques in NLP have not been thoroughly explored.


Kobayashi (2018) showed that replacing words with other words that were predicted from the sentence context using a bi-directional language model yielded a 0.5% gain on five datasets.

training a variational auto-encoder or bidirectional LSTM language model is a lot of work.


Therefore, we translate the source language, English, to the target language for the training corpus to reconstruct source language sentences and learn source and target languages jointly.

one English word might have multiple corresponding translations.


The previous cross-lingual settings were only for European languages, which share similar alphabets.

many languages use non-Latin orthography.