Past work on bias in hate speech datasets has exclusively focused on finding and removing bias against explicit identity mentions (e.g., woman, atheist, queer; Park and Fung, 2017; Dixon et al., 2018).

our work shows how insensitivity to dialect can lead to discrimination against minorities, even without explicit identity mentions.


In the English source sentence, the nurse's gender is unknown, while the coreference link with "her" identifies the "doctor" as a female.

the Spanish target sentence uses morphological features for gender: "el doctor" (male), versus "la enfermera" (female).


One of the weaknesses of these approaches, however, is that they do not take word ordering into account during the learning process.

word-based approaches based on RNNs that consider sequence information have been presented, but they are not competitive in terms of speed or quality of the embeddings (Mikolov et al., 2010;Mikolov and Zweig, 2012;Mesnil et al., 2013).


Finally, Press and Wolf (2017) introduced a model, based on word2vec, where the embeddings are extracted from the output topmost weight matrix, instead of the input one, showing that those representations are also valid word embeddings.

to the above approaches, each of which aims to learn representations of lexical items, sense embeddings represent individual word senses as separate vectors.


SGNS does not, on average, make the vast majority of words any more gendered in the vector space than they are in the training corpus; individual words may be slightly more or less gendered due to reconstruction error.

for words that are genderstereotyped (e.g., 'nurse') or gender-specific by definition (e.g., 'queen'), SGNS amplifies the gender association in the training corpus.


To allow a fair comparison with prior work, our experiments in this paper focus on gender association.

our claims extend to other types of word associations as well, which we leave as future work.


In practice, this issue often goes unnoticed because each word in the attribute set, at least for gender association, has a counterpart that appears with roughly equal frequency in most training corpora (e.g., 'man' vs. 'woman', 'boy' vs. 'girl').

this is not guaranteed to hold, especially for more nebulous attribute sets (e.g., 'pleasant' vs. 'unpleasant' words).


Given that much of the vocabulary falls into this category, this means that the embedding model does not systematically change the genderedness of most words.

because of reconstruction error, individual words may be more or less gendered in the embedding space, simply due to chance.


In our debiased embedding space, 94.9% of gender-appropriate analogies with a strength of at least 0.5 are preserved in the embedding space while only 36.7% of gender-biased analogies are.


Using RIPA, we found that SGNS does not, on average, make most words any more gendered in the embedding space than they are in the training corpus.

for words that are gender-biased or gender-specific by definition, SGNS amplifies the genderedness in the corpus.


Arguably, the gendered use of pregnant is benign-it is not due to cultural bias that women are more often described as pregnant, but rather because women bear children.

differences in the use of other adjectives (or verbs) may be more pernicious.


D S may span many diverse topics, while D T focuses on one or few, so there may be large overall drift from D S to D T too.

a judicious subset D S ⊂ D S may exist that would be excellent for augmenting D T .


Howard and Ruder (2018) propose to fine-tune a whole language model using careful differential learning rates.

epoch-based termination may be inadequate.


It is clear that pre-training the contextual embeddings on relevant target corpus helps in the downstream classification task.

the gains of SrcSel:R over Tgt is not clear.


For example, the angle between embeddings of (car, hypernymy, vehicle) and (car, synonymy, auto) is large.

that of (car, synonymy, automobile) and (car, synonymy, auto) is small.


(2017) present the model Attract-Repel to improve qualities of word embeddings for synonymy recognition.

they focus on one particular lexical relation, not capable of distinguishing multiple types of lexical relations.


Based on J m , if (x i , y i ) has the lexical relation type r m , the norm of M m x i − y i is likely to be small.

the norms of M n x i − y i (1 ≤ n ≤ |R|, n = m) are likely to be large.


This makes it a suitable objective for aligning the vector spaces of x, y in the latent space.

to the discriminative directed methods in (Mikolov et al., 2013a;Smith et al., 2017;Xing et al., 2015), IBFA has the capacity to model noise.


Parallel corpora are rare, and even when they do exist, they often only exist between specific pairs of languages.

the documentation of a language often begins with the creation of several important documents, including a dictionary of key terms, and translations of religious texts.


Also beginning with aligned Bible data, they recover verbal lemmas by leveraging multi-lingual alignments.

where they are only interested in recovering the lemma, we simultaneously induce detailed morphological features of the words in the target language, over a wider range of verbal and nominal morphology, and deploy a new set of machine learning techniques to do so.


For example, German nouns ending in "-ung" are very likely to pluralize with an "-en" suffix, but the projection baseline discovers no correct "-ung" pairs.

"-en" is a common plural suffix in German, and the systems systematically strip the "-en" from "-ungen" nouns, although often lower in the hypothesis list.


Both baselines struggle to  produce the correct lemma -nouns are about 4 times as likely to observe null-inflection as verbs, and even plural nouns tend to drift significantly from their lemmas, to the point that another citation form has a smaller edit-distance.

we note little difference between nouns and verbs for any of our systems -in fact, our verbal system prior to reranking is slightly better than the nominal system.


We claim that the Bible is a suitable resource for learning the morphology of low-resource languages, but due to the necessity of gold morphological dictionaries, many of our evaluation languages cannot be considered low-resource.

the only available resources we assume to exist are a translated Bible and a bilingual dictionary.


The Transformer is adept at parallelizing of performing (multi-head) and stacking (multi-layer) SANs to learn the sentence representation to predict translation, and has delivered state-of-the-art performance on various translation tasks (Bojar et al., 2018;Marie et al., 2018).

these positional embeddings focus on sequentially encoding order relations between words, and does not explicitly consider reordering information in a sentence, which may degrade the performance of Transformer translation systems.


Word ordering is an important issue in translation.

it has not been extensively studied in NMT.


It can be seen that pre-norm Transformer obtains the same BLEU score as TA without the requirement of complicated attention design.

dLCL in both post-norm and pre-norm cases outperform TA.


Nevertheless, our system with a 30-layer encoder is still faster than Transformer-Big, because the encoding process is independent of beam size, and runs only once.

the decoder suffers from severe autoregressive problems.


We attribute it to post-norm Transformer being more sensitive to the large learning rate.

in the case of either a 6-layer encoder or a 20-layer encoder, the pre-norm Transformer benefits from the larger batch and learning rate.


The model using FastText embeddings is shown to be more robust across the datasets, although it also fails to outperform the diverse decoding baseline in WMT14 dataset.

syntax-based models achieve much higher diversity in both datasets.


We can see that the candidate translations produced by beam search has only minor grammatical differences.

the translation results sampled with the syntactic coding model have drastically different grammars.


The cosine similarity of a sentence pair is calculated as the dot product of their representations: which is bounded in the [-1, 1] range.

the threshold to decide when to accept a pair is not straightforward and might depend on the language pair and the corpus Artetxe and Schwenk, 2018).


Previous works have reduced sequence lengths to make training more tractable through fixed-length downsampling.


Over the last two decades, there has been extensive study targeting unsupervised constituency parsing (Klein and Manning, 2002, 2004, 2005Bod, 2006a,b;Ponvert et al., 2011) and dependency parsing (Klein and Manning, 2004;Smith and Eisner, 2006;Spitkovsky et al., 2010;Han et al., 2017).

all of these approaches are based on linguistic annotations.


It is worth noting that the PRPN baseline reaches this performance without any information from images.

the performance of PRPN is less stable than that of VG-NSL across random initializations.


h h h h h h h h h h @ @ @ @ @ @ @ @ @ @ Three white sinks in a bathroom  Figure 7: A failure example by Benepar, where it fails to parse the noun phrase "three white sinks in a bathroom under mirrors" -according to human commonsense, it is much more common for sinks, rather than a bathroom, to be under mirrors.

most of the constituents (e.g., "three white sinks" and "under mirrors") are still successfully extracted by Benepar.


(2018) improved performance for several models.

while many of these augmented instructions have clear starting or ending descriptions, the middle portions are often disconnected from the path they are paired with (see  for in depth analysis of augmented path instructions).


To avoid the intensive labor involved in dense annotations, (Huang et al., 2018) and (Zhou et al., 2018) considered the problem of weaklysupervised video grounding where only aligned video-sentence pairs are provided without any fine-grained regional annotations.

they both ground only a noun or pronoun in a static frame of the video.


Also, to avoid affecting previously learned  Step-2 knowledge, we constrain θ 1,k and ψ 2,k to be orthogonal (Condition 2.2).

strictly imposing this condition into the objective function is not feasible (Bousmalis et al., 2016), hence we add a penalizing term into the objective function as an approximation to the orthogonality condition: Both Condition 2.1 and 2.2 are mutually dependent, because for two matrices' product to be zero, they share basis vectors between them, i.e., for an n-dimensional space, there are n basis vectors and if p of those vectors are assigned to one matrix, then the rest of the n − p vectors (or subset) should be assigned to the other matrix.


Modern entity linking systems rely on large collections of documents specifically annotated for the task (e.g., AIDA CoNLL).

we propose an approach which exploits only naturally occurring information: unlabeled documents and Wikipedia.


For instance, in Figure 3, for document "Brexit", we link entity Brexit to all other entities.

we do not link United Kingdom to Greek withdrawal from the eurozone as they are more than l entities apart.


11 Our model is accurate for PER, achieving accuracy of about 97%, only 0.53% lower than the supervised model.

annotated data appears beneficial for other named-entity types.


This result may be interpreted as suggesting that humanannotated data is not beneficial for entity linking, given that we have Wikipedia and web links.

we believe that the two sources of information are likely to be complementary.


Misra et al., 2018;Chaplot et al., 2018;Mei et al., 2016).

neural networks' powerful abilities to induce complex representations have come at the cost of data efficiency.


To solve the problem, training data selection (TDS) has been proven to be a prospective solution for domain adaptation in leveraging appropriate data.

conventional TDS methods normally requires a predefined threshold which is neither easy to set nor can be applied across tasks, and models are trained separately with the TDS process.


More recently, Generative Adversarial Net (GAN) based methods (Zang and Wan, 2017;Yu et al., 2017;Xu et al., 2018a) have been proposed to enhance the generation of long, diverse and novel text.

they still focus on word-level generation, and neglect the importance of topical and syntactic characteristics from natural languages.


to incorporate topic information for essay generation.

the model performance is not satisfactory.


Previous work (Feng et al., 2018) only adopts BLEU (Papineni et al., 2002) score based on ngram overlap to perform evaluation.

it is unreasonable to only use BLEU for evaluation because TEG is an extremely flexible task.


This limitation leads to poor coherence and topic-consistency.

the proposed model succeeds in generating novel high-quality text that closely surrounds the semantics of all input topics.


(2018) present a planand-write framework with two planning strategies to fully leverage storyline.

story generation and the TEG task focus on different goals.


For example, the word "okay" has a positive intensity around 0.6, "good" is around 0.7, and "great" is around 0.8.

when involving to the previous generated words, the sentiment intensity of current generated word may be totally different.


(2) Our model can more precisely control the sentiment intensity from human scores on sentiment, and it can also obtain both best results in sentiment mean absolute error (MAE) and relative sentiment rank (MRRR).

automatic   SC-Seq2Seq gets the second best MaE score while Revised-VaE + L extra gets the second best MRRR score.


This task aims to reverse the sentiment polarity of a sentence but keep its content unchanged without parallel data (Fu et al., 2018;Tsvetkov et al., 2018;Xu et al., 2018;Lample et al., 2019).

there are few researches focus on the fine-grained control of sentiment.


Recently with the rise of neural networks, many methods generate text in an endto-end manner Wiseman et al., 2017;Bhowmik and de Melo, 2018).

they pay little attention to the grammatical structure of the output which may be ignored in generating long sentences, but it is crucial in generating short noun compounds like type descriptions.


A possible solution is to apply beam search to enlarge the searching space at the first stage.

in our preliminary experiments, when the beam size is small, the diversity of predicted key facts is low, and also does not help to improve the accuracy.


In the unsupervised paradigm, Paetzold and Specia 2016proposed an unsupervised lexical simplification technique that replaces complex words in the input with simpler synonyms, which are extracted and disambiguated using word embeddings.

this work, unlike ours only addresses lexical simplification and cannot be trivially extended for other forms of simplification such as splitting and rephrasing.


Pre-trained word embeddings are often seen to have positive impact on sequence-to-sequence frameworks (Cho et al., 2014a;.

traditional embeddings are not good at capturing relations like synonymy (Tissier et al., 2017), which are essential for simplification.


The latent representations of an input sentence z x and a syntactic tree template z y are fed into SIVAE-i, and the syntax of the generated sentence conforms with the explicitly selected target template.

linearized syntactic sequences are relatively long (as shown in Table 1) and long templates are more likely to mismatch particular input sentences, which may result in nonsensical paraphrase outputs.


(2018) generate paraphrases in a deep generative architecture.

all these methods assume the existence of some parallel paraphrase corpora while unsupervised paraphrase generation has been little explored.


Previous work used multi-level LSTM encoders (Yang et al., 2016) or hierarchical autoencoders (Li et al., 2015a) to learn hierarchical representations for long text or defined a stochastic latent variable for each sentence at decoding time (Serban et al., 2017).

our model encodes the entire paragraph into one single latent variable.


As shown in Table 9, the original sentences have been successfully manipulated to positive sentiment with the simple attribute vector operation.

the specific contents of the reviews are not fully retained.


Here, we present the method for optimizing I p e (x,y) , which can also be applied to I p d (x,y) .

to the parameter sharing techniques in most multi-task learning work (Collobert et al., 2011;Ando and Zhang, 2005), parameter θ for the parser and parameter φ for generator are independent in our framework.


If the selected entity e t is new (e t ∈ E t−1 ), the hidden state of the tracking model is updated with the embeddingē of entity e t as input.

if entity e t has already appeared in the past (e t ∈ E t−1 ) but is not identical to the previous one (e t = e t−1 ), we use h ENT s (i.e., the memory state when this entity last appeared) to fully exploit the local history of this entity.


(2019)'s model has no tracking module unlike our model, which mitigates redundant references and therefore rarely contains erroneous relations.

when complicated expressions such as parallel structures are used our model also generates erroneous relations as illustrated by the underlined sentences describing the two players who scored the same points.


In recent years, automatic question generation (QG), which aims to generate natural questions based on a certain type of data sources including structured knowledge bases (Serban et al., 2016b;Guo et al., 2018) and unstructured texts (Rus et al., 2010;Heilman and Smith, 2010;Du et al., 2017;Du and Cardie, 2018), has been widely studied.

previous works mainly focus on generating standalone and independent questions based on a given passage.


Figure 3: Example questions generated by human (i.e., original questions denoted as OQ), NQG and our ReDR on CoQA.

nQG generates much more questions consisting of implicit coreference markers like "Where?"


Automatic Evaluation Table 2 summarizes the results: both GCN CITED TEXT SPANS and TALKSUMM-ONLY models, are not able to obtain better performance than ABSTRACT 8 .

for the Hybrid approach, where the abstract is augmented with sentences from the summaries emitted by the models, our TALKSUMM-HYBRID outperforms both GCN HYBRID 2 and ABSTRACT.


Pasunuru and Bansal (2018) develop a loss-function based on whether salient segments are included in a summary.

the optimization of RL-based models can be difficult to tune and slow to train.


Abstractive summary generated by baseline Bottom-Up Summarization is much more better, which indicates the effectiveness of modifications.

the generated summary only contains partial salient information of the document.


Under these circumstances, supervised neural network models have achieved wide success, using a large number of reference summaries (Wang and Ling, 2016;Ma et al., 2018).

a model trained on these summaries cannot be adopted in other domains, as salient phrases are not common across domains.


Because the abstractive approach generates a concise summary by omitting trivial phrases, it can lead to a better performance than those of the extractive ones.

for Movies & TV, our model is competitive with other unsupervised extractive approaches; TextRank and Opinosis.


Because our full model generates summaries via learning the latent discourse tree, it sometimes fails to construct a tree, and thus experiences a decline in performance for relatively short reviews.

for datasets with the number of sentences exceeding 30, our model achieves competitive or better performance than the supervised model.


(2018a) proposed to retrieve a related summary from the training set as soft template to assist with the summarization.

their approach tends to oversimplify the role of the template, by directly concatenating a template after the source article encoding.


Other extractive methods apply sequence tagging models (Luan et al., 2017;Gollapalli et al., 2017; to identify keyphrases.

extractive methods cannot produce absent keyphrases.


Joint models for sentence selection and fusion implicitly perform content planning (Martins and Smith, 2009;Berg-Kirkpatrick et al., 2011;Bing et al., 2015;Durrett et al., 2016) and there is limited control over which sentences are merged and how.

this work attempts to teach the system to determine if a sentence singleton or a pair should be selected to produce a summary sentence.


Moreover, these methods tend to have limited content coverage by selecting salient words.

recent years have witnessed the success of Natural Language Generation (NLG) models to generate abstractive summaries.


Vocabulary expansion may be used to address the different vocabularies in source and target domains, and adversarial domain adaptation (ADA) may be used to merge the embedded feature representations across domains.

aDa does not adapt the decoder in an encoder-decoder generation model.


We adapt the domain-adversarial method for feature alignment in an encoder proposed by (Ganin et al., 2016).

for text generation, a domain-independent representation from the encoder, as used in domain adaptation for classification, is not adequate.


Use of the second text discourages this so that both the encoder and decoder are trained on text from the target domain (enabling use of an expanded, joint vocabulary trained on both source and target) to learn its style and vocabulary.

the artificial titles will generally be different from the real titles, which may lead to lower summarization performance.


the same time, the labeled data domain is switched to the source domain, so that both the embedding and decoder domains are abruptly changed.

in ASADA the embedding is gradually adapted from the target domain to jointly embed the source and target (F1).


6 to the validation sample, the fraction of incorrect summaries increases from 26% to 29% (Table 2), demonstrating that the slight improvement on the validation data does not transfer to the test set.


current NLI models are not yet robust enough for our downstream task.

the stateof-the-art performance on common NLI datasets is already very close to human performance (Nikita and Bowman, 2019), suggesting that new datasets, such as the one presented here, are necessary to expose the models' remaining limitations.


the passage-question pairs to address) without training examples.

jia and Liang (2017) revealed that intentionally injected noise (e.g.


The existing way to solve the second problem is to encode general knowledge in vector space so that the encoding results can be used to enhance the lexical or contextual representations of words (Weissenborn et al., 2017;Mihaylov and Frank, 2018).

this is an implicit way to utilize general knowledge, since in this way we can neither understand nor control the functioning of general knowledge.


Prior work has shown that a pipeline of retriever, reader, and reranker can improve the overall performance.

the pipeline system is inefficient since the input is re-encoded within each module, and is unable to leverage upstream components to help downstream training.


Confirming our intuition, Table 2 shows us that QuAC has the highest percentage of GEN-ERAL questions.

coQA and SQuAD, which allowed the question-asker to look at the passage, are dominated by SPEcIFIc questions.


In the previous work, the sub tasks are performed using pipelined models (Nie et al., 2019;Yoneda et al., 2018).

our approach performs evidence extraction and answer prediction simultaneously by regarding FEVER as an explainable multi-hop QA task.


Owing to the large amounts of unlabeled data and the sufficiently deep architectures used during pre-training, advanced LMs such as BERT are able to capture complex linguistic phenomena, understanding language better than previously appreciated (Peters et al., 2018b;Goldberg, 2019).

as widely recognized, genuine reading comprehension requires not only language understanding, but also knowledge that supports sophisticated reasoning (Chen et al., 2016;Mihaylov and Frank, 2018;Bauer et al., 2018;Zhong et al., 2018).


Recently, lots of neural network-based models have been proposed and achieved promising results in OpenQA.

the success of these models relies on a massive volume of training data (usually in English), which is not available in many other languages, especially for those low-resource languages.


Both translate-train and translate-test methods rely heavily on the quality of the machine translation system.

the quality of the machine translation system varies in different language pairs, depending on the size of parallel data and the similarity of the language pair.


Other approximations are possible: for example we could use q φ (z | x) as an importance sampling distribution to estimate the first integral.

we found the above approximation to be efficient and effective in practice.


Furthermore, contextualized word representations learned from large-scale unlabeled texts under language model training loss Radford et al., 2018;Devlin et al., 2018), are proven to be extensively helpful for many NLP tasks including dependency parsing (Che et al., 2018;Kitaev and Klein, 2018).

parsing performance drops dramatically when processing texts that are different from the training data, known as the domain adaptation problem.


As our reviewers point out, the semi-supervised domain-adaptation scenario, tackled in this work, is less realistic than the unsupervised counterpart, due to need of labeled target-domain training data, which is usually extremely expensive.

we believe that this work can be equally valuable and useful when there exist only dozens or hundreds of labeled targetdomain training sentences, which may be a feasible compromise for realistic applications of parsing techniques, considering that, as discussed above, purely unsupervised domain adaptation makes very limited progress.


We can see that although PB-train is much smaller than BC-train, the PB-trained parser outperforms the BC-trained parser by about 8% on PB-dev, indicating the usefulness and importance of target-domain labeled data especially when two domains are very dissimilar.

the gap between the ZX-trained parser and the BC-trained is only about 2% in LAS, which we believe has a two-fold reason.


Following most of the recent work, we apply the PTB-SD representation converted by version 3.3.0 of the Stanford parser.

this dependency representation results in around 1% of phrases containing two or three head words.


A natural practice to perform the task is to scan through the query text using the dictionary and treat terms matched with a list of entries of the dictionary as the entities (Nadeau et al., 2006;Gerner et al., 2010;Liu et al., 2015;.

this practice requires very high quality named entity dictionaries that cover most of entities, otherwise it will fail with poor performance.


This yields a directed acyclic graph that expresses various relations among words.

the fact that SDP structures are directed acyclic graphs means that we cannot apply standard dependency parsing algorithms to SDP.


In the left path, the model resolves adjacent arcs first.

in the right path, distant arcs that rely on the global structure are resolved first.


Therefore, we obtain: The model then creates semantic dependency arcs by iterating over the sentence as follows: 1 1 This algorithm can introduce circles.

circles 1 For each word w i , select a head arc from T τ i .


Among various neural networks for sequence labeling, bi-directional RNNs (BiRNNs), especially BiLSTMs (Hochreiter and Schmidhuber, 1997) have become a dominant method on multiple benchmark datasets Chiu and Nichols, 2016;Lample et al., 2016;Peters et al., 2017).

there are several natural limitations of the BiLSTMs architecture.


Results: Left-and right-branching baselines are constructed by assigning 21 random labels 7 to constituents in purely left-and right-branching trees.

both branching baselines perform poorly in this evaluation, due to the fact that there is no straightforward way to assign labels to constituent spans that may correspond to how gold labels are organized.


Previous work (Seginer, 2007;Ponvert et al., 2011;Shain et al., 2016;Jin et al., 2018b) show that using data likelihood as both the objective for optimization and the criterion for model selection, either implicitly (in the case of Bayesian models) or explicitly (in the case of EM), gives good results on grammar induction.

it is also known that data likelihood is only weakly correlated with parsing accuracy, especially at convergence (Smith, 2006;Johnson et al., 2007;Jin et al., 2018a).


Induction models of constituency grammars or trees usually use data likelihood as both the objective and the model selection criterion (Seginer, 2007;Johnson et al., 2007;Ponvert et al., 2011;Shen et al., 2018), but the weak correlation between data likelihood and parsing accuracy hints at the non-optimality of this practice (Smith, 2006;Headden et al., 2009;Jin et al., 2018a).

many linguistic and psycholinguistic theories propose constraints either as properties of natural language grammar or as constraints on human processing and acquisition.


2018, all the above methods use crossdomain NER data only.

we leverage both NER data and raw data for both domains.


As noted in Section 1, the recovery of unbounded dependencies, including wh-object questions, is a 10 Fowlie and Koller (2017) previously demonstrated that MGs without head movement could be parsed in O(n 2k+3 ) worst case time, which was already a dramatic improvement over Harkema's original result.

stanojević (2019) shows that adding head movement to Fowlie and Koller's system increases complexity to O(n 2k+9 ).


Previous works on Twitter sarcasm detection focus on the text modality and propose many supervised approaches, including conventional machine learning methods with lexical features (Bouazizi and Ohtsuki, 2015;Ptáček et al., 2014), and deep learning methods (Wu et al., 2018;Baziotis et al., 2018).

detecting sarcasm with only text modality can never be certain of the true intention of the simple tweet "What a wonderful weather!"


The model with only text modality fails to detect sarcasm as for example (c), though the word so is repeated several times in example (c).

with image and attribute modalities, our proposed model correctly detects sarcasm in these tweets.


#planetfitness #hiddenfee #mrmet Attributes: ball holding shoes little white In the example, the insulting gesture in the picture is contrast to the phrase 'thanks for'.

the model is unable to obtain the common sense that this gesture is insulting.


Our work is built on the success of deep keyphrase generation models based on neural sequence-to-sequence (seq2seq) framework (Meng et al., 2017).

existing models, though effective on well-edited documents (e.g., scientific articles), will inevitably encounter the data sparsity issue when adapted to social media.


• T2: Here, someone is referring to another person's recollection.

this text contains all the linguistic markers associated with assault disclosure.


Hashtags are often employed on social media and beyond to add metadata to a textual utterance with the goal of increasing discoverability, aiding search, or providing additional semantics.

the semantic content of hashtags is not straightforward to infer as these represent ad-hoc conventions which frequently include multiple words joined together and can include abbreviations and unorthodox spellings.


Hashtags often carry very important information, such as emotion 1 Our toolkit along with the code and data are publicly available at https://github.com/mounicam/ hashtag_master  2017), sentiment (Mohammad et al., 2013), sarcasm (Bamman and Smith, 2015), and named entities (Finin et al., 2010;Ritter et al., 2011).

inferring the semantics of hashtags is nontrivial since many hashtags contain multiple tokens joined together, which frequently leads to multiple potential interpretations (e.g., lion head vs. lionhead).


STAN small is the most commonly used dataset in previous work.

after reexamination, we found annotation errors in 6.8% 5 of the hashtags in this dataset, which is significant given that the error rate of the state-of-theart models is only around 10%.


Pre-trained contextualized word embeddings (Peters et al., 2018;Devlin et al., 2019;Radford et al., 2018) have become increasingly common in natural language processing (NLP), improving stateof-the-art results in many standard NLP tasks.

beyond standard tasks, NLP tools are also vital to more open-ended exploratory tasks, particularly in social science.


10 We can see a clear separation between the female love interests and the male protagonists and antagonists, thus identifying similar roles in the same way as a persona model.

whereas the output of a persona model is distributions over personas and vocabulary, our system outputs scores along known dimensions of power, agency, and sentiment, which are easy to interpret and visualize.


In the raw scores, stand-out powerful people include businessman Warren Buffet and Pope Francis.

the only 3 women, Theresa May, Janet Yellen, and Angela Merkel, are underscored as compared to similarly ranked men.


As expected, popular players Serena Williams and Andy Murray have the highest sentiment scores and very high power scores.

novak Djokovic, who has notoriously been less popular than his peers, has the lowest sentiment score, but the second highest power score (after Williams).


Additionally, female players are typically portrayed with more positive sentiment (female average score = 0.58; male average score = 0.54), whereas male players are portrayed with higher power (female average score = 0.52; male average score = 0.57).

the difference in power disappears when we remove frequency from the metric and use only the regression scores, suggesting that the difference occurs because male players are mentioned more frequently.


Typically, given the claims, models are learned from auxiliary relevant sources such as news articles or social media responses for capturing words and linguistic units that might indicate viewpoint or language style towards the claim Rashkin et al., 2017;Popat et al., 2017;Dungs et al., 2018).

the factuality of a claim is independent of people's belief and subjective language use, and human perception is unconsciously prone to misinformation due to the common cognitive biases such as naive realism (Reed et al., 2013) and confirmation bias (Nickerson, 1998).


This may be suitable for news data as the salient news sentences often straightforwardly comment on the claim's veracity.

some claim verification tasks such as FEVER (Thorne et al., 2018a) are particularly defined to classify if the factual evidence from the source like Wikipedia, which rarely remark on the veracity of the mutated claim, can infer the claim as being supported, refuted or NEI.


Instead of ranking sentences into the top-k positions, we pay more attention on claim verification accuracy by embedding and aggregating the useful sentences as evidence like we have explained above.

such discrepancy inspires us to investigate in the future an end-to-end approach to jointly model evidence retrieval and claim verification in a unified framework based on our sentence-level attention mechanism.


$$) as a label, since 2 is the class of the restaurant that both authors visited most.

as can be seen from the column labels, the first reviewer visited restaurants belonging to all four classes, while the second one only visited restaurants of class 2: the second reviewer is clearly a less noisy data point.


Due to the nature of the labels (ranging across four classes related to increasing price), this could be seen as an ordinal regression problem.

following standard practice within the author profiling literature (Rangel Pardo et al., 2015;Rangel et al., 2016), especially regarding modelling age (where real values are binned into discrete classes), we treat this as a classification task.


Similar to the text classification results, performance on random and event splits are comparable.

there is a sharp drop in performance for time split.


Clearly the performance improves as more social links are available.

even with little social links provided in the latter case, our joint model propagates information effectively and results in an increase in performance compared to text classification.


Regarding the representativeness of our sample from the population of celebrities, we may cautiously claim to have obtained a wide cross-section of people of elevated status.

celebrities are excluded who do not use Twitter, whose account is not verified (which is exceedingly unlikely, the more famous they are), or who have no Wikipedia article about themselves.


In recent centuries, industrialization has considerably changed human society by providing a stimulus to economic growth and improved life quality.

the advancement is accompanied by the increase in air pollutant emissions and risks to public health.


We achieve higher accuracy (55.3%) as compared to tweets BOW (54.6%).

the model is less effective than using follow Bio BOW.


An end-to-end trainable dialog system requires thousands of dialogs for training.

the availability of the training data is usually limited as real users have to be involved to obtain the training dialogs.


Some domain adaptation work has been done on dialog states tracking (Mrkšić et al., 2015) and dialog policy learning (Vlasov et al., 2018) as well.

there is no recent work about domain adaptation for a seq2seq dialog system, except ZSDG Zhao and Eskénazi (2018).


Our full model produces more non-singleton coreference chains, suggesting greater coherence, and also gives different mentions of the same entity more diverse names.

both numbers are still lower than for human generated stories, indicating potential for future work.


The fusion model does not perform any implicit coreference to associate the allergy with his dog.

coreference entity fill produces a high quality completion.


Our method can complement and correct original MR with additional slot values described in the paired texts to effectively alleviate generating contradicted facts.

due to the imperfection of NLU model, our method may ignore part of slot values realized in utterances and produce some additional errors.


Previous work focuses on automatic commenting based solely on textual content.

in real-scenarios, online articles usually contain multiple modal contents.


(2015)) and copy mechanism (Gu et al., 2016) into a sequential generative architecture.

most models tended to confound the dialog history with KB tuples and simply stored them into one memory.


Thus the most straightforward method is to use T 0 as sem [x, Q, clues].

the last few layers in BERT are mainly in charge of transforming hidden representations for span predictions.


There is at most one answer span (y, start, end) in every paragraph, thus gt start ans is an one-hot vector where gt start ans [start] = 1.

multiple different next-hop spans might appear in one paragraph, so that gt start hop [start i ] = 1/k where k is the number of next-hop spans.


Once combined with information retrieval, our model finally gets the answer "Marijus Adomaitis" while the annotated ground truth is "Ten Walls".

when backtracking the reasoning process in cognitive graph, we find that the model has already reached "Ten Walls" and answers with his real name, which is acceptable and even more accurate.


Many neural models have been proposed to tackle the machine RC/QA problem (Seo et al., 2016;Xiong et al., 2016;Tay et al., 2018), and great success has been achieved, especially after the release of the BERT (Devlin et al., 2018).

current research mainly focuses on machine RC/QA on a single document or paragraph, and still lacks the ability to do reasoning across multiple documents when a single document is not enough to find the correct answer.


A lot of systems have been proposed to solve the multi-hop RC problem with these data sets (Sun et al., 2018;Wu et al., 2019).

these data sets require multi-hop reasoning over multiple sentences or multiple common knowledge while the problem we want to solve in this paper requires collecting evidences across multiple documents.


Such datasets emphasize the role of locating, matching, and aligning information between the question and the context.

some recent multi-document, multi-hop reading comprehension datasets, such as WikiHop and MedHop (Welbl et al., 2017), have been proposed to further assess MRC systems' ability to perform multi-hop reasoning, where the required evidence is scattered in a set of supporting documents.


Specifically, it encodes the leaf document of the reasoning chain while attending to its ancestral documents, and outputs ancestor-aware word representations for this leaf document, which are compared to the query to propose a candidate answer.

these two components above cannot handle questions that allow multiple possible reasoning chains that lead to different answers, as shown in Fig.


Our best system achieves 60.3 on the hidden test set 11 , outperforming all current models on the leaderboard.


The last few years have witnessed significant progress on text-based machine reading comprehension and question answering (MRC-QA) including cloze-style blank-filling tasks (Hermann et al., 2015), open-domain QA (Yang et al., 2015), answer span prediction (Rajpurkar et al., 2016(Rajpurkar et al., , 2018, and generative QA (Nguyen et al., 2016).

all of the above datasets are confined to a single-document context per question setup.


However, this 1-hop, similarity-based selection process would fail on multi-hop readingcomprehension datasets like WikiHop because the query subject and the answer could appear in different documents.

our Document Explorer can discover the document with the answer "Loon op Zand" (in Fig.


(2018) selected relevant sentences from long documents in a singledocument setup and achieved faster speed and robustness against adversarial corruption.

none of these models are built for multi-hop MRC where our EPAr system shows great effectiveness.


3) combines the control unit from the stateof-the-art multi-hop VQA model and the widelyadopted bi-attention mechanism from text-based QA to perform composite reasoning on the context and question.

bridge Entity Supervision even with the multi-hop architecture to capture a hopspecific distribution over the question, there is no supervision on the control unit's output distribution cv about which part of the question is important to the current reasoning step, thus preventing the control unit from learning the composite reasoning skill.


Intuitively, in context-based path encodings, limited and more fine-grained context is considered due to the use of specific entity locations.

the passage-based path encoder computes the path representations considering the entire passage representations (both passages which contain the head entity and tail entity respectively).


Empirically, WMD has improved the performance of NLP tasks (see §6), specifically sentence-level tasks, such as image caption generation (Kilickaya et al., 2017) and natural language inference (Sulea, 2017).

its cost grows prohibitively as the length of the documents increases, and the BOW approach can be problematic when documents become large as the relation between sentences is lost.


(2014) use automatic suggestions for de-identification of medical texts and find no change in inter-annotator agreement or annotation time.

to these works, we use a control group of two annotators, who never receive suggestions, and compare the performance of all annotators to previous annotations they performed without annotation suggestions.


Thus, there is little difference to reference phase S1, where SUGGESTION already yielded a 0.02 higher IAA.

below we discuss the helpfulness and time saving of suggestions even in MeD.


The previous section established that annotation suggestions have positive effects on annotating epistemic activities.

these suggestions were only possible since Schulz et al.


In fact, in the CUM online learning setup (bundle size 1), the model adjustment time is similar to, and after 100 documents higher than the time needed for annotation (1-2 minutes per text as shown in Table 5).

the adjustment times of CUM with a bundle size of 10 or higher and of RETRAIN, are lower than the time needed for annotating the respective bundle of texts.


Comparing between Deep Neural Network (DNN) models based on their performance on unseen data is crucial for the progress of the NLP field.

these models have a large number of hyper-parameters and, being non-convex, their convergence point depends on the random values chosen at initialization and during training.


the learning rate and the optimization algorithm) the range of feasible values reflects the intuitions of the model author, and the tuned value provides some insight about the model and the data.


In such cases we would not want to determine that X ∼ F is stochastically dominant over Y ∼ G because its CDF is strictly above the CDF of Y , and hence Y is stochastically larger than X.

according to this relaxation, X ∼ F is indeed stochastically larger than Y ∼ G. Almost Stochastic Dominance To overcome the limitations of the above straight forward approach, and define a relaxation of stochastic order, we turn to a definition that is based on the proportion of points in the domain of the participating distributions for which SO holds.


When min (F n , G m , α) = 0, algorithm A is stochastically dominant over B.

if min (F n , G m , α) ≥ 0.5, then F is not almost stochastically larger than G (with confidence level 1 − α) and hence we should accept the null hypothesis that algorithm A is not superior to algorithm B. del Barrio et al.


Note that the opposite direction does not always hold, i.e., it is easy to come up with an example where P (X ≥ Y ) > 0.5 but there is no stochastic order between the two random variables.

the opposite direction is true with an additional assumption that the CDFs do not cross one another (which we do not prove here).


Shallow surfacelevel metrics, such as BLEU and TER (Papineni et al., 2002;Snover et al., 2006), still predominate in practice, due in part to their reasonable correlation to human judgements, and their being parameter free, making them easily portable to new languages.

trained metrics (Song and Cohn, 2011;Stanojevic and Sima'an, 2014;Ma et al., 2017;Shimanaka et al., 2018), which are learned to match human evaluation data, have been shown to result in a large boost in performance.


We find that on the same number of training instances (3360), the model performs better on cleaner data compared to singly-annotated data (r = 0.57 vs 0.64).

when we have a choice between collecting multiple annotations for the same instances vs collecting annotations for additional instances, the second strategy leads to more gains.


Among neural models, our BILSTM+MEM and BILSTM+BIA models outperform other comparisons by successfully modeling users' previous messages and their alignment with the topics of ongoing conversations.

the opposite observation is drawn for BILSTM+CON and BILSTM+ATT.


secularism allows every citizen to freely profess his faith.


Several keywords from text (cat, game, winter) show types of content which are present in both image and text, but the image is merely an illustrating these concepts without adding additional information (Figure 5a).

the text is not represented in the image when it contains words specific of comments (when, lmao), questions (do, was), references (this) or ellipsis ('...'), all often referencing the content of the image as identified through data inspection.


Platforms such as Twitter have started to play a key role in elections (Politico, 2017) and have become widely used by public figures to disseminate their activities and opinions.

posts are rarely authored by the public figure who owns the account; rather, they are posted by staff who update followers on the thoughts, stances and activities of the owner.


This line of work exploits textual redundancy in massive streams predicting whether or not a document contains a new event as a clas-sification task.

we study the event schemas and extract detailed events.


To address this issue, the distant supervision method (Mintz et al., 2009) was proposed which annotated training data by heuristically aligning knowledge bases (KBs) and texts.

the long-tail problem in KBs (Xiong et al., 2018;  1: A data example of 5-way-5-shot relation classification in FewRel development set.


Similar to conventional prototypical networks (Snell et al., 2017), our proposed method calculates class prototype s via the representations of all support instances in this class, i.e., { s k } K k=1 .

instead of using a naive mean operation, we aggregate instance-level representations via attention over { s k } K k=1 , where each weight is derived from the instance matching score between s k and q.


Specifically, we adopt Kullback-Leibler (KL) divergence of both directions as the metric.

computing exact KL requires iterating over the whole entity pair space E × E, which is quite intractable.


As shown in Figure 2, the three baseline models could achieve moderate (0.1-0.5) positive correlation.

our model shows a stronger correlation (0.63) with human judgment, indicating that considering the probability over whole entity pair space helps to gain a similarity closer to human judgments.


Whenas, these Open RE methods adopt distantly supervised labels as golden relation types, suffering from both false positive and false negative problems on the one hand.

these methods still rely on the conventional similarity metrics mentioned above.


A promising method is to use rejection sampling by uniform sampling from the whole space, and only keep the synonymous ones judged by crowdworkers.

this is not practical either, as the synonymous pairs are sparse in the whole space, resulting in low efficiency.


2017, who use bandits to minimise the number of data queries required to calculate the F-scores of models.

this work does not consider the stochasticity of the resulting estimates or easily extend to other evaluation metrics.


This is encouraging, indicating that in these cases, i * and r are nearly "tied" in attention.

the picture of attention's interpretability grows somewhat more murky when we begin to consider the magnitudes of positive ∆JS values in Figure 3.


This is likely explained in part by the signal pertinent to the classification being distributed across a document (e.g., a "Sports" question in the Yahoo Answers dataset could signal "sports" in a few sentences, any one of which suffices to correctly categorize it).

given that these results are for the HAN models, which typically compute attention over ten or fewer sentences, this is surprising.


Ideally, we would then enumerate all possible subsets of that instance's components, observe whether the model's decision changed in response to removing each subset, and then report whether the size of the minimal decision-flipping subset was equal to the number of items that had needed to be removed to achieve a decision flip by following the ranking.

the exponential number of subsets for any given instance's sequence of components (word or sentence representations, in our case) makes such a strategy computationally prohibitive, and so we adopt a different approach.


diagnostic models answer a more specific question: to what extent a particular type of information can be extracted from a given representation.


One frequently resorts to largely qualitative evaluation: checking whether the conclusions reached via a particular approach have face validity and match pre-existing intuitions.

pre-existing intuitions are often not reliable when it comes to complex neural models applied to also very complex natural language data.


Parameter estimation for this model can be done by maximizing a lower bound E(φ, θ) on the loglikelihood of the data derived by application of Jensen's inequality: 2 (3) These latent rationales approach the first objective, namely, uncovering which parts of the input text contribute towards a decision.

note that an NN controls the Bernoulli parameters, thus nothing prevents this NN from selecting the whole of the input, thus defaulting to a standard text classifier.


Machine learning research has been focusing more and more on interpretability (Gilpin et al., 2018).

there are many nuances to interpretability (Lipton, 2016), and amongst them we focus on model transparency.


Empirically, previous work has found that LSTM language models use 200 context words on average (Khandelwal et al., 2018), indicating room for further improvement.

the direct connections between long-distance word pairs baked in attention mechanisms might ease optimization and enable the learning of long-term dependency (Bahdanau et al., 2014;Vaswani et al., 2017).


Previous work in the context of phrase-based statistical machine translation (Daumé III and Jagarlamudi, 2011) has noted that unseen (OOV) words account for a large portion of translation errors when switching to new domains.

this problem of OOV words in cross-domain transfer is under-examined in the context of NMT, where both training methods and experimental results will differ greatly.


Specifically, in this paper we tackle the task of data-based, unsupervised adaptation, where representative methods include creation of a pseudoparallel corpus by back-translation of in-domain monolingual target sentences (Sennrich et al., 2016a), or construction of a pseudo-parallel indomain corpus by copying monolingual target sentences to the source side (Currey et al., 2017).

while these methods have potential to strengthen the target-language decoder through addition of in-domain target data, they do not explicitly provide direct supervision of domainspecific words, which we argue is one of the major difficulties caused by domain shift.


The basic ideas are to store a handful of previous source or target sentences with context vectors (Jean et al., 2017;Wang et al., 2017a) or memory components (Maruf and Haffari, 2018;.


And to achieve localization, the coefficients corresponding to anchor points out of the neighbors of h M are set to zero.

it is hard to train in deep neural network using stochastic gradient methods.


The possible reason is that all the related work only leverage a small range of the documentlevel information, limited by model complexity    and time consuming.

our models are capable to express all information with more abstract representations.


Translation given by NMT is not readable.

m-RefNet generates the core verb "strengthened" and B-RefNet provides a more accurate collocation "stepped up patrols".


propose to modify the NMT with light-weight key-value memory to store the translation history.

due to the limitation of the memory size, the very short view on the previous (25 timesteps) is not sufficient to model the document-level contextual information.


The Reinforce-NAT is proposed on the basis that the top-k words can occupy the central part of the probability distribution.

it remains unknown which k is appropriate for us.


We can see that while on BLEU-vs-AL plots, their models perform similarly to our test-time wait-k for de→en and zh→en, and slightly better than our test-time wait-k for en→zh, which is reasonable as both use a full-sentence model at the very core.

on BLEU-vs-CW plots, their models have much worse CWs, which is also consistent with results in their paper (Gu, p.c.).


In the ideal case where the input and output sentences have equal length, the translation will finish k steps after the source sentence finishes, i.e., the tail length is also k. This is consistent with human interpreters who start and stop a few seconds after the speaker starts and stops.

input and output sentences generally have different lengths.


estimating θ h is difficult given their discrete nature.


(2016) to explain the decision of text classifier.

here we focus on selecting a few relevant tokens from a source sequence in a translation task.


Not surprisingly, data augmentation significantly improves the robustness of NMT models to homophone noises.

the noises in training data seem to hurt the performance of the baseline model (from 45.97 to 43.94), and its effect on our model seems to be much smaller, probably because our model mainly uses the phonetic information.


Meanwhile, there have been many successes of transfer learning for NLP: models such as CoVe (McCann et al., 2017), ELMo (Peters et al., 2018), OpenAI GPT (Radford et al., 2018), ULM-FiT (Howard and Ruder, 2018), and BERT (Devlin et al., 2019) obtain powerful representations by training large-scale language models and use them to improve performance in many sentence-level and word-level tasks.

a language generation task such as APE presents additional challenges.


They also used segment embeddings (along with word and position embeddings) to differentiate between a pair of sentences in different languages.

this is only used in one of the pre-training phases of the language model (translation language modelling) and not in the downstream task.


Once we have the translation options for tokens in the source vocabulary we can perform a word by word translation of the source into Translationese.

a naive translation of each source token to its top translation option without considering the context is not the best way to go.


We performed experiments in a zero-shot setting, showing that the copy behaviour is triggered at test time with terms that were never seen in training.

to constrained decoding, we have also observed that the method exhibits flexible use of terminology as in some cases the terms are used in their provided form while other times inflection is performed.


By default, the hidden states of each word are hierarchically calculated by attending to all words in the sentence, which assembles global information.

several studies pointed out that taking all signals into account may lead to overlooking neighboring information (e.g.


As seen, TRANSFORMER which seeks more global information outperforms other models in both the "WC" and "CoIn" tasks.

modeling locality is beneficial to "SeLn", "ToCo", "BShif" and "Tense" tasks.


It is based on the estimation of a probability distribution over all input words for each target word.

source and target words are in different representation space, and they still have to go through a long information processing procedure that may lead to the source words are incorrectly translated into the target words.


(2017) incorporated a reconstructor module into NMT, which reconstructs the input source sentence from the hidden layer of the output target sentence to enhance source representation.

in previous studies, the training objective function was usually based on word-level and lacked explicit sentencelevel relationships .


Compared with Row 1 and 4, loss mse + enhanced (Row 3 and 6) increases little parameters about 0.6M and 2.1M, train and decode speed drop very little.

it has greatly improved the translation performance.


In absence of such huge amount of parallel data, NMT systems tend to perform poorly (Koehn and Knowles, 2017).

nMT without using any parallel data such as bilingual translations, bilingual dictionary or comparable translations, has recently become reality and opened up exciting opportunities for future research Artetxe et al., 2018;Yang et al., 2018).


To translate between many languages using bilingual version of unsupervised NMT, we require an encoder and one  or two (Artetxe et al., 2018) decoders for each pair of languages.

we may not need separate decoders depending on the source language.


In supervised multilingual NMT, specifically for one-to-many translation directions, this consistency is absent in some existing works (Dong et al., 2015;Firat et al., 2016;Johnson et al., 2017).

in this work, we find that using shared encoder with fixed cross-lingual embedding improves performance in all the translation directions.


In English it means "The vice president of Trade Development Council" NMT can be factorized in character (Costa-Jussa and Fonollosa, 2016), word (Sutskever et al., 2014), or subword (Sennrich et al., 2015) level.

only using 1-best segmentation as inputs limits NMT encoders to express source sequences sufficiently and reliably.


(2016) propose languageadversarial training that does not directly depend on parallel corpora, but instead only requires a set of bilingual word embeddings (BWEs).

the multilingual transfer setting, although less explored, has also been studied (McDonald et al., 2011;Naseem et al., 2012;Täckström et al., 2013;Hajmohammadi et al., 2014;Zhang and Barzilay, 2015;Guo et al., 2016), showing improved performance compared to using labeled data from one source language as in bilingual transfer.


The idea is to have a set of language expert networks, one per source language, each responsible for learning language-specific features for that source language during training.

instead of hard-switching between the experts, each sample uses a convex combination of all experts, dictated by an expert gate.


In particular, the trainon-trans(lation) method translates the entire English training set into each target language which are in turn used to train a supervised system on the target language.

the test-ontrans(lation) method trains an English sequence tagger, and utilizes MT to translate the test set of each target language into English in order to make predictions.


On the Amazon dataset, it can be seen that when transferring to German or French (from the remaining three), the Japanese expert is less utilized compared to the European languages.

it is interesting that when transferring to Japanese, the French and English experts are used more than the German one, and the exact reason remains to be investigated.


The expert gate is a linear transformation (matrix) of size 128 × N , where N is the number of source languages.

the architecture of the task specific predictor C depends on the task.


The LSTMs − → f and ← − f are shared among multiple languages and capture a common language structure.

the word embeddings E and linear projection W are specific to each language .


In particular, many of the German words are mapped near the centre of the figure and make a large cluster.

the word embeddings trained by our model are not clustered by language, indicating that our model successfully maps word embeddings into a common space.


For POS tagging, the two most important features are dataset size and the TTR distance.

the lack of rich dataset-dependent features for the EL task leads to the geographic and syntactic distance being most influential.


On the one hand, cognate identification has been studied within linguistic typology and historical linguistics.

computational linguists have been researching methods for cognate production.


Then, we take these similarities as the weights to calculate an attentive vector for the entire graph G 2 by weighted summing all the entity embeddings of G 2 .ē We calculate matching vectors for all entities in both G 1 and G 2 by using a multi-perspective cosine matching function f m at each matching step (See Appendix A for more details): Graph-Level (Global) Matching Layer Intuitively, the above matching vectors (m att s) capture how each entity in G 1 (G 2 ) can be matched by the topic graph in the other language.

they are local matching states and are not sufficient to measure the global graph similarity.


It is motivated by the triangular NMT systems with pseudo target in the teacher-student networks.

we use pseudo source and apply different teacher-student networks.


In both cases, we first solve the translation problem, and the task is transformed to the monolingual setting.

while conceptually simple, the performance of this modular approach is fundamentally limited by the quality of machine translation.


(2018) showed that these spaces are, in general, far from being isomorphic, and thus they result in suboptimal or degenerated unsupervised mappings.

supervised methods that jointly train BWE from scratch (Upadhyay et al., 2016), on parallel or comparable corpora, do not have such limits since no pre-existing embedding spaces and no mapping function are involved.


These methods jointly train BWE by exploiting bilingual and monolingual contexts of words, materialized by sentence or document pairs, to learn a single BWE space.

they require large bilingual resources for training.


Bilingual lexicon induction (BLI) is by far the most popular evaluation task for BWE used by previous work in spite of its limits (Glavas et al., 2019).

to previous work, we used much larger test sets 10 for each language pair.


Again, BIVEC and SENTID performed similarly.

note that here USMT is merely an evaluation task: the improvement observed at step 0 are practically useless for USMT, since we can often gain much larger improvements through refinement as described in Section 2.2.


We also observed a lower accuracy when using original English, presumably due to the use of much smaller data than for training VECMAP.

when training monolingual word embeddings using fastText on the same English data used for training BIVEC, we observed that fastText underperforms BIVEC.


For instance, in our Hearst Graph the relation (male horse, is-a, equine) is missing.

since we correctly model that (male horse, is-a, horse) and (horse, is-a, equine), by transitivity, we also infer (male horse, is-a, equine), which SVD fails to do.


It is because the OOV issue is not severe for the char-based model and thus does not affect the performance much.

as we remove more and more training examples, the shrinking training dataset creates a bigger problem.


Assuming that the words have a uniform distribution, the paraphrase of C can then be written as an unweighted sum of its context vectors.

this uniformity assumption is unrealistic -word frequencies obey a Zipf distribution, which is Pareto (Piantadosi, 2014).


Therefore, we must also require that x 1 , y 1 , x 2 , y 2 be coplanar.

we do not need the word embeddings themselves to verify coplanarity; when there is no reconstruction error, we can express it as a constraint over M, the matrix that is implicitly factorized by the embedding model (see Definition 5).


This claim has since been repeated in other work (Arora et al., 2016).

for example, according to this conjecture, the analogy (king,queen)::(man,woman) holds iff for every word w in the vocabulary as noted earlier, this idea was neither derived from empirical results nor rigorous theory, and there has been no work to suggest that it would hold for models other than GloVe, which was designed around it.


Compositionality is one of the strongest assumptions in semantics, stating that the meaning of larger units can be derived from their smaller parts and their contextual relation.

for idiomatic phrases, this assumption does not hold true as the meaning of the whole phrase may not be related to their parts in a straightforward fashion.


In addition, this supervised approach requires an additional resource of ∼ 70k known noun phrases from Wikipedia for training.

compare their best models with all these baseline models and show that their models outperform across all the respective datasets.


These embeddings, therefore, are not favorable in applications that demand strong synonym identification.

supervised or semi-supervised representation leaning requires annotated corpus, such as paraphrastic sentences or natural language inference data (Conneau et al., 2017;Wieting and Gimpel, 2017;Subramanian et al., 2018;Cer et al., 2018).


The difference is that we focus on learning representation for multi-word concept names, hence the contextual and conceptual constraints are essential, in addition to the synonymous similarity.

most retrofitting approaches mainly aim to improve word representations.


The knowledge can be used to infer quality representations for new synonyms.

similar to skip-gram baselines, BNE faces serious challenges if the names are unpopular and contain words that do not reflect their conceptual meanings.


As shown in Table 2, SG W +WMD outperforms Jaccard baseline (in MAP score), mainly because of its ability to capture semantic matching.


However, both baselines are non-parametric.

bNE+SG W learns additional knowledge about the synonym matching by using synonyms sets in UMLS as training data.


Hand crafted lexical databases, such as Word-Net (Miller, 1995), have been built and maintained to be used in NLP and other fields containing antonyms, synonyms and other lexical semantic relations.

its construction and maintenance takes a considerable human effort and it is difficult to achieve a broad coverage.


At first glance, synonymy and antonymy can be seen as binary relations between words.

based on empirical results 2 Edmundson defined synonymy and antonymy as ternary relations in order to consider the multiple senses of the words, as follows: xS i y ≡ x synonym of y according to sense i xA i y ≡ x antonym of y according to sense i Note that the senses of the words are represented in the relationship rather than in the words themselves.


For instance in Figure 1, a relevant context word discover for Mars is missed if the chosen window size is less than 3.

a large window size might allow irrelevant words to influence word embeddings negatively.


(2016) are restricted to handling symmetric relations like synonymy and antonymy.

although recently proposed (Alsuhaibani et al., 2018) is capable of handling asymmetric information, it still requires manually defined relation strength function which can be labor intensive and suboptimal.


This allowed us to compare our model with existing approaches that use von Mises-Fisher distributions for document modelling.

to our method, these models are based on topic models (e.g.


Unsupervised word embeddings have become a popular approach of word representation in NLP tasks.

there are limitations to the semantics represented by unsupervised embeddings, and inadequate fine-tuning of embeddings can lead to suboptimal performance.


Models such as skip-gram (Mikolov et al., 2013a) and Glove (Pennington et al., 2014) capture the statistics of a large corpus and have good properties that corresponds to the semantics of words (Mikolov et al., 2013b).

there are certain problems with unsupervised word embeddings, such as the difficulty in modeling some fine-grained word semantics.


There have been many follow-ups on this concept of identifying and utilizing patterns to identify hypernym pairs (Caraballo, 1999;Mann, 2002;Snow et al., 2005Snow et al., , 2006.

by restricting the sentences of interest to only those which match patterns, even very large datasets with very loose pattern matching will often return small co-occurrence numbers, especially for more indirectly connected hypernym pairs.


The most consistently used evaluation approach is comparison of the summaries produces against reference summaries via automatic measures such as ROUGE (Lin, 2004) and its variants.

automatic measures are unlikely to be sufficient to measure performance in summarization (Schluter, 2017), also known for other tasks in which the goal is to generate natural language (Novikova et al., 2017).


The relative assessment is often done using the paired comparison (Thurstone, 1994) or the best-worst scaling (Woodworth and G, 1991;Louviere et al., 2015), to improve inter-annotator agreement.

absolute assessment of summarization (Li et al., 2018b;Song et al., 2018;Kryściński et al., 2018;Hsu et al., 2018;Hardy and Vlachos, 2018) is often done using the Likert rating scale (Likert, 1932) where a summary is assessed on a numerical scale.


(2018b,c) also falls in this category, as the questions were written using the reference summary.

summarization datasets are limited to a single reference summary per document (Sandhaus, 2008;Hermann et al., 2015;Grusky et al., 2018;Narayan et al., 2018b) thus evaluations using them is prone to reference bias (Louis and Nenkova, 2013), also a known issue in machine translation evaluation (Fomicheva and Specia, 2016).


The second component, which evaluates the notions of "Precision" and "Recall" requires the highlights from the first one to be conducted.

the highlight annotation needs to happen only once per document, and it can be reused to evaluate many system summaries, unlike the Pyramid approach (Nenkova and Passonneau, 2004) that requires additional expert annotation for every system summary being evaluated.


Perhaps unsurprisingly human-authored summaries were considered best, whereas, TCONVS2S was ranked 2nd, followed by PT-GEN.

the performance difference in TCONVS2S and PTGEN is greatly amplified when they are evaluated against document with highlights (6.48 and 5.54 Precision and Recall points) compared to when evaluated against the original documents (3.98 and 1.83 Precision and Recall points).


To solve the out-of-vocabulary (OOV) problem, conventional Seq2Seq models utilize a copy mechanism (Gu et al., 2016) that selects a word from source (complex) sentence directly with a trainable pointer.

editNTS has the ability to copy OOV words into the simplified sentences by directly learning to predict KeeP on them in complex sentences.


These external rules can provide reliable guidance about which words to modify, resulting in higher add/keep F1 scores (Table 5-a).

our model is inclined to generate shorter sentences, which leads to high F1 scores on delete operations 8 .


The proposed model can be trained end-to-end by maximizing the conditional probability (3).

learning from scratch may not be informative for the separator and aggregator to disentangle the paraphrase patterns in an optimal way.


Based on the observation that the sentence templates generated by DNPG tend to be more general and domaininsensitive, we consider directly performing the sentential paraphrase in the target domain as a solution.

the language model of the source and target domains may differ, we therefore finetune the separator of DNPG so that it can identify the granularity of the sentence in the target domain more accurately.


Meanwhile, we notice that there is a considerable amount of work on domain adaptation for neural machine translation, another classic sequence-to-sequence learning task.

most of them require parallel data in the target domain (Wang et al., 2017a,b).


Its goal is to improve the readability of a text, making information easier to comprehend for people with reduced literacy, such as non-native speakers (Paetzold and Specia, 2016), aphasics (Carroll et al., 1998), dyslexics (Rello et al., 2013) or deaf persons (Inui et al., 2003).

not only human readers may benefit from TS.


As noted iň Stajner and Glavaš (2017), data-driven approaches outperform rule-based systems in the area of lexical simplification (Glavaš andŠtajner, 2015;Paetzold and Specia, 2016;Nisioi et al., 2017;Zhang and Lapata, 2017).

the state-of-the-art syntactic simplification approaches are rule-based (Siddharthan and Mandya, 2014;Ferrés et al., 2016;Saggion et al., 2015), providing more grammatical output and covering a wider range of syn-tactic transformation operations, however, at the cost of being very conservative, often to the extent of not making any changes at all.


A complex input sentence is transformed into a semantic hierarchy of simplified sentences in the form of minimal, self-contained propositions that are linked via rhetorical relations.

to above-mentioned end-to-end neural approaches, we followed a more systematic approach.


When training the best-performing model of Aharoni and Goldberg (2018) on this new split-and-rephrase dataset, they achieve a strong improvement over prior best results from Aharoni and Goldberg (2018).

due to the uniform use of a single split per source sentence in the training set, each input sentence is broken down into two output sentences only.


Since the latter commonly express minor information, we denote them context sentences.

the former are of equal status and typically depict the key information contained in the input.


With a score of 1.30 on the Wikilarge sample sentences, it is far ahead of the baseline approaches, with HYBRID (0.86) coming closest.

this system receives the lowest scores for G and M. RegenT obtains the highest score for G (4.64), while YATS is the best-performing approach in terms of M (4.60).


There are some examples in MNLI that contradict the heuristics in ways that are not easily explained away by other heuristics; see Appendix A for examples.

such cases are likely too rare to discourage a model from learning these heuristics.


Such successes suggest that BERT is able to learn from some specific subcases that it should rule out the broader heuristics; in this case, the nonwithheld cases plausibly informed BERT not to indiscriminately follow the constituent heuristic, encouraging it to instead base its judgments on the specific adverbs in question (e.g., certainly vs. probably).

the models did not always transfer successfully; e.g., BERT had 0% accuracy on entailed passive examples when such examples were withheld, likely because the training set still included many non-entailed passive examples, meaning that BERT may have learned to assume that all sentences with passive premises are cases of non-entailment.


Candidate generation (Recall@64) is poor in the Low Overlap category.

the ranking model performs in par with other hard categories for these mentions.


The reliance on this trick illustrates the point we make in the Introduction: syntactic distance has the advantage of being a continuous value, which can be computed as an attention score in a differentiable model.

this comes at a price: the PRPN does not model trees or tree-building operations directly.


In the setting without punctuation, the PRPN sets an initial policy that agrees fairly well with right-branching, and this rightbranching bias is reinforced by imitation learning and policy refinement.

in the setting with punctuation, the agreement with rightbranching changes in the opposite way.


It has been argued that NLI as currently formulated is not a difficult task (Poliak et al., 2018); this is presumably why models can perform well across a range of different tree structures, only some of which are syntactically plausible.

this does not imply that the Tree-LSTM will learn nothing when trained with NLI.


In order to address gender bias in part-of-speech (POS) tagging and dependency parsing, we first require an adequate size data set labeled for a) syntax along with b) gender information of the authors.

existing data sets fail to meet both criteria: data sets with gender information are either too small to train on, lack syntactic information, or are restricted to social media; sufficiently large syntactic data sets are not labeled with gender information and rely (at least in part) on news genre corpora such as the Wall Street Journal (WSJ).


Inducing Chinese sentiment lexicons (Wang and Ku, 2016) needs properly tokenized corpora, which is not a hard requirement in Swedish.

we aim to design a method applicable to typologically diverse languages and we apply it to 1500+ languages.


Columns (i) and (ii) of Table 6 show that REG ( §3.4) delivers results comparable to Densifier (ORTH) when using the same set of generic training words (GEN) in lexicon induction.

our method is more efficient -no need to compute the expensive SVD after every batch update.


There have been methods that classify each phrase independently (Li et al., 2015;McCann et al., 2017).

sentiments over hierarchical phrases can have dependencies.


In the first sentence, the phrase "seemed static" itself bares the neutral sentiment.

it has a negative sentiment in the context.


Some of these models can be used to generate sentiment-tagged synthetic text.

most of them are not directly suitable for generating bilingual code-mixed text, due to the unavailability of sufficient volume of gold-tagged codemixed text.


We can also use Euclidean distance as d i,j .

this method requires multilingual word embeddings for every word to calculate the distance.


As a fine-grained sentiment analysis task, Aspect Sentiment Classification (ASC) aims to predict sentiment polarities (e.g., positive, negative, neutral) towards given particular aspects from a text and has been drawing more and more interests in natural language processing and computational linguistics over the past few years (Jiang et al., 2011;Tang et al., 2016b;.

most of the existing studies on ASC focus on individual non-interactive reviews, such as customer reviews (Pontiki et al., 2014) and tweets (Mitchell et al., 2013;Vo and Zhang, 2015;Dong et al., 2014).


A well-behaved approach to ASC-QA should match each question and answer bidirectionally so as to correctly determine the sentiment polarity towards a specific aspect.

different from common QA matching tasks such as question-answering (Shen et al., 2018a), ASC-QA focuses on extracting sentiment information towards a specific aspect and may suffer from much aspect-irrelevant noisy information.


Automatic and human evaluations show that an abstractive model trained with a multi-task objective outperforms conventional Seq2Seq, language modeling, as well as a strong extractive baseline.

our best model is still far from human performance since raters prefer gold responses in over 86% of cases, leaving ample opportunity for future improvement.


We leverage evidence queried from the web for each question.

to previous datasets where the human written answer could be found with lexical overlap methods (Weissenborn et al., 2017), ELI5 poses a significant challenge in siphoning out important information, as no single sentence or phrase contains the full answer.


We show this approach outperforms conventional Seq2Seq and language modeling, as well as a strong extractive baseline based on BidAF (Seo et al., 2017) but generalized to multi-sentence output.

our best-performing model is still far from the quality of human written answers, with raters preferring the gold answers 86% of the time.


Language models are trained to predict all tokens in the question, web source, and answer.

the standard Seq2Seq model only receives training signal from predicting the answer which is much less than the language model gets.


The BidAF model is the strongest (23.5), better than TFIDF between the question and the support document to select sentences.

these approaches are limited by the support document, as an oracle computed on the full web sources achieves 54.8.


We find only 19% of multi-task model answers are fully accurate; even if the model output answers the question, it can generate a sentence with an incorrect statement.

the extractive model copies sentences from humanwritten text.


However, answering visual questions requires not only information about the visual content but also common knowledge, which is usually too hard to directly learn from only a limited number of images with human annotated answers as supervision.

comparatively little previous VQA research has worked on enriching the knowledge base.


Both of them provide significant improvements to the baseline model.

there is still a reasonable gap between generated and human captions.


Human evaluation with GT captions also shows better performance than with GenP captions as seen in Table 4.

the results from the man-  Table 3: TextQA with GT model outperforms TextQA with GenP (we run each model five times with different seeds and average the scores.


More precisely, we share the features (dimensions) between the paired source and target embeddings (vectors).

in contrast to the previous studies, we also model the private features of the word embedding to preserve the private characteristics of words for source and target languages.


The proposed method has a limitation in that each word can only be paired with one corresponding word.

synonym is a quite common phenomenon in natural language processing tasks.


In the vanilla model, as shown in Figure 6(a), only the similar monolingual embeddings are clustered, such as the English words "died" and "killed", and the Chinese words "zhuxi" (president) and "zongtong" (presi-dent).

in the proposed method, no matter whether the similar source and target words are paired or not, they tend to cluster together; as shown in Figure 6(b) and 6(c).


Abusive behavior is an omnibus term that often includes harassment, threats, racial slurs, sexism, unexpected pornographic content, and insults-all of which can be directed at other users or at whole communities Nobata et al., 2016).

nLP has largely considered a far narrower scope of what constitutes abuse through its selection of which types of behavior to recognize (Waseem et al., 2017;Schmidt and Wiegand, 2017;Fortuna and nunes, 2018).


Such community-specific norms and context are important to take into account, as NLP researchers are doubling down on context-sensitive approaches to define (e.g., Chandrasekharan and Gilbert, 2019) and detect abuse (e.g., .

not all community norms are socially acceptable within the broader world.


Giving a dialogue agent this ability would enable it to continuously improve and adapt over its lifetime, rather than requiring additional annotation costs for each and every improvement.

naively training a dialogue agent on its own conversations yields poor results.


Hashimoto and Sassano (2018) used user responses to detect mistakes made by a deployed virtual assistant, showing that model mistakes can be identified in chit-chat, weather, or web search domains.

they did not explore how to use these identified mistakes to improve the model further; their agent was not equipped to feed itself.


4 The boost in quality is naturally most pronounced when the HH DIALOGUE training set is small (i.e., where the learning curve is steepest), yielding an increase of up to 9.4 accuracy points, a 31% improvement.

even when the entire PER-SONACHAT dataset of 131k examples is used-a much larger dataset than what is available for most dialogue tasks-adding deployment examples is still able to provide an additional 1.6 points of accuracy on what is otherwise a very flat region of  For each column, the model using all three data types (last row) is significantly better than all the others, and the best model using only one type of self-feeding (FEEDBACK examples or HB DIALOGUE examples) is better than the supervised baseline in the first row (p < 0.05).


A straightforward method to introduce such a classifier is to build a sentence-level emotion discriminator as follows: where W ∈ R K×d is a weight matrix and K denotes the number of emotion categories.

it is infeasible to enumerate all possible sequences as the search space is exponential to the size of vocabulary, and the length of Y is not known in advance.


Such systems work well if all possible combinations of user inputs and conditions are considered in the training stage (Paek and Pieraccini, 2008;Wang et al., 2018).

as shown Figure 1: An example of task-oriented dialogue system.


For each element h n in bi-RNNs outputs, we compute a scalar selfattention score as follows: The final utterance representation E(x) is the weighted sum of bi-RNNs outputs: (3) After getting the encoding of each sentence in C t , we input these sentence embeddings to another GRU-based RNNs to obtain the context embedding E(C t ) as follows: In the existing work (Williams et al., 2017;Bordes et al., 2016;Li et al., 2017), after getting the context representation, the dialogue system will give a response y t to the user based on p(y t |C t ).

the dialogue system may give unreasonable responses if unexpected queries happen.


4(a), the system will refuse to respond if JSD avg is higher than a threshold τ 1 .

the dialogue model tends to give close weights to all response candidates in the early stage of training, as shown in Fig.


To simulate the new unconsidered user needs, one possible method is to delete some question types in the training set of existing datasets (e.g., bAbI tasks (Bordes et al., 2016)) and test these questions in the testing phase.

the dialogue context plays an important role in the response selection.


It is possible for the dialogue model to retrieve responses directly without any preprocessing.

the fact that nearly all utterances contain entity information would lead to a slow model convergence.


These advances in task-oriented dia-logue systems have resulted in impressive gains in performance.

prior work has mainly focused on building task-oriented dialogue systems in a closed environment.


The intended outcome is that the agent produces utterances consistent with its given persona.

these models still face the consistency issue, as shown in Figure 1.


pick aŷ such that: Unlike Markovian processes, no sub-exponential algorithm exists to find the optimal decoded sequence, and thus we instead use approximations.

arg-max The simplest approach to decoding a likely sequence is to greedily select a word at each timestep:ŷ because this deterministic approach typically yields repetitive and short output sequences, and does not permit generating multiple samples, it is rarely used in language modelling.


The goal of diverse decoding strategies is to generate high-quality candidate sequences which span as much of the space of valid outputs as possible.

we find there to be a marked trade-off between diversity and quality.


Top-10 random sampling has the highest fluency, coherence, and interestingness, as well as significantly lower perplexity than other random sampling methods.

this trend did not extend to image captioning, where top-10 random sampling results in both worse SPICE scores and lower diversity measures than setting the temperature to 0.7.


Related contents of the candidates are appropriately integrated into the response, but the model is discouraged as the response is different from the ground-truth.

rather than just provide materials for the generation, N-best response candidates also contain references for evaluating responses.


We propose a Retrieval-Enhanced Adversarial Training method for neural response generation in dialogue systems.

to existing approaches, our REAT method directly uses response candidates from retrieval-based systems to improve the discriminator in adversarial training.


To deal with the generic response problem, various methods have been proposed, including diversity-promoting objective function (Li et al., 2016a), enhanced beam search (Shao et al., 2016), latent dialogue mechanism (Zhou et al., , 2018a, Variational Autoencoders (VAEs) based models Serban et al., 2017), etc.

these methods still view multiple responses as independent ones and fail to model multiple responses jointly.


incorporate topic information to generate informative responses.

these models suffer from the deterministic structure when generating multiple diverse responses.


This is a step towards having pretraining objectives that explicitly consider and leverage discourse-level relationships.

it is still unclear whether language modeling is the most effective method of pretrained language representation, especially for tasks that need to exploit multi-turn dependencies, e.g.


10 The figure indicates that the conversations often extend over days, or even more than a month apart (note the point in the topright corner).

our annotations rarely contain links beyond an hour, and the output of our model rarely contains links longer than 2 hours.


The discriminator in GAN is often used to evaluate the generated utterances and guide dialogue learning.

these methods mainly focus on the surface information of generated utterances to guide the dialogue learning, and fail to consider the utterance connection within the dialogue history.


Existing neuralbased dialogue systems only consider this signal in a weak and implicit way, where they use hierarchical encoders to model the dialogue history (Sordoni et al., 2015a;Serban et al., 2016;Li et al., 2017;Serban et al., 2017;Xing et al., 2018).

we argue that these methods are mainly designed to model the overall semantic context information of the dialogue history but not good at modeling intermediate sequential order.


Rather than measuring the relative probabilities of past forms across different verbs, CR@5 considers the relative rankings of different past forms for each verb.

cR@5 also yielded unstable results: 39-47% on A&H's data, and 29-44% on K&c's data, as shown in Figure 3.


In the previous experiment, we saw that individual models often rank implausible past tenses higher than plausible ones.

we see here that on aggregate nearly all the model's proposed past tenses are those suggested by A&H.


φ A is therefore sensitive to the actual SA scores, and to the popularity of mentioned concepts.

φ 1 is only sensitive to changes in the set of activated concepts.


On the one hand, this indicates that our experiments are not able to confirm the forward semantic priming hypothesis.

given the good results of AEoS, our experiments confirm the backwards priming hypothesis and sentence wrap-up.


End-to-end training with Deep Neural Networks (DNN) is a currently popular method for metaphor identification.

standard sequence tagging models do not explicitly take advantage of linguistic theories of metaphor identification.


This approach is not tailormade for metaphors; it is the same procedure to that used in other sequence tagging tasks, such as Part-of-Speech (PoS) tagging (Plank et al., 2016) and Named Entity Recognition (NER) (Lample et al., 2016).

we have available linguistic theories of metaphor identification, which have not yet been exploited with Deep Neural Network (DNN) models.


This is necessary because it will be concatenated with the purple attentive context representation, in encoding space; we found that performance is better when both meanings are in the encoding space.

the RNN HG does concatenate vectors from two different spaces; this works because they are representations of the same word, rather than word versus context.


RNN HG performs slightly worse than RNN MHCA.

it exceeds RNN MHCA by 0.3% on the VUA VERB track (F1=70.8%).


In this approach, we take 2 × 150 dimension BiLSTM hidden states so that h t and g t are aligned in dimensionality.

such an approach yields 73.7%, 70.0%, 78.9% and 71.8% F1 scores on VUA ALL POS, VUA VERB, MOH-X and TroFi datasets, which is worse than the concatenation approach (RNN HG) in Table 2.


It supports our arguments that the noise of treating non-target words as literals in TroFi negatively impact our models' ability to learn the difference between literals and metaphors.

all words in VUA news are annotated, so that the advantages of our models are more obvious.


Metaphors may be misclassified as literal by RNN HG.

rNN MHCA may flag the clash between literals and their contexts, if there are many metaphors in the contexts, so that literal target words may be misclassified as metaphoric.


Diachronic word embeddings have been widely used in detecting temporal changes.

existing methods face the meaning conflation deficiency by representing a word as a single vector at each time period.


It is well known that word meaning can be represented with a range of senses.

existing methods only assign one embedding to a 1840 1860 1880 1900 1920 1940 1960 1980   word for a time period, thus they face challenges in representing senses and tracking the change of them.


(2018) propose global anchor method for detecting linguistic shifts and domain adaptation.

the above methods could only assign one neural embedding to a word at each time period, which cannot model the change of the word senses.


We thus measure the discrepancy between the conditional distribution p(z B t+1 |m A t ) and the marginal distribution p(z B t+1 ) not taking m A t into account.

we want to assess agent B's behaviour under other possible received messages m A t .


In their setup, fully symmetric agents are encouraged to use flexible, multi-turn communication as a problem-solving tool.

the independent complexities of navigation make the environment somewhat cumbersome if the aim is to study emergent communication.


In informal experiments, we found shallow CNNs incapable to handle even the simplest random split.

it is hard to train very deep LSTMs, and it is not clear that the latter models need the same depth CNNs require to "view" long sequences.


(The prior simply corresponds to add-λ smoothing.)


2017;, with the possible exception of Phillips and Pearl 2014a,b).

very little previous research compares the performance of a wide range of algorithms using diverse and cognitively plausible segmentation methods within a large set of typologically diverse languages and closely matched corpora, with unified coding criteria for linguistic units.


If chance is defined as the highest of the two baselines (p=0, 1/6), 1 algorithm performed above chance in all 8 languages (DiBS).

if we relax this criterion, AG, FTPa, FTPr, MIr and MIa also performed above chance for nearly all languages.


An error analysis would be beyond the scope of this paper.

three categories of incorrect cases have been measured and can be found online.


First, no algorithm performed systematically below chance level in our study.

we cannot say that they all performed above chance for all languages either.


More deeply embedded sentences appear to be shorter (in number of words), and this is in accordance with the hypothesis that they impose a heavier processing load than shallower clauses.

surprisingly, when sentence complexity (in number of clauses) is accounted for, there is no clear bias against deeper embeddings.


Modern deep learning algorithms often do away with feature engineering and learn latent representations directly from raw data that are given as input to Deep Neural Networks (DNNs) McCann et al., 2017;Peters et al., 2018).

it has been shown that linguistic knowledge (manually or semi-automatically encoded into lexicons and knowledge bases) can significantly improve DNN performance for Natural Language Processing (NLP) tasks, such as natural language inference (Mrkšić et al., 2017), language modelling (Ahn et al., 2016), named entity recognition (Ghaddar and Langlais, 2018) and relation extraction (Vashishth et al., 2018).


Transfer learning results: For each of the re-  maining pages in our data we first utilize our pretrained model from the last step.

we train the dense layers with randomly selected 20% datapoints from the page to be tested.