Pseudo-label retrieval is done on a subset of onehundredth of the unpaired data (yielding 1000 samples).

to further validate our method in the semisupervised setup, we compare different backbone architectures  in our framework, where their methods are developed for fullysupervised methods.


To validate our model, we follow Karpathy splits [Karpathy and Fei-Fei, 2015], which have been broadly used in image captioning literature.

we mainly focus on image captioning and unpaired data handling literature.


Ours (1%) a group of sheep standing next to a fence.

by leveraging unpaired data, our method can generate more fluent and accurate captions, compared to Paired Only trained on the same amount of paired data.


Each dialog is made up of an image, a caption from MS-COCO dataset and 10 QA pairs.

in this section, we visualize the inference mechanism of our proposed model.


single image/single sentence co-occurrence statistics.

our approach for this difficult setting is rankingbased: we train algorithms to score image collec-tions and sentence collections that truly co-occur more highly than image collections and sentence collections that do not co-occur.


The similarity functions described in §3.2 rely on document-level, structural information, i.e., for a single image in a document, the other images in a document affect the overall similarity (and vice versa for sentences).

because no explicit multimodal annotation is available at training time, we refer to this task as unsupervised.


Using a similar approach, 8257 negative samples are chosen at random intervals where the last sentence is not immediately followed by a laughter marker.

results of our experiments indicate that humor can be better modeled if all three modalities are used together.


Transcriptions also include reliably annotated markers for audience behavior.

similarly, Figure 2.b shows the distribution of number of words per context sentence.


From the force alignment, we extract the timing annotations of context and punchline on word level.

for the majority of time, the camera stays focused on the presenter.


In cases where the dataset contains, apart from snippetlevel, also token-level annotations, the attention module can be directly supervised: we add a term ... to the model's loss that minimizes the error of the attention scores, with respect to token-level annotations.

• We introduce two novel tasks pertaining to shallow movie understanding that can be tackled in the context of television series data.


The presence of more than one related tasks makes our setup ideal for testing our model in a multi-task setting.

even the multi-head supervised attentive model, does not score as well as the multiview (non-attentive) model.


This indicates that differential neurons in brain are activated for lexical selection of action and object words.

the details of these two CNNs can be seen as follows: • 2D CNN We use ResNet-152  as the 2D CNN model.


First, a recent study by Richardson and Davies (2018) has shown that the majority of the public is reluctant to step inside an AV.

on average there are 4.27 objects with the same category as the referred object per image and on average there are 10.70 objects per image.


Moreover, the original nuScenes dataset considers the specific setting of urban scene understanding.

the non-RPN systems only have to encode the full image once and then reason over this embedding.


We decided to use this list because it contains words and phrases about events and people related to the image, which might give us an insight about what the image contains and what it is about.

we can see that selecting the top 4 to 5 features works best, yielding 65.4%, 75.1%, 84.1% and 92.5%.


Unfortunately, the number of claims that need to be fact-checked is several orders of magnitude larger than what humans can handle manually.

note that these top features for the two experiments are different: we use the scores in the AP(S) column in Table 1 for the Snopes dataset, and the AP(S+R) column for the Snopes+Reuters dataset.


Besides the baseline models, we also compare our method with the HACRN model (Zhao et al., 2018).

and the inference will also stop when it has arrived at the beginning of the dialog history.


while the predicted trajectory distribution contains all information necessary to reach the correct cards and goal location, the other three distributions further improve performance.


We first train each stage of the model separately, and then fine-tune them jointly.

appendix F describes cascaded evaluation formally.


Instruction-level metrics ignore error propagation, and do not accurately reflect the system's performance.

move in a 3D environment and collect valid sets of cards to earn points.


Different notions of context have resulted in different levels of success with downstream NLP tasks.

(2019), surpassing the baseline without making use of bounding boxes.


It has long been understood that the meaning of a word is systematically and predictably linked to the context in which it occurs (e.g., Firth 1957;Harris 1954;Deerwester et al.

videoBERT cannot answer questions, nor does the model consider bounding boxes.


Prior evaluation metrics typically considered several aspects, including content relevance, information correctness, grammaticality, and if expressions are human-like (Hodosh et al., 2013;Bernardi et al., 2016).

before testing, the input texts were lowercased and tokenized by using the ptbtokenizer.py script from the same tool package.


And I don't care if you are or not...you can say whatever the fuck you want.

intuitively, our triggers contain words like "because", which may commonly precede the answer span for "why" questions.


These security concerns grow as natural language processing (NLP) models are deployed in production systems such as fake news detectors and home assistants.

other trigger tokens have high PMI, e.g., the top PMI token before the answer to "why" questions is indeed "because".


The same idea has been used recently used for evaluating GANs for image generation (Shmelkov et al., 2018).

we will consider three families of measures: • H-divergence based metrics: based on the capacity of another classification model to distinguish between samples from D s and D t .


In their work, there is a single attention mechanism which is controlled by a fixed α.

we use a mini-batch size of 8192 tokens and warm up the learning rate linearly until 20k steps, after which it decays according to an inverse square root schedule.


Concretely, we replace the row normalization π in Equation 1 by This change leads to sparse attention weights, as long as α > 1; in particular, α = 1.5 is a sensible starting point (Peters et al., 2019).

this is the first work to allow such atten-  -entmax Figure 9: Interrogation-detecting heads in the three models.


Since the posterior is intractable, we demonstrate how a variational inference procedure can be used to perform both learning and accurate font reconstruction.

critically, each style latent variable is shared by all characters within a font, while character embeddings are shared by characters of the same type across all fonts.


The following abbreviations are used to represent various components: • C i : convolution filters with i filters of 5 × 5, 2 pixel zero-padding, stride of 1, dilation of 1 • I : instance normalization • T i,j,k : transpose convolution with i filters of 5×5, 2 pixel zero-padding, stride of j, k pixel output padding, dilation of 1, where kernel and bias are the output of an MLP (described below) • H : reshape to 26 × 256 × 8 × 8 A.2 Network Architecture Our fully-connected encoder is: Our convolutional encoder is: Our fully-connected decoder is: Our transpose convolutional decoder is: MLP to compute a transpose convolutional parameter of size j is:

the font) explaining lower-level visual features such as boldness, texture, and serifs.


That is, both "He knows that the answer is 5" and "He does not know that the answer is 5" imply that "The answer is 5".

understanding veridicality requires semantic subtlety and is still an open problem for computational models of natural language inference (NLI) .


For example, in the sentence "He does not know that the answer is 5", "know" is veridical with respect to "The answer is 5", since a speaker cannot felicitously say the former sentence unless they believe the latter proposition to be true.

we then generate two p, h pairs: the sentence and the complement as-is S, C , and the negated sentence plus the complement ¬S, C .


(2018) evaluated neural models' ability to carry out inferences in line with these signatures, making use of templatized "semantically bleached" stimuli (e.g.

"think" would not be veridical here, since "He does not think that the answer is 5" is felicitous whether or not it is taken to be true that "The answer is 5".


As ordering the sequence is more nuanced than sifting basic from non-basic, one might expect consistently lower correlations.

compound generation takes into account language-specific knowledge of which components and glues are common in that language.


To well address this, we come up with an attention-based neural network to model contextual information.

there are several golden annotated instances that we disagree with.


To capture such informative features, we then map the semantic role tag of each word to a real-valued vector x sr of dimension d sr using a embedding matrix SR ∈ R dsr×|Vsr| , where V sr is the set of semantic role label vocabulary.

we compare the performances between the models adding semantic role information in embedding layer and those only using the word embeddings.


The entitybased approaches capture attentional structure, the syntax-based approaches consider intention, and the organizational structure is largely captured by models that consider discourse relations and content (topic) distribution.

only articles with more than 10 sentences are included in our dataset.


Also, the SSO models can capture the global dependency among input sentences via the paragraph vector.

the attention layer repeatedly decides the next sentence from sentence-pair interactions between the current predicted sentence and each paragraph-independent candidate {s i } 5 1 .


As shown in Figure 4, at the 3rd step, the output vector (c 3 ) of the 3rd position is fed into the final attention layer as like in conventional pointer decoders.

this allows us to explore whether the topic latent vectors actually help the sentence ordering task.


Modeling the coherence among sentences to compute their gold order is one of the fundamental tasks in Natural Language Processing (NLP) with many applications such as document modeling (Narayan et al., 2018a), extractive document summarization (Jadhav and Rajan, 2018;Nallapati et al., 2017), question answering Liu et al., 2017), conversational analysis (Zeng et al., 2018), automated text generation (Guo et al., 2018), and image captioning (Anderson et al., 2018).

positional Accuracy (pAcc in %): The positional accuracy is defined as the ratio of the matched sentence-level absolute positions between the predicted and correct orders as follows: where o s j denotes the absolute position of sentence s j in the correct sequence S j .


We create sentence vectors s i for all sentences s i with the attention-based bi-LSTMs.

similar to traditional models, they also compress sentence vectors into a single fixed-length vector via average pooling.


Since our decoder directly receives the set of sentence vectors regardless of their input order via attention, our encoder is free from the constraint that all sentences must be compressed into a single fixedlength vector as a paragraph representation.

since our TGCM could elaborately capture both local and global dependencies simultaneously, TGCM showed the best sentence ordering performance among all comparative models on all datasets.


Previous approaches to RST parsing (Ji and Eisenstein, 2014;Feng and Hirst, 2014a;Joty et al., 2015;Braud et al., 2017) have used locally normalized discriminative models.

if e is an EDU consisting of the word sequence w 1:k , then where w t is the word embedding of w t .


We eliminated these from the Perret 2016 data set that we used in our comparative experiments below, as these sto-ries were obviously not a correct representation of what was going on in the game at the relevant point.

a dialogue begins at the beginning of a player's turn, and ends at the end of that player's turn.


For dialogues with multiple interlocutors, extraction of their discourse structures provides useful semantic infor-mation to the "downstream" models used, for example, in the production of intelligent meeting managers or the analysis of user interactions in online fora.

the higher order dependencies significantly improved the generative model's results on the full StAC corpus (see table 1).


Acc   and at article level, with the exception of the experiment where the only features used are references, which are article-level features.

after tagging a text segment with parts of speech, we encode the result as part-of-speech sequences and pass them through a convolutional layer of 50 filters and kernel size 4, followed by a max pooling layer.


For example, "A man came near to her tried to be physical with her .".

the BiLStM model comprises two layers of one directional LStM.


Because the context of the two consecutive words in the original text are only off by one position, it will be difficult for the CNN model to detect the difference.

the joint learning method boosted the performance even higher.


A directed edge in the network therefore represents a potential influence between words.

structured data extraction from news have also been used for stock price prediction using techniques of information retrieval in Xie et al., 2013;Ding et al., 2015;Chang et al., 2016;Ding et al., 2016).


For labeling the newly generated sentences, we used word embeddings as described in (Mikolov et al., 2013).

we used the following setting, mentioned in (Soldaini and Goharian, 2016) as having best performance: Similarity threshold = 0.9, Semantic types = [SignorSymptom, DiseaseorSyndrome, F inding, N eoplastic P rocess] • CRF (Baseline).


Results show that our approach achieves superior or comparable performance with significantly less training data (reduced by 75%) than state-of-the-art training methods.

existing deep generative models are not designed for generating class-specific data instances.


The conditional probability over words, i.e., P (w i |f (z; θ)) is modeled by a multinomial logistic regression: The parameters of the neural network, i.e., θ, are learned by maximizing the the log likelihood of the observed sentence x.

to shallow models such as Skip-Gram (Mikolov et al., 2013) which also embeds into latent spaces, deep generative models have been shown to capture the implicit semantics and structure of the underlying data more effectively.


neglecting the utility of the text embedding while manipulating it, may destroy the semantic meaning of the text data for the given task.


Parameter Analysis: Our proposed method RLTA has an important parameter α to change the level of privacy and utility.

some existing work shows the degraded quality of textual information (Anandan et al., 2012;saygin et al., 2006).


the given task, and 2) prevent leakage of private-attribute information by developing a reinforcement learning environment which incorporates a private-attribute inference attacker and manipulates the initial given text embedding accordingly to fool the attacker.

after training, we use the encoder's output as the text representation v u (Cho et al., 2014).


To our best knowledge, we are the first to explicitly give the model guidance of parent and sibling nodes.

we present a top-down hierarchical sequence-to-tree (seq2tree) model which explicitly models the tree structure.


In the incomplete KB setting, we simulate KB-based QA on an incomplete KB by randomly discarding some of the triples in the KB: specifically, we randomly drop a fact from the knowledge base with probability p = 50%.

we focus on tasks in which questions require compositional (sometimes called "multi-hop") reasoning, and a setting in which the KB is incomplete, and hence must be supplemented with information extracted from text.


(2) All the answers of ReCoRD are contained in the paragraphs and are assumed to be entities.

c: She would have managed to get to the elevator quicker with some assistance.


Multiple-choice QA allows agents to support clear, dataset-curated possible answers.

finding the correct answer to a given question may require reading large volumes of text or understanding complex arguments.


As a lower bound, we train an answer-only model to evaluate how effectively the QA model is using the passage sentences it is given.

we would not expect agent evidence to transfer to other models if models only exploit method-specific patterns.


Compared with RKN, which only uses the paragraph-question relevance, our completed ranker can further utilize the paragraph-paragraph relevance to aggregate the evidence among relevant paragraphs, thereby increasing their confidence scores.

given the paragraph probabilities Pr sen ∈ R n , the ranker is trained by minimizing the loss: For the reader, we first select k paragraphs based on the RS→WS strategy and then obtain the final scores Pr s i and Pr e i for each paragraph, where Pr s i (t)/Pr e i (t) represent the start/end scores of the t-th word in the i-th paragraph.


This model, called BlockHolE, represents relations as block circulant matrices.

computing circular correlation requires O(n log n) time if FFT is employed.


Nevertheless, we observe that there are still two important research issues (RIs) which are not processed well or even neglected.

this task has attracted more attention in recent years (Serban et al., 2016;Elsahar et al., 2018).


Take the question "Which sexes do King Harold, Queen Lillian and Arthur Pendragon possess" as an example, a spurious logical form only retrieves the genders of "King Harold" and "Queen Lillian", while it gets correct answers for the question.

hence, we separately evaluated the entity linking task on the test set.


To tackle issues mentioned above, we propose a novel multi-task semantic parsing framework for KB-QA.

and, the early stage execution is performed to filter out illegal logical forms that lead to empty intermediate result.


Along with the development of this dataset, a variety of neural MRC models have been proposed, such as BiDAF (Seo et al., 2016), R-NET (Wang et al., 2017), ReasonNet (Shen et al., 2017), DCN (Xiong et al., 2016), QANet (Yu et al., 2018), SAN , etc.

after that, we train MRC models on this language-mixed dataset to endow them with the multilingual comprehension capacity.


We then ask our bilingual crowdsourced workers to create questions and answers on these collected passages.

lingual Information Retrieval and Question Answering, 13(1):27-40.


Several neural network models have been proposed and tested on these datasets, such as ASReader (Kadlec et al., 2016), StanfordAttentiveReader (Chen et al., 2016, AoAReader (Cui et al., 2017), etc.

the intuitive approaches to CLQA are to translate the questions into the document's language (Sutcliffe et al., 2005;de Pablo-Sánchez et al., 2005;Aceves-Pérez et al., 2007), which, however, suffers from translation errors.


feathers wings [-1.8] , nests [-3.1] , feathers [-3.2] , died [-3.7] , eggs [-3.9] HasPrerequisite Typing requires .

we present an in-depth analysis of the relational knowledge already present (without fine-tuning) in a wide range of state-of-theart pretrained language models.


Two sets of edges are considered in this work: pointing from v i to v j will be added to the graph if n( , which is denoted as solid arrow in Figure 1. , which is denoted as dashed arrow in Figure 1.

please refer to the paper for more details due to space limitation.


Second, we find semantically-drifted questions are usually unanswerable by the given answer.

first, in a paragraph, usually, there are several facts relating to the answer and can be used to ask questions.


Semantics-Reinforced Models As shown in Table 3, when using QAP and QQP separately, all metrics are significantly (p < 0.001) improved over our baseline and all metrics except ROUGE-L are significantly (p < 0.05) improved over the models using traditional metrics as rewards.

these metrics sometimes fail to evaluate equally to question paraphrases and thus provide inaccurate rewards.


This dataset contains two subsets: movie descriptions collected from Wikipedia (SelfRC) and from IMDB (ParaphraseRC).

we hypothesize that this is because the generated question-answer pairs are often noisy and inaccurate, and directly fine-tuning the answer module with synthetic data may hurt the performance, which is also observed in Sachan and Xing (2018), especially when a well-performed MRC model is used as the baseline.


The problem of synthesizing answers across different domains will be investigated in future work.

both studies assumed that annotated data in the target domain (either questions or question-answer pairs) are available.


It makes the generated answer semantically correct and comprehensive.

there have also been a few studies on incorporating knowledge into QA models without passage reading.


KEAG is capable of leveraging symbolic knowledge from a knowledge base as it generates each word in an answer.

due to the size of a knowledge base and the large amount of unnecessary information, we need an effective way of extracting a set of candidate facts which provide novel information while being related to a given question and passage.


This is the key insight that we are going to exploit.

training the discriminator is performed on the RACE (Reading Comprehension Dataset) which is also collected from English Examinations (Lai et al., 2017).


This decision step is accomplished by a simple feed-forward network (Figure 3).

in the next sections, various discriminators are illustrated, and, after that, we present how they are combined to produce the final ranking model.


The structure of the dataset is perfect for our situation.

the end-to-end performance of question answering (QA) systems decreases considerably.


We also investigated the ability of the Attentive Ranker to sort the retrieved documents as compared to document embeddings -which may be used to measure the similarity between a question and a candidate document.

we have made the ranked documents public and further research can benefit from it.


The dataset contains 1572 documents in total, devided into Train (1102 docs, 32.7k questions), Dev (115 documents, 3.5k questions) and Test (355 documents, 10.5k questions) sets.

this supports our hypothesis that discourse information is required for generalizing to longer contexts.


It is clear that all models show improvement over length 800-1000.

we enhance self-attention with knowledge about entity coreference (Coref), their participation in events (SRL) and the relation between events in narrative discourse (Shallow Discourse Parsing (Xue et al., 2016), DR).


the system gains 74.62 on the evidence F1, 22 points greater that of the second system, demonstrating its ability on semantic retrieval.


In all the cases, we found small drops in both performance and justification scores across both the datasets, with the removal of either C(A) or C(Q) having the largest impact.

table 2 reports comprehensive results on MultiRC, including both overall QA performance, measured using F1 m , F1 a , and EM0, as well as justification quality, measured using standard precision (P), recall (R), and F1.


Recent work on open-domain question answering largely follow this retrieve-and-read approach, and focus on improving the information retrieval component with question answering performance in consideration (Nishida et al., 2018;Kratzwald and Feuerriegel, 2018;Nogueira et al., 2019).

span offset scores are predicted from each paragraph independently before finally aggregated and normalized with a global softmax operation to produce probabilities over spans.


By using the vanilla BLEU, it produces a much heavily penalized score of 0.065.

we believe that the generated SQL statement is a much better choice in this case.


Converting natural-language questions to SQL codes (NL2pSQL) is an active area of research in natural language processing.

to this end, we first collect all the QA pairs tagged with SQL.


We first collect well-formed SQL statements among SQL statements written by developers in SO using MySQL parser (GuduSoftware, 2018), as MySQL is the most popular SQL dialect among developers (StackOverflowSurvey, 2018).

we transformed all the multi-line SQL statements into a single line by replacing the line break with a white space.


Since the prediction result for each query substructure is independent, the score for query structure S i is measured by joint probability, which is Assume that should be 1 in the ideal condition.

sINA (shekarpour et al., 2015) and NLIWOD conduct query generation by predefined rules and existing templates.


S j = (V T j , T j ) should be a query substructure of S i according to the definition.

they may suffer from the lack of training data, especially for questions with rarely appeared structures.


Take birthP lace and bornLocation as examples.

mSR is the average of SR on T r : Lower mSR denotes that it's more difficult to learn r, because the entities connected with relation r may have more aspects.


The result also indicates that our model is well extensible and can easily incorporate off-the-shelf IE tools.

in the 1hopsubgraphs, GUpdater beats IE-gold + R-GCN by 0.42 in Added Acc, 0.65 in Deleted Acc, indicating that GUpdater can well capture both explicit and implicit information in the news snippets.


To the best of our knowledge, we are the first to introduce GAIL into the field of knowledge graph reasoning.

each negative triple is generated by replacing the answer entity t with a faked one t ′ given a positive triple (h, r, t).


In summary, our contributions are threefold: • We present a plug-and-play framework based on GAIL to enhance existing RL-based reasoning in KGs by learning reasoning policies and reward functions through imitating demonstrations.

compared to NELL-995, FB15K-237 is more challenging and closer to real-world scenarios, where its facts are created from FB15K (Bordes et al., 2013) with redundant relations removed.


Given a positive entity pair ⟨e head , e tail ⟩, we introduce a relational set R h which contains all relations directly connected to e head .

such evidential paths are obtained by random walks, which inevitably introduces inferior or even noisy paths.


There are two main tasks in knowledge graph reasoning, namely link prediction and fact prediction.

given a relational path x = {r 1 , r 2 , ...r t , ...}, the path embedding x is encoded as: where each relation r t is mapped into a real-valued embedding r t ∈ R k pre-trained by TransE (Bordes et al., 2013).


We acknowledge anonymous reviewers for their valuable comments.

such an approach may lead to Figure 1: Overview of our framework DIVINE, where the generator can be any of the existing policy-based agent for knowledge graph reasoning and the discriminator can be considered as a self-adaptive reward function.


SciTail We compare our model with the following previous models on SciTail dataset, and show the results in Table 3.

the cross sentence attention is oriented to the original semantic representations of another sentence, so as to be able to capture inherent semantics by relying on the fixed matching target.


Most recently, PaSKoGE model (Jia et al., 2018) is proposed for KG embedding by minimizing a pathspecific margin-based loss function.

to better deal with complex relations like 1-to-N, N-to-1 and N-to-N, transH (Wang et al., 2014) and transR (Lin et al., 2015b) employ relation-specific hyperplanes and relation-specific spaces respectively to separate triples according their corresponding relation.


Moreover, STransE (Nguyen et al., 2016) extends TransR by introducing two projection matrices for the head entity and the tail entity, respectively.

instead of employing the hyperplane like TransH, TransR proposes a matrix W r ∈R m×n to project entity vectors into a relation-specific space.


To better deal with complex relations like 1-to-N, N-to-1 and N-to-N, TransH (Wang et al., 2014) and TransR (Lin et al., 2015b) employ relation-specific hyperplanes and relation-specific spaces respectively to separate triples according their corresponding relation.

the concrete forms of the variables in Equation 2are shown as follows, where W p s=n ∈R d×d indicates the projection matrix for path p s=n , which aims to project the tail entity in a path to the space of p s=n .


In the future, we will explore the following research directions: (1) we will study the applications of the proposed models in various domains, like personalized recommendation (Liu et al., 2018); (2) we will explore other techniques to fuse the ordered relation information from different paths (Liu et al., 2019).

they pay less attention to the order of relations in a path when learning representations of the path.


Following previous work on KG completion (Bordes et al., 2011), we use Hits@K and mean reciprocal rank (MRR) to evaluate the effectiveness of the KGR and OKGR.

it is desirable to design a method that can filter out irrelevant facts for augmenting the reasoning model.


To facilitate this, a If-Then commonsense reasoning dataset Atomic is proposed, together with an RNN-based Seq2Seq model to conduct such reasoning.

while con- ditional variational autoencoder (CVAE) (Sohn et al., 2015) is an extension of VAE on the conditional generation problem.


Label: contradiction lem (MacCartney and Manning, 2008;Heilman and Smith, 2010).

in a local inference module, we update the sentence representation by using the local inference information, based on the attention of the other sentence.


We filter the verbs and adjectives through a list of sentiment words (Hu and Liu, 2004) to ensure we do not choose REs that would obviously significantly alter the text's sentiment.

our second model makes use of the transformer architecture, and our implementation replicates Vaswani et al.


This is useful for reasons such as marketing, overall enjoyment of interaction, and mental health therapy.

note that for nouns, the outputs from GWn-STEM and nWn-STEM are equivalent.


The news coverage of events often contains not one but multiple incompatible accounts of what happened.

similarly the entity he is likely anaphoric to Jamal Khashoggi.


The process breaks down to three steps: first, we pair c j with each of the sentences t k ∈ T and compute a sentence-sentence relational vector v j,k which is the sum over all the word-word bi-linear contraction (the bi-linear contraction of two word embeddings Then, we average over the sentencesentence relation vectors to obtain a summary vector p j = average  Since our models' supervision comes in the form of whole scenarios, supervising our iterative insertion clustering model is not completely straightforward.

for instance, for the example in figure 1, the most readable update can be made by inserting He never exited the consulate but died there to the right of Jamal Khashoggi was murdered.


When given a very small proportion of prior alignment like 10%, the results of AlignEA and GCN-Align decrease enormously, but our KECG still achieves outstanding results.

this will be further discussed in Section 4.4.


At a high level, control tasks have: structure: The output for a word token is a deterministic function of the word type 1 .

finally, we showed how selectivity can be used to provide added context to probing results, demonstrating that marginal differences in part-of-speech tagging accuracy between ELMo1 and ELMo2 correspond to large differences in selectivity, and similarly, the even though ELMo2 achieves similar part-of-speech tagging accuracy to a random representation baseline, ELMo2 achieves it with much higher selectivity.


The possible behaviors C(v) are as follows: attach to self: Always attach tokens of this type to themselves.

to explore the relationship between representations, probes, and task accuracies, we defined control tasks, which by construction can only be learned by the probe itself.


The output is not constrained to be a tree for evaluation; each prediction is evaluated independently.

a good probe (one that provides insights into the linguistic properties of a representation) should be what we call selective, achieving high linguistic task accuracy and low control task accuracy (see Figure 2).


Although we do want tag t i to consider context-e.g., to distinguish between noun and verb uses of word i-we want "most" of t i 's information to come from word i itself.

as Figure 4 shows, averaging over the 9 languages, the reconstruction retains 71% of POS information (and as high as 80% on Spanish and French).


s ⇠ is not used at test time, but only as part of our training objective.

vIB goes beyond mere dimensionality reduction to a fixed lower dimensionality, since it also avoids unnecessary use of the dimensions that are available in the compressed representation, blurring unneeded capacity via randomness.


To avoid this, variational IB replaces p ✓ (t) with some variational distribution r (t).

y is independent of T given X, as in the graphical model T !


By making information about the global sentence context available in local word representations, this encoder can be assumed to mitigate error propagation for transition-based parsers and to widen the feature scope beyond individual word pairs for graph-based parsers.

we call the transition-based and graph-based systems enhanced with ELMo TR+E and GR+E.


The output of the last layer h L,k is then employed in conjunction with a softmax layer to predict the next token at k + 1.

(2018) posit that the best way to take advantage of ELMo's representational power is to compute a linear combination of BiLSTM layers: where s j is a softmax-normalized task-specific parameter and γ is a task-specific scalar.


However, more accurate feature learning using continuous representations and nonlinear models has allowed parsing architectures to be simplified.

similarly, graph-based parsers again rely on first-order models and obtain no improvements from using higher-order models (Kiperwasser and Goldberg, 2016;.


First of all, although the baseline parsers exhibit the familiar pattern of accuracy decreasing with sentence length, it is not the transition-based but the graph-based parser that is more accurate on short sentences and degrades faster.

first of all, although the baseline parsers exhibit the familiar pattern of accuracy decreasing with sentence length, it is not the transition-based but the graph-based parser that is more accurate on short sentences and degrades faster.


With regard to dependency relations, the baseline graph-based parser has better precision and recall than the baseline transition-based parser for the relations of coordination (conj), which is consistent with the old study, as well as clausal subjects (csubj) and clausal complements (ccomp), which are relations that involve verbs in clausal structures.

although the transition-based parser still seems to suffer from search errors, as shown by the results on dependency length and distance to the root, it no longer seems to suffer from error propagation in the sense that earlier errors make later errors more probable.


The ZPar performs better than MaltParser and MSTParser, partic due to the richest feature representation.

while the standalone ELMo models were trained on the tokenized WikiDump and CommonCrawl for each language respectively, the BERT model was trained only on the former, albeit simultaneously for 104 languages.


Edges are decoded by computing the minimum spanning tree on the weight matrix (the lowest sum of edge distances).

this effectively redistributes probability mass to all other layers, forcing the network to incorporate the information content of all BERt layers.


Fact: If a liquid disappears then that liquid probably evaporated Gap: Kool-aid is a liquid Answer: Evaporation Figure 3: Knowledge gap between the question (Koolaid) and the fact (liquid).

we considered {50, 100, 200} dimensional representations, λ ∈ {0.1, 1, 10}, retrieving {10, 20} knowledge tuples and {[x -y; x*y], [x, y]} combination functions for during the development of the model.


The core science fact metal lets heat to travel through points to metal as the correct answer, but it is not one of the 4 answer choices.

our knowledge-gap guided approach to QA is orthogonal to the underlying model.


the confidence of triple classification).

as shown in Table 2, we first use the official split to compare our model with the baseline methods reported on the paper and leaderboard.


In our case, we have path-level and concept-pair-level attention for learning to contextually model graph representations.

we propose to represent graphs with respect to the paths between question concepts C q and answer concepts C a .


Our method is extensible to any language with an existing open-source NMT, leveraging only its attention scores (Bahdanau et al., 2014).

• RQ2: Does our model generally work on more noisy environment?


Second, (2) we modify a passage by removing a sentence containing the answer span, so that the modified passage is the subset of original one, but unanswerable.

this suffers low-recall, considering we found only 53.6% spans among whole translated Q-A pairs by the high-precision method.


In this paper, we highlight the learning challenge and show that our learning method, independent of the model architecture, can give a significant gain.

following Clark and Gardner (2018), we filter a subset of segments in TRIVIAQA through Tf-IDf similarity between a segment and a question to maintain a reasonable length.


Unless otherwise specified, visual features representing the scenes are extracted via a fixed Convolutional Neural Network (CNN) fed with images resized to 224 x 224 pixels.

both FiLM and CNN+LSTM+SA experience a drop in performance compared to POS 1 /POS, confirming that computing the threshold function over a subset of objects is the most challenging task.


consider words like 'small' and 'minuscule' as synonyms.

in SET+POS-hard applying the min/max function to the reference set would lead to chance level.


(2019) compare language models' probabilities for pairs of minimally different sentences differing in grammatical acceptability.

all but one licensor-NPI pair follows a 2⇥2⇥2 paradigm, which manipulates three boolean NPI licensing features: licensor presence, NPI presence, and the occurrence of an NPI within a licensor's scope.


(2019) use Boolean acceptability judgments inspired by methodologies in generative linguistics.

this result is in direct contrast with the results from Wilcox et al.


We find that BERT has significant knowledge of these features, but its success varies widely across different experimental methods.

these results vary widely across the different methods we compare.


The licensor (whether) or licensor replacement (that) is in bold.

the other methods show a more nuanced picture.


Here, we find positive masculine expectation in the m and m and f and m conditions, and negative masculine expectation in the f and f condition, as expected.

lar results, with all models except LSTM (FTB) demonstrating significant plural prediction in all cases.


despite being fundamentally based on argument, /r/ChangeMyView has a reputation for being remarkably civil and productive (CMV moderators, 2019a), e.g., a journalist wrote "In a culture of brittle talking points that we guard with our lives, Change My View is a source of motion and surprise" (Heffernan, 2018).


Although our findings are certainly related to the persuasion process, we focus on understanding the self-described reasons for persuasion, instead of the structure of arguments or the factors that drive effective persuasion.

to the best of our knowledge, our work is the first to investigate the phenomenon of pointers in explanations.


We develop features to capture the properties of a word in the explanandum, and show that our proposed features not only have relatively strong predictive power on the echoing of a word in an explanation, but also enhance neural methods of generating explanations.

explanations are essential for understanding and learning (Keil, 2006).


We also demonstrate the important role of nouns in shaping explanations.

we believe that developing a computational understanding of everyday explanations is crucial for explainable AI.


IMHO, Calorie-counting is a crock what you have to look at is how wholesome are the foods you are eating.

3:PREMISE:SUPPORT:0 The example illustrates that the argumentative component at index 0 is a claim, and is followed by attacking one's own claim, and in turn attacking that premise (an example of the rhetorical move known as prolepsis or prebuttal).


Argument mining on discussion forums (online discussion) Most computational work related to argumentation as a process has focused on the detection of agreement and disagreement in online interactions (Abbott et al., 2011;Sridhar et al., 2015;Rosenthal and McKeown, 2015;Walker et al., 2012).

we collect 95,406 threads from the full CMV subreddit between 2013-02-16 and 2018-09-05 and find pairs of posts where the quoted text in the response is an exact match for the original post (removing threads that overlap with the labeled data).


We observe very similar agreement of 0.91 (Chinese→English) and 0.92 (Only English) when the ties were excluded, with the overall agreement being 0.90 between the six participants.

figure 3 shows one such lowagreement example.


We conclude that the knowledge regularizer plays a key role in achieving the state-of-the-art performance and the knowledge layer must be used together with knowledge regularization in our framework.

we propose to tightly integrate these constraints into the discourse relation inference process by manipulating hidden word representations to reflect relations between words, and meanwhile balance attentions to contexts and constraints through adding a knowledge regularization term in the final objective function.


attribute encoder is based on attributespecific words obtained from Section 2.2, which not only stores these words into attribute-specific vocabulary memory, but also utilizes multi-layer perceptron to merge them into an attribute embedding.


Review summarization aims to generate a condensed summary for a review or multiple reviews 1 .

for attribute information, we can first get users' demographic information such as age and gender from the website.


The classical summary decoder combines the context vector c t and the decoder state s t , and then feeds the merged vector into a linear layer to produce the vocabulary distribution.

we present two cases from TripAtt that show the effect of attribute on review summarization in Figure 5.


It shows that all these strategies improve the performance of re- view summarization.

it only obtains 11.59 ROUGE-1, 2.41 ROUGE-2, and 10.21 ROUGE-L F1 scores.


Benefiting from the success of neural sequence models in other NLP tasks, Cheng and Lapata (2016) propose a novel approach to extractive summarization based on neural networks and continuous sentence features, which outperforms traditional methods on the Dai-lyMail dataset.

and the improvement is even greater when we only consider long documents.


Finally we retrieve one-hop facts from Wikidata.

even in such case, neural models still tend to make mistakes for sparse co-occurrence pairs as we will show in the experiments section.


(6)  13:Let M = t 14: if Done then 15: Break the inner loop 16: else 17: Update the action set A = A \{at} 18: Generate summary SUM b = (Sa 1 , .


Our main contributions are as follows: • We propose an extractive summarization method that simulates human being's reading cognitive process.

• We use a hierarchical neural network for rough reading which consists of a neural net to encode the whole document and another one to capture features in paragraphs.


Experiments on two real-world datasets demonstrate that our proposed model can significantly outperform the state-of-the-art extractive methods on summary quality, coverage and non-redundancy.

bidirectional LSTMs (BiL-STMs) on word-and sentence-level are first used to encode a document with N sentences into Second, a global featureS ∈ R ds is computed as an average of all the sentence vectors.


(3) How fluent, grammatical the summary is?

pseudo-Filter is the dataset randomly sampled from Non-Filter version and is of the same size as Filter version.


(2) From the perspective of information compression, machine translation can be regarded as a special kind of cross-lingual summarization with a compression ratio of 1:1.

each property is assessed with a score from 1 (worst) to 5 (best).


(2018) have constructed a multimodal summarization dataset MSMO where the text input and output are similar to that in CNN/Dailymail.

the current machine translation (MT) is not perfect, which results in the error propagation problem.


The reason is that training on sensational samples alone discards around 80% of the whole training set that is also helpful for maintaining relevance and a good language model.

maximizing the noisy sensationalism reward will generate unnatural phrases instead of sensational headlines.


More detailed statistics, including other relations and performance with respect to headdirectionality, can be found in the supplementary material.

we address that issue in the following section.


Recent work improves upon adversarial approaches by additionally leveraging the idea of back translation (dos Santos et al., 2018;Logeswaran et al., 2018;Lample et al., 2019;Prabhumoye et al., 2018).

the ability to generate or rewrite a certain text with some attributes controlled or transferred to meet pragmatic goals is widely needed in applications such as dialogue generation (Oraby et al., 2018), author obfuscation (Kacmarcik and Gamon, 2006;Juola and Vescovi, 2011), and written communication assistance (Pavlick and tetreault, 2016).


2019, we asked human judges to evaluate outputs of different models in terms of content preservation, grammaticality and attribute change correctness on a Likert scale from 1 to 5.


For future work, we plan to test our method in other summarisation tasks (e.g.

we show that our learned rewards have significantly higher correlation with human ratings than previous approaches.


ROUGE between the generated summary and the input document (without using any reference summaries).

we consider three text encoders to vectorise x and y x .


Then we set m guide t to 0 if x t is a stop word or is inside the answer phrase.

summarization or question generation exhibit one-to-many relationships because * Most work done during internship at Clova AI.


The discriminator of GAN compares the entire real and synthetic sentences, instead of individual words, which should in principle alleviates the exposurebias issue (Yu et al., 2017;Li et al., 2017a;Guo et al., 2018).

we assess the performance of our GAP model and compare it with previous methods.


Recent work (Larsen et al., 2015; also considers combining autoencoders or variational autoencoders with GAN to demonstrate superior performance on image generation.

far, we can compute the loss from two paths of the encoder.


It was his second doubledouble over his last three games … The only other Wizard to reach double -digit points was Kris Humphries , who came off the bench for 13 points ( 4 -8 FG , 5 -6 FT ) and five rebounds in 26 minutes …  Figure 3 shows an example generated by our model.

because the rigid structure of texts ensures the correct grammaticality and no repetition in template-based system's output.


Also, tables contain the date when the match happened and we let k denote the date of the record.

since the templates are stilted and lack variability compared to others, it was deemed less coherent than the others by the raters.


Inspired by Luxburg (2007), we constrain the subtopic embedding to be orthogonal with each other: (3) The purpose of introducing subtopics is not to make the summary diverse.

we obtain a document representation d with a hierarchical RNN.


by incorporating readers' comments, STDS and RAVAESum achieve better results than STDS/NC and RAVAESum/NC.


Recall that we claim the sequential context information affects sentence salience.

recall that we construct a document embedding from news sentence representations S, which are generated from a word sequence.


For subtopic representation, we also utilize S, and incorporate the comment sentence embedding Y .

they apply conditional Markov Random Walk and HItS on clusters, separately.


STDS and STDS/NC take both elements into consideration and perform much better than these baselines.

we only take a single sentence without context information to build subtopic representations, as in a typical topic model.


In this paper, we propose an unsupervised MDS solution, which deals with both sequential context information in a single document and the correlations among documents.

many studies attempt to construct well connected graphs, e.g., group sentences into clusters (Wan and Yang, 2008;Banerjee et al., 2015) and construct dense features with distributed embeddings (Yasunaga et al., 2017).


From a practical perspective, it is reasonable to assume that such an entity profile exists for common entities such as popular locations and celebrities, as well as for targeted entities of interest that should be handled by an in-domain NLG system.

we use the gold-standard content for the demonstrative and description categories from the webNLG dataset, which provides an upper bound on the potential performance of this system.


To address this issue, we can regularize D g by adding entropy or we propose to scale the gradient of D g by 1/D g .

the final score for each system is the average score for all outputs.


Therefore, we hope to generate a variety of paraphrases while ensuring quality.

the generator discriminator is pretrained using data generated by the pretrained generators.


The final representation r i is employed in a sequence input of a bidirectional LSTM.

ggNNs also do not require to constrain parameters to ensure convergence.


quality of the generated sentences regarding S2S.

they strongly rely on named entities anonymisation.


Multitask learning has a benefit in that the shared encoder captures common features among tasks; in addition, the encoder focuses more on relevant and beneficial features, and disregards irrelevant and noisy features (Ruder, 2017).

moreover, L 1 , L 2 and L 3 signify the loss of the classification, key phrase generation and headline generation tasks, respectively.


Generally, the generation task need to focus on a wider range of input words than the classification tasks.

job seekers can easily retrieve the job advertisements they desire by reading them.


figure 1 shows an example of feedback comments on preposition use.


On the one hand, it would require a successful parsing to recognize the sources of the errors.

this limitation is particularly problematic for beginner to intermediate learners.


In that case, the retrieved feedback comment will likely be inappropriate no matter how high its similarity is (in the above example, a feedback comment concerning in is applied to at).

we then tested a rule-based and two neural-based baselines on the dataset, showing that retrieval-based methods were promising.


Actual examples are: I/People (don't) agree the/this statement/opinion/that/thinking and (more) harmful (not only) for anyone/people/smokers/nonsmokers.

they encounter the tremendous difficulty of covering a wide variety of errors and of maintaining a large set of rules.


As shown in Figure 2, our criteria can select answer-relevant relations (waved in Figure 2), which is especially useful for sentences with extraneous information.

then we formalize the task under the encoder-decoder framework with gated attention and dual copy mechanism.


Such capability improves the interpretability of our model because the model is given not only what to be asked (i.e., the answer) but also the related fact (i.e., the answer-relevant relation) to be covered in the question.

• Hybrid (Sun et al., 2018) is a hybrid model which considers the answer embedding for the question word generation and the position of context words for modeling the relative distance between the context words and the answer.


In this paper, we proposed RevGAN that automatically generates personalized product reviews from product embeddings as opposed to labels, which could output results targeting on specific products and users.

experimental results support that our proposed model indeed achieves significantly better generation results compared to the prior literature.


The learning rates for generator and conditional discriminator are fixed at 5e-5 and 1e-5 respectively.

regarding longtext generation tasks, for all the models above, the computation complexity might be too high, thus failing to provide satisfying results.


scores render the ability of the various metrics to reproduce human preferences (in terms of readability and relevance).


Second, to capture the dependencies on s <t , we explicitly model the dependencies among local latent variables by inputting z s t−1 to the sentence decoder, so that z s t is conditioned on z s <t and is expected to model smooth transitions in a long text.

generally speaking, to write a long text, a human writer first arranges contents and discourse structure (i.e., high-level planning) and then realizes the surface form of each individual part (low-level realization).


At the time of writing this paper, these models are the top performing models on Yelp, Amazon and Captions, with the D&R model of  showing stateof-art performance.

we ask annotators to rate each pair of generated sentences given the source sentence, on content preservation, style transfer strength, fluency, and overall success.


1 Prime Min Bertie Ahern of Ireland calls for general election on May 24.

moreover, entity mentions connecting sentences have also been used to extract non-adjacent yet coherent sentences (Siddharthan et al., 2011;Parveen et al., 2016).


We found that absolute grammaticality judgments were hard to achieve on Mechanical Turk; Turkers' ratings of grammaticality were very noisy and they did not consistently rate true article sentences above obviously noised variants.

we implement a deletion-based BiLSTM model for sentence compression (wang et al., 2017) and run the model on top of our extraction output.


where h i is a representation of the i-th sentence in the document.

we turn to other methods as described in the next two paragraphs.


Details of the human evaluation instruction are included in Appendix A.3.

typically, an encoder encodes the input x i to a semantic representation c i , while a decoder controls or modifies the stylistic property and decodes the sentence x i based on c i and the pre-specific style l i .


The auto-encoding reconstruction objective of the source domain is: where the encoder-decoder model (E, D) is shared in both domains.

we would like to thank the reviewers for their constructive comments.


The test accuracy of all these classifiers used for evaluation are reported in Appendix A.1.

the corresponding objective can be written as:  Figure 1: Illustration of the proposed DAST-C (left) and DAST (right) model.


Over the past couple of years, several variants of the encodeattend-decode model have been proposed.

we observe that the increase in Q-Metric for refined questions is because the RefNet model can correct/add the relevant Named Entities in the question.


Table 4 shows a comparison of existing and proposed summarization systems on the set of corpora in §5 except for Newsroom 12 .

lin and Bilmes (2011) claim that many existing summarization systems are instances of mixtures of such sub-aspect functions; for example, maximum marginal relevance (MMR) (Carbonell and Goldstein, 1998) can be seen as an combination of diversity and importance functions.


Social posts and news articles are biased to the position aspect while the other two aspects appear less relevant.

• AMI (Carletta et al., 2005): is documented meeting minutes from a hundred hours of recordings and their abstractive summaries.


Alternative approaches for enrichment exists, of course, but we wonder how worthwhile further efforts would be.

we refer to linguistic literature to argue that proper nouns, having no lexical meaning but rather just a referential function, cannot reliably be used in the evaluation of word-level translation systems.


In the original paper, early stopping is done by training for 50 epochs, and the best model regarding development accuracy is applied to the test set.

for this task, all languages are both development and target languages.


For example, Beam-10 yields 15.9% fewer search errors (absolute) than greedy decoding (57.68% vs. 73.58%), but Beam-100 improves search only slightly (53.62% search errors) despite being 10 times slower than beam-10.

exact search may not be practical, but it allowed us to discover deficiencies in widely used NMT models.


This paper systematically studies this temporal commonsense problem.

given the lack of evaluation standards and datasets for temporal commonsense, our second contribution is the development of a new dataset dedicated for it, MCTACO (short for multiple choice temporal common-sense).


To the best of our knowledge, there are no other available systems for the "stationarity", "typical time", and "frequency" phenomena studied here.

all other systems drop by almost 30% from F 1 to EM, possibly another piece of evidence that they only associate surface forms instead of using one representation for temporal commonsense to classify all candidates.


We observe that accuracy is stable across the different subsets for the single LM and GPT-2 117M with full scoring.

the proliferation of artificial-intelligence technologies that interact with human users (e.g., dialogue systems, recommendation systems, information retrieval tools) has led to renewed interest in common-sense reasoning (CSR).


(2018), is a rule-based system that uses search engines to gather evidence for the candidate resolutions without relying on the entities themselves.

their performance drops significantly on the non-associative subset, when information related to the candidates themselves does not give away the answer.


We carefully analyze the generated results of Pun-GAN with low overall scores in human evaluation.

the generation probability of the whole sentence x is formulated as To give a warm start to the generator, we pretrain it using the same general training corpus in the original paper.


In this work, we propose to incorporate language modeling as an auxiliary task to help QG via multi-task learning.

some other works try to generate questions from a text without answers as input (subramanian et al., 2018;.


Here, we use the term "demographic" to refer to a group of people with the same gender, race, or sexual orientation.

we present a systematic study of biases in natural language generation (NLG) by analyzing text generated from prompts that contain mentions of different demographic groups.


We demonstrated the utility of our approach in a cross-domain evaluation of textual entailment for fact verification.

first, we note that indeed the fully lexicalized model, which performs well in-domain, transfers very poorly to a new domain.


Basic NER: Token sequences which are labeled as NEs are replaced by the corresponding label, e.g., location, person.

since our approach is implemented as a data pre-processing step, it can potentially help any neural method that learns from text and is likely to be used out of domain.


Overlap Aware NER (OA-NER): This technique additionally captures the lexical overlap between the claim and evidence sentences with entity ids.

we split the training dataset into two partitions, with 40,904 records for training and 9,068 records for development.


Finally, the composed sentence vector is used for sentiment prediction.

these methods belong to the biased model, where the words in the tail of a sentence are more heavily emphasized than those in the header for building sentence representations.


In addition, our approach is easy to train and can be used as a drop-in complement for any text-spotting algorithm that outputs a ranking of word hypotheses.


Tritraining (Zhu, 2006) uses 3 classifiers to vote their classification results and labels them if all the classifiers agree with the prediction.

after the early-stopping in meta-epoch, we simply add all the unlabeled data with its pseudo-labels to training set, which finally brings significant performance gain.


Whereas mass processing of articles was once largely limited to keyword searches and citation crawling, modern natural language processing techniques can now deeply search for specific and broad concepts, explore relationships, and automatically extract useful information from text.

further, if two proposed regions overlap, their size and position can help determine which is more likely to fit a true document region more closely.


The architecture of the neural topic model is shown in Figure (1) and we describe the model in more details below.

with the sampledθ , for each document d ∈ N d , we can estimate the Evidence Lower Bound (ELBO) with a Monte Carlo approximation using L independent samples: By minimising the ELBO in Eq.


Examples include neural topic models which are typically built upon variational autoencoder (VAE) with an objective of minimising the error of reconstructing original documents based on the learned latent topic vectors.

we evaluate our model on the 20 Newsgroups † consisting of 18K documents, and NIPS (Tan  et al., 2017) consisting of 6.6k documents.


Here, the action aims to select words with high coherence scores and filter background words.

the topic features have only 30 dimensions so the performance is limited in comparison with CNN and RNN.


By training translation models using varying amounts of training data, we are able to explore performance across a wide range of resource levels and examine the reliability of performance patterns.

there is no established guidance on how to optimize the resulting MT-IR system.


They can capture semantic and syntactic properties of words as linear substructures, allowing relationships to be expressed as geometric translations (Mikolov et al., 2013b).

it is surprising that linear maps also outperform geometric translations, given that they are ultimately learned using translation vectors.


So far, we have obtained the adjective-noun pairs and value polarity relations for nouns.

syntaxsQLNet and sQL-Net solve the text-to-sQL task using a sequenceto-set structure.


Bosc and Vincent (2018) proposed a method that learns the long short-term memory (LSTM) (Hochreiter and Schmidhuber, 1997) that encodes a word definition into an embedding.

while the baseline CPAE outperformed our model on the two out of five datasets pertaining to word relatedness, our model consistently outperformed the baseline on the similarity benchmarks.


Relationships between words captured in embeddings are studied in terms of similarity or relatedness (Hill et al., 2015).

the model with the proposed method chose the correct words at both the generalized class and the details.


As far as semi-automatic approaches are concerned, Mihalcea and Moldovan (2001) conceived eXtended WordNet, a resource providing disambiguated glosses by means of a classification ensemble combined with human supervision.

supervision depends heavily on large quantities of reliable sense-annotated data which -especially for languages other than English -are poorly available.


further, more interpretability would be obtained by our method due to the structurally sparse attention distribution.


Our model outperforms BERT BASE by a stable margin especially in SST-2, SenTube-A, SenTube-T, and SciTail, with the improvements of 1.2%, 2.1%, 1.7%, and 1.5%,   respectively.

the attention matrices of models with different setups are visualized in Figure 1.


UMH (Alaux et al., 2019), was also evaluated on this benchmark.

to gradient based methods, our approach avoids word sampling and hyper-parameters that need to be tuned.


2) LSTM-AutoEncoder (Ryu et al., 2017): Recent work on OOD detection that uses only ID examples to train an autoencoder for OOD detection.

we use squared hinge loss and L2 regularization.


We expect optimizing only on L in and L ood will lead to lower confidences on ID classification, because the system tends to mistakenly reduce the scale of F in order to minimize the loss for OOD examples.

for example, Table 1 shows some of the utterances a chat-bot builder provided for training.


Some entities, such as R & B, are changed to lower case incorrectly, especially if not recognized as a proper noun.

this paper focuses on harnessing pre-trained neural networks with rulebased systems.


Following the previous works, we compute the BLEU score (Papineni et al., 2002) used in machine translation.

we follow previous works and apply three automatic evaluation metrics and one human evaluation for three indicative aspects: attribute compatibility, content preservation, and fluency.


To this end, we compute the softmax normalization over all incoming edges for node i: It provides us a way to compare all incoming edges in the same manner, rather than making a local decision via a bundle of edges from node j.

this model reduces the architecture search problem to learn continuous variables {α i,j k }, which can be implemented using efficient gradient descent methods.


In this paper, we study differentiable neural architecture search (NAS) methods for natural language processing.

we follow the same setup as in language modeling but with a different learning rate (0.1) and a different hidden layer size (256).


, e n } and a set of relations R = {r 0 , .

note that results with full softmax (Appendix D) demonstrate that our implementation of baselines is very competitive.Our implementation of ComplEx performs significantly better than ConvE (Dettmers et al., 2018) on two out of the three datasets, and come close to results of Lacroix et al.


When pretraining is combined with annealing, PPL improves substantially.

the girl is drinking water with a bucket .


BIOBERT is trained on PubMed abstracts and PMC full text articles, and CLIN-ICALBERT is trained on clinical text from the MIMIC-III database .

this corpus consists of 18% papers from the computer science domain and 82% from the broad biomedical domain.


We found that all of the models, including humans, relied more on the punchline of the joke in their predictions (Table 2).

when we look at the general population's opinion as well, we find a stark difference between their preferences and those of the Reddit users (Table  2).


To enhance the capability of domain adaptation for the sentence generator, we propose to first interpret dialogue acts in natural language so that new dialogue acts can be understood by the generator.

table 1 gives some examples of atomic templates used in DStC 2&3 dataset.


PaLM provides an intuitive way to inject structural inductive bias into the language model-by supervising the attention distribution.

their contribution is orthogonal to ours and not head-to-head comparable to the models in the  punctuation removal.


We observe no significant trend of favoring one branching direction over the other.

meaningful span representations are crucial in span-based tasks (Lee et al., 2017;Peng et al., 2018c;Swayamdipta et al., 2018, inter 1 We experiment with a strong LSTm implementation for language modeling (merity et al., 2018), see §3.


Our split has less training data and more test instances in the "Hard" category and less in "Easy" and "Medium".

(2018a)'s model on their English dataset but under our split.


(2018b), we make training, development and test sets split in a way that no database overlaps in them as shown in Table 1.

the characterbased model is less sensitive to question sentences, which is likely because characters are less sparse compared with words.


Thus, the parameters of the gating GCN are trained from the relevance loss and the usual decoding loss, a ML objective over the gold sequence of decisions that output the query y. Discriminative re-ranking Global gating provides a more accurate model for softly predicting the correct subset of DB constants.

experimental setup We train and evaluate on SPIDeR (Yu et al., 2018b), which contains 7,000/1,034/2,147 train/development/test examples, using the same pre-processing as .


we report the performance in probing tasks in Table 2.


We observe that before finetuning, the attention patterns on [SEP] tokens and periods is almost identical between sentences.

in this work, we show that pretrained language models, BERT (Devlin et al., 2018) in particular, can be used for this task to capture contextual dependencies without the need for hierarchical encoding nor a CRF.


It is clear that our approach outperforms BERT+TRANSFORMER.

we found it better to train our model to directly predict the ROUGE scores, and the loss function we used is Mean Square Error.


First, we reward the agent when it correctly guesses the answer after communication r self after = 1 y * =ŷ 1 , where 1 is an indicator function, in order to encourage it to incorporate information received from the other agent.

we consider two linguistic communities of population sizes N 1 and N 2 , which are trained independently from each other as fully-connected communities and have developed separate communication protocols.


This helped the annotators learn how to approach the task.

there is wide support for the idea that women experience more condescension than men do (Hall and Braunwald, 1981;Harris, 1993;McKechnie et al., 1998;Cortina et al., 2002;trix and Psenka, 2003), as reflected in the recent lexical innovation mansplaining, which can be roughly paraphrased as 'a man condescending to a woman'.


Condescending language use is caustic; it can bring dialogues to an end and bifurcate communities.

we then used Expectation-Maximization, as in Dempster et al.


The results are largely consistent with the extractive model scores, with CONTENTSELEC-TOR performing the best, the topic causing no statistically significant difference, and a large gap remaining between current models and a perfect system (the heuristic labels).

because we are interested in topic-focused summarization, in this work, it is also assumed that a high-level topic is provided as input, such as in Figure 1.


we believe this concept warrants further exploration in future work.


Core to our approach is the principle of the Information Bottleneck (Tishby et al., 1999), producing a summary for information X optimized to predict some other relevant information Y.

this choice is motivated by linguistic cohesion, in which we expect more broadly relevant information to be common between consecutive sentences, while less relevant information and details are often not carried forward.


In Figure 1, the alignment matrix will only have A 0,6,8 = 1 and A 1,2,3 = 1 which indicates that the first slot is aligned with "nation of Turkey", and the second slot is aligned with "silver medals".

the original dataset is annotated with SQL queries, but we only use the execution result for training.


For instance, programs like "select( previous( next( previous(argmax [all rows], column:silver) column:bronze)" are unlikely to have a corresponding question.

similar ideas of using abstract meaning representations have been explored with fully-supervised semantic parsers (Dong and Lapata, 2018;Catherine Finegan-Dollak and Radev, 2018) and in other related tasks (Goldman et al., 2018;Herzig and Berant, 2018;Nye et al., 2019).


Considering the large search space of possible programs, an alignment model that takes into account the full range of correspondences between program operations and question spans would be very expensive.

we instead optimize the following objective: where E[A] are the marginals of A with respect to p (A|x, t, h).


4 We find that 83.6% of questions in WIK-ITABLEQUESTIONS are covered by at least one consistent program.

we set the maximal number of production rules for generating abstract programs to 6 and 9, which leads to search time of around 7 and 10 hours for WIKISQL and WIKITABLEQUES-TIONS respectively, using a single CPU.


It is interesting to explore the use of some revision mechanisms when the initial steps go wrong.

an updated graph G t is produced and G t will be processed for the next time step.


We use a standard semantic parser provided by AllenNLP (Gardner et al., 2017), based on a sequence-to-sequence model (Sutskever et al., 2014).

we additionally pruned equivalent examples: if we generated a logical form with the structure A B, we pruned B A.


Others used a variational autoencoder by modeling unlabeled utterances (Kociský et al., 2016) or logical forms (Yin et al., 2018) as latent variables.

on two datasets, our method leads to 70.6 accuracy on average on the true distribution, compared to 51.3 in paraphrasing-based data collection.


After the paraphrase process was completed, we manually fixed typos in all generated para-  phrases.

conversing with a virtual assistant in natural language is one of the most exciting current applications of semantic parsing, the task of mapping natural language utterances to executable logical forms (Zelle and Mooney, 1996;Zettlemoyer and collins, 2005;Liang et al., 2011).


Lawrence and Riezler (2018) and Berant et al.

we train s t+1 from all the examples generated in iterations 0 • • • t. in every iteration more and more examples are labeled, and a better scoring function is trained.


We design ablation studies as follow: 1) w/o C: without word-character Containing graph(C-graph).

by using a single NVIDIA GeForce GTX 1080 Ti G-PU, We randomly select 10 training and testing epoch as samples.


For example, as shown in Figure 1, the boundaries and semantic knowledge of the selfmatched word " ¬::" (Beijing Airport) can help the character ":"(airplane) to predict an "I-LOC" tag, instead of "O" or "B-LOC" tags.

this work is also supported by the CCF-tencent Open Research Fund.


Since aligning word-character lattice structure for batch training is usually non-trivial, the lattice LSTM (Zhang and Yang, 2018) suffers from slow speeds in training and testing.

the output of j-th layer is a new set of node features, NF (j+1) = {f 1 , f 2 , ..., f N }.


The left side shows the overall architecture, including an encoding layer, a graph layer, a fusion layer, and a decoding layer.

t-graph performs poorly without cooperating with other graphs.


On DS datasets, the four methods targeting label distribution shift achieve consistent performance improvement, with averagely 3.83 F1 improvement on KBP and 1.72 on NYT.

this phenomenon verifies that shifted label distribution is making a negative influence on model performance.


We observe that, on human-annotated dataset, neural relation extraction models outperform feature-based models by notable gaps, but these gaps diminish when the same models are applied to DS datasets-neural models merely achieve performance comparable with feature-based models.

in the train set of KBP, 74.25% instances are annotated as NONE and 85.67% in the test set.


Low-Resource NER Following Cotterell and Duh (2017), we emulate truly low-resource condition with 100 sentences for training.

our models are optimized by mini-batch stochastic gradient descent (SGD) with learning rate 0.01 and batch size 10.


Currently, research efforts have derived useful discrete features from dependency structures (Sasano and Kurohashi, 2008;Cucchiarelli and Velardi, 2001;Ling and Weld, 2012) or structural constraints (Jie et al., 2017) to help the NER task.

furthermore, every entity type (i.e., row) has a most related dependency relation (with more than 17% occurrences).


Note that, the bilingual data needed for this approach is coarsely taken from the same domain.

task-specific models use all three splits for training, hyper-parameter tuning and evaluation, respectively, while models trained on bilingual data only use the test split for evaluation (so  table 2: AUC performance for our approach in comparison to the task-specific models directly trained on the evaluation datasets (results are comparable to supervised state-of-theart).


The main thesis underlying this work is that texts composed by members of a specific cultural group are distinct from those composed by members of another.

the translation and document representation components of the compound transformation are complementary in the sense that when translation is of high quality, a naive document representation suffices.


For each training size n, we sampled n training instances to train the models and tested on the entire dev/test set.

it should also be noted that this model misses the sentiment polarity inversion expressed by the word despite, due to its simple sequence-agnostic nature.


The iteration number iter used in dynamic routing algorithm is 3.

next, an Induction Module executes a dynamic routing procedure, in which the matrix transformation can be seen as a map from the sample space to the class space, and then the generation of the class representation is all depending on the routing-by-agreement procedure other than any parameters, which renders a robust induction ability to the proposed model to deal with unseen classes.


However, this sample-wise comparison may be severely disturbed by the various expressions in the same class.

munkhdalai and Yu (2017) proposed the meta Network, which learnt the metalevel knowledge across tasks and shifted its inductive biases via fast parameterization for rapid generalization.


A prediction is incorrect if it disagrees with what is known to be true.

second, we develop a systematic framework for mitigating inconsistency in models by compiling the invariants into a differentiable loss functions using t-norms (Klement et al., 2013;Gupta and Qi, 1991) to soften logic.


With the notion of errors, we can now focus on how to train models to minimize them.


5 Here, we will focus on the product t-norm to allow comparisons to previous work: as we will see in the next section, the product t-norm strictly generalizes the widely used cross entropy loss.

we count the percentage of unlabeled examples where the symmetry/transitivity loss is positive.


Indeed, the output that copies input gives maximal BLEU yet clearly fails in terms of the style transfer.

to generate plausible sentences with specific semantic and stylistic features every sentence is conditioned on a representation vector z which is concatenated with a particular code c that specifies desired attribute, see Figure 3.


Consequently, latent features are encouraged to cooperate and behave diversely to capture meaningful information for each sentence.

we consider three public datasets, the Penn Treebank (PTB) (Marcus et al., 1993;Bowman et al., 2015), Yahoo, and Yelp corpora He et al., 2019).


In addition, we apply three SOTA document vectorization methods -doc2vec (Le and Mikolov, 2014), InferSent (Conneau et al., 2017), and BERT (Devlin et al., 2019) -and use the extracted vectors to train a linear regressor to predict emotion distributions.

considerable research efforts on small sample learning are emerging recently (Lake et al., 2015;Shu et al., 2018).


Recently, a decent number of deep learning based models have been proposed for text generation.

in tasks that are set up to judge individual pieces of generated text (e.g., reviews, translations, summaries, captions, fake news) where there exists human-written ground-truth, it is better to use word-overlap metrics instead of adversarial evaluators.


Human annotation is able to assess the quality of text more directly than task based evaluation.

notably, maximizing the adversarial error is consistent to the objective of the generator in generative adversarial networks.


Training SBERT in the 10-fold cross-validation setup gives a performance that is nearly on-par with BERT.

smart batching achieves a speed-up of 89% on CPU and 48% on GPU.


These tasks include large-scale seman-tic similarity comparison, clustering, and information retrieval via semantic search.

it requires that both sentences are fed into the network, which causes a massive computational overhead: Finding the most similar pair in a collection of 10,000 sentences requires about 50 million inference computations (~65 hours) with BERT.


To explain the ERM framework for bipartite ranking, in this section, let us assume that we are given positive data drawn from the class-conditional probability densities p(x|y = 1) and p(x|y = −1), respectively.

they can also pick an optimization method that is suitable for their model such as Adam (Kingma and Ba, 2014) or AMS-Grad (Reddi et al., 2018).


If the class priors are not identical, we suggest not to use these metrics since it can be misleading because a good F 1 -measure in training data cannot guarantee a good F 1 -measure in the test data.

a good pseudo-labeling is an algorithm that can divide the data with a large gap (i.e., θ − θ is large).


Greek is becoming the best university in the world.

(Erhan et al., 2010;Dahl et al., 2012) showed the effectiveness of pre-training for tasks such as speech recognition.


The state-of-the-art performances have been significantly advanced for classification and sequence labeling tasks, such as natural language inference (Bowman et al., 2015), named-entity recognition, SQuAD question answering (Rajpurkar et al., 2016) etc.

results on four datasets show that PoDA can improve model performance over strong baselines without using any task-specific techniques and significantly speed up convergence.


However, most of the clusters are very small: 91.7% of the clusters contain only 2-5 questions.

because of this, most of the works focus on few simple dialog intents and fail to explore the realistic complexity of user intent space (Williams et al., 2013;budzianowski et al., 2018).


We use the neural encoder trained with the autoencoding objective to initialize the two utterance encoders in AV-KMEANS.

most of the clusters are very small: 91.7% of the clusters contain only 2-5 questions.


we generated Nyström representation of the Compositionally Smoothed Partial Tree Kernel function (Annesi et al., 2014) consistently with (Croce et al., 2017).


The Layer-wise Relevance Propagation assigns to each dimension, or feature, x d a relevance score R < 0 correspond to evidence in favor or against, respectively, the output classification.

more formally, let f : R d → R + be a function that quantifies, for example, the probability of x ∈ R d being in a certain class.


The consequence is that the variational posteriors would be more diverse in the latent space characterizing different input sequences, while the KL regularizer restricts the posteriors to match the Gaussian prior, with less "holes" in between where the decoder cannot be trained.

the estimation and maximization of the MI in the high-dimensional space are difficult.


Acknowledgements: We thank Prof. Vineeth Balasubramium, IIT Hyderabad, India for the many helpful suggestions and discussions.

diversity-based query strategies (Sener and Savarese, 2018) are used to address this issue, by selecting a representative subset of the data.


We analyze three algorithmic factors of relevance to sampling bias: (a) Initial set selection (b) Query size, and, (c) Query strategy.

it seems to have a inductive bias for class boundaries, similar to the above works.


2 For example, our VQA-CP ) bias-only model (see Section 5.2) uses the question type as input, because the correlations between question types and answers is very different in the train set than the test set (e.g., 2 is a common answer to "How many..." questions on the train set, but is rare for such questions on the test set).

for TriviaQA we use a 256 dimensional fully connected layer and 128 dimensional LSTMs, with highway connections between each BiL-STM (Srivastava et al., 2015) and a recurrent dropout rate of 0.2.


we consider biases beyond partial-input cases (Feng et al., 2019), and show our method is superior on VQA-CP.


Then build an augmented vector for each premise word by concatenating the word's embedding, the context vector, and the elementwise product of the two.

it is the features we will use in our bias-only model.


a series of differentiable transformations h k : where z k is the vector of activations in the k-th layer and the final output z K consists of the logits for each class.

the specification against which we verify is that a text classification model should preserve its prediction under character (or synonym) substitutions in a character (or word) based model.


If we only allow a character to be replaced by another character nearby on the keyboard, already for this short sentence we need to exhaustively search over 2,951 possible perturbations.

the true class logit should be greater or equal than those for all other classes y, which means the prediction remains constant.


Transforming it into a three-player game helps to improve the performance of the evaluation set while lower the accuracy of the complement predictor.

table 5 shows the performance of subjective evaluations.


Generator: The generator extracts R and R c by generating the rationale mask, z(•), as shown in Eqs.

3 This problem happens because they have no control of the words unselected by R. Intuitively, in the presence of degeneration, some key predictors in X will be left unselected by R. by looking at the predictive power of R c , we can determine if degeneration occurs.


For our sampling-based objectives, we use k = 1024 samples, and the unigram distribution as p n .

2 may be simplified as following: We can further accelerate learning by using importance sampling on the second term, and because of the logarithm applied to the sum, we obtain an objective that is equivalent to composing the approximated softmax with γ-divergence, as can be seen in Appendix A.4.


The large vocabulary sizes encountered in training corpora arguably stem from the fact that the frequency distribution of words in a corpus of natural language follows Zipf's law (Powers, 1998).

tracking these values during training 13 shows that they all behave very similarly to perplexity.


There has been methods that aim to speed up neural CRF (Tu and Gimpel, 2018), and to solve the Markov constraint of neural CRF.

our model outperforms all the baselines on all the languages.


We do not use adversarial training (Goodfellow et al., 2015) because it would require running an adversarial search procedure at each training step, which would be prohibitively slow.

their models were still be fooled by running a more expensive search procedure at test time.


This is an unrealistic setting that limits the applicability of such models in real world scenarios.

the performance of the simple baselines on this dataset is shown in table 4.


• Domain of learning task: Indicates the domain of the current classification task.

we discretize this ratio into three values, corresponding to the upper (HIGH), lower (LOw) and middle two interquartile ranges (MEDIUM) for the value of the ratio over all explanations in our data.


A notable issue in learning from explanations, which we do not model here is that a teacher's multiple explanations of a concept can have a large variance in their utility to a learner.

automated learners should be capable of learning from a blend of observations, explanations and clarification.


(2018), who train loglinear classifiers (with parameters θ) using natural language explanations of the individual classes and unlabeled data.

from the learner's perspective, over time the size of the unlabeled data decreases, and the labeled examples increase (see Equation 2).


To alleviate the concerns to some extent, we consider the case in which the leakage is in the induced negative words distribution.

this setup lends itself to a discriminative training approach, which we demonstrate to work better than generative language modeling.


Language models are traditionally evaluated with perplexity.

we tackle these three issues: we propose an ASRmotivated evaluation setup which is decoupled from an ASR system and the choice of vocabulary, and provide an evaluation dataset for English-Spanish code-switching.


(2018b) construct graphs from procedural text to track entity position to answer when and if entities are created, destroyed, or moved.

the node weight and edge weight are equivalent to the number of merge operations + 1.


We describe how symbolic graph representations of knowledge can be constructed from text.

we build graphs from substantially longer multi-document input and use them for multi-sentence text generation.


(2017) leverage the source-side monolingual data to train the NMT system by learning reward func-4200 tion in a reinforcement learning framework.

in this work, we study how to use both the source-side and targetside monolingual data for NMT, and propose an effective strategy leveraging both of them.


One of knowledge graph completion tasks is link prediction, predicting new triples based on existing ones.

entities only have one triple during training will make MetaR unable to learn good representations for them, because entity embeddings heavily rely on triples related to them in MetaR.


modaira, 2000;Quiñonero-Candela et al., 2009;Daume III, 2007;Ben-David et al., 2010;Blitzer et al., 2011;Pryzant et al., 2017) or domaininvariant features (Ganin and Lempitsky, 2015;Tzeng et al., 2014).

many DRO problems admit efficient batch optimization procedures based on Lagrangian duality .


Topic CVaR robustness beyond subpopulation shift.

unlike our work, these approaches use topics at test time by inferring the domain from the input variable x.


Finally, language modeling objectives have previously been used for domain adaptation of text classifiers (Ziser and Reichart, 2018), but this prior work has focused on representation learning from scratch, rather than adaptation of a pretrained contextualized embedding model.

we are also interested to more thoroughly explore how to combine domain-adaptive and task-specific fine-tuning within the framework of continual learning (Yogatama et al., 2019), with the goal of balancing between these apparently conflicting objectives.


Another source of out-of-vocabulary words is the addition of a silent e to the end of many words.

interestingly, domain-adaptive fine-tuning has no impact on the performance on the original tagging task.


On the other hand, the last example is very difficult for humans (row 4), possibly due to the relatively neutral text.

with the methods proposed here those outputs can be used to learn the latent parameters of the data to focus in on what exactly is working well and what isn't with respect to the models being tested and the data used to train them.


PIE2 However , there are always two sides to the stories .

the alternative of predicting insert independently at each gap with a null token added to Σ a performs 2.7 F 0.5 points poorly (table 6 row 4 vs row 2).


And a higher BLEU score implies a better translation quality.

flowSeq has much larger gains in the decoding speed w.r.t.


However, they are all relatively small, often created only as a test set.

the Winograd Schema is given as the "premise".


Example: (cheek, brow, red) is sensory and (grapes, wine, fruit) is logical.

the current model is able to identify "skyscraper" as being taller than an "apartment", but fails to identify neither as taller than a "giraffe".


Even fine-tuning the pre-trained model with taskspecific dataset may take several hours to finish one epoch.

ideally, we should pre-train BERT 6 [Large] and BERT 6 [Base] from scratch, and use the weights learned from the pretraining step for weight initialization in KD training.


We also propose two different strategies for the distillation process: (i) PKD-Last: the student learns from the last k layers of the teacher, under the assumption that the top layers of the original network contain the most informative knowledge to teach the student; and (ii) PKD-Skip: the student learns from every k layers of the teacher, suggesting that the lower layers of the teacher network also contain important information and should be passed along for incremental distillation.

the distance between the teacher's prediction and the student's prediction can be defined as: where c is a class label and C denotes the set of class labels.


the food is good , but the food is good this place is a great place to go for lunch .

the optimal variational posterior q * ∈ Q is then the one that minimizes the KL divergence Based on this, variational autoencoder (VAE) (Kingma and Welling, 2014) is proposed as a latent generative model that seeks to learn a posterior of the latent codes by minimizing the KL divergence between the true joint density p θ (x, z) the variational joint density q φ (z, x).


VAE is able to learn a continuous space of latent random variables which are useful for a lot of classification and generation tasks.

we assume that the members of variational family Q are dimensional-wise independent, meaning that the posterior q can be writ- The simplicity of this form makes the estimation of ELBO very easy.


by changing the kernel construction in Section 2.2.2, we can define a larger space for composing attention.


Formally, a sequence with f i ∈ F being the nontemporal feature at time i and t i ∈ T as an temporal feature (or we called it positional embedding).

incorporating positional embedding into the attention mechanism may still improve performance.


the ones that could be informative) varies from 32% (MRPC) to 61% (QQP) depending on the task.

manual inspection of self-attention maps for both basic pre-trained and fine-tuned BERT models suggested that there is a limited set of selfattention map types that are repeatedly encoded across different heads.


(2019) extended this work to using multiple layers and tasks, supporting the claim that BERT's intermediate layers capture rich linguistic information.

the gain from disabling a single head is different for different tasks, ranging from the minimum absolute gain of 0.1% for StS-B, to the maximum of 1.2% for MRPC (see Figure 8).


Frankle and Carbin (2018) showed that widely used complex architectures suffer from overparameterization, and can be significantly reduced in size without a loss in performance.

it is important to note that across all tasks and datasets, disabling some heads leads to an increase in performance.


The second baseline, referred as "NMT + Document Translation", is to treat the weakly aligned doc-   10.9 14.6 ---- NMT (Lample et al., 2018) 17.2 21.0 19.7 20.0 21.2 19.5 PBSMT (Lample et al., 2018) 17.9 22.9 --22.0 23.7 PBSMT + NMT (Lample et al., 2018) 20  uments as two long sentences and use them as a bilingual sentence pair to train NMT models.

2) We leverage the topic consistency of two weakly paired documents and learn the sentence translation model by constraining the word distribution-level alignments.


Different language versions of Wikipedia pages about the same entity/event are usually created by different people speaking different native languages, and therefore most sentences in two weakly aligned documents are not aligned.

the challenge is how to mine such sentence pairs from those document pairs.


Ablation Study Our method leverages weakly paired documents in two ways.

if we translate an article from English to French sentence-by-sentence, the word distribution of the translated article should be generally similar to the word distribution of the corresponding article in French.


We employ policy gradient methods to train the first agent with the target language loglikelihood as reward.

+LM ein alter mann in einer jacke beobachtet einen tisch .


(2018) show that up to a certain layer performance of representations obtained from a deep LM improves on a constituent labeling task, but then decreases, while with representations obtained from an MT encoder performance continues to improve up to the highest layer.

for MLM, representations initially acquire information about the context around the token, partially forgetting the token identity and producing a more generalized token representation.


Future work should move beyond the restrictive assumption by exploring new methods that can, e.g., 1) increase the isomorphism between monolingual spaces (Zhang et al., 2019) by distinguishing between language-specific and language-pairinvariant subspaces; 2) learn effective non-linear or multiple local projections between monolingual spaces similar to the preliminary work of Nakashole (2018); 3) similar to Vulić and Korhonen (2016) and Lubin et al.

even in these settings when the comparison to the weakly supervised FULL-SUPER+SYM is completely fair (i.e.,  (c) Swedish→ L2 Figure 3: A comparison of BLI scores on "easy" (i.e., similar) language pairs between the fully UNSUPERVISED model and a weakly supervised model (seed dictionary size |D 0 | = 200 or |D 0 | = 500) which relies on the self-learning procedure with the symmetry constraint (FULL+SL+SYM).


Current unsupervised adversarial approaches show that it is possible to build a mapping matrix that aligns two sets of monolingual word embeddings without high quality parallel data, such as a dictionary or a sentence-aligned corpus.

(2018) and Chen and Cardie (2018), when setting β to less than 0.01, the orthogonalization usually performs well.


Our approach is based on the intuition that mapping across distant languages is better done at the concept level than at the word level.

the performance of directly trained models is limited by their vocabulary size.


A small distance reflects a high probability for an entity pair to be aligned as equivalent entities.

for DBP100K, the dimensionalities are set to 100, 50, and 50, respectively.


As shown in Table 2, MAN and HMAN consistently outperform all baselines in all scenarios, especially HMAN.

we adopt SGD to update parameters and the numbers of epochs are set to 2,000 and 50,000 for MAN and HMAN, respectively.