Long short-term memories ("LSTMs": Hochreiter and Schmidhuber (1997)), a particular variant of RNN, have become particularly popular, and been successfully applied to a large number of tasks: speech recognition (Graves et al., 2013), sequence tagging (Huang et al., 2015), document categorisation (Yang et al., 2016), and machine translation .

although CRFs are adept at capturing local structure, the problem does not naturally suit a linear sequential structure, i.e.


Using the scoring function in Equation 7, we calculate the score of the sequence y normalised by the sum of scores of all possible sequencesỹ, and this becomes the probability of the true sequence: We train the model to maximise the probability of the gold label sequence with the following loss function: where p(y (n) |x (n) ) is calculated using the forward-backward algorithm.


The grid is summarized into a vector of transition probabilities.

2017propose a convolutional neural network formulation for AA tasks (detailed in Section 3).


Grammar Error (GE) To detect GEs, we use the LanguageTool proofreading program 5 to detect all GEs (e.g., redundant phrases and typos) in all training set justifications.

these results are obtained when n is set to 12.


Style Our fourth baseline captures aspects of an argument's style.

given an argument, we employ the following steps to extract references and clean up internal citations.


This task recently draws increasing research concerns and is helpful to many downstream applications, such as user and product recommendation.

we train the word embeddings through using the training and developing sets of each dataset with word2vec tool (Mikolov et al., 2013).


Semantic information maybe provides opposite opinions in different contexts.

for sentiment classification with CNN, LSTM vectors, we proposed a 3-layer neural network which efficiently takes advantages of these vectors.


Simverb-3500 (Gerz et al., 2016) was introduced to provide researchers with a testbed for verb relations, a specific yet important class of words that was less common in earlier word-level similarity data sets.

it appears that similarity, relatedness, and motivational alignment are more highly correlated with one another than perceived actor congruence.


The features are generated for each segment 9 for tokens in the training data x (i) .

it can be shown, more generally that the order restriction in (7), (8) leads directly to where the order between any two edit operations in δ (i) , called σ k , σ l is free only in the case of multiple insertion acting on the same position of a string, i.e.


3%, .43-, '(, 5+'/4 transition score matrix A mn , which indicates the score of jumping from m-th tag to the n-th tag.

we directly use ground-truth segments as inputs and predict a DA label for these segments.


The experimental results show that these factors indeed influence the emotional trend.

early stopping is carried out on validation set during training.


However, it is impractical to ask users to provide explicit feedbacks when the agents' responses displease them.


Both Q −2 and Q −1 contains negative emotions and possibly lead to dissatisfaction.

as described in the introduction, the task is to predict the impending dissatisfaction given n + 1 round context.


PARC3, derived from the Penn Discourse Tree-Bank, consists of Wall Street Journal news articles in which attributions have been manually annotated.

to approximate the perceptions of the general public, we use crowdsourced human judgments in creating the ground truth verifiability-scored dataset.


Spot checks of 100 pronoun-containing attributions in PARC3 showed that this produced reliable, grammatical interpolations.

one important way in which attributions can differ, particularly with respect to news reporting, is in their verifiability, the ease with which an attribution's fidelity to the source can be checked.


We see that our predictions mostly accord with the personality literature, and that domain adaptation often strengthens the correlations that we expected and weakens the correlations that had been predicted from language, but were not expected.

the results suggest that tSDA works well when the county-level scores are further aggregated to the state level (Rentfrow and Jokela, 2016).


We know that personality at the level of individuals is relatively stable over time; average personality in a county level should be extremely stable from year to year.

tSDA corrects for the different word distributions between Facebook and twitter and for the varying word distributions across counties by adjusting target side word frequencies; no changes to the trained model are made.


These two learners are complementary in the two-path bootstrapping system.

meanwhile, Twitter user @purplhaze42 is a self-proclaimed anti-racist and anti-Zionist.


In order to ensure our annotators have a complete understanding of online hate speech, we asked two annotators to first discuss over a very detailed annotation guideline of hate speech, then annotate separately.

among them, only 45 phrases were seen in existing hate slur databases while the other terms, 261 phrases in total, were only identified in real-world tweets.


several recent works have focused on learning generative models of product reviews, either to generate reviews per se, or as a means of learning user and item attributes (Lipton et al., 2015;Radford et al., 2017;Dong et al., 2017;.


A sentence is regarded as a question if the last character is "?"

the tables show the number of times each method on a row was evaluated higher than another method on a column.


Lead Question is a strong baseline in particular.

in this study, the vanilla encoder-decoder and the attention models use Long Short-Term Memory (Hochreiter and Schmidhuber, 1997), and Gated Recurrent Unit (GRU) (Cho et al., 2014) is used in the model with the copying mechanism.


There was no length constraint in terms of the number of characters or words.

each instance was evaluated by 3 evaluators.


In Example 3 in Table 4, the pronoun "it" in the main question "Do you like it?"

the model failed to decode the sequence.


The evaluation criteria were "grammaticality" and "focus", which are based on the criteria used in DUC.

we assumed that a better summary would contains more focused content in a shorter output.


As the METEOR and ROUGE evaluations show, this is not a problem for the final result.

if n concepts are selected, n units of flow are sent from the root over the edges of the graph and each selected concept consumes one of them.


This eases the burden of the decoder by shifting the labeling task to the selector.

it may seem redundant since the RNN gated unit already has the sophisticated gating mechanism such as the GRU unit  and the LSTM unit (Hochreiter and Schmidhuber, 1997).


We can motivate improvements to Algorithm 1 by viewing it as a greedy approximation to the optimization problem which chooses a set of scored predictions according to: Given the view that CAEVO is providing a solution to the objective in Equation 1 using Algorithm 1, it is straightforward to see possible directions for improvement.

in the past, labeled corpora used for training and evaluation contained only small subsets of pairs of events and times.


DT, SVM, RF, GB and AB stand for Decision Tree, Support Vector Machine, Random Forest, Gradient Boosting, and AdaBoost, respectively.

environmental influences, which are usually called Gene-environment interaction (Gxe), have been considered as important factors and have extensively been researched in biology.


From this observation, we anticipate that if we address the GxE task focusing on the number by combining the two models, the performance will exceed the current best score, 0.426.

the static RNN decoder outperforms other models.


In our paper, we propose a simple PMI-based approach, which utilizes local mutual frequencies in the course corpus, to calculate the phraseness score for each candidate.

our experimental results verify that over 98% of the candidates have vector representations in this way.


The choice of the fake identities was based on two practical considerations.

in the future, with largerscale data collections, we will be able to work with an extended set of gender options.


By contrast, we explore the discriminatory capability of the unstructured clinical narratives to infer the possible diagnoses.

such models heavily rely on large labeled data, and lack the ability to capture inherent ambiguities and complexities of a clinical scenario.


In the following, an overview of the participants of the user study, the study design as well as the results are presented.

as discussed in Section 5.3.1, there exist significant interactions of generation method, nationality and original statement on the preference of elaborateness/indirectness.


Here, CG statements are harder to understand than HG ones.

section 3 gives an overview of the Ds our approach is employed in.


In this work, we propose to follow the work of Bamman et al.

6 For each tweet, (1) we search for a location mention from GeoNames 7 in the text message, then (2), we verify if the user enabled the geo-coding function of his/her device, and (3) we look for location information from the stored user's profile.


Similarly to Kwok and Wang (2013), this study focuses on a specific community and is of limited scope.

this data set can be thought as a hard test for classifiers as non racist tweets may contain slurs unlike most works so far, which assess their models based on the hypothesis that non racist tweets usually contain general vocabulary and do not exhibit any critical content.


rule-based means the conventional rulebased method proposed by Sasano et al.


Since " (totemo, such)" and " (tanoshii, fun)" are Out Of Vocabulary (OOVs), the traditional system cannot generate the correct word segments or POS tags.

twitter data contain manually annotated estimates the word segmentation and POS tagging, we have to assess whether or not adding the normalization candidates negatively affects a system.


Figure 8 shows the average evaluation results on the simulated data sets.

the system returns true if an original copy is found in the database; otherwise, the system returns false and adds the document to the database.


The system returns true if an original copy is found in the database; otherwise, the system returns false and adds the document to the database.

(A branching is a subgraph where each vertex has an in-degree of at most 1.)


SQuAD: SQuAD is a machine comprehension dataset constructed on 536 Wikipedia articles (23K paragraphs), with more than 100,000 questions.

although these improvements are not as large as those we achieved with multiple-turn reasoning, they are are still considerable and imply that robust representations of words is an important building block to strong RC models.


The document score ds is defined by Term Frequency-Inverse Document Frequency (TF-IDF), which is written as where tf (w ′ , d) is the frequency of the word w ′ in the document d, idf (w ′ ) is the inverse document frequency of the word w ′ , and ℓ(d) is the length of d. The VQA feature uses document-local statistics (except for IDF) and counts only in the topk search results.

and  demonstrated that the system performance was boosted by multi-step inference that combines, e.g., taxonomic knowledge ("N.Y. is in the northern hemisphere") and general law ("The summer solstice in the northern hemisphere is in June").


The test sentences (i.e., hs) included those taken from NCTUA world history exams and hence the latter task setting is close to ours.

the solver needs a more rigorous inference about temporal relations than about the matching of other NEs.


In this work, we develop the dataset DailyDialog which is high-quality, multi-turn and manually labeled.

we examine our Dai-  lyDialog dataset by how many conversations are ending or positive emotions (i.e., happy), and find 3,675 (28.0%) "happy" dialogues.


a women sings I The kid is sliding down a tan plastic slide.

(2015) which was their best performing individual neural model.


We believe this is because the sentences in FN+ contain language that is rarely seen.

a series of injections are used to battle a type of cancer in patients because patients have a special type of drug which counteracts this sickness.


This has multiple advantages: it allows us complete control over the learning process, and we can train on larger and more diverse corpora.

they have some drawbacks that limit their usefulness for specific, low-resource domains.


The dimensions of w j−1 andŝ j−1 are d. The attention score α j (i) is calculated as follows: The context vector c j for generating a targetlanguage sentence is calculated by The attentional vectorŝ j is calculated by using the context vector as follows: and then using the state of this hidden layer, the probability of the output word y j is given by where W c and W s represent weight matrices 3 .

the vertical axis represents a source sentence.


In fact, the German sentence is an idiomatic translation of the English sentence.

padó and Lapata (2005), as one of the earliest studies on annotation projection for SRL using parallel resources, apply different heuristics and techniques to improve the quality of their model by focusing on having better word and constituent alignments.


Note that PBMT 1-best results are equivalent for PBMT in × NMT in and PBMT in × NMT out since the same PBMT model is used and NMT is not relevant.

a very deep lattice with a large beam begins to approach the unconstrained search space of standard decoding in NMT.


Many JSL words and syllables, which are basic units that compose words, are newly coined to meet these requirements, e.g., (Japanese Federation of the Deaf, 2011).

contacts indicate whether and when both hands have contact in the syllable execution.


Therefore, such bounds should be used to decide the number of dimensions, instead of trial and error.

in this paper, we show that the dimension should instead be chosen based on corpus statistics.


This is because once the lower bound is reached, the errors introduced due to the violation of equality constraint are removed.

one needs to think of embedding algorithm's capability to capture these differences effectively, which is governed by its hyperparameters.


The dataset contains 74 restaurant reviews typed with phrase suggestions.

we can obtain an unbiased estimate of the reward that h ✓ would incur using importance sampling:R the variance of this estimate can be unbounded because the importance weights h ✓ (y i |x i )/p i can be arbitrarily large for small p i .


We formulate label assignment problem as a resource allocation problem, where we maximize total value of assigned labels.

exploiting this directional and conditional dependency between the labels should allow us to predict a more coherent set of labels.


Therefore MLL algorithm should take advantage of this correlation embedded in the training data to always infer the label tree when mountain is one of the label.

• Yahoo News MLL Dataset (English) 1 : This is one of the few publicly available large scale datasets for MLL.


Our techniques involve solving simple constraint optimization problems over the outputs produced by any MLL approach and the result is a refined version of the input prediction by removing spurious labels and reordering the labels by utilizing the additional label-dependency side information.


We plan to investigate these matters in more detail in the future.

5 The first one 4 Experiments were done in a machine with an Intel Xeon E5-2687W 3.10GHz as CPU and a GTX TITAN X as GPU.


For instance, a bag-of-words (BOW) model uses one-hot encoding as vectors and it is still a strong baseline for many tasks.

averaging vectors discards any word order information from the original text, which can be fundamental for more involved NLP problems.


Unlike standard evaluation for IR, all the metrics are positively correlated in general.

the false negatives (FN) generated by the segmenter can be obtained by subtracting tP from PP, total number of words return by the segmenter.


Most stateof-the-art systems require tens of thousands of annotated sentences in order to obtain high performance.

we experiment on 15 languages from the crosslingual named entity dataset described in Pan et al.


Another large scale dataset, containing user generated image descriptions for 1M images, created by quering Flickr.

we follow Black's (1979) observation that a metaphor is essentially an interaction between two terms, creating an implication-complex to resolve two incompatible meanings.


AAS and MAS deal with many-to-many and oneto-many word alignments, respectively.

given H identified using the Hungarian algorithm (Kuhn, 1955), HAS is computed by averaging the similarities between embeddings of the aligned pairs of words.


As expected, we get much higher gains for Urdu compared to Telugu.

to demonstrate this, we obtain translations for embeddings of 3 languages 2 and show the results in table 1.


We do 10 iterations of retrofitting process for all our experiments because 10 iterations are enough for convergence (Faruqui et al., 2015) and also using the same value for all experiments avoids overfitting.

w i being the weights, α, β are set as : Further reduction in weights of noisy words can be done by taking powers of cosine similarity.


Roth and Lapata (2016) reported that F 1 decreased by 10 points or more when path embedding was excluded.

these values are the means of five different models trained with the same training data and hyperparameters.


Annotations with more than 1 token were split into a sequence of tokens (e.g., BAB/BUP to BAB, BUP).

the audio was manually transcribed and aligned with videos; the gestures were manually annotated and segmented according to video and audio recordings.


As with the results on gesture semantics, this suggests that multimodal meaning and meaning of iconic gesture relies heavily on speech, in accor- Figure 5: Featuring ranking according to coefficient values (weights assigned to the features, see (Lücking et al., 2010) for the details of the annotation scheme).

it remains unclear how to computationally derive the semantics of iconic gestures and build corresponding multimodal semantics together with the accompanying verbal content.


a simple way to have a flexible non-linear model over the data.

we follow closely the definition of GPs in Rasmussen and williams (2006).


Many efforts have been made to construct multi-dimensional affective lexicons, such as ANEW for English (Bradley and Lang, 1999;Warriner et al., 2013), CVAW for Chinese (Yu et al., 2016), and other languages (Montefinese et al., 2014;Imbir, 2015).

another method is to use sentiment related features, such as total count of sentiment tokens, total sentiment score, maximal sentiment score, etc.


between the VAD representation and word embedding, it is not clear which one is more effective for sentiment analysis.


Recognizing the commonalities between ACC and ATE task can boost the performance of both of them.

the system with window approach cannot be jointly trained with that using sentence window approach.


Although the majority approach does not model inter-relation and ambiguity of the signals, we assume that more signals, and thus longer prefixes, give better or the same prediction 6 .

this work investigates whether DRs can be identified incrementally based on human performance.


All recorded dialogues with the total length of 21 hours have been manually transcribed and annotated with speech acts and semantic labels at each turn level.

semantic Label Given a sequence of annotated intent tags and associated attributes for each history utterance, we employ a BLsTM to model the explicit semantics: where intent t is the vector after one-hot encoding for representing the annotated intent and the attribute features.


Neural conversation systems, typically using sequence-to-sequence (seq2seq) models, are showing promising progress recently.

λ in MMR scores was empirically set to 0.5; the beam size was 30.


These methods add a diversity punishment term to the scoring function in beam search; it is hard to balance this term with other components in the function.

the fine-grained selection improves the diversity and retains the quality of subsequences at each time of decoding, so the generated replies are of good quality and diversity.


It runs many services simultaneously and the dynamic combination of the 2 GHz Intel Core i7 processor and 16 GB RAM enables the acute ability to focus on concurrent tasks with minimal performance degradation.

we compare these methods in experiments: Basic, +Templated, +Structured and Full are ranked based on basic features, basic+templated features, basic+structured features and ba-sic+templated+structured features respectively; wordCount and AttriCount are rankers which sort candidates in the descending order of word count and attribute count respectively; OracleBLEU is an oracle ranker which always chooses the top candidate in term of BLEU as the answer (can be seen as the upper bound of ranking).


The proposed structured information and templated information are helpful for deciding what to say and how to say for description generation.

we extract attributes which mentioned in a reference description, and compare them with those in its corresponding generated descriptions.


Another solution is to use optimization techniques such as integer linear programming (ILP) to infer the scores of sentences with consideration of the quality of a whole summary (McDonald, 2007).

second, we propose a novel deep learning network for estimating Q-values used in RL.


This research was supported by Basic Science Research Program through the National Research Foundation of Korea(NRF) funded by the Ministry of Science, ICT & Future Planning(NRF-2015R1C1A2A01051685)

the differences of ROUGE scores between the models are relatively small.


They have also been developed for the update summarization task, with work such as (Li et al., 2015) about the weighting of concepts.

we adopted this method to represent sentences and used only the embeddings of unigrams since bigrams and phrases are generally not well covered by the existing pre-trained embeddings 1 .


We only considered sets A and B for all the datasets.

in practice, most proposed similarity measures for AS are subject to a time efficiency problem which tends to increase with the quality of the similarity measure.


In ML estimation,α m estimated from document D is determined asα Thus, ML estimated generative probability of w in the hyperspherical QLMs is formulated as where v is a normalized word embedding of v. In MAP estimation,α m estimated from document D is determined aŝ where C is document collection in which all retrieved documents are included.

wCwV used cosine distance between two document vectors composed by adding word embeddings with IDF weights (Brokos et al., 2016).


These setups are equivalent to the evaluation in Salakhutdinov and Hinton (2009); Larochelle and Lauly (2012).

the word similarity is calculated as where w is the word embedding normalized to a unit length for w. This paper proposes a novel probabilistic IR method based on hyperspherical QLM that leverages pre-trained word embeddings as random variables for directional probabilistic models.


Candidate Pruning: Calculation of score for all triplets from the dataset is an memory and time wise prohibitive operation.

question Encoding question is represented as sequence of words (x 1 , x 1 , ..., x |q| ).


We use SimpleQuestions 3 dataset introduced by (Bordes et al., 2015) for training and testing purposes.

the real challenge for leveraging such knowledge in practical applications consists of mapping natural language questions to their corresponding entries in thus enhanced KBs.


Visualization reveals that it is much easier for neural net models to identify and concentrate on the relevant regions of contexts using the dependency chain representation, which further verifies that additional event words in a sentential context are crucial to predict target event temporal status.

all the models were trained using RMSProp optimizer with the initial learning rate 0.001 and the same random seed.


Our proposed model uses LSTM model (Gers, 2001) as the basic classifier.

lSTM has no obvious model advantage in this set of training data.


Other text normalization task In this study, we evaluated our methods with Japanese dialect data.

when we sample dialect pattern m s from MatchedList, we use two types of p(m s |m t ).


Then, the (normalization) probability of t given dialect sentence s can be written as where θ represents a set of all model parameters in the encoder-decoder model, which are determined by the parameter-estimation process of a standard softmax cross-entropy loss minimization using training data.

we set p(m s |m t ) = F ms,mt /F mt , where F ms,mt is the joint frequency of (m s , m t ) and F mt is the frequency of m t in the extracted morphological conversion patterns of training data.


General purpose language identification tools such as TextCat (Cavnar and Trenkle, 1994) and langid.py (Lui and Baldwin, 2012) can identify 50-100 languages with accuracy of 86-99%.

by introducing additional texts of linked users, the accuracy of the proposed model improved by 1.67-5.56, except for Portuguese.


As expected, the performances were fundamentally inferior to the shared layers architecture.

for language variety identification, sparse traditional models have shown stronger performance than deep neural models (Medvedeva et al., 2017).


An explanation for such behaviour may be found on the fact that long sentences are also difficult translations.

with the attention mechanism (Bahdanau et al., 2014), a decoder is used to decide which part of the source sentence to pay attention to and predict corresponding word representation at time t together with history predictions before time t. Then, a softmax is used to restore the word representation to natural target words.


That is, all the instances are regarded as equal and used to train the NMT model equally.

note that the procedure restarts using the entire training set every 3 epochs.


We test several normalisation strategies: • by batch size (number of training instances) • by (target) sentence length • without normalisation (longer sentences were always assigned a higher perplexity) Experimental results show no significant performance differences for any of the strategies.

multiple GPUs or T-PUs 1 are needed which requires additional replication and combination costs.


Training efficiency has been a main concern by many researchers in the field of NMT.

we emulate a human spending additional energy on learning complex concepts.


SrcVoice represents the voice of the source side.

to be Active the morphology and the crystallinity of the dots depended on the temperature .


In bidirectional attention models, interface vectors are transformed vectors of h t with alignment model and target word, but we can still use h f t h b t .

if the optimal distance is sufficiently small, then this method will guide the training in early updates and preserve the true optimal with respect to cross entropy with restriction of generating negative sentences.


We rely on the article's MeSH terms only to select RCTs.

this abstract was taken from (Krogh et al., 2016) and several sentences have been removed for the sake of conciseness.


This also applies for CVs that have separate section headers for identification, such as Name, Email etc.

multiple industrial solutions, such as Text Kernel 1 , Burning Glass 2 and Sovren 3 , have attempted to solve the problem at hand, and are offered as commercial products.


For individuals, it is possible to add value by designing CV improvement and organization tools, enabling them to create more effective CVs specific to their career objectives as well as maintain the CVs easily over time.

for sections that are intuitively more complex, show some (meaningful) confusions across classes.


We see these results as supporting the assumption that natural user interfaces should respond to multimodal input, where possible, rather than just language alone.

there are 5371 sketch/photograph ensembles in our image retrieval evaluations.


The decoder predicts a word conditioned on the correct word sequence (y t−1 1 ) during training, whereas it does with the predicted word sequence (ŷ t−1 1 ) at test time.

the key difference from MLE is that the reward is not restricted to tokenlevel accuracy.


We propose a novel method to analyze Japanese review documents with exploiting the visual information of ideograms and logograms.

we chose 2,709 characters from 3,630 characters.


Despite this, we demonstrate improvements on NER F-scores with our multi-task model.

we are interested in leveraging recent advances in learning BWE from comparable corpora (Hermann and  Figure 1: Our multi-task framework, which trains bilingual word embeddings from comparable corpora while optimizing an NER objective on the high-resource language.


To resemble the existing CW resources (Shardlow, 2013;Paetzold and Specia, 2016a;Kauchak, 2013), we collected 500 sentences from Wikipedia.

the drop in the F-scores of the NEWS and WIKIPEDIA systems when moving from native to non-native datasets, could probably be attributed to a slightly lower inter-annotator agreement among nonnative than native annotators.


Most LS systems focus on simplifying news articles (Aluísio et al., 2008;Carroll et al., 1999;Saggion et al., 2015;Glavaš andŠtajner, 2015).

cross-group-genre Results (Table 6): Similar to the the cross-group experiments, the best results are achieved when tested on the datasets annotated by native speakers, indicating once again that the F-score is highly influenced by the inter-annotator agreement on the test set.


Since a wide variety of users, ranging from young to old, and from female to male, participates in an SNS, the learned responses are not guaranteed to be stylistically consistent (e.g.

this is the first work to focus on creating a stylistically consistent end-to-end DRG system and evaluating stylistic consistency in neural dialog response generation studies.


automatically collecting polite utterances from a large Twitter corpus with a filter, or generating stylistically consistent responses with a smaller or even no specific style corpus.

this may not be a good strategy, because the top N d most frequent words in a dialog corpus do not necessarily include words that are potentially useful for generating stylistically consistent responses.


Our model can also accurately explain many known acronyms.

instance: "that sir right there, is being quite adoucheous" Target: adoucheous Reference: a person acting in a conformative manner that causes social upset and violence.


The importance property of the summary considers how much relevant information present in a summary.

algorithm 1 a Greedy algorithm for maximizing the objective function Require: a minimization LP in standard form.


Recently, summarization has also been considered as a submodular function maximization (Lin andBilmes, 2010, 2011;Dasgupta et al., 2013) where greedy algorithms were adopted to achieve near optimal summaries.

to model this property, we introduce a new monotone nondecreasing submodular function based on the atomic concept.


Since the scoring function f (s) of our proposed summarizer is non-decreasing monotone submodular, we thus use the following greedy algorithm to obtain the near optimal solution.

abstractive summarization is a way of natural language generation and using this approach, it is possible to produce human-like summaries (Rush et al., 2015;Chopra et al., 2016;Wang and Ling, 2016).


We emphasize that question swapping is done only for the groups of questions used for training; the development and test triples are kept the same.

following the notation of Bahdanau et al.


The SemEval dataset (Nakov et al., 2016) was created from questions posted on the Qatar Living forum and has a different distribution and structure.

the results now show  table 5: Results w/ and w/o training on weighted external triples using LM-based selection.


Further heat maps demonstrating this can be found in the appendix.

we hypothesize that leftto-right works consistently well due to the left-toright nature of the English language.


This effect is greatly exacerbated when performed independently on each instance.

attention mechanisms do provide a look into the inner workings of a model, as they produce an easily-understandable weighting of hidden states.


In the NER task, agreement is measured across the entire sequence.

we learn all word and character level features from scratch, initializing with random embeddings.


sampled data to evaluate the comparative effectiveness of different AL strategies.

we simulate pool-based AL using labeled benchmark datasets by withholding document labels from the models.


In our evaluation, we compare the relative performance (accuracy or F1, as appropriate for the task) of the successor model trained with corpus D A to the scores achieved by training on comparable amounts of native and i.i.d.

for each test instance we calculate its cosine similarity to all other test instances, inducing a ranking.


Future work will involve incorporating a diverse set of domain specific KBs for specialized NLP applications.

large pretrained models such as ElMo (Peters et al., 2018), GPT (Radford et al., 2018), and BERT (Devlin et al., 2019) have significantly improved the state of the art for a wide range of NlP tasks.


We experiment with integrating both WordNet (Miller, 1995) and Wikipedia, thus explicitly adding word sense knowledge and facts about named entities (including those unseen at training time).

this results in general purpose knowledge enhanced representations that can be applied to a wide range of downstream tasks.


The best-performing PC static embeddings belong to the first layer of BERT, although those from the other layers of BERT and ELMo also outperform GloVe and FastText on most benchmarks.

in all layers of all three models, the contextualized word representations of all words are not isotropic: they are not uniformly distributed with respect to direction.


Replacing static word embeddings with contextualized word representations has yielded significant improvements on many NLP tasks.

for all three contextualizing models, PC static embeddings created from lower layers are more effective those created from upper layers.


ELMo creates contextualized representations of each token by concatenating the internal states of a 2-layer biLSTM trained on a bidirectional language modelling task (Peters et al., 2018).

recall from Definition 3 that the maximum explainable variance (MEV) of a word, for a given layer of a given model, is the proportion of variance in its contextualized representations that can be explained by their first principal component.


More recent work, namely deep neural language models such as ELMo (Peters et al., 2018) and BERT (Devlin et al., 2018), * Work partly done at the University of Toronto.

static representations are much easier to deploy.


Part of the appeal of cosine similarity perhaps lies in the simple geometric interpretation behind it.

while the abundance of various pooling operations may be intimidating, the resulting vectors are always subject to the many tools of univariate statistics.


Certainly, these approaches are empirically attractive: not only are they very simple computationally (e.g.

each row w (i) of W is a D-dimensional word vector.


By the linearity of expectation, we have that , and so the mean across w mean will also be close to zero at least for small k. In practice, this seems to hold even for moderate k in naturally occurring sentences, as seen in Figure 1.

as embeddings are ultimately just arrays of numbers, we are free to take alternative viewpoints other than the geometric ones, if they lead to illuminating insights or strong-performing methods.


Our model fares particularly well on the disambiguation of nouns and verbs.

(2018) proposed fastSense, a model inspired by fastText  which -rather than predicting context words -predicts word senses.


A typical reward function on policy learning consists of a small negative penalty at each turn to encourage a shorter session, and a large positive reward when the session ends successfully if the agent completes the user goal.


Ubuntu Corpus consists of English multi-turn conversations about technical support collected from chat logs of the Ubuntu forum.

we use the final state of GRU output h L as features and apply a single-layer perceptron to obtain score: where W and b are learnt parameters, σ(•) is sigmoid activation function.


In multi-turn dialogues, ensuring that the model is able to distinguish among turns is essential, especially when multiple emotion are present in different turns.

in order to understand whether or how MoEL can effectively improve other baselines, learn each emotion, and properly react to them, we conduct three different analyses: model response comparison, listener analysis, and visualization of the emotion distribution p. Model response comparison The top part of Table 4 compares the generated responses from MoEL and the two baselines on two different speaker emotional states.


2) The second assumes that the emotion to condition the generation on is given as input, but we often do not know which emotion is appropriate in order to generate an empathetic response.

i hate when that happens Figure 4: The visualization of attention on the listeners: The left side is the context followed by the responses generated by MoEL.


This is due to the first part of the context which conveys a sad emotion.

as shown in Figure 2, our context embedding E C is the positional sum of the word embedding E W , the positional embedding E P  and the dialogue state embedding E D .


In the first step, inspired by the observation that a response can usually be supported by a single KB row, we introduce the KB retriever to return the most relevant KB row, which is used to filter the irrelevant KB entities and encourage consistent generation.

the major difficulty of such a training scheme is that the discrete retrieval result is not differentiable and the training signal from the generation model cannot be passed to the parameters of the retriever.


the neighbourhood speaker information for each utterance in the graph.

a number of models has also been proposed for emotion recognition in multimodal data i.e.


The overall architecture of our KET model is illustrated in Figure 2.

the concept representation c(t) ∈ R d for token t is computed as where c k ∈ R d denotes the concept embedding of c k and α k denotes its attention weight.


Multiple emotion detection from texts has been previously addressed in (Zhou et al., 2016) which predicted multiple emotions with different intensities based on emotion distribution learning.

to further validate the effectiveness of eventdriven attention components, we compare IRER-EA with two sub-networks based on our architectures.


Ren-CECps corpus (Blogs) (Quan and Ren, 2010) is a Chinese data set containing 1,487 blogs annotated with eight basic emotions from writer's perspective, including Anger, Anxiety, Expect, Hate, Joy, Love, Sorrow and Surprise.

existing studies often ignore the latent event information.


To generate convincing and diverse justifications, we developed two models: (1) Ref2Seq which leverages historical justifications as references during generation; and (2) ACMLM, which is an aspect conditional model built on a pre-trained masked language model.

if we encounter a fine-grained aspect, we will replace it with a [MASK] token 30% of the time; while for other words, we will replace them with a [MASK] token 15% of the time.


In Table 5, we compare the sentiment prediction results of MILNET, CAMIL r , CAMIL s and CAMIL f ull .

in order to predict service satisfaction and identify sentiments of all customer utterances with available satisfaction labels, we propose a CAMiL model based on multiple instance learning approach.


In addition, we propose a context clue matching mechanism (CCMM) to match any customer utterance with the most related server utterances.

for each customer utterance C i , we give the attention weights βs t on all the server utterances (see formula 10).


Figure 1 illustrates that satisfaction polarity ("unsatisfied") is mostly embedded in the last few customer utterances (i.e., u 7 , u 9 and u 10 ) 3 .

we use back propagation to calculate the gradients of all the model parameters, and update them with Momentum optimizer (Qian, 1999).


Table 3 shows the comparison with previous work on the PGR testset, where our models are significantly better than the existing models.

using the 1-best dependency tree can result in low recall given an imperfect parser.


After running, Louvain might produce a number of singleton clusters with few instances.

the batch size is 100 selected from {25, 50, 100}.


Here, we adopt the complete-linkage criterion, which is more robust to extreme instances.

it is worth noting that these features are only used by baseline models.


Through multiple rounds of interactions between the primal and dual graphs, RDGCN can effectively incorporate more complex relation information into entity representations and achieve promising results for entity alignment.

following the previous works Wang et al., 2018b;Sun et al., 2018), we use 30% of the pre-aligned entity pairs as training data and 70% for testing.


Recently, embedding-based entity alignment methods were proposed to reduce human involvement.

we first pre-train the entity alignment model (Section 4.2) until its entity alignment performance has converged to be stable.


For WDtext, Con-Mask is a strong baseline and has a better result on Hits@10 and Hits@5, but it performs worse on Hits@1 and MRR compared to our framework.

recently, temporal convolution and attention have been considered to learn a common representation and pinpoint common experiences (Mishra et al., 2018).


To be more precise, a large proportion of relations have only a few facts in KGs.

to be more precise, a large proportion of relations have only a few facts in KGs.


Besides, the results of the two ablation baselines are significantly worse than our overall framework, and thus we can see both components play an important role in our framework.

existing works using textual descriptions have not tackled this issue effectively.


maximizing the potential of massive noisy data as well as highquality part, yet being efficient, is challenging.


We propose three features for non-entity samples: nearby entities (f 1 ), ever within entities (f 2 ) and term/document frequency (f 3 ).

they are usually limited by the extra knowledge resources that are effective only in specific languages or domains.


And our analysis shows that on average 0.72% and 3.56% relative improvement of 2-hop DCA-(SL/RL) over 1-hop DCA-(SL/RL) or baseline-SL (without DCA) is statistically significant (with P-value < 0.005).

as Figure  (4.c) shows, when |E| increases, the running time of these two global EL models increases shapely, while our DCa model grows linearly.


To reduce the high cost of computing spectral norm σ(W ) using singular value decomposition at each iteration, we follow (Yoshida and Miyato, 2017) and employ the power iteration method to estimate σ(W ) instead.

similarly, to extract the city-traffic related event, Anantharam et al.


Nevertheless, newsworthy events are often discussed by many tweets or online news articles.

for calculating the precision of the 4-tuple, we use following criteria: • (1) Do the entity/organization, location, date/person and keyword that we have extracted refer to the same event?


We denote the above measurement as the Pattern Mover Similarity (PMS) measurement.

the pattern mover similarity network (PMSN) is a unified model for adaptively scoring the similarity of entities or patterns to seed entities.


Besides, due to the sparse supervision problem, entity scoring is often unreliable, which in turn influences the delayed feedback estimation.

base on this intuition, we devise the reward function as follows: where E 0 is the set of known entities in root node, E is the set of new entities, SIM(e, E) is the similarity score of newly extracted entity e to known entities, σ(•) is the sigmoid function, and a is a "temperature" hyperparameter.


The pattern mover similarity network (PMSN) is a unified model for adaptively scoring the similarity of entities or patterns to seed entities.

this step evaluates generated patterns using sparse supervision and other sources of evidence, e.g., pattern embedding similarity.


The lower performance on TTL and FAC entities is likely due to the fact that the context patterns of TTL and FAC entities are similar to those of person and location names respectively, which makes them easily be regarded as special person names and location names respectively.

the whole MCTS algorithm looks like a tree structure (see Figure 2(b)), where the node s represents for known entities, i.e., both seed entities and previously extracted entities, and the edge,  (b) The Monte Carlo Tree Search for the pattern evaluation in a bootstrapping system for entity set expansion.


We apply GCN to construct multi-lingual structural representations for cross-lingual transfer learning.

here, A i,j = 1 denotes the presence of a directed edge from node i to node j in the dependency tree.


These approaches, however, incorporate language-specific characteristics and thus are costly in requiring substantial amount of annotations to adapt to a new language (Chen and Vincent, 2012;Blessing and Schütze, 2012;Li et al., 2012;Danilova et al., 2014;Agerri et al., 2016;Hsi et al., 2016;Feng et al., 2016).

two entity mentions are unlikely to be combined into one word in Arabic, thus Relation Extraction does not suffer from tokenization errors and corresponding POS features.


In all annotated sentences, 68, 124 are used for training, 22, 631 for validation and 15, 509 for testing.

• Clean-Sentence Clean-Label (CSCL): All sentences and all labels are clean (Figure 1(a)).


Distant supervision was proposed in (Mintz et al., 2009) to automatically generate large dataset through aligning the given knowledge base to text corpus.

some of them, including (Riedel et al., 2010;Zeng et al., 2015;Lin et al., 2016;Ji et al., 2017;Zeng et al., 2018;Feng et al., 2018;Wang et al., 2018b,a), formulate the task as a multi-instance learning problem where only one label is allowed for each bag.


Assume the event triggering as the starting node (the initial EDAG), there comes a series of path-expanding sub-tasks following a predefined event role order.

for each document and each event type, we pick one predicted record and one most similar ground-truth record (at least one of them is non-empty) from associated event tables without replacement to calculate event-role-specific true positive, false positive and false negative statistics until no record left.


(2) To a certain extent, the NPN model could alleviate the problem by utilizing hybrid representation learning and nugget generator in a fix-sized window.

for each character, the model should identify if it is a part of one trigger and correctly classify the trigger into one specific event type.


(2018) and Sohrab and Miwa (2018) with Fscore value on all categories.

our model can locate entities precisely by detecting boundaries using sequence labeling models.


Hard parameter sharing greatly reduces the risk of overfitting (Baxter, 1997) and increases the correlation of our boundary detection module and categorical label prediction module.

in our model, it is inconvenient and inefficient for the reason that we predict entity categorical labels after all boundary-relative regions have been detected.


The dataset covers over 31,000 sentences corresponding to over 590,000 tokens.

the hidden state of bidirectional LSTM can be expressed as following: where x t i is the token representation which is mentioned in section 3.1.


Recent works that automatically canonicalize OpenKGs, including (Galárraga et al., 2014) and CESI (Vashishth et al., 2018), pose canonicalization as a clustering task of the NPs and RPs.

we thank the anonymous reviewers for their constructive comments.


Previously, statistical models (Mintz et al., 2009;Hoffmann et al., 2011;Surdeanu et al., 2012) have used designed features, such as syntactic and lexical features, and have then been trained by logistic regression or expectation maximization.

the amount of CNN filter is set among {64, 128, 230, 256}.


Huang and Wang (2017) demonstrated that multi-layer ResCNNs network does achieve performance improvement by adding residual identity shortcuts, which aligns with the study that deeper CNN has positive effects on noisy NLP tasks (Conneau et al., 2017).

(4) State-of-the-art DSRE Noise Filter Systems: DSGAN (Qin et al., 2018) is a model to filter out noise instances.


Figure 1: Mislabeling issue example in DSRE: the entity pairs bag containing three sentences is labeled as person/company/founder.

in a minibatch, entity pair bags with conflicts in sentence selection are dropped, the remaining bags are used to update the network parameters.


We adopt the cross-entropy loss as the training objevtive function.

here we get a vector representation w i ∈ R kw for each word from a pre-trained word embedding matrix.


In this paper, we propose REDS2 2 , a new neural relation extraction method in the multiinstance learning paradigm, and design a hierarchical model structure to fuse information from 1-hop and 2-hop DS.

treating S T equally as S may mislead the prediction, especially when their sizes are extremely imbalanced.


In this task, entity representations can be obtained either by contextualized entity representations or descriptive entity representations.

we use contextualized entity representations to decode the hyperlinked Wikipedia description, and also use the descriptive entity representations to decode the linked context.


We conduct extensive experiments on five public datasets from various domains (summarized in Table 1 and detailed in Appendix A).

the cumulative reward from current time step to the end of an episode would cancel the intermediate rewards and thus reflect whether the current action improves the holistic label assignment or not.


Different from most of existing global HTC methods that rely on pre-specified features (Gopal and Yang, 2013) as input or build on specific models (Cai and Hofmann, 2004;Vens et al., 2008;Silla Jr and Freitas, 2009), our framework is trained in an end-to-end manner by leveraging a differentiable feature representation learning model as the base model.

at the time of inference, we greedily select labels with the highest probability asL x i .


The local loss of HiLAP-SL is defined as where T is the lowest label's 449 level of one example and O t estimates the binary cross entropy over the candidate labels C(l t ): Intuitively, HiLAP-SL works as if there were a set of local classifiers, although most of its parameters (except for the label embedding l) are shared by all the labels so that there is no need to train multiple classifiers.

the policy P puts x i at the root label in the beginning.


It demands us to extract the separate senses from the overall word representation to avoid the ambiguity.

firstly, we construct a matrix X i the columns of which is the normal vectors w of the hyperplanes for the ith word.


Our core idea that decomposing the semantic capsules by projecting on hyperplanes is a necessary complement to capsule network to tackle the polysemy problem in various natural language processing tasks.

existing capsule networks for natural language processing cannot model the polysemic words or phrases which contain multiple senses.


This result further confirms that the proposed adaptive attention fusion strategy is much helpful to learn the label-specific document representation for multi-label text classification.

then, we can obtain the the linear combination of the context words for each label with the aid of label-word attention score (A (s) ) as follows.


L takes adantage of label text to explicitly determine the semantic relation between documents and labels, however, label text is not easy to distinguish the difference between labels (e.g., Management vs. Management movies).

in order to seamlessly integrate the above two parts, an adaptive fusion strategy is designed, which can effectively output the comprehensive document representation to build multilabel text classifier.


To make full use of local and global information, Zhao et al.


Here we use GRU as the local feature extractor, and features produced for window x t h+1:t can be represented as: To maintain translation invariant, a max-overtime pooling layer is then applied to CNN or DRNN layer, the pooling result is regarded as the output of Encoder2: Set enc 1 as the global representation produced by Encoder1, required information for a certain window x t h+1:t with size h is defined as g t : where G is a function of interaction mode.

en-coder1 serves as a global information provider, while encoder2 performs as a local feature extractor and is directly fed into the classifier.


Our approach simply conditions on the embedding of the latent variable value and therefore does not add many parameters.

similar trends appear for the other datasets.


2016  The task of summarization is to compress long documents by identifying and extracting the most important information from the source documents.

2015; Post (2018) in machine translation, (Gkatzia and Mahamood, 2015;Reiter and Belz, 2009;Reiter, 2018) in natural language generation, Lee et al.


The former, which also makes use of the Neu-ralREG approach (Castro Ferreira et al., 2018a), tackles standard NLG tasks (discourse ordering, text structuring, lexicalization, referring expression generation and textual realization) in sequence, while the latter does not address these individual tasks, but directly tries to learn how to map RDF triples into corresponding output text.

the Transformer could have had difficulties caused by the task's design, where triples and sentences were segmented by tags (e.g.


We find that word mover based metrics combining BERT fine-tuned on MNLI have highest correlations with humans, outperforming all of the unsupervised metrics and even supervised metrics like RUSE and S 3 f ull .

its variants consider overlap of unigrams (-1), bigrams (-2), unigrams and skip bigrams with a maximum gap of 4 words (-SU4), longest common subsequences (-L) and its weighted version (-W-1.2), among others.


The major improvements come from contextualized BERT embeddings rather than word2vec and ELMo, and from fine-tuning BERT on large NLI datasets.

our goal in this paper is to devise an automated evaluation metric assigning a single holistic score to any system-generated text by comparing it against human references for content matching.


We fine-tune BERT on each of these, yielding different contextualized embeddings for our general evaluation metrics.

we also observed that soft alignments (MoverScore) marginally outperforms hard alignments (BERTScore).


We are grateful to Rik Koncel-Kedziorski and Hannaneh Hajishirzi for sharing their system outputs.

a biLSTM-based keyphrase reader, with hidden states h e k , is used to encode all keyphrases in M. We also insert entries of <STaRT> and <END> into M to facilitate learning to start and finish selection.


The meaning of this sentence is "China encourages foreign merchants to invest in agriculture".

secondly, the sRL performance gap between AU-TODEL and AUTO is large though they provide the same correct syntactic information.


Another drawback of VerbNet is its organization into Levin's classes (Levin, 1993), namely, 329 groups of verbs sharing the same syntactic behavior, independently of their meaning.

while PropBank roles do not provide clear explicit semantics, FrameNet roles are explicit but frame-specific (e.g., Ingestor and Ingestibles for the "Ingestion" frame, or Impactor and Impactee for the "Impact" frame).


yet, VerbNet suffers from low coverage, in that it includes only 6791 verbs, which makes it a suboptimal resource for wide-coverage SRL.


Furthermore, to  Levin's classes  329  Thematic roles  39  Senses  6,791  PropBank  Verbs  5,649  Proto-roles  6  Framesets  10,687  WordNet  ----Synsets  13,767  VerbAtlas  Frames  466  Semantic roles  25  Synsets  13,767   Table 1: Quantitative analysis of popular verbal resources.

propBank's major drawback is that its roles do not explicitly mark the type of semantic relation with the verb, instead they just enumerate the arguments (i.e., Arg0, Arg1, etc.).


In the SP task, the inputs are the target sentence together with 4 surrounding sentences.

to make correct predictions, the model needs to be aware of both typical orderings of events as well as how events are described in language.


We also experimented with adding hidden layers to the DiscoEval classification models.

all of our sentence embedding models use this loss.


Examples (e, f) illustrate unk and never labels.

possession is an asymmetric semantic relation between two entities, where one entity (the possessee) belongs to the other entity (the possessor) (Stassen, 2009).


devices (e.g., watch, guitar, cell phone) yield more possessions (alienable and control labels) and most of the possessions are alienable.

example (c) illustrates an alienable possession in which the author possesses the possessee (i.e., the bag) before, during and after tweeting.


We show that in addition to using past dialogue, these models are able to effectively use the state of the underlying world to condition their predictions.

ranking Candidates Each of the three tasks has a different method for determining candidates.


-full list in Appendix H) which were selected by us to pro-  vide both inspiration and cohesion to annotators.

the unseen test set is com-Category: Graveyard Description: Two-and-a-half walls of the finest, whitest stone stand here, weathered by the passing of countless seasons.


However, it is not satisfying since texts and images are forced to be in one-to-one correspondence.

following (Karpathy and Li, 2015;Carvalho et al., 2018), we use a max-margin ranking loss to ensure the gap between both terms is higher than a fixed margin γ (cf.


Many species are visually similar (e.g., Figure 1, top), making them difficult for a casual observer to label correctly.

it struggles on the two data subsets with highest visual similarity (VISUAL, SPECIES).


NYT contains formal, well written news stories from the New York Times Corpus.

in the end, a set of 14 handcrafted patterns are used to extract a predicate-argument triple from each utterance.


We first run several existing IE systems on the labeled data and use their extraction results as input features along with other rich features including word embedding, part-ofspeech embedding, syntactic role embedding and syntactic dependency information.

second, supervised IE systems require the target relations to be predetermined and learn to extract only the predefined relations.


One strategy for tuple matching would be to enforce an exact match by matching the boundaries of the extracted and benchmark tuples in text.

while most existing Open IE systems extract verbal relations, each of the systems focuses on different relational structures and extraction rules, resulting in heterogeneous results.


End-to-end systems Miwa and Bansal, 2016) are a promising solution for addressing error propagation.

table 3 presents the performance of our baseline systems compared to the human performance.


For example, the authors of DocRED report that 41% of facts require reasoning over multiple sentences in a document (Yao et al., 2019).

they do not provide an exhaustive annotation of facts, which is needed for end-to-end evaluation of KBP.


This performance has been generally attributed to the robust transfer of lexical semantics to downstream tasks.

2 To benchmark the performance, we compare to three rule-based baselines.


Across the three state changes, the model suffers a loss of performance in the movement cases.

we observe that such "post-conditioning" approaches don't perform significantly better than rule-based baselines on the tasks we study.


Although the first one has proven to be beneficial during pre-training to some NLP tasks, we want to check how much its influence is to our final translation performance.

as mentioned in Section 3.4, it is interesting to explore the different usage of pre-trained decoders in the MT task.


In particular, we can show that when y is fixed (observed), the {z n } N n=1 variables are d-separated (Bishop, 2006) i.e., are mutually independent.

the transformer model however fails to decode the later portions of the long input accurately.


In conventional EM algorithm for shallow probabilistic graphical model, the M-step is generally supposed to have closed-form solution.

the model cannot be trained by directly optimizing the data log-likelihood because of its non-convex property.


Tasks   Table 1 compares LaSyn versions against some of the state-of-the-art models on the IWSLT'14 dataset.

since deep neural network is differentiable, we can update θ by taking a gradient ascent step: The resulting algorithm belongs to the class of generalized EM algorithms and is guaranteed (for a sufficiently small learning rate η) to converge to a (local) optimum of the data log likelihood (Wu, 1983).


‡ ‡": significantly better than SAMPLE without CE (p < 0.01).

the evaluation metric is BLEU (Papineni et al., 2001) as calculated by the multi-bleu.perl script.


Serving as a weight assigned to each synthetic sentence pair, sentence-level confidence is expected to help to minimize the negative effect of estimating parameters on sentences with lower confidence.

as the synthetic corpora generated by the NMT model are inevitably noisy, translation errors can be propagated to subsequent steps and prone to hinder the performance (Fadaee and Monz, 2018;Poncelas et al., 2018).


Reliable uncertainty quantification is key to building a robust artificial intelligent system.

further enlarging the monolingual corpus hurts the translation performance.


It is appealing to find that afte 3 iterations, the distribution of the voting weightsα ij will converge to a sharp distribution and the values will be very close to 0 or 1.

a natural question was raised, Will carefully designed aggregation operations help the Enc-Dec paradigm to achieve the best performance?


To support cross-lingual applications, it is essential for us to integrate these language-specific KGs into a unified KG.

suppose we need to align the entities in KG 1 to KG 2 , for each entity e i in KG 1 , we rank each entity e j in KG 2 based on the similarity between e i and e j .


Since the conception of linear mapping between two spaces, there are only very small minor improvement in the supervised mapping function itself and largest improvements are obtained by updating the word translation retrieval in the shared space and normalization steps (Artetxe et al., 2018).

second, we adapt an existing technique for piecewise linear regression to instead perform piecewise linear mapping.


It also first trains the NMT model on out-of-domain training corpus, and then finetunes it using both out-of-domain and oversampling in-domain training corpora.

to this end, we propose an iterative dual domain adaptation framework for NMt.


In this group of experiments, we investigated the impacts of out-of-domain corpus size on our proposed framework.

we conjecture that during the process of knowledge distillation, by assigning non-zero probabilities to multiple words, the output distribution of teacher model is more smooth, leading to smaller variance in gradients (Hinton et al., 2015).


As shown in Figure 2(a), previous studies usually mix them into one out-of-domain corpus, which is applicable for the conventional one-to-one NMT domain adaptation.

the other is to use the mixed-domain training corpus to construct a unified NMt model for all domains.


we first inspected its impacts on the development sets.


In this paper, according to the Generate sequence: Compute KD loss: Compute NLL loss: Compute Agent loss: Model loss: 13 Update gradients for each agent; 14 end 15 until convergence; previous experiments, we find that a simple multitask learning technique without sharing any modules presents promising performance, In Algorithm 1, we describe the overall procedure of our approach.

we argue that our model can bring further improvement using their back-translation technique.


In this paper, we propose a universal yet effective learning method for training multiple agents.

according to the empirical achievements of previous studies on training with two agents, it is natural to consider the training with more than two agents, and to extend our study to the multi-agent scenario.


The resulting optimization is also called Procrustes problem.

thanks to the pivot language, we can pre-train a source encoder and a target decoder without changing the model architecture or training objective for NMt.


However, pivoting requires doubled decoding time and the translation errors are propagated or expanded via the two-step process.

experiments in WMT 2019 French→German and German→Czech tasks show that our methods significantly improve the final source→target translation performance, outperforming multilingual models by up to +2.6% BLeU.


For all transfer learning setups, we learned byte pair encoding (BPE) (Sennrich et al., 2016) for each language individually with 32k merge operations, except for cross-lingual encoder training with joint BPE only over source and pivot languages.

to other NMT transfer scenarios (Zoph et al., 2016;Nguyen and Chiang, 2017;Kocmi and Bojar, 2018), this principle has no language mismatch between transferor and  Figure 2: Step-wise pre-training.


(2014) greatly improved the initial sequence-tosequence architecture; rather than conditioning each target word on the final hidden unit of the encoder, each target word prediction is conditioned on a weighted average of all the encoder hidden units.

there have been several attempts to mitigate this discrepancy, mostly with a view to improving the quality of output translations (Alkhouli et al., 2016(Alkhouli et al., , 2018Chen et al., 2016;Liu et al., 2016) -though some work has focused specifically on alignment (Legrand et al., 2016;Zenkel et al., 2019).


Experimental results show that the joint model indeed improves performances on both ZP prediction and translation.

this is a key problem for the pipeline framework, since numerous errors would be propagated to the subsequent translation process.


The proposed models consistently outperform other models in all cases, demonstrating the superiority of the joint learning of ZP prediction and translation.

in addition, relying on external ZP prediction models in decoding makes these approaches unwieldy in practice, due to introducing more computation cost and pipeline complexity.


Bilingual Content Agreement Intuitively, the translated source contents should be semantically equivalent to the translated target contents, and so do untranslated contents.

in addition, two auxiliary learning signals facilitate GDR's acquiring of our expected functionality, other than implicit learning within the training process of the NMT model.


This issue may be attributed to the poor ability of NMT of recognizing the dynamic translated and untranslated contents.

we add additional Redundant Capsules Ω R (also known as "orphan capsules" in Sabour et al.


Such methods find the erroneous words by calculating the edit distance between the MT hypothesis and its corresponding reference.

this suggests wrong translation is more severe than missing word.


This work represents the preliminary work for more multi-faceted machine translation evaluation, focusing on multiple aspects instead of only a score or ranking, with the goal of push MT techniques to a higher standard.

here, we only briefly list the features adopted in our model as they are all commonly used standard features.


Evaluation We evaluate the effectiveness of estimating word importance by the translation performance decrease.

several researchers turn to expose systematic differences between human and NMT translations (Läubli et al., 2018;schwarzenberg et al., 2019), indicating the linguistic properties worthy of investigating.


We find that the syntactic information is almost independent of the word importance value.

we analyze the linguistic behaviors of words with the importance and show its potential to improve NMT models.


It confirms the existence of important words, which have greater impacts on translation performance.

we measure the word importance by attributing the NMT output to every input word through a gradient-based method.


Modern word embeddings encode character-level knowledge (Bojanowski et al., 2017), which should-in principle-enable the models to learn this behaviour; but morphological generalization has never been directly tested.

their ability to generalize decreases as the slots get less frequent and/or the paradigms get larger.


order of the machine-translated sentence.

the motivation is to reduce noise induced by problematic word alignments.


Previous work aims to build a well-formed tree (Tiedemann and Agić, 2016) from source dependencies, solving word alignment conflicts by heuristic rules.

the code and related data will be released publicly available under Apache License 2.0.


(2018) used UD Treebanks 2.1, which is not the most up-to-date version.

the structure loss L is the pointing loss for the pointer network: where θ denotes the model parameters, y <t represents the subtrees that have been generated by our parser at previous steps, and t is the number needed for parsing the whole sentence (i.e., number of words in dependency parsing and spans containing more than two EDUs in discourse parsing).


Also, the decoder state for "sell" is far apart from the one for "pens".

considering the performance has already exceeded the human agreement of 95.7 F 1 , this gain is remarkable.


In this paper, we propose a Hierarchical Pointer Network (H-PtrNet) parser to address the above mentioned limitations.

more relevant information could be diminished in a sequential decoder, especially for long range dependencies.


In this paper we aim to render semi-supervised learning for semantic role labeling as simple as possible, by eliminating the reliance on multiple external pre-processing tools.

figure 2 illustrates the auxiliary modules and the types of context they see.


Our model does not rely on external tools, and is generally applicable across semantic role representations based on dependencies or constituents (i.e., phrases or spans).

in the third block, we remove cross-view training from our model, and observe a 0.8% drop in F 1 over the full model.


We also apply CVT on the first hidden layer of the sentence learner to further improve the performances of auxiliary tasks, utilizing the views introduced in  for sequence tagging and dependency parsing.

we add the following four auxiliary prediction modules to the model: The "forward" module makes predictions without seeing the right context of the current word.


Some of the previous work also proposes sequence labeling models with shared parameters between languages for performing cross-lingual knowledge transfer (Lin et al., 2018;Cotterell and Duh, 2017;Yang et al., 2017;Ammar et al., 2016;Kim et al., 2017).

we also validate our approaches on a distant language pair, English-Chinese, and the results are competitive with previous methods which use large-scale parallel corpora.


N LL i is the negative log likelihood of language i in Eq (4).

we leverage the mean and variance of internal distributions for alignment.


This method achieved stateof-the-art performance on the four datasets.

the F1 score of the OntoNotes and Weibo datasets even suffered a serious reduction around 1.5% and 1.8%, respectively.


In our CM-Net, information exchanges are performed simultaneously with knowledge diffusions in both directions.

in the first CM-block, the hidden state h t is initialized with the corresponding word embedding.


The small value of a i implies that there is a breakpoint between w i and w i+1 , so the constituent prior C is obtained from the sequence a as follows.

one prior attempt formulated this problem as a reinforcement learning (RL) problem (Yogatama et al., 2017), where the unsupervised parser is an actor in RL and the parsing operations are regarded as its actions.


All values of a on top few layers are very close to 1, suggesting that those are not good breakpoints.

whether two words belonging to the same constituent is determined by "Constituent Prior" that guides the self-attention.


(2017) rely on BiLSTMs and improve initial predictions from an competitive baseline and obtain state-of-art-results on English-to-French translation.

our model slightly reduces the number of R violations, while He et al.


The basic factorized model got it wrong, assigning A1 to the argument 'state'.

this work is similar to energy network approach (Belanger and McCallum, 2016), while a global score function is provided, and approximate inference steps are used.


In the baseline confusion matrix, we see the errors are fairly balanced for all the roles we consider here.

learning those constraints in a soft way might be beneficial.


The forward pass for each document, n, starts from τ = 1 and computes: r − n,τ −1,ι e E ln T ι,j + ll n,τ (j), where r − n,0,ι = 1 where ι ='O' and 0 otherwise.

the VB algorithm is described in Algorithm 1, making use of update equations for the variational factors given below.


6 Despite this and their phylogenetic and geographical relatedness, they share very little vocabulary: only 6.5% of North Sámi tokens appear in Finnish data, and these words are either proper nouns or closed class words such as pronouns or conjunctions.

a popular approach is to train a parser on a related high-resource language and adapt it to the low-resource language.


The latter serves as   (Straka, 2018) for Galician and Uppsala  for Kazakh.

when the number of training data is the lowest (T 10 ), data augmentations improves performance up to 9.3% LAS.


For example, our framework improves the UAS score of Urdu by 15.7%, and Tamil by 7.3%.

here we use the oracle setting to reduce the noise introduced by corpus-statistics estimation errors.


Natural language processing (NLP) techniques have achieved remarkable performance in a variety of tasks when sufficient training data is available.

with the constrained inference, especially the postposition constraint (C2), the proposed inference algorithm bring significant improvement.


In the following, we derive the constrained inference algorithm for corpus-statistics constraints.

we explore several types of corpus linguistic statistics and compile them into corpus-wise constraints to guide the inference process during the test time.


We add two variations of our model ORACLE-DISCRIM and PRETRAIN-ENC for the transfer scenario to see the performance improvements gained from replacing different components of our model with an oracle/near oracle.

then a decision is made on whether this subtree fails to match the relevant words in the utterance and needs to be replaced.


Moreover, we present a detailed analysis of our proposed model and the performance of its different components through an oracle study.

the 'Adapt" module is further broken down into two modules "Aligner" and "Discriminator".


Unlike prior studies, the outputs of the auxiliary task do not directly determine the entity boundaries but rather serve as features for assisting the main task.

unlike prior studies, the outputs of the auxiliary task do not directly determine the entity boundaries but rather serve as features for assisting the main task.


As is shown in Figure 5, to enable TextCNN to predict scores for each word, and to make a fair comparison with SAC, we remove the pooling layer of TextCNN and adopt the same convolutional structure as SAC.

we chose CNN instead of RNN in SAC and we set the max kernel size to 3.


Let the set of transitions of a derivation be: T (y) = T (a 1 .

lexical frequency was added as a covariate of noninterest, to statistically factor out effects of general word frequency (generally used in psycholinguistics), that may correlate with other types of expectations.


We thank Sam Bowman from NYU University and Alon Talmor from Tel Aviv University for providing us the annotation information of the MNLI and COMMONSENSEQA datasets.

a common crowdsourcing practice is to recruit a small number of high-quality workers, and have them massively generate examples.


Hence, our goal is to learn a single classifier that can adapt to the output of any selector operating at any budget.

these approaches process the entire text and encode words and phrases in order to perform target tasks.


Given a commonsense head-relation-tail triple x = (h, r, t), we are interested in determining the validity of that tuple as a representation of a commonsense fact.

(2017) and Trinh and Le (2018) demonstrate a similar approach to using language models for tasks requiring commonsense, such as the Story Cloze Task and the Winograd Schema Challenge, respectively (Mostafazadeh et al., 2016;Levesque et al., 2012).


As the goal of our experiments is to demonstrate the ability of our approach to reduce the number of parameters, we only consider rational baselines: the same rational RNNs trained without group lasso.

our heavily regularized models perform substantially better than the unigram baselines, gaining between 1-2% absolute improvements in four out of five cases.


We use a slightly different notation in this section: for a word w the ith component of its embedding is given by E w,i .

a list of occupation names 3 by Bolukbasi et al.


In this paper, we propose to apply metalearning algorithms in general language represen-tations learning.

to this end, we perform transfer learning experiments on a new natural language inference dataset, namely Scitail.


At the same time, because f is invertible the average number of bits that can be encoded is given by the entropy H(q).

the small but positive KL value for arithmetic coding with k = 300 comes from the slight distributional difference when the long tail is truncated.


To ensure that these texts seem more natural to human eavesdroppers we introduce two parameters to trade off quality for compression: we modulate the LM distribution by a temperature τ , and we truncate the distribution to a top-k tokens at each position to ensure generations do not include tokens from the long tail.

the compression objective is to maximize H(q).


ROUGE is widely used to automatically evaluate summarization systems.

we build upon this line of work and show that cosine similarity between the reference and summary embedding works well, and better than ROUGE on recent datasets, for comparing single document summarization systems.


Another difference between DUC'01 and DUC'02 is that the number of evaluated articles for each system is considerably larger than that in DUC'01.

rOUGE precision is a more suitable metric for evaluating the extent of unnecessary content and verbosity.


Additionally, we show the example of summaries generated by our model and baseline model in Table 3.

abstractive summarization (Rush et al., 2015;Nallapati et al., 2016;See et al., 2017;Chen and Bansal, 2018) generates summaries word-by-word after digesting the main content of the document.


In this study, we investigated several aspects of incorporating pseudo data for GEC.

aspect (i): multiple methods for generating pseudo data D p are available (Section 3).


Our MTM has higher F1 than other models in both intra-and cross-lingual evaluations, while discovering coherent topics and meaningful topic links.

joint inference also allows insights from high-resource languages to uncover low-resource language patterns.


On the relation extraction side, the recently developed deep learning approaches have also proved useful, by successfully combining information about the meaning of the arguments, their context and their grammatical connections (Zeng et al., 2014;Liu et al., 2015;dos Santos et al., 2015).

we further analyze the performance of the relation classifier in two scenarios: i) when the sentence includes information regarding a single person; and ii) when a given sentence includes information about two or more different persons.


five times more simple sentences than complex ones.

we have presented a dataset of natural language answers to a questionnaire designed to obtain a patient's medical history.


Table 6: The precision, recall, and F-score of our models for entity and relation classification.

aMT's complex sentences are much denser in information than the complex sentences in the GCS data -14.2 vs 6.1 relations per sentence.


While most existing systems provide semantically correct responses given goals to present, they struggle to match the variation and fluency in the human language.

for example, it outperforms Slug (Juraska et al., 2018), the best model in E2E-NLG competition, by 2.2% in BLEU score.


The dimension 2 (dim2) usually generates longer and informative responses.

following the techniques described above, ELBO then has the form: where and the KL-divergence term can be derived as * : where α, β are parameters of Dirichlet distribution q φ (z|x) and p(z) respectively, K is the dimension of z, and ψ is the Digamma function.


Recurrent neural networks (RNNs) (Bengio et al., 2003) have achieved great success on many natural language processing tasks.

dir-VHREd is significantly better than others.


At the scarce data levels (10% and 30%) , the pseudo-labelling model is not producing pseudo training points that help improve DST predictions.

the first is the DSt joint goal accuracy, defined as an average joint accuracy over all slots per turn (Williams et al., 2016).


Since FT-AttRNN is only trained on new data, it is oftentimes overwhelmed by new knowledge and results in forgetting the old knowledge.

fT-Cp-AttRNN can be treated as a naive solution to achieve both goals by almost duplicating the model again and again.


Each turker is given 100 utterances from training set in one dataset; as well as the number of groups G for this dataset.

the performance of FT-Lr-AttRNN is even lower than FT-AttRNN most of the time.


(2015) proposed adding contextual signals to the joint IC-SL task.


Following (Wu et al., 2017), we employ R n @ks, mean average precision (MAP), mean reciprocal rank (MRR) (Voorhees et al., 1999) and precision at position 1 (P@1) as evaluation metrics.

we do see improvement on the two data sets.


This makes the prediction of those slots more challenging for the system.

gCAS's act-slot pair performance is far from perfect.


In-scope accuracy suffers for all models using the undersampling scheme when compared to training on the full dataset using the oos-train and oos-threshold approaches shown in Table 2.

our dataset also covers 150 intent classes over 10 domains, capturing the breadth that a production taskoriented agent must handle.


Similar experiments were conducted for evaluating unknown intent discovery models in Lin and Xu (2019).

out-of-scope recall improves compared to oos-train on Full but not OOS+.


Suppose the original utterance is "What is the offer?

we show that this input-aware model performs on par with the input-agnostic one (where the controller outputs do not depend on the source inputs), and may need more epochs to expose the model to the many diverse policies it generates.


we can see the number of auxiliary languages as a mean to qualitatively measure closeness of languages, by looking on the average number of auxiliary languages used for each source-target pair.


The RL time is the average time needed to obtain the three RL models in Figure 2. generated action sequences for simultaneous translation, which leads to faster training and better policies than previous methods, without the need to retrain the underlying NMT model.

wait-training uses 8 GPUs, while others use 1 GPU.


fore all WRITE actions usually has large latency.

in the following experiments, we report results with 4 http://www.statmt.org/wmt15/translation-task.html = 50 and = 3.


The multilingual version of BERT (which is trained on Wikipedia articles from 100 languages and equipped with a 110,000 shared wordpiece vocabulary) has also demonstrated the ability to perform 'zeroresource' cross-lingual classification on the XNLI dataset (Conneau et al., 2018).

if language-adversarial training encourages language-independent features, then the English documents and their translations should be close in the embedding space.


For +RPEHead (or +MPRHead), with the increasing in the dimension of d r , BLEU scores are gradually increasing, but BLEU scores begin to decrease when d r is more than 320 (or 256).

we propose a recurrent positional embedding approach based on part of word embedding instead of numerical indices of words to capture order dependencies between words in a sentence.


In the Transformer network architecture, positional embeddings are used to encode order dependencies into the input representation.

(5), the final sentence representation S M is represented as: Note that for both RPEHead and MPRHead models, the RPEs will be jointly learned with the existing Transformer architecture.


Indeed, its curves in the two figures show either a stable (It-En) or even a slightly downward trend (De-En) that confirms the known limitations of NMT to preserve sentiment traits of the source sentences (see Section 1).

to condition the generic models and obtain the Reinforce and MO-Reinforce systems, we use the polarity-labeled German/Italian tweets in our de-velopment sets, with reward as defined in Section 2.


The most popular such method is known as byte-pair encoding (BPE), which merges iteratively the most frequent pair of symbols.

for ConvS2S, we used 4 layers and an embedding size of 256, dropout of 0.2, an initial step-size of 0.25.


3 A popular inference-time approach is constrained decoding (Anderson et al., 2016;Hokamp and Liu, 2017;Hasler et al., 2018;Post and Vilar, 2018), which modifies beam search to require that user-specified words or phrases to be present in the output hypotheses.

table 3 summarizes key exploratory results for the baseline, Ct, and CD approaches on the development set.


(2016) use translation probabilities from a lexicon (like SMT phrase tables) in conjunction with NMT probabilities.

with statistical machine translation (SMT; Koehn et al., 2007), where there are several established methods of incorporating external knowledge, 1 recent work is still examining how best to incorporate bilingual lexicons into NMT systems.


Semi-supervised approaches for NMT are often based on automatically creating pseudoparallel sentences through methods such as backtranslation (Irvine and Callison-Burch, 2013;Sennrich et al., 2016) or adding an auxiliary autoencoding task on monolingual data (Cheng et al., 2016;Currey et al., 2017).

we duplicate the number of parallel sentences by 5 times in the training data augmented with the reordered pairs.


The first method is to use a fixed-latency policy, such as the wait-k policy .

world bank plans to remit and reduce debts of po-or-est countries Figure 5: Chinese-to-English example on dev set.


Therefore, absolute position (Vaswani et al., 2017) or relative position (Shaw et al., 2018) are generally used to capture the sequential order of words in the sentence.

we conduct probing tasks 3 (Conneau et al., 2018) to evaluate structure knowledge embedded in the encoder output in the variations of the Base model that are trained on En⇒De translation task.


Results for #1 demonstrate that NMT systems trained on 18k parallel sentences can achieve only poor results for Bn and Ja, whereas reasonably high BLEU scores (> 20) are achieved for the other target languages.

we can safely conclude that the gain is due to multilingualism.


(2018a) that Transformers perform WSD better than RNNS2Ss.

gT is mainly funded by the Chinese Scholarship Council (NO.


Ghader and Monz (2017) have shown that nouns have different attention distributions from other word types.

neural machine translation (nMT) models (Kalchbrenner and Blunsom, 2013;Sutskever et al., 2014;Cho et al., 2014;Bahdanau et al., 2015;Luong et al., 2015) have access to the whole source sentence for the prediction of each word, which intuitively allows them to perform word sense disambiguation (WSD) better than previous phrase-based methods, and Rios et al.


It is also noticed from the table that 'Cascade' multi-task model achieves better performance than 'Noncascade' model, which implies that the unimpeded information from morpheme processing to POS tagging helps predict accurate POS tags.

it is inferred from the error analysis that developing a more accurate morpheme processing model is required to improve the performance of the proposed Korean morphological analyzer.


Such an approach reduces the number of POS tags and is helpful in segmenting eojeols into morphemes precisely.

in Korean, n i=1 where is a concatenate operator.


Most languages, especially those that are severely impacted by diacritics, rarely have such amounts of diacritized datasets.

the performance of Zalmout and Habash (2017)'s model falls in between BiLStM and A-tCN.


All in all, the decision whether to use A-TCN or BiLSTM is a tradeoff between accuracy and efficiency.

orife (2018) uses seq2seq modeling which generate diacritized sentences that are not of the same length as the input and can generate words not present in the original sentence (hallucinations).


Until recently, few approaches based on crosslingual joint training and cross-lingual supervised pre-training for DNNs have been proposed.

to the best of our knowledge, data selection has not yet been explored for DNNbased CLTL in SLU.


The VSE baselines in the first five rows are trained with English and German descriptions independently.

recent advancement in VSE models explores methods to enrich the English-Image corpora.


In particular, we (i) leverage ultra-fine-grained semantic labels (e.g., "golden gate bridge" vs. "bridge") for featurization (Juan et al., 2019); and, (ii) focus on scenarios in which object detection modules trained on Visual Genome (VG) (Krishna et al., 2017) are applied to out-of-domain images: image captioning on the Conceptual Captions dataset (Sharma et al., 2018), and VQA on the VizWiz dataset (Gurari et al., 2018).

moreover, this 1.8% improvement is a weighted average across answer types; the per-answer-type numbers indicate that 8 evalai.cloudcv.org/web/challenges/challengepage/102/overview our approach achieves even better improvements on two of the more difficult answer types, "number" (+4.5%) and "rest" (+3.3%).


Object detection plays an important role in current solutions to vision and language tasks like image captioning and visual question answering.

most notably, the domain of images can be very different from Visual Genome, unlike in popular benchmarks such as mSCOCO (Lin et al., 2014).


Nevertheless, these features are considerably complementary.

nevertheless, these features are considerably complementary.


Prior metrics (e.g., BLEU4) only provide an overall quality score, which is difficult to infer specific description mistakes in a caption.


we measure relevance using standard cosine similarity.


we propose weakly Supervised Language Localization Networks (wSLLN) which requires only video-sentence pairs during training with no information of where the activities temporally occur.


To our knowledge, this kind of approaches have not been applied to syntactic parsing.

we use a hard-sharing architecture where the BILSTMs are fully shared and followed by an independent feedforward layer (followed by a softmax) to predict the corresponding label for each of the tasks.


4 Multitask learning We previously argued that gaze-annotated data is unlikely to be available at inference time.

this parser obtains similar results to competitive transition-and graph-based parsers such as BISt (Kiperwasser and Goldberg, 2016) and can be taken as a strong baseline to test the effect of eye-movement data for dependency parsing.


If o i is positive, then the head of w i is the o i -th token to the right that has the part-of-speech tag p i .

we denote LSTM θ (w) as the abstraction of a long short-term memory network, that processes an input sentence w=[w 1 , ..., w n ] to generate an output of hidden representations a BILSTM can be seen as BILSTM In this work we will stack two layers of BILSTMs before decoding the output.


ELMo CI uses only the context-insensitive character embeddings produced by ELMo.

a model trained with 25 codes 6 achieves greater than 60% recall at labeling the ground truth trees for WSJ-10, indicating the codes represent some syntactic patterns although not as effectively as when using K-means.


This suggests that 1,000 utterances are not enough to achieve optimal results.

the baseline model tags only three items in our test data as reparandum, while the finetuned model identifies thirty-one such instances.


Training only using 1,000 annotated in-domain ConvBank data reaches 75.65% and 58.29% accuracy on unlabeled and labeled edges respectively.

models trained on datasets based on news articles and web data do not perform well on spoken human-machine dialog, and currently available annotation schemes do not adapt well to dialog data.


First, we reinterpret the ad-hoc tree score in previous spanbased parsing work (Stern et al., 2017;Gaddy et al., 2018;Kitaev and Klein, 2018) as a joint distribution over the labels.

global decoding methods (Durrett and Klein, 2015;Lee et al., 2016) can incorporate non-local features and decode a tree with the maximum global score.


With capsule networks, the words in a context source sentence is taken as lowlevel capsules and the information of different perspectives is treated as high-level capsules.

(2017) to build the partwhole relationship through the iterative routing procedure.


In this method, the context sentences are considered in the form of hierarchical attentions retrieved by the current generation.

we take the sentence-aligned document-delimited News Commentary v11 corpus 2 as our training set.


We found that the growth of Meteor and BLEU scores are opposite.

we choose three historical sentences as our final setting.


Results on the sequence-level Transformer and our DocNMTs show that the captured contextual features provide helpful semantic information for enhancing the translation quality.

the choice of utilizing how many historical sentences is a trade-off between both scores.


However, fine-tuning requires training and maintaining a separate model for every language, for every domain.

bLEU scores are computed on the checkpoint with the best validation performance, on true-cased output and references.


While LHUC is competitive with full fine-tuning and light-weight adapters for extremely small fractions, the lack of capacity limits the applicability of the approach when larger quantities of adaptation data are available.

experiments on domain adaptation demonstrate that our proposed approach is on par with full fine-tuning on various domains, dataset sizes and model capacities.


As the state-of-the-art Transformer model is welldesigned in the model structure, an analysis of the integration in Transformer is thus necessary.

and the context information assigned to the j th word in the sentence S i is detailed as: where α i,j is the attention weight assigned to the word and d ctx i,j is the corresponding context information distributed to the word.


Experimental results on two Chinese reading comprehension dataset CMRC 2018 (Cui et al., 2019) and DRCD (Shao et al., 2018) show that by utilizing English resources could substantially improve system performance and the proposed Dual BERT achieves state-of-the-art performances on both datasets, and even surpass human performance on some metrics.

the settings of the proposed approaches are listed below in detail.


Specifically, we first tokenize the question and  Figure 2: An illustration of MTMSN architecture.

the passage using the WordPiece vocabulary (Wu et al., 2016), and then generate the input sequence by concatenating a [CLS] token, the tokenized question, a [SEP] token, the tokenized passage, and a final [SEP] token.


This paper considers the reading comprehension task in which some discrete-reasoning abilities are needed to correctly answer questions.

we also provide an in-depth ablation study to show the effectiveness of our proposed methods, analyze performance breakdown by different answer types, and give some qualitative examples as well as error analysis.


The impressive performance on the negation type confirms our judgement, and suggests that the model is able to find most of negation operations.

various kinds of prediction strategies are required to successfully find the answers.


For instance, they could not successfully transfer models between technical and other nontechnical domains.

hyperparameters, and evaluation measures.


A counterexample is Quora, which only contains question titles.

this can be beneficial for a large number of cQA forums that do not contain enough annotated duplicate questions or question-answer pairs to use existing training methods.


MQAN also includes pointer-generator networks (See et al., 2017), which allow it to copy tokens from the input text depending on the attention distribution of an earlier layer.

if not otherwise noted, we use the same number of training instances as in the original training splits with duplicates.


There are several variants of GCCA (Kettenring, 1971); we follow Bach and Jordan (2002) and solve a multiview version of Equation 1: For stability, we add τ σ j I j to every covariance matrix Σ j,j , where τ is a hyperparameter (here: τ = 0.1), I j is the identity matrix and σ j is the average variance of x j .

1 To overcome this problem, two avenues have been explored: The first is supervised domainadversarial training on a label-rich source forum (Shah et al., 2018), which works best when 10 0 10 1 10 2 10 3 10 4 10 5 10 6 #D #Q source and target domains are related.