While different approaches to semantics postulate different basic units and different properties for them, they have in common that meaning arises from a set of independent and discrete units.

the semantic units assumption is general and has minimal commitment to the actual nature of semantics.


For a summary S represented by P S : is maximized for a uniform probability distribution when every semantic unit is present only once in S: ∀(i, j), P S (ω i ) = P S (ω j ).

we define Redundancy, our first quantity relevant to summarization, via entropy: Since H max = log |Ω| is a constant indepedent of S, we can simply write: Redundancy in Previous Works: By definition, entropy encompasses the notion of maximum coverage.


This is comparable to the combination of Informativeness and Redundancy in our framework when semantic units are ngrams.

previous approaches to Informativeness generally craft an alternate background distribution to model the a-priori importance of units.


Furthermore, the summary should be non-redundant (i.e., high entropy).

these two requirements are unified by the Kullback MDI principle: the least biased summary S * that best approximates the distribution P D K is the solution of: we note θ I as the quantity that scores summaries: Interpretation of P D K : P D K can be viewed as an importance-encoding distribution because it encodes the relative importance of semantic units and gives an overall target for the summary.


Indeed, each quantity is a summary scoring function and could, therefore, be evaluated based on its ability to correlate with human judgments (Lin and Hovy, 2003).

we also report the performances of the summary scoring functions from several standard baselines: Edmundson (Edmundson, 1969) which scores sentences based on 4 methods: term frequency, presence of cue-words, overlap with title and position of the sentence.


In this work, we argued for the development of theoretical models of Importance and proposed one such framework.

we investigated a theoretical formulation of the notion of Importance.


A total of 20 editors contribute to 85% of the total summaries on newser.com.

we believe that this dataset allows for the summarization of diverse source documents and summaries.


To maintain semantic similarity and avoid human detection, it requires human efforts such as searching related facts online for insertion.

subsequent research are mainly based on the word substitution strategy so as to avoid artificial fabrications and achieve automatic generations.


Since there are four categories for AG's News, the classification accuracy of workers on this dataset is significantly lower than that on IMDB (binary classification tasks).

we did not try human evaluation on Yahoo!


Still, step costs c(n, n ) will never be negative, which makes g(n) monotonic but not necessarily differentiable.

the heuristic function will not be fully consistent and may even overestimate.


In practice, JS ∆ change is not a monotonic function and steepest-ascent hill climbing does not guarantee an overall lowest-cost path.

we applied each operator to two occurrences of the top ten n-grams and selected from these (up to 140 successors) six randomly for expansion.


This makes it difficult to assess the overall obfuscation with a global measure like JS ∆ .

only few sentences were actually obfuscated with most of the text left untouched.


Since our study aims at deciding the fate of the paper, we take predicting recommendation score and overall decision as the objectives of our investigation.

our proposal to augment sentiment of reviews to the deep neural architecture seems intuitive.


From Figure 3 and Table 4, we see that the sentiment activations are strongly correlated (negatively) with 5 We call them as Sentiment Activations the actual and predicted recommendation scores.

we hypothesize that our model draws considerable strength if the review text has proper sentiment embedded in it.


The literal usage score is computed as the average similarity between the words in the sentence and the words in the literal usage representation.

this score is a real value between 0 and 1 (where 1 is literal and 0 is figurative).


We obtain an F-score of (a) 76.46% when only the literal usage score is used, and (b) 69.72% when the LDA-based estimator is also used.

we use the literal usage score along with the figurative usage features for our experiments.


Second, not all news articles contain topic labels, since it is very expensive and timeconsuming to manually annotate the massive news articles emerging everyday.

it is not suitable to directly incorporate the topic labels of news as model input.


For example, in the news title "90th Birthday of Mickey mouse", the local contexts of "mouse" such as "Mickey" is useful for inferring it is a comic character name.

we use CNN to learn contextual word representations by capturing local contexts.


For example, if a user browses many news with the topic "sport", then she may be interested in sports.

exploiting the news topics has the potential to improve the representations of news and users.


However, not all news in online news platforms contain topic labels, since it is very costly and time-consuming to annotate the massive news emerging everyday.

instead of incorporating news topics as model input, we propose to learn topic-aware news encoder which can extract topic information from news titles by jointly training it with an auxiliary news topic classification task, as shown in Fig.


We find if λ is too small, the performance of our approach is not optimal, since the useful topic information is not fully exploited.

the performance improves when λ increases from 0.


This can be done through attention in the rumor-specific layer, since it has already encoded the user credibility information through UE embedding.

we use an attention-based LSTM to give more attention to the important tweets.


LDA model assigns a score based on the importance of representation of the words within each topic.

the value of assigned scores to topics indicates which ones dominate throughout the text and create the feature set as s i = [s i1 , .


With that said, we hereby define the second feature set for each text as where n is the number of words in the text i, ν i,j is the vector representation of j-th word of language model with dimension of p ν (here set to 100 based on experiment).

far, we have defined features which need to be trained using the ads we already had.


The rationale behind this composite feature set is to allow for the use of textual context as well as the simpler features.

we have the final feature vector defined as as , with the dimension of p = k + p ν + 768.


The representor parameters are denoted θ rep .

the objective function (Equation 6) becomes: This representor (θ rep ) coordinates the semantic presentation of multiple languages in a closely related universal level, which also increases the utilization of commonality for different languages.


On the other hand, using the sum of the length of all segments could lead to false positives.

we only rely on the longest segment and use the size parameter of the average filter to balance the fragmentation.


The empirical findings show that the performance of UNMT is significantly affected by the performance of UBWE.

we propose two methods that train UNMT with UBWE agreement.


The denoising auto-encoder acts as a language model that has been trained in one language and does not consider the final goal of translating between two languages.

backtranslation (Sennrich et al., 2016) was adapted to train translation systems in a true translation setting based on monolingual corpora.


In other words, Baseline-fix prevents the degradation of UBWE accuracy; however, the fixed embedding also prevents UBWE from further improving UNMT training.

the fixed UBWE does not enhance the performance of UNMT.


In addition, word-based embedding are easy to combine with UBWE technology.

we do not adopt BPE in the proposed method.


The generator G 1 is trained to minimize log(1−D 1 (G 1 (enc x ))).

the two-player minimax game (Goodfellow et al., 2014) with value function V (G 1 , D 1 ) is optimized as: (12) D 2 and G 2 are similar to D 1 and G 1 .


In addition, Japanese and English are different language families and their word orderings are quite different.

the performance of Ja-En UNMT is too poor to further empirical study if only pure monolingual data are used.


By doing so, every language is forced to translate to all the other languages.

i(i; z) is effectively close to 0 from the beginning, preventing the model from capturing the spurious correlation between i and z.


However, values of φ and ϕ may not be on the same scale or even from the same family of distributions.

despite its simplicity, C mix co may not be able to enforce either curriculum sufficiently.


We find that NMT indeed captures better alignment for CFS words than the alignment for CFT words, and FAST ALIGN generates much better alignment than NMT for CFT words.

this fact indicates that CFT words are the reason why NMT generate worse alignment than FAST ALIGN.


In contrast, PD only relies on prediction from a neural model to define the relevance, it has been successfully used to understand and interpret a neural model (Zintgraf et al., 2017).

in the rest of this subsection, we try to understand NMT by using PD both qualitatively and quantitatively.


After visualizing the alignment of y i by PD, we find that its alignment in red color is not correct compared to the ground-truth alignment in green color.

the NMT model can not capture the sufficient context to accurately predict the reference word y i and thereby generates an incorrect word 'construction'.


It firstly reveals that attention may not capture word alignment for an NMT model with multiple attentional layers.

it proposes two methods (explicit model and prediction difference) to acquire word alignment which are agnostic to specific NMT models.


Intuitively, NAT abandon feeding previous predicted words into decoder state at the next time step, but directly copy source encoded representation (Gu et al., 2017;Lee et al., 2018;Guo et al., 2018;Wang et al., 2019) as inputs of the decoder.

the generation of the NAT models does not condition on previous prediction.


In contrast, NAT models generate tokens in parallel, thus the target-side dependency is indirect and weak.

the decoder of a NAT model has to handle the translation task conditioned on less and weaker information compared with its AT counterpart, thus leading to inferior accuracy.


Specifically, We propose to employ a knowledgeable AT demonstrator to supervise every decoding state of NAT across different time steps and lay-ers.

imitate-NAT leads to remarkable improvements and largely closes the performance gap between NAT and AT on several benchmark datasets.


Meanwhile, a fixed MT system that was trained to expect complete sentences may impose a low ceiling on any adaptive schedule that uses it.

we propose to train an adaptive schedule jointly with the underlying neural machine translation (NMT) system.


Meanwhile, we observe that a sentence may have multiple entities and relation mentions, and the patterns in which the entities appear in a sentence may contain useful semantic information that can be utilized to distinguish between positive and negative instances.

we further incorporate the embeddings of character-wise/word-wise BIO tag from the named entity recognition task into character/word embeddings to enrich the input representation.


The patterns in which these entities appear in the sentence can provide useful semantic information to distinguish between positive and negative instances.

we exploit the character-wise/word-wise BIO (Beginning, Inside, Outside) tag used in the named entity recognition (NER) task to enrich the input representation.


For graph edges, we connect every relation node to its two entity nodes instead of directly connecting any entity (relation) nodes.


The passages in this task is usually much longer compared to ACE.

we adopt the following pre-processing step -for the entity pair in each relation, we assume the tokens related to their relation labeling are always within a range from the fifth token ahead of the pair to the fifth token after it.


(2018), introducing language modeling as an auxiliary objective during fine-tuning improves generalization and leads to faster convergence.


Intuitively, we apply KL (Kullback-Leibler divergence) as the optimized function, which describes the differences between distributions: What is more, the Equation 2 can be further reduced as following: where loss a represents the loss of attention regularization.

because a m contains fixed values, the equation is equal to we adapt loss a into classification loss loss c to regularize attention learning.


And we guess some relation patterns in training data are too rare to make the model learn to attend them.

after we add attention regularization to the model, the recall increases by about 10% with only 2% decline in precision.


The pre-trained PCNN also achieves a significant improvement, and after further denoising by RL 2 , CNN+RL 2 finally obtain 57.89% in F1, which is still 3% lower than the performance of our model.

we consider that starting the model with a small but clean dataset might be a choice for noise reduction.


We find relation pattern is an important feature but is rarely captured by the previous model trained on noisy data.

we design attention regulation to help the model learn the locating of relation patterns.


However, after aggregating available patterns at the dataset level, there inevitably exist redundant ones.

we design a pattern hierarchy to merge redundant patterns.


Moreover, the number of instances from which the pattern can be induced is closely related to the pattern representativeness.

we follow the decreasing order of this number to select top n r most representative patterns for human annotation.


This low precision means that a large portion of DS-generated positive labels do not indicate the target relation.

this issue inevitably causes the NRE model to absorb some irrelevant patterns.


(2016) conducted the unsupervised parameter estimation based on unlabeled data by solving Different from the general DP that treats each LF with the equal prior, we have strong priors that patterns produced by DIAG-NRE are either supportive or unsupportive of the target relation with high probabilities.

in our case, we directly employ the small labeled set S L obtained at the pattern-refinement stage to estimate (α, β) by solving where the closed-form solutions are for each i ∈ {1, • • • , m}.


Notably, we observe that when training on data with large labeling noises, different parameter initializations can heavily influence the extraction performance of trained models.

as mentioned in the main body, to clearly and fairly show the actual impact of different types of training labels, we restart the training of NRE models with 5 random seeds, ranging from 0 to 4, for each case and report the averaged scores.


A fully connected layer with a twoclass softmax function is used to determine the quality of entity proposals: where W p ∈ R 2D sl ×2R and b p ∈ R 2R are weights and the bias for the entity proposal layer; s k contains 2R scores including R scores for being an entity and R scores for not being an entity at position k. The cross-entropy loss is employed in the Detector as follows: where y r k is the label for proposal type r at position k and s r k is the probability of being an entity for proposal type r at position k. It is worth mentioning that, most entity proposals are negative proposals.

to balance the influence of positive proposals and negative proposals in the loss function, we keep all positive proposals and use down-sampling for negative proposals when calculating the loss L p .


As shown in Figure 1, we obtain the rule province(x, y) ∧ dialect(y, z) ⇒ dialect(x, z) from the informative KG2, then transfer it to KG1 based on the aligned relation province and dialect.

in KG1, we find the suitable triplets province(Jilin City, Jilin) ∧ dialect(Jilin, Northeastern Mandarin), thus obtain a new triplet dialect(Jilin City, Northeastern Mandarin).


Similarly, we obtain the loss L r for KG G .

the overall loss function for multi-channel GNN is as follows: In this section, we conduct experiments on five publicly available datasets involving both different language pairs and sources.


Chinese language is inherently ambiguous since the granularity of words is less well defined than other languages (such as English).

massive wrongly matched entities can be generated with the use of gazetteers.


In this paper, we will experiment with both word level and character level language modeling.

tokens can be either words or characters.


(3) as a regularization term, where λ P DR is a positive weighting coefficient, to construct the following new loss function for the language model.

equivalently PDR can also be viewed as a method of defining an augmented loss function for language modeling.


While the baseline model achieved a validation (test) perplexity of 44.3 (43.1), on applying PDR, the model achieved a perplexity of 44.0 (42.5).

pDR is relatively less effective on larger datasets, a fact also observed for other regularization techniques on such datasets (Yang et al., 2017).


The model with only PDR turned on achieves 2.4 and 5.1 better validation perplexity on PTB and WT2 as compared to the model without any regularization.

biasing the LSTM by decoding the distribution of past tokens from the predicted next-token distribution can indeed act as a regularizer leading to better generalization performance.


However, some sequences have multiple possible segmentations, and a model considering a single one might underestimate the true probability of the sequence.

it is important to marginalize over the set of segmentations to obtain the true probability of a sequence (van Merriënboer et al., 2017;Buckman and Neubig, 2018).


In the experiments, we use n-grams of size up to 4, excluding n-grams appearing less than 200 times (1000 times on text8) to limit the size of the vocabulary.

segmentations which contain out-of-vocabulary n-grams have a probability equal to zero.


23, Z must be such that p j sums to 1, yielding the well-known softmax expression.

in the case of sparsemax, note that for any p ∈ △ d , we have B Backward pass for generalized sparse attention mappings When a mapping π Ω is used inside the computation graph of a neural network, the Jacobian of the mapping has the important role of showing how to propagate error information, necessary when training with gradient methods.


If S(ρ) > 1, ∆(ρ) < 0, so there must exist an optimal ρ satisfying S(ρ) ∈ [0, 1].

(48) has the two solutions τ ± (ρ) = M (ρ) ± 1−S(ρ) ρ .


But M (ρ) = z ρ only if z 1 = • • • = z ρ , in which case S(ρ) = 0 (contradiction).

τ ⋆ = τ (ρ) = M (ρ) − 1−S(ρ) ρ for some ρ verifying (46).


Pick any ρ 1 < ρ 2 , both verifying (46).

ρ 1 + 1 ≤ ρ 2 and z ρ 1 +1 ≤ thus τ (ρ 1 ) = τ (ρ 2 ), and so any ρ verifying (46) satisfies τ ⋆ = τ (ρ), concluding the proof.


To the best of our knowledge, there is no prior work that uses pre-trained BERT for machine translation.

the Transformer model is employed for neural machine translation task.


We conjecture that self-attentive models rely heavily on attention scores, and changing the word with the highest or lowest attention score could substantially undermine the model's prediction.

this attack method exploits and also investigates the attention scores as a potential source of vulnerability.


To enforce monotonicity, hard attention with conditionally independent alignment decisions is not enough: The model needs to know the previous alignment position when determining the current alignment position.

we allow the transition distribution to condition on previous one alignment p(a i | a i−1 , y <i , x) and it becomes a 1 st -order neuralized HMM.


For simplicity, we drop y <i and x in p(y i | a i ) and p(a i | a i−1 ).

for completeness, we include the recursive definition of the forward probability: computation of the likelihood in our Decoding at test time, however, is hard and we resort to a greedy scheme, described in Alg.


Nevertheless, integrating these traditional gates inevitably increases computational overhead which is accumulated along token positions due to the sequen-tial nature of RNNs.

the weak parallelization of RNNs makes the benefits from improved model capacity expensive in terms of computational efficiency.


Partial Routing As opposed to the entire output space on data sets, the sub-output space corresponding to individual examples is rather small, i.e., only few labels are assigned to one document in text classification, for example.

it is redundant to route low-level capsules to the entire output space for each example in the training stage, which motivated us to present a partial routing algorithm with constrained output spaces, such that our NAS function is described as: where D + and D − denote the sets of real (positive) and randomly selected (negative) outputs for each example, respectively.


The major focus of this work is to investigate the scalability of our approach on datasets with a large output space, and generalizability in low-resource settings with few training examples.

we validate our capsule-based approach on two specific NLP tasks: (i) multi-label text classification with a large label scale; (ii) question answering with a data imbalance issue.


Such issue arises particularly in TREC QA with only 12%   positive answers.

we use this dataset to investigate the generalizability of our approach.


Transfer learning aims to improve classification by learning these K tasks in parallel.

our objective is to learn a sentence (or pair) representation x k per task k, but take into account the correlation among related tasks.


,z is fed into the classifier, and the overall loss of CASE is defined as follows: Selective Refinement Lastly, CASE aims to refine only when the perturbation can refine the translation.


On the other hand, the presence of noise or incomplete values breaks these natural correlations and leads to higher rank tensor representations.

we can use tensor rank minimization to learn tensors that more accurately represent the true correlations and latent structures in multimodal data, thereby alleviating imperfection in the input.


As compared to TFN which uses a single outer product to obtain a multimodal tensor of rank one, T2FN creates a tensor of high rank (upper bounded by T ).

the notion of rank naturally emerges when we reason about the properties of M. We first state several observations about the rank of multimodal representation M: 1) r noisy : The rank of M is maximized when data entries are sampled from i.i.d.


The pinyin as the official romanization representation for Chinese provides a solution that maps Chinese character to a string of Latin alphabets so that each character has a letter writing form of its own and users can type pinyin in terms of Latin letters to input Chinese characters into a computer.

converting pinyin to Chinese characters is the most basic module of all pinyinbased IMEs.


Different from machine translation job, users' inputting choice will always indicate the 'correct' prediction right after IME returns the list of its P2C conversion results.

iME working mode implies an online property, we will let our neural P2C model also work and evaluate in such a way.


Note that as we have discussed above, segmentation over pinyin sequence is also necessary to alleviate the ambiguity of pinyin-to-character mapping.

the task here for IME online working actually requires an online word segmentation algorithm if we want to keep the aim of open vocabulary learning.


As the bilingual vocabulary V consists of the pinyin and Chinese word pair of all the words that ever appeared, it is natural to use a prefix maximum matching algorithm to obtain a sorted list of relevant candidate translations D(x) = [Ch 1 , Ch 2 , ...] for the source pinyin.

we generate a target vocabulary V s for a sentence x = (P y 1 , P y 2 , ...) by merging all the candidates of all pinyin.


If the rare words are not initialized properly, they would also bias the whole word embeddings.

we more incline to make character-level embedding to represent a rare word, and build CWE embeddings for others.


This early placement allows the model to have access to the morphological features while decoding the rest of the characters in the word.

our morphology data m = m 1 , m 2 , ..., m n is a sequence of labeled pairs in the form m i = (x, y) where x is a character and y is a set of morphology tags for that character.


3 Because of this, our distant supervision setting on MWC is not comparable with Mielke and Eisner (2019), which uses the second vocabulary setting.

we re-train our character LM baselines and multitasked models in this vocabulary setting (Table 6).


Both hard and soft debiasing methods ignore gender-definitional words during the subsequent debiasing process, and focus only on words that are not predicted as genderdefinitional by the classifier.

if the classifier erroneously predicts a stereotypical word as a gender-definitional word, it would not get debiased.


However, such a re-annotation exercise would be costly and inconsistent with the original ratings.

we resort to a proxy where we reassign the human rating for the original word-pair to its derived opposite gender version.


Indeed, there is no existing annotated corpus of paired sentences that could be used to train a supervised model.

we take an unsupervised 3 approach using dependency trees, lemmata, part-of-speech (POS) tags, and morpho-syntactic tags from Universal Dependencies corpora (UD; Nivre et al., 2018).


To the best of our knowledge, this task has not been studied previously.

there is no existing annotated corpus of paired sentences that can be used as "ground truth."


Sentiment analysis makes up a large portion of current NLP systems.

preventing negative sentiment from mixing with sensitive attributes (i.e.


Our findings show that existing approaches to toxic language detection have racial biases, and that text alone does not determine offensiveness.

we encourage paying greater attention to the confounding effects of dialect and a speaker's social identity (e.g., race) so as to avoid unintended negative impacts.


For us humans, ambiguity is not a problem, since we use common knowledge to fill in the gaps and understand each other.

a computational model suited to understanding natural language and working side by side with humans should be capable of dealing with ambiguity to a certain extent (Navigli, 2018).


for w to be equally associated with both sets of attribute words, not only must w be unbiased with respect to {(x, y)} by Definition 1, but words x and y must also occur with equal frequency in the corpus.


In contrast, we use syntactic collocations to explicitly represent gendered relationships between individual words.

we are able make definitive claims about these relationships, thereby enabling us to answer sociolinguistic research questions.


'Charge' and 'potential' are more related in a Physics corpus than in Wikipedia.

pinning can lead to poor target task performance in case of serious sense mismatch.


However, there are plenty of high-quality broad-coverage sources like Wikipedia that includes plenty of Physics documents that could gainfully supplement D T .

we seek to include target-relevant documents from a generic source corpus D S , even if the dominant sense of a word in D S does not match that in D T .


To ensure that term pairs with the same lexical relation type have similar SphereRE vectors, the problem of optimizing J g can be reformulated by maximizing the probability of predicting the neighbors of (x i , y i ) given its SphereRE vector r i .

we define a new objective function J g to re- place J g based on the negative log likelihood: A remaining problem is to define the neighborhood N b(x i , y i ) properly, to preserve the hyperspherical similarity property of the distance func- In this work, we introduce a weight factor w i,j ∈ [0, 1] w.r.t.


By utilising the symmetry of Equation 2, we can show the converse is true as well -maximising the joint probability also minimises the reconstruction loss given the latent projections E p(z|y k ) [z].

this forces the latent embeddings of x k and y k to be close in the latent space.


From the joint p(m, z|θ), again using rules from (Petersen et al., 2008) Sections (8.1.2) we get For the case of two views, the joint probability can be factored as where  This follows the same form as regular factor analysis, but with a block-diagonal constraint on Ψ.

by Equations (5) and (6) of (Ghahramani et al., 1996), we apply EM as follows.


However, these positional embeddings focus on sequentially encoding order relations between words, and does not explicitly consider reordering information in a sentence, which may degrade the performance of Transformer translation systems.

the reordering problem in NMT has not been studied extensively, especially in Transformer.


(1): where j is the word's position index in the sentence and i is the number of dimensions of the position index.

there is a sequence of positional embeddings: Each pe j is then added to the corresponding word embedding x j as an combined embedding v j : Finally, a sequence of embeddings is the initialized sentence representation H 0 .


d k is the dimension size of the query and key vectors.

the output of the N -th layer H N is the final sentence representation for machine translation.


Although it is only applied to the encoder (or decoder), it can still obtain information about the target (or source) from the decoder (or encoder) by neural network feedback.

the proposed reordering mechanism makes use of information concerning both the source and the target.


While combining the dedicated NFR with the baseline NMT systems yielded the best results for both language pairs, the unified NFR systems achieve comparable BLEU gains over the baseline NMT systems.

the unified NFR systems offer yet a simpler alternative to the baseline NMT systems due to their ability to translate all source sentences.


In our primitive experiments, we observed that training with larger batches and learning rates worked well for deep models.

all the results of deep models are reported with batch = 8192, lr = 2×10 −3 and warmup = 16,000 unless otherwise stated.


While the association of the adjective (white) could be induced from the visual grounding of phrases, whether the preposition (on) should be associated with a white cat or the lawn is more challenging to induce.

we impose an inductive bias to guide the learner to correctly associate prepositions with their complements, determiners with corresponding noun phrases, and complementizers with the corresponding relative clauses.


If say, the predicted path covers only half of the reference path (i.e., P C = 0.5), then we expect the optimal length of the predicted path to be half of the length of the reference path.

ePL is given by: LS for a predicted path P is optimal only if PL(P ) is equal to the expected optimal lengthit is penalized when the predicted path length is shorter or longer than the expected path length: (4) There is a clear parallel between the terms of CLS and SPL.


2 There are not major differences between goal oriented and fidelity oriented agents, highlighting the problematic nature of R2R paths with respect to instruction following: essentially, rewards that only take into account the goal implicitly signals path conformity-by the construction of the dataset itself.

an agent optimized to reach the 2 Our goal oriented results match the RCM benchmark on validation unseen but are lower on validation seen.


The computation of dynamic relational attention is mathematically equivalent to attention over all visual "relationships".

our method provides a direct way to model visual relationships in language.


2 The PhotoBook task is symmetric, i.e., participants do not have predefined roles such as instruction giver and follower, or questioner and answerer.

both participants can freely and naturally contribute to the conversation, leading to more natural dialogues.


Each highlighted image is displayed exactly five times throughout a game while the display of images and the order of rounds is randomised to prevent participants from detecting any patterns.

of this carefully designed setup, dialogues in the PhotoBook dataset contain multiple descriptions of each of the target images and thus provide a valuable resource for investigating participant cooperation, and specifically collaborative referring expression generation and resolution with respect to the conversation's common ground.


task efficiency as calculated by points per minute significantly increases with each game round.


We hypothesise that later segments within a reference chain might be more difficult to resolve, because they rely on referring expressions previously established by the dialogue participants.

a model that is able to keep track of the common ground should be less affected by this effect.


While most highlighted images recur five times during a game, they can be highlighted for both participants in the same round.

any given image is highlighted in an average of 3.42 rounds of a game (see Table 5 for the highlighting schema).


Through the goal-oriented nature of participants' interactions in the PhotoBook dataset, we do not only collect image descriptions but rather the full, collaborative process of establishing, grounding and refining referring expressions throughout the subsequent rounds of the PhotoBook task.

we capture a wide range of dialogue acts such as clarification questions, corrections, extensions, (self-)repairs as well as interactions concerning game mechanics.


When there is not a clear delineation between domains, the inferred domain posterior is likely to be uncertain, and the approach reduces to an ensemble technique.

we introduce the second modelling approach as Continuous domain identifiers (CSDA), inspired by the way in which LDA models the documents as mixtures of several topics (Blei et al., 2003).


It is well known that neural methods do best when given extremely large amounts of data.

much of AI and NLP community has focused on making larger and larger datasets, but we believe it is equally important to go the other direction and explore methods that help performance with little data.


With this framework, RL ensures the TDS process being conducted without requiring a predefined threshold and can automatically select the best instances in the training data as well as learn task-and domainspecific representations for them according to the target domain.

useful information from the source domain is properly organized and represented and the redundant or noisy data are avoided in training the target domain specific models.


(1) is conducted in a sequential manner based on two adjacent data bags B j−1 and via parameters Ψ of the feature extractor updated bŷ B j−1 .

the state transition probability P(s |s, a) is determined by stochastic optimization and other randomness in training, e.g., dropout (Srivastava et al., 2014).


Normally, the transferable knowledge across domains are derived from some certain data, while others contribute less and are costly to be learned from (Axelrod et al., 2011;Ruder and Plank, 2017).

previous studies conduct domain adaptation through selecting relative and informative source data according to the nature of the target domain, via entropy-based methods , Bayesian optimization (Ruder and Plank, 2017), etc.


Training and Prediction To optimize both entity and relation representations, following TransE (Bordes et al., 2013), we assume the relation between two entities can be interpreted as translations operated on the entity representations, namely h + r ≈ t if (h, r, t) holds.

for each tuple (e h i , r i , e t i ), we can compute their distance score: 2 .


Generator For a particular word w, it may occur multiple times in the reference title or in multiple related entities.

at each decoding step i, for each word w, we aggregate its attention weights from the reference attention and memory attention distributions: P i τ = m|wm=w α im and P i e = m|w∈em β im respectively.


The CVAE and Seq2seq model with the rhetorically controlled decoder should be trained jointly.

the overall loss L is a linear combination of the KL term L KL , the classification loss of the rhetoric predictor cross entropy (CE) L predictorCE , the generation loss of the rhetorical controlled decoder cross entropy L decoderCE , and the word type classifier (word type distribution predictor) cross entropy L word classifier : The technique of KL cost annealing can address the optimization challenges of vanishing latent variables in this encoder-decoder architecture.


generate essays that are more novel and diverse.

we equip the decoder with a memory mechanism to effectively incorporate commonsense knowledge from ConceptNet.


Consistency: An ideal essay should closely surround the semantics of all input topics.

we pre-train a multi-label classifier to evaluate topic-consistency of the output.


In other words, we can only access the corpora which are labeled with fine-grained sentiment ratings or intensity values.

in the FTST task, we can not train a generative model via ground truth outputs.


That is to say, the sentiment intensity of each word at time-step t should be decided by both the sentiment representation E s and the current decoder state s t .

we define a sentiment intensity prediction function g(E s , s t ) as follows: where W s is a trainable parameter, and sigmoid is used to scale the predicted intensity value to [0, 1].


Intuitively, if the model performs well in content preservation, it is easy to back-reconstruct the source input x.

we design the reward for content preservation to be the probability of the model reconstructing x based on the generated textŷ and the source sentiment intensity value v x .


Specifically, the model first generates a sequenceŷ based on the input text x and the target sentiment intensity value v y , and then reconstructs the source input x based onŷ and the source sentiment intensity value v x .

the gradient of the cycle reconstruction loss is defined as: (12) where S refers to the Seq2SeniSeq model.


review in the test dataset, crowd-workers are required to write five references with sentiment intensity value from V = [0.1, 0.3, 0.5, 0.7, 0.9].

the BLEU (Papineni et al., 2002) score between the human reference and the corresponding generated text of the same sentiment intensity can evaluate the content preservation performance.


The head-modifier template is universal as the head-modifier rule exists in any noun compound in English, even in Chinese (Hippisley et al., 2005).

the templates are applicable for open domain KGs, with no need to design new templates for entities from other KGs.


The Stage 1 in our model learns the templates from training data, but there are no existing templates for the template generation training.

we acquire head-modifier templates by using a dependency parser provided by Stanford CoreNLP (Manning et al., 2014).


During dependency parsing, the root of a noun compound is always a head word of the type description.

we acquire heads by finding the root and its parallel terms.


However, this dataset is not only too small to reveal the subtlety of models, but it's also relatively imbalanced with too many human entities based on the property instance of.

we propose a new and larger dataset Wiki200K, which consists of 200K entities more evenly sampled from Wikidata dated 2018-10-01.


For a type description, it is crucial to make sure that the head word is the right type of entity.

in order to give an approximate estimate of the data fidelity regarding head words, we also evaluate the head word's accuracy in the output.


Specifically, in Wikidata, the values in instance of (P31) and subclass of (P279) are usually suitable types for an entity, though not every entity has these properties and these types could be too coarse-grained like human.

after dependency parsing, we count the head words in the output with heads from corresponding ground truth and values of corresponding infobox properties, then gives an accuracy of the heads of output.


The surface realization model is used to organize and complete the text given the key facts.

it is possible to construct the pseudo parallel data by removing the skeleton of the text and reserving only the key facts.


We also want to know the effect of the denoising data augmentation.

we remove the denoising data augmentation from our model, and compare with the full model.


Although this model has robust performance on generation, it doesn't allow us to syntactically control the generated sentences by freely changing the syntactic tree template in z y .

we propose SIVAE-i (Section 2.2), which generates sentences and syntactic trees assuming the priors p(z x ) and p(z y ) are independent.


However, linearized syntactic sequences are relatively long (as shown in Table 1) and long templates are more likely to mismatch particular input sentences, which may result in nonsensical paraphrase outputs.

we use simplified syntactic sequences as templates, by taking the top two levels of the linearized constituency trees.


Subsequent sentence representations, namely the plan vectors, are generated with the sentence-level LSTM in a sequential manner: The latent code z can be considered as a paragraph-level abstraction, relating to information about the semantics of each generated subsequence.

we input z at each time step of the sentence-level LSTM, to predict the sentence representation.


We can represent the joint posterior distribution of the two latent variables as 3 : Concerning the generative network, the latent variable at the bottom is sampled conditioned on the one at the top.

we have: , the second term of the VAE objective, then becomes the KL divergence between joint posterior and prior distributions of the two latent variables.


This may be due to the fact that the latent variable priors of the ml-VAE-D model are inferred from the data, rather than a fixed standard Gaussian distribution.

the model is endowed with more flexibility to encode informative semantic features in the latent variables, yet matching their posterior distributions to the corresponding priors.


A hierarchy of stochastic layers is employed, where the priors of the latent variables are learned from the data.

more informative latent codes are manifested, and the generated samples also exhibit superior quality relative to those from several baseline methods.


The lower bound deduction process is as following: where KL(• •)( 0) is the Kullback-Leibler (KL) divergence.

to maximize I p e (x,y) , we can instead maximize its lower bound of L e DIM .


The lower bound of the unsupervised objective E x∼Ux log p(x) is a function of θ and φ.

updating this unsupervised objective will jointly optimize the parser and the generator.


Intuitively, jointly updating both the primal and dual models allows a better learned dual model to provide high-quality learning signals, leading to an improved lower bound for the primal.

freezing parameters of the dual model has a negative impact on the learning signal quality, which affects primal model learning.


The models may mention an incorrect data record, e.g., Kawhi Leonard added 19 points: the summary should mention LaMarcus Aldridge, who scored 19 points.

we need a model that finds salient parts, tracks transitions of salient parts, and expresses information faithful to the input.


Their method consists of two stages corresponding to content planning and surface realization.

by incorporating writer information to each of the two stages, we can clearly see which part of the model to which the writer information contributes to.


In this research, we proposed a new data-to-text model that produces a summary text while tracking the salient information that imitates a humanwriting process.

our model outperformed the existing models in all evaluation measures.


In addition, to encourage ReDR to generate meaningful and interesting questions, ideally, one may employ humans to provide feedback, but as widely acknowledged, involving humans in the loop for training models can be very costly.

in this paper, we leverage a popular and effective reading comprehension (or QA) model (Chen et al., 2017) to predict the answer to a generated question and use its answer quality (which can be seen as a proxy for real human feedback) as rewards to fine-tune our model based on a reinforcement learning mechanism (Williams, 1992).


Oftentimes the conversation history is very informative and complicated, and one single layer of reasoning may be insufficient to comprehend the subtle relationship among the rationale, the conversation history, and the to-be-generated question.

we propose a dynamic reasoning procedure to iteratively update the encoding representation.


The transcript itself cannot serve as a good summary for the corresponding paper, as it constitutes only one modality of the talk (which also consists of slides, for example), and hence cannot stand by itself and form a coherent written text.

to create an extractive paper summary based on the transcript, we model the alignment between spoken words and sentences in the paper, assuming the following generative process: During the talk, the speaker generates words for describing verbally sentences from the paper, one word at each time step.


Then, at the next time-step, the speaker either stays with the same sentence, or moves on to describing another sentence, and so on.

given the transcript, we aim to retrieve those "source" sentences and use them as the summary.


Concretely, we design the focusattention mechanism to improve the capability of capturing the local context information and further encode the document comprehensively.

the basic model with focus-attention mechanism is expected to achieve improvement in producing summaries with continuous salient segments.


The child sentences provide an explanation in terms of the size (1st) and thickness (2nd), or provide the background (4th).

we assume that reviews can generally be described as a multi-root non-projective discourse tree, in which the summary is the root, and the sentences construct each node.


As for the former, the decoder learns a language model to generate grammatical sentences by reconstructing the document sentences.

the model can appropriately decode the summary embedding toŷ 0 .


12, a ij is given by: StrSum generates the summary under the large influence of the child sentences of the root.

sentences that are not related to the rating (e.g., the 4th in Figure 1) also affect the summary and can be considered noise.


, n}) are calculated by: The updated scoreā 0j is used to calculate the summary embeddingŝ 0 instead of Eq.16.

the generated summary reflects the sentences with a higher marginal probability of dependency on the root, while focusing on the main review point.


For Fast Rerank, since there is no ground truth available, we use ROUGE-1 3 (Lin and Hovy, 2003) to evaluate the saliency of a candidate template with respect to the gold summary of current source article.

the loss function is defined as: where s is a score predicted by Equation 9, and N is the product of the training set size, D, and the number of retrieved templates for each article.


Furthermore, we introduce a new evaluation method that incorporates name variations of the ground-truth keyphrases using the Wikipedia knowledge base.

our evaluation method can more robustly evaluate the quality of predicted keyphrases.


, y a,|Y a | } to denote the sets of present and absent ground-truth keyphrases, respectively.

the ground-truth keyphrases set can be expressed as Y = Y p ∪ Y a .


When in the present (absent) keyphrase prediction stage, we want the baseline B t to be the initial return obtained by the greedy sequenceȳ in its present (absent) keyphrase prediction stage.

we first letT P andT be the decoding steps where the greedy search algorithm generates the token and EOS token, respectively.


In such a case, a ground-truth keyphrase can be matched by a present predicted keyphrase or an absent predicted keyphrase.

this ground-truth keyphrase should be treated as both a present ground-truth keyphrase and an absent ground-truth keyphrase, as shown in the following definition.


We found that some of the ground-truth keyphrases have included an acronym at the end of the string, e.g.,"principal component analysis (pca)".

we adopt the following simple rule to extract an acronym from a ground-truth keyphrase.


In addition, we propose a novel evaluation method which incorporates name variations of the ground-truth keyphrases.

it can more robustly evaluate the quality of generated keyphrases.


Sentences A and B each contain certain valuable information, and an appropriate way to merge them exists.

a sentence pair can be scored higher than a singleton given the content it carries and compatibility of its consisting sentences.


Instead of grammatical, wellsegmented sentences, the input is often composed of ill-formed utterances.


For example, in Table 1, PGN fails to capture the keywords remote control, trendy and user-friendly.

we propose a multi-modal hierarchical attention mechanism across topic segments, utterances, and words.


To demonstrate the effectiveness of VFOA attention, we rank the utterances in terms of VFOA, and achieve 45.8% accuracy of selecting salient utterances based on the annotation of (Shang et al., 2018) 2 .

the model learns that when the speaker receives higher VFOA, the utterances of that speaker is more important.


For example, the project manager pays more attention to the user interface designer in 'trend watching' segment, while focuses more on the marketing expert in another segment about 'project budget'.

the VFOA feature not only benefits the summarization decoder, but also improves the performance of topic segmentation.


Lexical substitution might lead to nonfluency C but significantly increases the number of novel words.

the next steps focus on creating pseudo parallel data to boost the fluency of C while attempting to maintain the rate of novel words.


We expect that our model can automatically find out these important tokens.

we propose a gated token selection mechanism and equip it with the self-attention module.


To facilitate the explorations and innovations in this area, many MRC datasets have been established, such as SQuAD (Rajpurkar et al., 2016), MS MARCO (Nguyen et al., 2016), and Trivi-aQA (Joshi et al., 2017).

many pioneering MRC models have been proposed, such as BiDAF (Seo et al., 2016), R-NET (Wang et al., 2017), and QANet (Yu et al., 2018).


However, due to the complexity and diversity of natural languages, only a part of the extracted results can serve as useful general knowledge, while the rest of them are useless for the prediction of answer spans, and the proportion of the useless part always rises when κ is set larger.


Although KAR is equipped with both categories, its most remarkable feature is that it explicitly uses the general knowledge extracted by the data enrichment method to assist its attention mechanisms.

we separately name the attention mechanisms of KAR as knowledge aided mutual attention and knowledge aided self attention.


For each word, we use the pre-trained GloVe (Pennington et al., 2014) word vector as its word embedding, and obtain its character embedding with a Convolutional Neural Network (CNN) (Kim, 2014).

for both the passage and the question, we pass the concatenation of the word embeddings and the character embeddings through a shared dense layer with ReLU activation, whose output dimensionality is d. we obtain the passage lexicon embeddings L P ∈ R d×n and the question lexicon embeddings L Q ∈ R d×m .


Finally we construct an answer span prediction matrix O = uptri(o s o e ) ∈ R n×n , where uptri(X) represents the upper triangular matrix of a matrix X.

for the training, we minimize −log(O as,ae ) on each training example whose labeled answer span is [a s , a e ]; for the inference, we separately take the row index and column index of the maximum element in O as a s and a e .


However, since context embeddings contain high-level information, we believe that introducing the preextracted general knowledge into the calculation of such similarities will make the results more reasonable.

we modify the above similarity function to the following form: where c * x represents the enhanced context embedding of a word x.


However, we believe that this is both unnecessary and distracting, since each passage word has nothing to do with many of the other passage words.

we use the preextracted general knowledge to guarantee that the fusion of coarse memories for each passage word will only involve a precise subset of the other passage words.


For the question, our model uses another identical layer and obtains α q t ∈ R J and c q t ∈ R d .

p q and p p are the copy distributions over the extended vocabulary: where k(l) means the passage index corresponding to the l-th word in the concatenated passages.


Unlike previous pipelined approaches, RE 3 QA shares contextualized text representation across different components, and is carefully designed to use high-quality upstream outputs (e.g., retrieved context or candidate answers) for directly supervising downstream modules (e.g., the reader or the reranker).

the whole network can be trained end-to-end to avoid the context inconsistency problem.


However, the reader is still trained on the initial context retrieved using IR techniques.

the reader could face a context inconsistency problem once the neural retriever is used.


For example, the average number of segments per instance in TriviaQA-unfiltered is 20 even after pruning, while the total number of Transformer blocks is 12 or 24.

we propose to rank all segments using early-summarized hidden representations as a mechanism for efficiently retrieving few top-ranked segments.


However, since there is no constraint to predict an unique span for an answer string, multiple candidates may refer to the same text.

other than the first correct span, all other spans on the same text would be false positives.


A likely reason for this observation may be that sharing highlevel features with the retriever could disturb the reading prediction.

the above results demonstrate that an early-stopped retriever with a relatively small J is able to reach a good trade-off between efficiency and effectiveness.


As shown in Table 9, we first notice that the topranked predictions have highly-relevant semantics and share the same linguistic pattern.

the top-4 candidates contain very similar reading scores from 11.1 to 11.3, which matches the observations of Clark and Gardner (2018).


On the one hand, despite the confusing candidate "Kowloon" has the highest retrieving and reading scores, the reranker assigns a larger confidence on the candidate "Macau".


Search Vector Derivation Recall that our retrieval algorithm is based on executing a MIPS in the paragraph encoding space.

to derive such a search vector from the question encoding q, we observe that: the final search vector of a question Q is q s = w 1 + w 2 q + w 3 • q.


Yes/No Prediction We argue that one can decide whether the answer to a given question should be span-based or yes/no-based without looking at any context at all.

we first create a fixed-size vector representing the question using max-pooling over the first bidirectional GRU's states of the question.


Hence, C may not include two gold paragraphs, and in that case, A S and E cannot be extracted.

the oracle model does not achieve 100 % accuracy.


Our model uses the results of evidence extraction as a guide for selecting the answer, but it is not a pipeline model of evidence extraction and answer selection.

we evaluated a pipeline model that selects the answer string A S only from the extracted evidence sentences, where the outputs of the answer layer corresponding to nonevidence sentences are masked with the prediction of the evidence extraction.


For all above three approaches, the target-domain labeled data would be overwhelmed by the source-domain data during training if directly combined, since there usually exists a very big gap in their scale.

we employ the simple corpus weighting strategy (Li et al., 2014) as a useful trick.


Division span structure adds token H in front of the category to distinguish whether the phrase is on the left or right of the head.

the head is the last one of the category with H which is marked with a box.


The objective function is the hinge loss, For dependency labels, following (Dozat and Manning, 2017), the classifier takes head and its children as features.

we minimize the negative log probability of the correct dependency label l i for the child-parent pair (x i , h i ) implemented as cross-entropy loss: the overall loss is sum of the objectives: As our joint span is defined in a recursive way, to score the root joint span has been equally scoring all spans and dependencies in the HPSG tree.


The comparison in Table 1 indicates that the best performing setting comes from 12 selfattention layers, and more than 12 layers shows almost no promotion even reduces the accuracy.

the rest experiments are done with 12 layers of the self-attention encoder.


Despite the usefulness of HPSG in practice and its theoretical linguistic background, our model achieves new state-of-the-art results on both Chinese and English benchmark treebanks of both parsing tasks.

this work is more than proposing a high-performance parsing model by exploring the relation between constituent and dependency structures.


Note that E X,Y=1 (f (x), 1) can be effectively estimated using positive data.

the main problem of PU learning is how to estimate E X,Y=0 (f (x), 0) without using negative labeled data.


Then, its corresponding risk defined inR is If is not upper bounded, to achieve a small value of From this analysis, we can expect that, when using a unbounded loss function and a flexible classifier,R will dramatically decrease to a far below zero value.

in this work, we force to be bounded by replacing the common unbounded cross entropy loss function with the mean absolute error, resulting in a bounded unbiased positiveunlabeled learning (buPU) algorithm.


It may only label the beginning (B), the internal (I), or the last (E) word Algorithm 1 Data Labeing using the Dictionary 1: Input: named entity dictionary D e , a sentence s = {w 1 , • • • , w n }, and the context size k 2: Result: partial labeled sentence 3: if j == 0 then 11: of an entity.

we cannot distinguish which type, B, I, or E, the labeled entity word belongs to.


The set of possible transitions T τ i for the i-th word at time step τ is a subset of possible transitions T i that satisfy two constraints: (i) no arcs can be reflexive, i.e., w i cannot be an argument of itself, and (ii) the new arc must not be a member of the set of arcs A τ comprising the partial parse graph y τ constructed at time step τ .

we obtain: The model then creates semantic dependency arcs by iterating over the sentence as follows: 1 1 This algorithm can introduce circles.


WordPiece is adopted to tokenize sequence in BERT, which may cut a word into pieces, such as converting "Johanson" into "Johan ##son".

additional efforts should be taken to maintain alignments between input tokens and their corresponding labels.


Intuitively, the amount of overlap is the weakest for this type of entities between raw text from source and target domains.

the results show the effectiveness of our method in deriving domain contrast with respect to NER from cross-domain language modeling.


However, the main difference is that, instead of extracting highorder features on only one intermediate tree, the update of GNN node vectors is able to inspect all intermediate trees.

it may reduce the influence of a suboptimal intermediate parsing result.


But as the dependency tree edges have directions, a word plays a different role regarding it is the head or the dependent in an edge.

instead of using one vector representation, we employ two vectors to distinguish the two roles (Dozat and Manning, 2017).


Similarly, for encoding grandchildren of j in σ t (j, i) (also denoted by k), we need the new dependent representation of node i examine its neighbors when they act as children of i.


We allow each w l to match to at most one w h .

we identify pairs of words (w h , w l ), where w h and w l are very similar words but with polar opposite power scores.


While articles about the #MeToo movement portray men like Weinstein as unpowerful, we can speculate that the corpora used to train ELMo and BERT portray them as powerful.

in a corpus where traditional power roles have been inverted, the embeddings extracted from ELMo and BERT perform worse than random, as they are biased towards the power structures in the data they are trained on.


http://www.pewinternet.org/2018/03/ 01/social-media-use-in-2018/ claims being produced on daily basis.

it is an urgent need to automate the process and ease the human burden in assessing the veracity of claims (Thorne and Vlachos, 2018).


Using such a random sample would require us to filter out a large percentage of the total data collected, making the data collection process inefficient.

in order to target only those tweets that are rich in human activity content, we formulate a set of queries that allows us to use the Twitter Search API to find instances of users tweeting about common human activities.


In reality, it is very likely that users have participated in activities that belong to a huge variety of clusters, regardless of which activities were actually reported on social media.

it should be sufficient for a model to give a relatively high probability to any activity that has been reported by a user, even if there is no report of the user having performed an activity from the cluster with the highest probability for that user.


In particular, we hypothesise that some of our target classes are typified by certain writing styles which differ in their formality and the extent to which they approach informal speech.

we aim to capture the difference between a plainer writing style, with few or no interjections, without abbreviations and/or emojis; and a writing style which more closely approximates speech, making substantial use of exclamation marks and emojis for emphasis, abbreviations, possibly incorrect spelling of words to approximate phonetic form and broad use of direct speech.


However, we observe that in reality only a few users utilize this feature.

requiring sentence-level spoiler annotations from users is not a successful approach to comprehensive fine-grained spoiler annotation.


This is complementary to the self-reporting approach, but may have scalability issues as it is relatively difficult to engage sufficient consumers in a timely fashion.

we seek to address the lack of completeness exhibited by self-reporting and crowdsourcing.


Comparing these 28,454 true matches with those obtained by our matching heuristic, we distinguish three cases: (1) 20,579 are linked correctly, (2) 124 are linked incorrectly (0.6% error rate), and (3) 7,751 are not linked (27.7% miss rate).

our heuristic achieves a very high precision of 0.994 at a reasonably high recall of 0.723.


2 Dataset Creation 2.1 Definition for "Constructiveness" According to the dictionary, 4 "constructive" means "having or intended to have a useful or beneficial purpose."

we expect constructive comments to provide insight and encourage healthy discussion.


This definition realizes a graded numeric score that harnesses the individual variation due to subjective judgements in the conditions, such as "new idea" and "rare experience."

the C-score indicates how many people think that a comment is constructive with the goal of sufficiently satisfying as many users as possible.


However, the advancement is accompanied by the increase in air pollutant emissions and risks to public health.

predicting real-time air quality information (AQI), such as the concentration of PM 2.5 , has attracted more and more attention.


Nevertheless, these sensors are expensive in not only installation but also maintenance.

exploiting sensors for air quality prediction may be too costly for most of the cities.


After determining the corpus representation m all , the final features x(l, t) for air quality prediction can be constructed by concatenating m all and the historical measurements H(l, t).

the final features incorporate the knowl-edge of existing observations and the crowd power on social media.


Based on the reasoning that the social network information is passed between main IDs mainly through some common follow IDs, the follow IDs that only connect to very few main IDs will have minimum functionality in information flow.

we decide to filter the graph by keeping the follow IDs with more than 10 connections to the main IDs.


From the experiments on tweets, we find that using the bag-ofwords features achieve comparable performance to using the bag-of-clusters features.

we opt for the bag-of-words representation in subsequent experiments.


However, the availability of the training data is usually limited as real users have to be involved to obtain the training dialogs.

adapting existing rich-resource data to new domains with limited resource is an essential task in dialog system research.


Then in the adaptation step, since the basic internal representation has already been captured, the model is sensitive to the unique features of the new domain.

one or a few gradient steps and minimum amount of data are required to optimize the model to the new domain.


Most recent threads are used as test set.

we have 11, 356 threads or OPs (217, 057 arguments) for training, 1, 774 (33, 318 arguments) for validation, and 1, 703 (36, 777 arguments) for test.


1 Comments of online articles can provide rich supplementary information, which reduces the difficulty of understanding the article and enhances interactions between users.

achieving automatic commenting is necessary since it can contribute to improving user experience and increasing the activeness of social media platforms.


Data Analysis High-quality testing set is necessary for faithful automatic evaluation.

we randomly selected 200 samples from the testing set for quality evaluation.


As revealed by adversarial tests (Jia and Liang, 2017), models for single paragraph QA tend to seek answers in sentences matched by the question, which does not involve complex reasoning.

multi-hop QA becomes the next frontier to conquer (Yang et al., 2018).


This framework is a simple compromise between single paragraph QA and scalable information retrieval, compared to human's ability to breeze through reasoning with knowledge in massive-capacity memory (Wang et al., 2003).

insights on the solutions to these challenges can be drawn from the cognitive process of humans.


In this paper, we assume that System 2 conducts deep learning based instead of rule-based reasoning by computing hidden representations X of nodes.

system 1 is also required to summarize para[x] into a semantic vector as initial hidden representation when extracting spans.


For example "New York City" is more possible to be the answer of a where question than "2019", while next-hop entities are often the entities whose description matches statements in the question.

we predict answer spans and next-hop spans separately.


Semantics Generation As mentioned above, outputs of BERT at position 0 have the ability to summarize the sequence.

the most straightforward method is to use T 0 as sem [x, Q, clues].


We directly sum the scores from candidate nodes and entity nodes as the final scores over multiple candidates.

the output score vector a ∈ R C×1 gives a distribution over all candidates.


Candidates that are type-consistent with the answer and share the same relation in query with the answer are included, resulting in a set of candidates.

wIKIHOP is a multi-choice style reading comprehension data set.


However, most of the questions in these datasets can be answered by using only a single sentence or passage.

systems designed for these tasks may not be able to compose knowledge from multiple sentences or passages, a key aspect of natural language understanding.


We are interested in multi-sentence texts, both machine-and humangenerated.

we consider subsets of two corpora that have been judged by humans: a collection of automatically generated summaries of articles in the CNN/Daily Mail news dataset (alongside reference summaries; see Section 4.1; Chaganty et al., 2018;Hermann et al., 2015;Nallapati et al., 2016) and student essays from the Hewlett Foundation's Automated Student Assessment Prize (Section 4.2).


Hence, their training was deterministic and the number of configurations a model could have was rather small -decisions about model design were usually limited to feature selection and the selection of one of a few loss functions.

when one model performed better than another on unseen data it was safe to argue that the winning model was generally better, especially when the results were statistically significant (Dror et al., 2018), and when the effect of multiple hypothesis testing was taken into account in cases of evaluation with multiple datasets (Dror et al., 2017).


When δ = 0, roughly half of the disagreements should favor S 1 and the other half should favor S 2 .

under the null hypothesis, n 1>2 ∼ Bin(n, .5) where n = n 1>2 + n 2>1 .


At the same time, the best tagger is 1.16% below the oracle ensemble.

we were interested in disagreements between taggers.


Manual MT evaluation is subjective and difficult, and it is not possible even for a diligent human to be entirely consistent on a continuous scale.

any human annotations are noisy by nature.


This results in the severe imbalance over instances of re-entry and non re-entry (negative samples where users do not come back) on both datasets.

strategies should be adopted for alleviating the data imbalance issue, as done in Eq.


It indicates the sparse user activity in conversations, where most users engage in a conversation only once or twice.

predicting user re-entries only with context will not perform well, and the complementary information underlying user history should be leveraged.


The hate speech/counternarrative pairs have been collected through nichesourcing to three different NGOs from United Kingdom, France and Italy.

both the responses and the hate speech itself are expert-based and composed by operators, specifically trained to oppose online hate speech.


In line with the idea of artificially augmenting pairs, and since in our dataset we have many responses for few hate speeches, we produced two manual paraphrases of each hate speech and paired them with the counter-narratives of the original one.

we increased the number of our pairs by three times in each language.


Images are currently present in a significant fraction of tweets and tweets with images get double the engagement of those without (Buffer, 2016).

in addition to text, images have become key components of tweets.


We perform such updates at regular intervals, but they can only be meaningfully applied to the adjudicated gold standard, not individual annotations.

our reliability is likely underestimated by the analysis above.


(2008) notice a lack of consistence in how sarcasm is defined by people of different socio-cultural backgrounds.

an utterance that is intended as sarcastic by its author might not be perceived as such by audiences of different backgrounds (Rockwell and Theriot, 2001a).


1: for each news cluster c ∈ N do 2: Sample a latent event type vector t ∼ Normal(α) 3: for each entity e ∈ Ec do 4: Sample a slot s ∼ Multinomial(MLP(t; θ)) 5: Sample a head h ∼ Multinomial(1, λs) 6: Sample a feature vector f ∼ Normal(βs) 7: end for 8: end for A limitation of ODEE-F is that sampling slot assignment s from a global uniform distribution does not sufficiently model the fact that different events may have different slot distributions.

in Figure 2(b), we further sample a latent event type vector t ∈ R n for each news cluster from a global normal distribution parameterized by α.


One intu-Algorithm 3 ODEE-FER 1: for each news cluster c ∈ N do 2: Sample a latent event type vector t ∼ Normal(α) 3: for each entity e ∈ Ec do 4: Sample a slot s ∼ Multinomial(MLP(t; θ)) 5: Sample a head h ∼ Multinomial(1, λs) 6: Sample a feature vector f ∼ Normal(βs) 7: Sample a redundancy ratio r ∼ Normal(γs) 8: end for 9: end for ition is that occurrence frequency is a straightforward measure for word-level redundancy.

in Figure 2(c), we additionally bring in the normalized occurrence frequency of a coreferential slot realization as an observed latent variable r ∼ Normal(γ s ).


One possible reason is that the local matching process enhances the interaction between a query instance and a support set when calculating s and q.

simple Euclidean distance between them may not be able to describe the complex correlation and dependency between them.


However, computing exact KL requires iterating over the whole entity pair space E × E, which is quite intractable.

we further provide a sampling-based method to approximate the similarity score over the entity pair space for computational efficiency.


We hope to design a simple and clear experiment setup to conduct error analysis for relational prediction.

we consider a typical method TransE (Bordes et al., 2013) as the subject as well as FB15K (Bordes et al., 2013) as the dataset.


Only α 2 of the relation statements inD explicitly name both of the entities that participate in the relation.

minimizing L(D) requires f θ to do more than simply identifying named entities in r. We hypothesize that training onD will result in a f θ that encodes the semantic relation between the two possibly elided entity spans.


Although such assumptions are common in the model evaluation literature (Reimers and Gurevych, 2018) and for statistical testing in NLP (Dror et al., 2018), they could be problematic for the bounded metrics common in NLP.

we also experimented with modelling the logit transformation of our evaluations, mapping our evaluation metric to the whole real line.


Although we are primarily interested in learning µ m , we must also learn σ 2 m in order to make confidence guarantees about the optimality of our selected model.

as well as keeping track of the sample means for the evaluations of each modelμ m , we also keep track of the sample variancesŜ m and counters T m of the number of times each model has been evaluated.


When structure matching is strict (λ = 1), only trained BERT and Infersent capture syntactic information according to RSA; however, RSA REGRESS still shows moderate correlation for BoW and the untrained versions of BERT and Infersent.

rSA rEGrESS is less sensitive to the value of λ than rSA since changing it from 1 2 to 1 does not alter results in a qualitative sense.


This affine transformation leaves the cdf unchanged, i.e.

we can obtain samples from this generalisedsupport Kumaraswamy by sampling from a uniform distribution U(0, 1), applying the inverse transform (19), then shifting and scaling the sample according to (20, left).


The term ∂L ∂h depends on a differentiable observation model and poses no challenge; the term ∂h ∂t is the derivative of the hard-sigmoid function, which is 0 for t < 0 or t > 1, 1 for 0 < t < 1, and undefined for t ∈ {0, 1}; the term ∂t ∂k = r − l follows directly from (20, left); the term ∂k ∂u = ∂ ∂u F −1 K (u; a, b) depends on the Kumaraswamy inverse cdf (19) and also poses no challenge.

the only two discontinuities happen for t ∈ {0, 1}, which is a 0 measure set under the stretched Kumaraswamy: we say this reparameterisation is differentiable almost everywhere, a useful property which essentially circumvents the discontinuity points of the rectifier.


(2018) is performed on separated fixed-length segments of a few hundred characters, without any information flow across segments.

of the fixed context length, the model cannot capture any longer-term dependency beyond the predefined context length.


First, the largest possible dependency length is upper bounded by the segment length, which is a few hundred on character-level language modeling (Al-Rfou et al., 2018).

although the self-attention mechanism is less affected by the vanishing gradient problem compared to RNNs, the vanilla model is not able to fully exploit this optimization advantage.


However, notice that the recurrent dependency between h n ⌧ +1 and h n 1 ⌧ shifts one layer downwards  per-segment, which differs from the same-layer recurrence in conventional RNN-LMs.

the largest possible dependency length grows linearly w.r.t.


In theory, we can cache as many previous segments as the GPU memory allows, and reuse all of them as the extra context when processing the current segment.

we can cache a predefined length-M old hidden states spanning (possibly) multiple segments, and refer to them as the memory m n ⌧ 2 R M ⇥d , due to a clear connection to the memory augmented neural networks (Graves et al., 2014;Weston et al., 2014).


Notice that, both E s⌧ and E s ⌧ +1 are associated with the same positional encoding U 1:L .

the model has no information to distinguish the positional difference between x ⌧,j and x ⌧ +1,j for any j = 1, .


One Billion Word does not preserve any longterm dependency because sentences have been shuffled.

this dataset mainly tests the ability of modeling only short-term dependency.


A reference can either provide simple Chinese terms such as "daizhe rongyu" or directly offer the corresponding English translation "with honor".

if a good quality reference book which covers various translation scenes is provided, it can definitely improve the source canjia dongaohui de faguo yundongyuan zaiyu fanhui bali.


However, it is impossible to store the weight and bias for every q t computed within the training data.

we approximate the weight and bias function in Eq.18 using local coordinate coding as: where v j ∈ C is an anchor point, W v j and b v j are trainable parameters corresponding to v j , and γ j (q t ) is the weight function, computed as: Similar to M-RefNet, the score is computed by the tri-nonlinear function as: Here, f s (q t ) can be regarded as an approximation of e(y t ) based on all the sentence pairs in the training data.


Therefore, we approximate the weight and bias function in Eq.18 using local coordinate coding as: where v j ∈ C is an anchor point, W v j and b v j are trainable parameters corresponding to v j , and γ j (q t ) is the weight function, computed as: Similar to M-RefNet, the score is computed by the tri-nonlinear function as: Here, f s (q t ) can be regarded as an approximation of e(y t ) based on all the sentence pairs in the training data.

we feed the function value to the decoder state to guide sentence generation: The optimal weight matrices and anchor points are obtained by minimizing the hinge loss for each sentence pair (x, y) as: Stage-wise training strategies have been proven to be efficient when system is relative complicated by plenty of recent work (Maruf and Haffari, 2018;.


During inference, all parameters related to LCC are fixed.

the work can be regarded as a static approach, compared with the conventional document-level NMT.


The prominent feature of the autoregressive model is that it requires the target side historical information in the decoding procedure.

target words are generated in the one-by-one style.


Since one of the released BERT models was jointly pre-trained on 104 languages, 2 we use this multilingual BERT pre-trained model to encode the bilingual input pair of APE.

the whole encoder of our APE model is the multilingual BERT: we encode both src and mt in the same encoder and use the segment embeddings to differentiate between languages ( Figure 1).


Prior studies reveal that modeling locality in lower layers can achieve better performance (Shen et al., 2018;Yu et al., 2018;Yang et al., 2018).

we merely apply the locality model at the lowest two layers of the encoder.


We design a Sentence Agreement Loss L mse to measure the distance between the source and target sentence-level vectors: Finally, our goal is to improve translation with shortening the distance in sentence-level.

the final objective of our model is composed of two parts, the formula is as follows: Sentence-level agreement helps make the targetside sentence representation closer to the source.


We also use the same method to combine H enc and EH enc : In this way, we can enhance the source representation with a sentence-level representation closer to the target-side.

the updated translation training objective is: the final objective is as follows: 4 Experiments For Chinese-English (ZH-EN) translation, our training data for the translation task consists of 1.25M Chinese-English sentence pairs extracted from LDC corpora 1 .


Meanwhile, the selfattention mechanism is good at learning the dependency between characters in parallel, which can partially compare and learn information from multiple segmentations (Cherry et al., 2018).

it is challenging to directly apply the lattice structure to Transformer.


Previous work (McDonald et al., 2011) indeed showed that transferring from multiple source languages could result in significant performance improvement.

in this work, we focus on the multi-source CLTL scenario, also known as multilingual transfer learning (MLTL), to further boost the target language performance.


As mentioned in §1, using only the shared features for MLTL imposes an overly strong con-straint and many useful features may be wiped out by adversarial training if they are shared only between the target language and a subset of source languages.

we propose to use a mixture-of-experts (MoE) model (Shazeer et al., 2017;Gu et al., 2018) to learn the private features.


However, instead of hard-switching between the experts, each sample uses a convex combination of all experts, dictated by an expert gate.

at test time, the trained expert gate can decide the optimal expert weights for the unseen target language based on its similarity to the source languages.


In other words, the more accurate the language prediction is, the more the correct expert gets used.

j g is used to encourage samples from a certain source language to use the correct expert, and each expert is hence learning languagespecific features for that language.


Finally, the expert gates G use token-level NLL loss in (2), which translates to the negative loglikelihood of using the correct language expert for each token in a sample.

the objectives that C, D and G minimize are, respectively: where h w in (5) is the BiLSTM hidden representation in F p as shown in Figure 2.


This corroborates our hypothesis that MAN only leverages features that are invariant across all languages for CLTL, and it learns such features better than weight sharing.

when transferring to German or Spanish, which is similar to a subset of source languages, the performance of  MAN degrades significantly.


For German, however, all nouns are capitalized, and the capitalization features learned on the other three languages would lead to poor results.

for German only, we lowercase all characters in systems that adopt CharCNN.


In these sentences, each English word corresponds to each Spanish one in the same order.

these equivalent words would have similar representations so that the shared language models can encode the English and Spanish sentences effectively.


However, it is not always true that all languages in a single language family share the same linguistic properties (Ahmad et al., 2018).

another strategy is to select transfer languages based on the typological properties that are relevant to the specific NLP task, such as word ordering for parsing tasks (Ammar et al., 2016;Ahmad et al., 2018).


However, as noted in the introduction, this bruteforce method for finding optimal transfer languages is not practical: if resources for many languages are available a priori, it is computationally expensive to train all of the models, and in many cases these resources are not-available a priori and need to be gathered from various sources before even starting experimentation.

we turn to formulating our goal as a ranking task: given an NLP task, a low-resource task language t, and a list of J available high-resource transfer languages a 1 , a 2 , .


The data for the entity linking task consists only of named entities, so the TTR is typically close to 1 for all languages.

we do not include TTR related features for the EL task.


Such data was not usable to test our method that performs transliteration on its own.

we created a dataset of 40 concepts with fully annotated sets of cognate groups.


The main challenge of the decipherment task is the lack of strong supervision signal that guides standard machine translation algorithms.

the proposed architecture has to effectively utilize known patterns in language change to guide the decipherment process.


These KGs encode rich monolingual knowledge but lack the cross-lingual links to bridge the language gap.

the cross-lingual KG alignment task, which automatically matches entities in a multilingual KG, is proposed to address this problem.


We then learn the cross-lingual word embeddings from Swahili to English, from Tagalog to English, and from Somali to English.

all three languages are in the same word embedding space.


2 For every word i, we iteratively transform each word vector x i by first making the vectors unit length, and then making them mean zero, Equation 5and (6) project the embedding matrix X to the set of embeddings that satisfy Equation (3) and (4).

our method is a form of alternating projection (Bauschke and Borwein, 1996), an algorithm to find a point in the intersection of two closed sets by alternatively projecting onto one of the two sets.


In summary, the two constraint sets that Iterative Normalization projects onto satisfy Assumption 1.

iterative Normalization converges following the analysis of Zhu and Li (2018).


We encourage W to perform correct word alignment so that the denoised translated sentences are fluent and grammatically correct.

the training objective is to minimize the negative expected reward, which is formulated as follows: where z t is the output of denoising auto-encoder with t as the input, R(z t ) is the reward evaluating the fluency of z t , and p(t|s) is the probability of W outputs t by performing word-to-word translation on s. We introduce them in detail as follows.


If z t is fluent and grammatically correct, the corresponding reward R(z t ) is relatively large.

the reward R(z t ) can be used as feedback to guide the training of W. Since operation of word-to-word translation is discrete, we use REINFORCE algorithm (Williams, 1992) to estimate the gradient of Eq.


(2016) show that simpler alignment models favor low-resource languages.


This also makes sense intuitively since nearby languages should prefer the source model more than distant languages.

we use two different sets of regularization parameters for nearby languages and distant languages, respectively.


On dependency parsing task, however, our method does not demonstrate significant improvement by the use of mBERT, while mBERT greatly helps the discriminative baseline.

although our method still outperforms the discriminative baseline on four very distant languages, the baseline demonstrates superior performance on other languages when using mBERT.


This unintuitive observation comes from the use of USMT to generate synthetic data: L1 words not covered by the phrase table are directly copied in the translations.

such L1 words are introduced into the L2 vocabulary even if they do not appear in the L2 monolingual data used to train VECMAP, artifically increasing the coverage ratio 11 of the lexicon.


For (king,queen)::(man,woman), the canonical analogy in the literature, the transformation corresponds to changing the gender.

to formalize the definition of an analogy, we will refer to it as a transformation.


The product W * C T is the matrix in (3); each of its three rows is the difference between two rows of M (e.g., M a,• −M y,• ).

we can translate coplanarity in the embedding space to the coplanarity of M's row vectors.


An inner product with the zero vector is always 0, so we can infer from the SGNS identity in (2) that PMI( / 0, •) − log k = 0 for every word in the vocabulary.

the vectors x, y, z, / 0 are all coplanar, and we know from the csPMI theorem that if a linear analogy holds over the csPMI of the sum and one word is equal to the log probability of the other word shifted by a model-specific constant.


For two words x, y, assume without loss of generality that p(x) > p(y).

by 12: addition automatically down-weights the more frequent word.


From (9), we know that for some λ , α, γ ∈ R, csPMI(x, y) = λ γ + α, where γ = − x − y 2 2 .

rearranging this identity, we get the squared Euclidean distance between two word vectors is simply a linear function of the negative csPMI.


Note that, the distributions of distributional similarity scores and proposed combined scores are significantly different (according to the z-test (Fisher, 1932)).

while falling back to the distributional similarity scores we scale up the scores by the proportion of normalized means of the two distributions.


For example, 'hard disk' and 'disk' have the same hypernym 'storage device'; similarly 'soft drink' and 'drink' have 'product'; 'hard drive' and 'drive' have 'device'.

these non-compositional cases are different from entirely opaque expressions like 'couch potato', 'hot dog' where none of the hypernyms of the noun phrases match with the hypernyms of any of the constituent words.


This mismatch between surface forms is problematic, resulting in difficulties pertaining to learning effective representations.

this has tremendous implications such as rendering downstream applications inefficacious and/or potentially unreliable.


FastText (Bojanowski et al., 2017) leverages this idea by considering character n-grams instead of words.

the model can derive representations for names that contain unseen words.


We intentionally leave the chemical concepts out for out-domain evaluation.

approximately 16 thousand synonym sets (associated to that number of disease concepts) are collected for training.


Words which are related, but not syn-onymous, may thus have very dissimilar relational word vectors.

we test our proposed models on a number of different supervised tasks for which accurately capturing relational information is crucial to improve performance.


Now, unlike CBOW which takes the sum of the context embedding of words in C wi to predict w i , we apply directed Graph Convolution Network (as defined in Section 3) on G s with context embeddings of words in s as input features.

for each word w i in s, we obtain a representation h k+1 i after k-layers of GCN using Equation 1 which we reproduce below for ease of readability (with one exception as described below).


A structured regularization is applied to the embeddings to ensure they are tuned in an incremental way.

the tuned word embeddings become better word representations by absorbing semantic information from supervision without "forgetting."


For instance, durative adverbials map unbounded predicates onto extended no-change predicates, and the progressive maps dynamic predicates of all kinds onto stative ones.

the aspectual class of a full clause or sentence may differ from the one of its main verb (plus its arguments); thus, annotating aspect at the clause or sentence level differs from our annotation task.


3, where L is the number of hidden layers).

a word is predicted using both its left and right context jointly, akin to the context2vec architecture (Melamud et al., 2016) but differently from, e.g., the BiLSTM architecture used for ELMo (Peters et al., 2018).


The specificity of our model is that it learns a given graph similarity metric, which is not provided in these datasets.

we use only the graphs from these datasets, computing the shortest path distances between all pairs of nodes using the algorithm of Johnson (1977).


Furthermore, KG2Vec leverages the most updated information from comprehensive knowledge bases (Wikipedia & Wiktionary).

kG2Vec can be applied to training embeddings of newly emerging OOV words.


Take Figure 1(b) as an example, although tough may fit in the context as well as powerful, it changes the contextualized representation of the sentence more than powerful.

it is not so good as powerful for the substitution.


To address this issue, we propose a novel approach for manual evaluation, HIGHlight-based Reference-less Evaluation of Summarization (HIGHRES), in which summaries are assessed by multiple annotators against the source document via manually highlighted salient content in the latter.

summary assessment on the source document by human judges is facilitated, while the highlights can be used for evaluating multiple systems.


It turns out that the sequence of edit operations z constructed by Section 3.2 is deterministic given x and y (an example of of z can be seen in Table 2).

editNTS can learn to simplify by modelling the conditional distribution P (z|x) with a programmer, an interpreter and an edit pointer:  At time step t, the programmer decides an edit operation z t on the word x kt , which is assigned by the edit pointer, based on the following contexts: 1) the summary of partially edited text y 1:j t−1 , 2) the previously generated edit operations z 1:t−1 , 3) and the complex input sentence x.


We resolve this by two approaches during training: 1) we associate the inverse of edit label frequencies as the weights to calculate the loss; 2) the model only executes DELETE when there is an explicit DELETE prediction.

if the system outputs STOP before finish editing the whole complex sequence, our system will automatically pad KEEP until the end of the sentence, ensuring the system outputs remain conservative with respect to the complex sequences.


In terms of the percentage of unchanged sentences, one can see that MT-based models have much higher rates of unchanged sentences than the reference.

the models learned a safe but undesirable strategy of copying the sources sentences directly.


However, learning from scratch may not be informative for the separator and aggregator to disentangle the paraphrase patterns in an optimal way.

we induce weak supervision to guide the training of the model.


Linguistic complexity stems from the use of either a difficult vocabulary or sentence structure.

tS is classified into two categories: lexical simplification and syntactic simplification.


Another major drawback of the structural TS approaches described so far is that they do not preserve the semantic links between the individual split components, resulting in a set of incoherent utterances.

important contextual information is lost, impeding the interpretability of the output for downstream semantic tasks.


However, due to the uniform use of a single split per source sentence in the training set, each input sentence is broken down into two output sentences only.

the resulting simplified sentences are still comparatively long and complex.


In contrast, the former are of equal status and typically depict the key information contained in the input.

they are called core sentences in our approach.


Language models (LMs) based on sequential LSTMs (Hochreiter and Schmidhuber, 1997) have numerous practical applications, but it has also been shown that they do not always develop accurate syntactic generalisations (Marvin and Linzen, 2018).

one strategy for improving LSTMs is to change their biases to facilitate more linguistically valid generalisations.


Parts of the river valley often meander), while the set of singular verbs has, meanders, and is are assigned much lower probability mass since they are syntactically illicit.

as long as the RNNG makes the accurate structural generalisations (and we have shown that it largely does in §3), every training instance provides the student LSTM with a wealth of information about all the possible legitimate continuations according to the predictions of the hierarchical teacher, thereby making it easier for the student to learn the appropriate hierarchical constraints and generalisations.


(b) The model is first trained with step-by-step supervision, and then Gumbel-Softmax is applied to refine the policy.

sT-Gumbel is difficult to train (similar to RL), and may be stuck in poor local optima, resulting in low self-agreement for multiple random initializations (Williams et al., 2018a).


Nevertheless, this straightforward approach also faces scalability challenges because it requires training an independent parser for each language, which results in over 1.8 billion parameters for ten languages.

we consider a single parser with parameters shared across languages and finetuned jointly.


is less than 0.5% of the total parameters.

this joint training entails a 10x reduction in model size.


We conduct separate experiments for Bokmål and Nynorsk, which are not distinguished by GT.


Our tree communication models (TCM) take a trained tree LSTM as an initial state, performing information exchange using graph neural network (GNN).

h j is dependent on all related neighborhood nodes rather than only descendant nodes: where r j is the set of all relevant nodes of node j.


Manually labeled code-switched text, especially involving minority languages, is extremely rare.

the best monolingual methods perform relatively poorly on code-switched text.


However, neither is a clear winner and the gains are small.


The decoder is also a RNN which generates the target sequence Y based on X, and predicts the next label y t based on the context vector c t and all previous labels [y 1 , y 2 , ..., y t−1 ] predicted by the  same decoder.

the joint probability of the target sequence is defined as: where .., y t−1 ] and the conditional probability of label y t can be modeled by the decoder, and defined as: where is the target vocabulary size, and s d is the hidden state size of decoder.


However, existing neural network based sequence labeling methods first compute the label scores of each word simultaneously, and obtain the globally optimized label sequence (Collobert et al., 2011).

they do not know the label of previous word when computing the label scores for the current word.