For this knowledge, we create a knowledge base by counting how many times a predicate-argument tuple appears in a corpus and use the resulted number to represent the preference strength.

we use the English Wikipedia 9 as the base corpus for such counting.


Along with the increase of t, both the max and the average number of pruned candidates drop quickly, so that the space complexity of the model can be reduced accordingly.

there are as much as 80% candidates can be filtered out when t = 10 −1 .


In this paper, we proposed a two-layer model for pronoun coreference resolution, where the first layer encodes contextual information and the second layer leverages external knowledge.

a knowledge attention mechanism is proposed to selectively leverage features from different knowledge sources.


These scores depend to a large degree upon the contextual information of the pronoun (e.g., they) and the candidate antecedent (e.g., councilmen).

it requires to capture the semantic meaning of the pronoun and the candidate antecedent based on the sentences where they occur, respectively.


The results on parsing reveal similar performances for a spectrum of heterogeneous configurations.

while the chosen configuration is still important, its impact on performance of state-of-the-art dependency parsers (Kiperwasser and Goldberg, 2016; is decreased, as such parsers are heavily parametrized multi-component methods (e.g., besides word embeddings they rely on biLSTMs, intra-sentence attention, character representations).


Figure 4 to 7 show the results on all downstream tasks across languages when each test data point is categorized according to some specific component.

figure 4 shows the results when only different segmentation methods are considered, and figure 5, 6 and 7 show the results on varying composition functions, position embedding types and whether including the word token embedding, respectively.


In this paper, we develop topic modeling with knowledge graph embedding (TMKGE), a Bayesian nonparametric model to employ knowledge graph (KG) embedding in the context of topic modeling, for extracting more coherent topics.

we build a hierarchical Dirichlet process (HDP) based model to flexibly borrow information from KG to improve the interpretability of topics.


Another useful form of knowledge for topic discoveries is based on word semantics Chemudugunta et al., 2008;Hu et al., 2014;Jagarlamudi et al., 2012;Doshi-Velez et al., 2015).

word embedding (Pennington et al., 2014;Goldberg and Levy, 2014), in which bag of words are transformed into vector representations so that contexts are embedded into those word vectors, are used as semantic regularities to enhance topic models (Nguyen et al., 2015;Li et al., 2016;Das et al., 2015;Batmanghelich et al., 2016).


To improve the stability of online variational inference, we use a mini-batch of documents to compute the natural gradients.

the contribution of the single document d in (4) is replaced by sum of contributions of documents in the minibatch S, and the factor D is replaced by D/|S|.


The proposed method allows for flexible sharing of information between documents and knowledge graph.

tMKGE avoids forcing the words and entities to identical latent factors, thus making it a suitable framework for scenarios where only partial relational information are available.


In some cases one could estimate the densities even for off-the-shelf embeddings, but this still requires access to the training data and the usefulness of such post-factum densities is limited (Vilnis and McCallum, 2014).

these approaches are not very helpful to practitioners who are accustomed to using highquality pre-trained word embeddings directly.


While it's tempting to invoke the Central Limit Theorem (at least for longer sentences), under our formalism, averaging word vectors corresponds to averaging scalar random variables used to represent words, which are neither independent nor identically distributed.

there are no easy guarantees of normality for such sentence vectors.


This may bias a classifier trained on the Spanish or German datasets towards learning MWEs and long individual words (if length is a feature) are complex.

this observation may help explain why adding English as a training language decreased the performance of our crosslingual system when testing on French and Spanish (where all MWEs are complex).


Such adaptive convolution will model the interactions between the two types of input not only more naturally but also more effectively.

given a triple, the vector representation of the subject is reshaped and fed as input to a convolutional layer, while that of the relation is split and reshaped into a set of filters.


During optimization, we use dropout (Srivastava et al., 2014) to prevent overfitting.

we use dropout on the reshaped subject representations, the convolutional feature maps, and the projected vectors after the fully-connected layer.


We further investigate parameter efficiency of ConvR against ConvE on FB15k-237.

we tune the number of filters c ∈ {20, 40, 60, 80, 100} and the filter size h × w ∈ {2 × 2, 3 × 3, 4 × 4, 5 × 5}, fix the other hyperparameters to their optimal configurations (see Table 2 for details), and show how the performance of ConvR (on the test set) will change as the number of parameters varies.


(2) ConvR consistently and substantially outperforms the best performing ConvE with all the configurations listed in Table 5.

3even the most efficient configuration (i.e., c = 20 and h × w = 2 × 2) offers a 7% increase in MRR and a 6% increase in Hits@10, with its parameter number only 88% as large as that of ConvE.


Experimental results on multiple benchmark knowledge bases show that ConvR achieves significant and consistent improvements against a variety of baselines.

it is not only more effective but also more efficient than state-of-the-art convolutional models, offering a 7% increase in MRR and a 6% increase in Hits@10, while saving 12% in parameter storage.


So, we construct our queries based on four key variables, (relation, head, tail, t i ).

we combine the embeddings (pretrained by TransE (Bordes et al., 2013)) of head and tail and project the combination through an affine matrix Φ q ∈ R De * Dr , where D e and D r denote the dimension of relation and entity embedding, respectively.


The performance of comparative experiment is reported by precision-recall (PR) curve.

we sort the prediction scores of the model in 2 The details of construction of WIKI-TIME can be found in Appendix A. CNN ONE (Zeng et al., 2015).


Unlike some previous works, our framework does not require semantic correspondence between classes in a training stage and classes in an inference stage.

the seen and unseen classes can be clearly different.


In contrast, the zero-shot classifier is a binary classifier with a sigmoid output.

it takes a text document x i and a class c as inputs and predicts the confidence p(ŷ i = c|x i ).


Phase 2: To see how well the augmented feature v w,c work in ZSL, we ran the zero-shot classifier with different combinations of inputs.

five combinations of v w , v c , and v w,c were tested with documents from unseen classes only (traditional ZSL).


Additionally, we employ the node2vec approach as a baseline.

we use node2vec to learn the word-node representations of the graph constructed as per Section 4.


To circumvent this limitation, we need to map the word representations of TDSMs to a shared vector space.

we hypothesize that TDSMs capture meaningful variations in usage of polysemous words, while the relative semantic distance between monosemous words is preserved.


Then our objective is to extract a list of semantic anchors A that minimizes the Euclidean distance between the two different similarity distributions.

for every word i we calculate the average semantic distribution across all topics: where s i g , s i k is the i-th row of the S g and S k similarity matrix, respectively, representing the similarity distribution between word i and every other word in the vocabulary V .


The transformation matrix M k ∈ R d×d that projects the first space to the latter is learned via solving the following constraint optimization problem: The induction of multiple topic embeddings in the unified vector space is achieved via applying Equation 3 to each TDSM.

given a word and its k-th topic distributed representation x k ∈ R d , we compute its projected representation x k ∈ R d as follows: Starting from the set of aligned topic embeddings {x k } K k=1 for each word, we learn a Gaussian Mixture Model with N components, where closely positioned topic embeddings are assigned to the same component.


Following the evaluation guidelines proposed in literature, we employ the AvgSimC and MaxSimC contextual metrics, firstly discussed in Reisinger and Mooney (2010).

given the wordpair (w, w ), and their provided contexts (c, c ) we define: Following the notation used in 3.2, K is the number of topics returned by the trained LDA model, x j is the word embedding trained on the subcorpus corresponding to the j-th topic after being projected to the unified vector space, p(j|w, c) denotes the posterior probability of topic j returned by LDA given as input the context c of word w, d denotes the cosine similarity between the two input representations and finallyx (w) = u argmax 1≤j≤K p(j|w,c) (w) is the vector representation of word w that corresponds to the topic with the maximum posterior for c. Intuitively, a higher score in MaxSimC indicates the existence of more robust multi-topic word representations.


201869 Furthermore, we observe that GMM smoothing has a different effect on the MaxSimC and AvgSimC metrics.

for AvgSimC we consistently report lower results when GMM smoothing is applied for different number of components.


We observe that our model performs better than the baseline across all metrics for both averaging approaches (AvgC D , Avg D ), while the usage of dominant topics appears to have lower performance (MaxC D ).

we get an improvement of 2 − 2.5% on topic-based average and 0.5 − 1% on simple average combination compared to using Global-DSM.


After the mappings, we see on the right that the area under a word's distribution is indicative of its degree of polysemy.

we observe that the variance of the learned representations becomes larger for the cases of polysemous words such as Table 4: Examples of polysemous words and the change of meaning between different topic domains.


Although the word vectors and evaluations used in this study are insensitive to part of speech and sense distinctions, the number and order of groupings affects the retrofitting procedure.

as noted by Speer and Chin (2016), the results depend on the order in which the groupings are iterated over.


This is not surprising, and follows from the design principles underlying each resource: while PPDB and WordNet specifically group synonyms, FrameNet groups words which evoke the same semantic frame.

some frames intentionally contain antonyms.


Indeed, we often see probing models rivaling or exceeding the performance of (often carefully tuned and task-specific) state-of-theart models.

the linear probing model surpasses the published state of the art for grammatical error detection and preposition supersense identification (both role and function).


We evaluate how well CWR features perform the pretraining task-bidirectional language modeling.

we take the pretrained representations for each layer and relearn the language model softmax classifiers used to predict the next and previous token.


We focus on fully differentiable objectives that can be plugged into an automatic differentiation system and efficiently optimized by SGD.


Define analogous estimates based on the k-th minibatch bŷ If l N denotes an objective function computed from all N samples in the training data and l k denotes the same objective computed from B k , a condition on the correctness of SGD is that the gradient of l k (with respect to model parameters) is consistent with the gradient of l N on average: where denotes the bias of the stochastic gradient estimator.

any loss of the form l N = (1/K) k l k that decomposes over independent minibatches (e.g., the cross-entropy loss for supervised classification) satisfies (8) with = 0.


• We use many-to-one accuracy as a primary evaluation metric.

we map each induced label to the most frequently coinciding ground-truth POS tag in the annotated data and report the resulting accuracy.


In this work, we explore a technique for handling intractable marginalization while also injecting inductive bias.

we employ amortized variational inference (Kingma and Welling, 2014;Rezende et al., 2014;Mnih and Gregor, 2014) with a structured inference network.


Variational inference lets us tractably optimize a lower bound on the log marginal likelihood, while employing a structured inference network encourages non-trivial structure.

a con-ditional random field (CRF) constituency parser (Finkel et al., 2008;Durrett and Klein, 2015), which makes significant independence assumptions, acts as a guide on the generative model to learn meaningful trees through regularizing the posterior (Ganchev et al., 2010).


A solution is to make the baseline inputdependent.

we use the self-critical training (SCT) baseline of Rennie et al.


This allows dedicated optimisation schemes for each module.

we found that it is important to have an unbiased estimator of the parser gradients and to allow multiple gradient steps with PPO.


Despite the domain mismatch between the WikiText and CoNLL corpora, the WikiText coverage of the entity words that exist in the CoNLL dataset is high.

most of the person, location and organization entity words that appear in CoNLL either have a Wikipedia section, or are mentioned in a Wiki article.


Under the standard encoder-decoder architecture, an encoder first maps the source-language input sentence into a sequence of hidden vectors, and a decoder then incrementally predicts the target output sentence.

we should notice that several recent models (Vaswani et al., 2017;Zheng et al., 2017;Cheng et al., 2018) which have been shown to be more powerful can also serve as our baseline, since these models focus on very different aspects of NMT, which could be potentially complementary with our focus of syntax integration.


The results are just for reference since experimental details could be very different.

we list the relative improvements over the corresponding baseline models by integrating syntax structures, which are calculated according to their papers.


By not using the 1-best parser outputs, our approach can reduce the error propagation problem, thus bring larger improvements with syntax.

we find that the increases of BLEU scores are smaller than that of Chinese-English translation by integrating syntactic features.


The SAWR approach can also outperform the baseline system significantly.

we find that our SAWR approach is much more effective than the Tree-RNN and Tree-Linearization approaches.


(2015)) now represents the state-of-the-art adapted in most machine translation systems (Wu et al., 2016;Crego et al., 2016;Bojar et al., 2017a), largely due to its ability to benefit from end-to-end training on massive amounts of data.

recentlyintroduced self-attentional Transformer architectures (Vaswani et al., 2017) are rapidly becoming the de-facto standard in NMT, having demonstrated both superior performance and training speed compared to previous architectures using recurrent neural networks (RNNs; (Kalchbrenner and Blunsom, 2013;Sutskever et al., 2014)).


In this paper, we propose a radically different approach for unsupervised NMT-extract-edit, a powerful alternative to back-translation (see Figure 1).

to train the source-to-target translation model, we first extract potential parallel sentence candidates in the target language space given a source language sentence.


The comparative translation loss can avoid the model suffering from the noise while taking advantage of more information.

it is more likely to project the source sentence into the probable region in the target language space with more sentences serving as the anchor points, and the comparative learning scheme iteratively approximates towards more accurate target points.


In the multilingual setting, we would like to learn to translate in all directions having access to only few parallel bilingual corpora.

we would like to learn a collection of models, We can assume that models are independent and choose to learn them by maximizing the following objective: In the statistics literature, this estimation approach is called maximum composite likelihood (Besag, 1975;Lindsay, 1988) as it composes the objective out of (sometimes weighted) terms that represent conditional sub-likelihoods (in our example, P θ (x j | x i )).


Definition 1 (Expected Zero-shot Consistency) Let E s and E 0 be supervised and zero-shot tasks, respectively.

let (•) be a non-negative loss function and M be a model with maximum expected supervised loss bounded by some ε > 0 : we say that a machine translation system is zero-shot consistent if low error on supervised tasks implies a low error on zero-shot tasks in expectation (i.e., the system generalizes).


To introduce agreement-based objective, we use the graph from Figure 2 that defines translation tasks between 4 languages (En, Es, Fr, Ru).

consider the composite likelihood objective (3) for a pair of En − Fr sentences, (x En , x Fr ): where we introduced latent translations into Spanish (Es) and Russian (Ru) and marginalized them out (by virtually summing over all sequences in the corresponding languages).


Variables can only be observed only in pairs (shaded in the graph).

the full log likelihood lowerbounds the agreement objective (up to a constant log Z).


Remark 5 Note that the other terms in the full likelihood also have a non-trivial purpose: (a) the terms ), encourage the model to correctly reconstruct x 1 and x 2 when back-translating from unobserved languages, L 3 and L 4 , and (b) terms P θ (x 3 | x 4 ), P θ (x 4 | x 3 ) enforce consistency between the latent representations.

full likelihood accounts for a combination of agreement, back-translation, and latent consistency.


Experimental results show that the additional recurrence encoder, implemented with either RNN or ARN, consistently improves translation performance, demonstrating the necessity of modeling recurrence for Transformer.

the ARN implementation outperforms its RNN counterpart, which confirms the strength of ARN.


Then we introduce the integration of recurrence encoder into the Transformer.

two strategies are presented to fuse the representations produced by the recurrence encoder and the conventional encoder.


Attentive Recurrent Network (ARN) We can also extend RNN by recurring over a set of feature vectors extracted with an attention model, which allows the model to learn a compact, abstractive feature vectors over the input sequence.

the ARN performs T recurrent steps on the attentive output of the input representation H n−1 r : The output representations H n ARN = {h n 1 , .


Accordingly, we feed the information of the additional recurrence encoder into the decoder of Transformer.

we serve an additional attention layer R n d as the fourth sub-layer in each decoder block to perform attention over the output of the recurrence encoder H N r .


Concerning BIARN models, reducing the layers consistently improves performance.

the 1-Layer BIARN achieves the best performances in both translation quality and training speed.


Training with RL usually has 2 steps: supervised pretraining and policy gradient reinforcement learning (Williams and Zweig, 2016;Dhingra et al., 2017;Li et al., 2016).

the supervised learning step maximizes the log likelihood on the training dialogs, where θ is the model parameter: Then the following RL step uses policy gradients, e.g.


The basic idea is to use an ROC-style curve to visualize the tradeoff between achieving higher reward and being faithful to human language.

at each checkpoint i over the course of RL training, we record two measures: (1) the PPL of a given model on the test data p i = PPL(θ i ) and (2) this model's average cumulative task reward in the test environment R t i .


In order to quantitatively verify the correspondence between direction and diversity, we visualize the distribution of cosine similarities among multiple references for each context for a set of 1000 random samples drawn from the test dataset.

for a context x k and its associated reference responses [y k,0 , y k,1 , • • • ], we compute the cosine similarity between In Figure 3, we compare the distribution of our model with that of MTask, which does not employ our regularization terms.


However, to attain perfect disentanglement, the KB words should be copied, while the language should be generated.

any word in the response that is present in the BOSS KB memory should have a low g s .


It has greater capacity to model complex dependencies, and can handle context variation and polysemy problems harassing many NLP tasks.

it replaces vector w T ∈ R 1×da in additive compatibility function (Eq.


B) Source2token self-attention mechanism (Liu et al., 2016;Lin et al., 2017;Shen et al., 2018a) is designed for sentence embedding or sequence compression, which is based on the importance of each token x i to the entire source sequence x for a specific task.

it removes the query q from the compatibility function f (x i , q) when computing the alignment score.


To verify the capability of MTSA in generating context-aware representation of each token, we compare it with baselines on semantic role labeling (SRL) task, which aims to tag each token from an input sequence with a label for its semantic role.

given a sentence, the goal of SRL is to identify the arguments of each target verb into semantic roles, which can benefit many downstream NLP tasks.


6 The annotators were not provided with knowledge from any external lexical resource (such as WordNet).

the number of senses and the sense distinctions of the word (in the target sense inventory) were unknown to the annotators.


3), we formulate this problem as a classification task.

given a beforerebuttal review, its corresponding author response and other peer reviews, we try to predict whether the reviewer will increase (INC), decrease (DEC) or keep (KEEP) her overall score after the rebuttal.


• But we do not see this as a weakness of our approach.

• why we do not do that in the first submission.


• (2.2) ### = ### * ###, where ### is a matrix of n samples with ### features followed by ### features, hence the size of ### is ###.

• (2.3) our coverage is ### times larger than theirs, so our proposed system can deal much better with the noise when learning ###.


• We conduct t-test and get the p value as ###, which shows good agreement.

• we will strive to improve the presentation quality and to make the draft more readable and better organized for more potential readers.


A situation similar to this might also occur in GEC.

evaluation in GEC has relied heavily on the CoNLL-2014 benchmark, which implies that the field is overdeveloping on this dataset.


We essentially follow the idea used in a previous study (Junczys-Dowmunt and Grundkiewicz, 2016), with some key differences.

we only use English Wikipedia for language model training and only the NUS Corpus of Learner English (NUCLE) and the Lang-8 Learner Corpora (Lang-8) for translation model training to make the experimental settings equal in all models.


Our pre-processing and experimental setup is similar to that reported previously (Chollampatt and Ng, 2018).

a subset of NUCLE (5.4K) is utilized as the development data for selecting the model; the remaining subset (1.3M) is utilized as the training data.


Another interesting observation is that H-LSTM-CRF performs better in unsupervised and semi-supervised settings (i.e., with less target labels).

h-LSTM-CRF adapts better than h-LSTM with small target datasets by exploiting the tag dependencies in the source.


The classification of the defamatory object has a mediocre performance.

an insufficient coverage of group names and names of individuals in the dataset seem to be the main cause as the no classes usually perform considerably better than the yes classes.


Analysis The results show that student assistants solve this task considerably better than crowd workers.

the recognition of references to specific group poses the biggest challenges for crowd workers, which also explains why this group performs much more poorly than the student assistants.


We collected 2.5M tweets using the Twitter's streaming API over a period from 2015-01-01 to 2017-12-31 using an initial set of keywords, henceforth referred as seed keywords to retrieve cyber security related tweets.

we use the main group names of cyber security taxonomy described in Le Sceller et al.


a standard form available in a dictionary.

we use UniMelb (Han et al., 2012), UTDallas (Liu et al., 2011) datasets.


Using the annotated corpus described in Section 2.2, we now develop classifiers that detect threats reported online and analyze users' opinions toward their severity.

given a named entity and tweet, e, t , our goal is to estimate the probability the tweet describes a cybersecurity threat towards the entity, p threat (y| e, t ) and also the probability that the threat is severe, p severe (y| e, t ).


We now explore the effectiveness of this model for forecasting.

we aim to answer the following questions: (1) To what extent do users' opinions about threat severity expressed online align with expert judgments?


may contain more than one name entity): Tweet volume baseline: Intuitively, the number of tweets and retweets can indicate people's concern about a specific event.

the severity for threat CVE i according to the volume model is defined by the cardinality of T CVE i : In our first set of experiments, we compare our forecasted threat severity scores against CVSS ratings from the NVD.


Representation learning algorithms can reveal intrinsic low-dimensional structure in data (Rumelhart et al., 1986;Bengio et al., 2013;LeCun et al., 2015).

deep averaging networks (DANs) are effective for text classification (Shen et al., 2018;Arora et al., 2017;Wieting et al., 2016;Iyyer et al., 2015).


As Figures 1(d) and 1(e) show, DAN's macro-F1 classification performance considerably decreases as divergence (or text length) increases for IMDb and Reddit datasets; note that we sort and bin instances based on their divergence values and report average macro-F1 for each bin.

as the trend lines in Figures 1(d) and 1(e) show, the average macro-F1 performance drops from 0.86 and 0.82 on shorter IMDb and Reddit posts to 0.82 and 0.71 on their longer posts respectively.


To tackle this issue, we further integrate cluster-level representations, at the cost of some further information loss, to create representations of lower dimension for inputs.

given all input documents with k cluster-level representations A ∈ R d×k for each document, we develop an autoencoder with one hidden layer that integrates these cluster-level representations to create a final representation for each document, vector a ∈ R d×m where m is the dimensionality reduction parameter and m × d is length of the representation (final layer of the encoder) and is smaller than d × k for m < k. Training a single-layer autoencoder corresponds to optimizing the learning parameters to minimize the overall loss between inputs and their reconstructions.


It is calculated using the mean intra-cluster distance and the mean nearestcluster distance for each sample.

the mean distance between each embedding and all other embeddings in the same cluster (mc), and the mean distance between the embedding and all other embeddings in the next nearest cluster (the nearest cluster that the embedding is not part of) (mn) are used to measure SC for the embedding: .


In contrast, here we target several different kinds of offensive content.

we model the task hierarchically, identifying the type and the target of offensive messages in social media.


Micro  Table 5: Comparison with different traditional classifiers in AD:Control classification task.

our model has higher accuracy than the best traditional classifier, MLP.


We show that consensus networks, with the noise modality and cooperative optimization, improve upon traditional neural network baselines given the same features.

with all 413 lin-guistic features, our models outperform fullyconnected neural networks and other traditional classifiers used by state-of-the-art papers.


Utterances addressing the main appealing problem which is yet to be resolved, including client's internal factors or their behaviors related to the problems.

the utterances include cognition, emotion, physiological reaction, and diagnostic features of the problem and desire to be changed.


Utterances describing insights, cognition of small and big changes in internal factors or behaviors.

an utterance at the point where the appealing problem is being resolved.


NER is highly sensitive to word representations, because the model needs to make token level decisions.

each token needs to be assigned a proper label.


Results in Table 3 show that, although all pretrained word vectors and LMs can improve the performance of the target model, the improvement varies in different target data sets.

no single source is suitable for all target NER data  Table 4: Correlation coefficients between similarity measures and the effectiveness of pretrained models.


We use Spearmans' correlation coefficient as a scoring function.

for each sentence we create two vectors comprising counts of how many times each token was annotated by crowd and expert workers, respectively, and calculate the correlation between these.


Our method shows improvements over a plain encoder-decoder framework, which reportedly achieves state-of-the-art performance on lemmatization and morphological analysis (Bergmanis and Goldwater, 2018;Peters et al., 2018).


We choose three fundamental NLP tasks that serve as a good representative of various properties inherent in a language, ranging from morphology (word structure), syntax (grammar), and semantics (meaning).

we experiment with morphological tagging for German, Czech, Russian, and English, 3 lexical semantic tagging for English and German, and syntactic tagging via CCG supertagging for English.


We studied the impact of using different representation units-words, characters, BPE units, and morphological segments-on the representations learned by seq2seq models trained for neural machine translation.

we evaluated the performance of such representations on core natural language processing tasks modeling morphology, syntax, and semantics.


Simply presenting improved results does not entirely satiate our curiosity: we would also like to understand why our model performs better.

we have assumed an additional level of supervision-namely, the annotation of morphological tags.


In addition to strong numbers, we tried to explain when and why our model does better.

we show a significant correlation between our scores and the number of tokens and tags present in a treebank.


We can view this model as a form of exponentialfamily matrix factorisation (Collins et al., 2001).

our model seeks to explain a binary matrix of parameters.


In this work, we use a language modelling objective to pre-train language embeddings; we train a character-level neural language model with a distributed language embedding of language .

we use the model ofÖstling and Tiedemann (2017), visualised in Figure 4.


Typological KBs such as WALS are known to be incomplete.

not all parameters π i are observed for all languages.


In this work, we present a principled study of the morphosyntactic information encoded in flat word clusters induced exclusively from classbased language models via Brown Clustering and Exchange algorithm.

we focus on how well these approaches derive clusters that represent Parts of Speech as a measure of the morphosyntactic information encoded.


In our experiments, purity measures the percentage of vocabulary that is labeled correctly.

purity does not depend on word frequency.


Unlike purity, values of AdjMI cannot be interpreted to say anything about the percentage of points that have been properly allocated.

a value of, say 0.3, does not indicate that 30% of the points have been properly separated.


Thus, words with multiple roles, such as denominal verbs or deverbal nouns, cannot be differentiated by the algorithms when operating on corpora from languages where the such morphological derivations are performed without employing suffixes or prefixes.

if the lexical form does not change, neither Brown nor Exchange can identify which tokens represent what grammatical role.


Contributions Our study provides a theoretical analysis of crosslingual transfer learning in topic models.

we first formulate on-site transfer as circular validation, and derive an upper bound based on PAC-Bayesian theories (Section 2).


showed that knowledge transfer across languages happens through priors.

assume the source document is generated from θ (d S ) ∼ Dir(α), and has a sufficient statistics n d S ∈ N K where each cell n k|d S is the count of topic k in document d S .


In crosslingual document classification, a document classifier, h, is trained on documents from one language, and h is then applied to documents from another language.

after training bilingual topic models, we have K bilingual word distributions { ϕ .


Additionally, let P x T ,zx S be the conditional probability for the to-ken x T being assigned to topic z x S : Another assumption we made is once the source language is converged, we keep the states of it fixed.

z (t) , and only sample the target language.


(2016) and for dependency parsing by Stenetorp (2013) and , among others.

showed that composing representations of subtrees using recursive neural networks can be beneficial for transition-based dependency parsing.


As is consistent with the literature de Lhoneux et al., 2017a;Smith et al., 2018b), using characterbased word representations and/or POS tags consistently improves parsing accuracy but has a different impact in different languages and the benefits of both methods are not cumulative: using the two combined is not much better than using either on its own.

character models are an efficient way to obtain large improvements in morphologically rich languages.


The simplest way to do this is to make arguments dependents of their functors, similar to Koller and Kuhlmann (2009).

a word v with the (indexed) category X|Y α  Table 2: Functional categories which in dependency conversion become dependents of their first argument, rather than the other way around, depending on the treebank-specific conventions.


To mitigate the hubness problem, we adopt the Cross-Domain Similarity Local Scaling (CSLS) metric proposed in (Conneau et al., 2017) that penalizes the similarity score of the hubs.

given two mapped embeddings W xy x denoted x and y, CSLS first computes the average cosine similarity of x and y for their k nearest neighbors denoted r T (x ) and r S (y) in the other language respectively, then the corrected similarity measure CSLS(•, •) is defined as: where cos(•, •) is the cosine similarity.


Our experimental results demonstrate the effectiveness of this approach for zero-shot and few-shot learning of dependency parsing.

our method consistently outperforms the previous state-of-the-art on 6 tested languages, yielding an improvement of 6.8 LAS points on average.


As above, contextindependent anchors are informing this process.

we introduce an alignment component to the loss function of the language model, pushing the anchors to be closer in the joint space.


on the Kazakh tree-bank from the recent CoNLL 2018 shared task with only 38 trees for training, the model yields 5 LAS points gain over the top result (Smith et al., 2018a).


2018;  and use a first-order graphbased model.

we adopt the neural edge-scoring architecture from Dozat and Manning (2017); , which is based on Kiperwasser and Goldberg (2016).


The focus of our paper is on early rumour detection.

our aim is to identify rumours as early as possible, while keeping a reasonable detection accuracy.


Through reinforcement learning ERD is able to learn the minimum number of posts required to identify a rumour.

eRD can dynamically determine the appropriate checkpoint for different rumours, and this feature is the core novelty of our methodology.


Next, to further fuse representations distilled by the bi-attention on each encoder, we design a merge layer, a multilayer perceptron (MLP) activated by hyperbolic function: where Note that either v p or v c conveys the information from both posts and conversations, but with a different emphasis.

v p mainly retains the contexts of posts with the auxiliary information from conversations, while v c does the opposite.


For Twitter dataset, the improvements are not that large, which confirms again that hashtag annotation on Twitter is more difficult due to the noisier data characteristics.

compared to SEQ2SEQ, OUR MODEL achieves an additional performance gain in producing new hashtags by leveraging conversations with the bi-attention.


As can be seen, only our model successfully generates "aus open", the gold standard.

it not only ranks the correct answer as the top prediction, but also outputs other semantically similar hashtags, e.g., sport-related terms like "bbc football", "arsenal", and "murray".


Our work is also closely related to neural language generation, where the encoder-decoder framework (Sutskever et al., 2014) acts as a springboard for many sequence generation models.

we are inspired by the keyphrase generation studies for scientific articles (Meng et al., 2017;Ye and Wang, 2018;Chen et al., 2018Chen et al., , 2019, incorporating word extraction and generation using a seq2seq model with copy mechanism.


It is worth pointing out that the learned character embeddings of SELMo carry almost no visual information, as illustrated in Table 2.

except for a few very standard cases, nearest neighbors of characters do not visually resemble the orginal characters, even when trained on the 1 billion word benchmark (Chelba et al., 2013 VELMo: To obtain a visually informed variant of ELMo, we replace learned character embeddings with the ICEs and keep the character embeddings fixed during training.


17 We optimize using Adam (Kingma and Ba, 2014) with LR=.001 for 20 epochs, apply dropout with p = .2, select the model checkpoint that performs best over the validation set, and cross-validate the model's dimension (128 vs. 256) and the number of layers (1 vs. 2) separately for each crossvalidation split.

18 : for the title and text body separately, length, type-token ratio, rate of first-person pronouns, rate of secondperson pronouns, rate of question-marks, rate of capitalization, and Vader sentiment (Hutto and Gilbert, 2014).


We augment the post-time features with early-discussion feature sets by giving our algorithms access to comments from increasing observation periods.

we train linear classifiers by combining our best post-time feature set (TEXT+TIME) with features derived from comment trees available after t minutes, and sweep t from t = 15 to t = 180 minutes in 15 minute intervals.


To incorporate shallow linguistic knowledge, we process the news article collection for each nation pairs to extract (1) verbal predicates and (2) nouns  and proper nouns from each sentence.

we use a dependency parser to detect verbal predicates and their subjects and objects, and only include predicates for which both subjects and objects were found.


The intuition behind our model follows RMN, i.e., inferring relation embeddings by reconstruct-ing textual information in the embedding space.

we learn a fixed set of relation embeddings and use a convex combination of these relation embeddings to reconstruct information in sentences that mention both entities.


In addition to the comparison with RMN, we also checked a simpler baseline by replacing the descriptor weights with term-frequency values of each verbal predicates for each document.

the term-frequency of a predicate p in document d is count(p) length (d) .


We, therefore, aim to develop techniques to generate section titles in low-resource environments.

we present an extractive pipeline for section title generation by first selecting the most salient sentence and then applying deletion-based compression.


We define an idealized evaluation metric based on whether p model is close to a reference distribution p ref , which is generally human-generated.

1 consider a random variable y drawn from either the reference or the model based on an indicator z ∼ Bernoulli 1 2 : Define L * to be twice the lowest possible error over any discriminator f that attempts to determine z based on x and y: Obstacles.


The optimal discriminator simply predicts and z = 0 otherwise.

the decision boundary is given by More generally, we can obtain this equality with a wider class of φ.


We opt to use nearest neighbors classifiers as they are simple, require no training, and can asymptotically capture arbitrary continuous decision boundaries.

we set k = 16 and define neighbors using L 2 distances over the feature vectors φ(x, y) scaled componentwise to have unit variance.


Unfortunately, n-gram diversity is computed across contexts by counting the number of unique n-grams generated, and so does not measure a model's ability to generate multiple valid utterances at any single context.

a model which only outputs a single memorized utterance per context (e.g., via memorization or retrieval) can still have high n-gram diversity as long as the memorized sentences differ across contexts.


leading to state-of-the-art performance.

we identify and study eight aspects of conversation that can be measured by human judgments, while varying four types of low-level attributes that can be algorithmically controlled in neural models; see Figure 1.


Controllable neural text generation Researchers have proposed several approaches to control aspects of RNN-based natural language generation such as sentiment, length, speaker style and tense Ficler and Goldberg, 2017;Ghazvininejad et al., 2017;Hu et al., 2017;Kikuchi et al., 2016;Peng et al., 2018;Wang et al., 2017).

several works use control to tackle the same common sequence-to-sequence problems we address here (particularly genericness and unrelated output), in the context of single-turn response generation (Baheti et al., 2018;Li et al., 2016aLi et al., , 2017aShen et al., 2017;Zhang et al., 2018a;.


We find that this approach works well -for example, the sequence-to-sequence model is generally good at detecting when to ask a question.

this is easier than the alternative: developing a separate process to decide, for each utterance, whether to ask a question.


By applying a negative weight to these features, we can reduce repetition.

if the weight is −∞, our method is equivalent to n-gram blocking as described by Kulikov et al.


To control response-relatedness with weighted decoding, we use the decoding feature resp rel: where word emb(w) is the GloVe embedding for the word w, sent emb( ) is the sentence embedding for the partner's last utterance (note is part of the context x), and cos sim is the cosine similarity between the two.

the sentence embedding sent emb(s) for an utterance s is a Input: Do you go get coffee often Baseline Response: I do, when I am not playing the piano.


engagingness improvements over the greedy and beam-search baseline models.

we find that controlling for multi-turn (self) repetition is important and should be incorporated alongside other attribute control methods.


Our strategy is to model puns as deviations from normality.

we mine seed sentences (sentences with the potential to be transformed into puns) from a large, generic corpus, and edit them to satisfy the three requirements above.


We show that the local-global surprisal scores strongly correlate with human ratings of funniness, and all of our systems outperform the baselines based on human evaluation.

rETrIEVE+SWAP+TOPIC (henceforth SUrGEN) achieves the highest success rate and average funniness score among all systems.


There is slight inconsistency between results on SEMEVAL and KAO.

for puns and non-puns, the distinctiveness metric shows a significant correlation with human ratings on KAO but not on SEMEVAL.


The latter takes as input the initial sentence representations s rather than sentence embeddings v k−1 from the previous layer.

new embeddings v k are computed from scratch relying on the structure from the previous layer.


To gain further insight into the structures learned by SUMO, we inspected the trees it produces.

we used the Chu-Liu-Edmonds algorithm (Chu and Liu, 1965;Edmonds, 1967) to extract the maximum spanning tree from the atten-tion scores.


We also examine the trees learned from different SUMO variants (with different numbers of iterations) in order to establish whether the iterative process yields better structures.

we compared the dependency trees obtained from our model to those produced by a discourse parser (Zhao and Huang, 2017) trained on a corpus which combines annotations from the RST treebank (Carlson et al., 2003) and the Penn Treebank (Marcus et al., 1993).


The counter c t is decremented at every time step as in Section 4.2.

when the counter is zero, we only allow the model to copy or generate stop words.


When we allow the model to determine its own output, we find that "EDIT + COUNT" performs the best.

this model does well at understanding which part of the context to select, and even does better than other unconstrained models at selecting appropriate substitutions.


To explore the role played by conversation dynamics in deceptive behaviors, we collected conversations where participants acted deceptively.

we opted for identifying public sources where the veracity or falsehood of conversation participants is known.


These features are generated from the non-verbal behaviors described in Section 3.2 and represented as percentages.

the different behavioral values for a category (e.g., Head) in a verbal utterance are counted and represented as percentages.


In this scenario, we explore whether the availability of data by the same individual can help to improve the detection of deceptive behavior.

this experiment builds personalized deception models for the host using the different sets of features representing verbal and non-verbal behavior.


Currently, the dataset provides feature sets drawn from three modalities (verbal and non-verbal, as well as dialogue) but can be further analyzed to extract additional features from other modalities such as speech.

the dialogue has the potential to add many more layers of information by systematically analyzing verbal, nonverbal, and speech patterns between the participants.


For future work, we are investigating alternatives to MACE, which was designed for categorical annotations rather than the sequence labelling of our task.

we are looking into the Bayesian method of Simpson and Gurevych (2018), which takes advantage of the sequential dependencies between BIO tags, and works more robustly with noisy, subjective data such as ours.


where, F represents all the events in first sentence, independent variable b 1 (i) indicates the minimum number of causal relations that an event in F participates in, M represents the set of event mention pairs that can be mapped to the same sentence and l ij is a leakage variable that allows distant event mentions in F receiving a very high confidence value p ij to have a causal relation.

we encourage causal links between two event mentions that are in nearby sentences or can be mapped to the same sentence using coreference links 7 .


Thus, clauses having one of these temporal discourse relations may also favor causal relations between events in them.

we model the above three dependencies between discourse relations and causation through constraints 11 and the objective function 12. x ij ≥ 1 x ij ≤ 0 we enforce events in clauses with the contingency discourse relation to have at least one event pair with causal relation.


The idea of Word2Vec can be transferred to the level of sentences as well.

the socalled Skip Thought Vector model (STV) (Kiros et al., 2015) derives a vector representation of the current sentence by predicting the surrounding sentences.


(Song and Roth, 2015) propose an alternative approach to applying the cosine measure to the two word vector centroids for ESA word embeddings.

they establish a bipartite graph consisting of the best matching vector components by solving a linear optimization problem.


This proof can easily be extended to the general case, too.

we show that the similarity estimate does not change, if two columns of the first matrix are exchanged, which can be expressed by postmultiplying this matrix with a permutation matrix P. By employing symmetry and induction this proof can be applied to arbitrary sequence permutations and to the second argument matrix as well.


In this paper, we will describe an alternative approach based on unsupervised natural language processing.

our business partner operates a commercial youth platform for the Swiss market, where registered members get access to third-party offers such as discounts and special events like concerts or castings.


We experimented with different keyword list sizes but obtained the best results with rather few and therefore precise keywords.

we used the following number of keywords for the individual milieus: • Action Sportsman: 3 • Young Performer: 4 • Hedonist: 7 • Conservative Youth: 4 • Progressive Postmodern Youth: 6 In order to provide a gold standard, three professional marketers from different youth marketing companies annotated independently the best matching youth milieus for every contest answer.


We need to show that: For that we leverage the Cauchy-Schwarz inequality that states that for an inner product space H we have for all x, y ∈ H, where equality holds if, and only if, x is a scalar multiple of y.

the function f (Y, X) = tr(Y X) is such an inner product.


We propose a Glocal (global-local portmanteau) technique which incorporates both components directly over the word-graph.

we contribute a neural model that elegantly incorporates the random walk scores, while incorporating parameters to fit keyphrase labels.


Therefore, we can use it for a bottom-up (max-violating) inference procedure, which first finds the best candidate item to be put at the lowest position of the rank and proceeds filling the positions in the ascending order.

we start with the last N th position of the rank and put there the minimum weighted item: According to the decomposition in Eq.


In this paper, we proposed new structured prediction algorithms for ranking problems.

we designed (i) a new loss function that leads to the direct optimization of MAP; and (ii) two new algorithms, based on LSP and LSSVM solvers, to optimize it.


In our paper, we demonstrate a similar approach with neural networks.

our neural model attends over sentences to distinguish between sentences that are likely to express some relation between the entities and sentences that do not.


Our goal is to predict which relation types are expressed between a pair of entities (e 1 , e 2 ), given all sentences in which both entities are mentioned in a large collection of unlabeled documents.

following previous work on distant supervision, we use known tuples (e 1 , r, e 2 ) in a knowledge base K to automatically annotate sentences where both entities are mentioned with the relation type r. we group all sentences s with one or more mentions of an entity pair (e 1 , e 2 ) into a bag of sentences B e 1 ,e 2 , then automatically annotate this bag with the set of relation types L distant = {r ∈ R : (e 1 , r, e 2 ) ∈ K}, where R is the set of relations we are interested in.


Results demonstrated that our WE WPI yielded significantly different results among metrics.

wE wPI was found to have significantly better results than those of wE at the segment level, as shown in Figures 8 and 9.


2018), an extended variant of cosine similarity that avoids the hubness problem (Radovanović et al., 2010;).

2 the element m ij at row i and column j of M corresponds to the CSLS value between the crosslingual vectors x CL i and z CL j of the i th source word and the j th target word respectively: and r Zk (z) calculate the average cosine similarity of a vector with its k nearest neighbors (measured by cosine) in the mapped spaces of X, Z, respectively.


In this work we present two primary methods of synthesizing natural noise, in accordance with the types of noise identified in prior work as naturally occurring in internet and social media based text (Eisenstein, 2013;Michel and Neubig, 2018).

we introduce a synthetic noise induction model which heuristically introduces types of noise unique to social media text and labeled back translation (Sennrich et al., 2015a), a data-driven method to emulate target noise.


For every token we choose to introduce the different types of noise with some probability on both French and English sides in 100k sentences of EP.

we fix the probabilities of error types as follows: spelling (0.04), profanity (0.007), grammar (0.015) and emoticons (0.002).


We impute the limitation to the "offline" property of these methods.

only employing checkpoints after training cannot affect the original training process.


As the formal language becomes more expressive, the execution space is more compressed as each state can be reached in more ways.

the formal language in SCENE contains more functions compared to the other domains, and so it benefits the most from execution-space search.


VBSIX is applicable to any task that supports graph-search exploration.

for tasks that can be formulated as an MDP with a deterministic transition function, which allow efficient execution of multiple partial trajectories.


Here, we study goal-directed object detection, where the set of possible valid objects is far greater than in the typical object detection problem.

we introduce the Visual Query Detection (VQD) task (see Fig.


), we view this problem as an interactive process, in which the human operator can observe the agents' response to their instruction and adjust it by providing advice, a form of online feedback.

the advice consists of a short sentence, simplifying the user's intent.


We present a novel method for mapping unrestricted text to knowledge graph entities by framing the task as a sequence-to-sequence problem.

given the encoded state of an input text, our decoder directly predicts paths in the knowledge graph, starting from the root and ending at the target node following hypernym-hyponym relationships.


1 Our model is based on a sequence-to-sequence neural network (Sutskever et al., 2014) coupled with an attention mechanism (Bahdanau et al., 2014).

we use an LSTM (Hochreiter and Schmidhuber, 1997) encoder to project the textual description into a vector space and an LSTM decoder to predict the sequence of entities that are relevant to this definition.


Table 4 shows that the models trained on MuST-C data achieve better results on the balanced test set in all the three tasks.

: i) a reduction of 10.1 WER points in ASR indicates a higher quality of audio-transcription alignments, ii) a BLEU increase of 0.56 points in MT indicates a similar quality for transcription-translation alignments, and iii) a BLEU increase of 3.31 points in SLT indicates a higher quality of audio-translation alignments.


We omitted several arcs for convenience; namely, every morphological tag m i depends on the entire sequence .

for contextual inflection, as follows: (1) the distribution is over interleaved sequences of one-to-one aligned inflected words and morphological tags, conditioned on a lemmatized sequence, all of length n. This distribution is drawn as a hybrid (directed-undirected) graphical model (Koller and Friedman, 2009) in Fig.


However, the differences between the mixed model and the two language-specific models not trained on a given test language are significant (p < 0.05).

the difference between Abs-mix and Abs-sw and the difference between Abs-mix and Abs-tl are significant on the Somali test set, the difference between Abs-mix and Absso and the difference between Abs-mix and Abstl are significant on the Swahili test set, and the difference between Abs-mix and Abs-so and the difference between Abs-mix and Abs-sw are significant on the Tagalog test set.


An NSD can be used to form the output sequence to improve NMT using the idea of multi-task learning.

we train the model to predict the NSD sequence.


Zero-shot recall is considerably degraded relative to the non-adapted baseline for both full and adaptation of a fixed subset of tensors (fixed and top).

terms never observed before during online training are translated correctly less often than they would be with an unadapted system, despite the data set's consistent domain.


As evident from comparing the curves for BLEU and R0, the BLEU score and the proposed metric give disparate signals for this data.

there are two distinct dips in the curves for R0 (as well as R0+1) and BLEU: 1.


We consider the problem of making efficient use of heterogeneous training data in neural machine translation (NMT).

given a training dataset with a sentence-level feature such as noise, we seek an optimal curriculum, or order for presenting examples to the system during training.


Our results show that this model successfully detects most shifts in our data, but it overgeneralizes.

the model gets confused with contextual changes due to speakers in the community often talking about particular people and events, which are frequent on short time spans.


Cuz I need a lean', used to talk about hiring players due to new hires typically leaning on a Liverpool symbol when posing for a photo right after signing for the club.

illustrative is the 'F5' example shown in Table 2.


Instead, we achieve to learn a personalized word embedding with the guidance of user-item interactions.

the importance of words is learned to match user preferences.


In this paper we present a conceptually simple and effective transfer learning approach that addresses the problem of catastrophic forgetting.

we combine the task-specific optimization function with an auxiliary language model objective, which is adjusted during the training process.


weighted sum of the task-specific loss L task and the auxiliary LM loss L LM , where γ is a weighting parameter to enable adaptation to the target task but at the same time keep the useful knowledge from the source task.


The performance improvement is more pronounced when there is a mismatch between the LM and task domains, i.e., the non-Twitter domain tasks.

for the PsychExp and SCv2 datasets, sequentially unfreezing yields significant improvement in F 1 building upon our intuition.


Empirical results that are shown in Table 2 indicate that by only adding the auxiliary language modeling objective, we obtain better results on all downstream tasks.

a comparison of the P-LM + aux model and the P-LM model shows that the performance of SiATL on classification tasks is improved by the auxiliary objective.


We show that this strategy is more effective than using pretrained emoji embeddings for tweet classification.

we obtain new state-of-the-art results in irony detection and sentiment analysis despite our neural network is simpler than previous proposals.


In this paper, we tackle both irony and sentiment analysis in tweets-two classification tasks.


We preprocess the input text following standard steps.

we tokenize with the NLTK's TweetTokenizer (Bird, 2006), lowercase all text, and use regular expressions to remove stop words, numbers, urls, consecutive repeated words and Twitter users (i.e., tokens whose first character is '@').


These results confirm that the personalized word embeddings trained through our methods successfully learned taskindependent personal semantic variations.

they were helpful even for solving tasks other than the review-target identification.


In the context of fake news, bias, and propaganda, we study two important but relatively under-explored problems: (i) trustworthiness estimation (on a 3-point scale) and (ii) political ideology detection (left/right bias on a 7-point scale) of entire news outlets, as opposed to evaluating individual articles.

we propose a multi-task ordinal regression framework that models the two problems jointly.


(2017) used multi-task ordinal regression for the task of fine-grained sentiment analysis.

they introduced an auxiliary coarse-grained task on a 3-point scale, and demonstrated that it can improve the results for sentiment analysis on the original 5-point scale.


Therefore, bias and factuality can be analogous to the facial AUs in (Walecki et al., 2016), and represent two aspects of news reporting, each being modeled on a multi-point ordinal scale.

we model bias on a 7-point scale (extreme-left, left, center-left, center, centerright, right, and extreme-right), and factuality on a 3-point scale (low, mixed, and high).


We can see that factuality is not very useful as an auxiliary task by itself (MAE=1.584 and MAE M =1.695).

a medium with low factuality could be extremely biased to either the right or to the left.


The results of these experiments, shown in parentheses in Table 3, indicate that adding factuality to any combination of auxiliary tasks consistently yields lower MAE scores.

modeling the combination of factuality+bias5-way+bias3-way yields the best results (MAE=1.475 and MAE M =1.623).


The contexts have the characteristic that each context contains a maximum of one pun (Miller et al., 2017).

there exists only one pun if the given sentence is detected as the one containing a pun.


We also observe that a pun tends to appear at the end of a sentence.

based on the statistics, we found that sentences with a pun that locate at the second half of the text account for around 88% and 92% in homographic and heterographic datasets, respectively.


We then train a number of standard baselines to predict upcoming actions, a task that requires to be aware of the context.

we test a number of generic models, from a simple logistic regression to neural models.


Argument mining studies the automatic detection of argumentative components and structure within discourse (Peldszus and Stede, 2013).


As observed in Figure 3, ACL reviews contain more requests but fewer facts than other venues.

we find that 94.6% of ACL reviews have at least one REQUEST proposition, compared to 81.5% for ICLR and 84.7% for UAI.


Our second graph is an extended version of the first (referred to as the extended graph) that additionally contains nodes representing the tweets of the authors.

in addition to the 1, 875 author nodes, the graph contains 16, 202 tweet nodes.


Node2vec extends the word2vec skipgram model (Mikolov et al., 2013) to graphs in order to create low-dimensional embeddings for nodes based on their position and neighborhood.

for a given graph with nodes V = {v 1 , v 2 , .


In contrast to node2vec, our method allows us to additionally propagate information with respect to whether tweets composed by authors and their communities are abusive or not.

as labels are available for a subset of nodes in our graph (i.e., the tweet nodes), we frame the task as a graphbased semi-supervised learning problem, allowing the model to distribute gradient information from the supervised loss on the labeled tweet nodes.


In our experiments, we apply a 2-layer GCN to the extended graph.

2 our GCN performs the following computation, yielding a softmax distribution over the 3 classes in the dataset for each of the nodes: We set the input feature vectors in F to be the binary bag-of-words representations of the nodes (following Kipf and Welling 2017); for author nodes, these representations are constructed over the entire set of their respective tweets.


We run every method 10 times with random initializations and stratified train-test splits.

in each run, the dataset is split into a randomly-sampled train set (90%) and test set (10%) with identical distributions of the 3 classes in each.


In this paper, we provide an analysis to highlight the importance of addressing the meaning conflation deficiency.

we show how distinguishing different meanings of a word can facilitate a more accurate semantic understanding of a state-of-the-art reverse dictionary system, reflected by substantial improvements in recall and generalisation power.


(2016) which leverages a standard neural architecture in order to map dictionary definitions to representations of the words defined by those definitions.

they proposed two neural architectures for mapping the definition of word t to its word embedding e t .


We mitigate the meaning conflation deficiency through a sense-specific mapping function that obtains distinct interpretations for individual definition, hence mapping them to different points in the target space: s t 1 , s t 2 , and s t 3 .

in our experiments we leveraged DeConf (Pilehvar and Collier, 2016).


This indicates a prospective strategy of expanding the triple embedding models to improve the ranking quality of the search personalization systems.

our MRR and Hits@1 scores are higher than those of TransE (with relative improvements of 14.5% and 22% over TransE, respectively).


In particular, our MRR and Hits@1 scores are higher than those of TransE (with relative improvements of 14.5% and 22% over TransE, respectively).

our CapsE achieves the highest performances in both MRR and Hits@1 (our improvements over all five baselines are statistically   (Vu et al., 2017).


Many machine learning tasks require structured outputs, and the goal is to assign values to a set of variables coherently.

the variables in a structure need to satisfy some global properties required by the task.


The general assignment problem requires assigning d agents to d 0 tasks such that the agent nodes and the task nodes form a bipartite graph (without loss of generality, assume d  d 0 ).

an agent can handle exactly one task, and each task can only be handled by at most one agent.


Here we adopt the MATRES dataset (Ning et al., 2018b) for its better inter-annotator agreement and relatively large size.

we use 35 documents as T 0 (the TimeBank-Dense section, 4 147 documents as U 0 (the TimeBank section minus those documents in T 0 ), and the Platinum section (a benchmark testset of 20 documents with 1K TempRels) as T test .


The structural constraints for SRC is that each argument can be assigned to exactly one semantic role, and the same role cannot appear twice for a single verb, so SRC is an assignment problem as in Example 2.

we use the Wall Street Journal (WSJ) part of Penn TreeBank III (Marcus et al., 1993).


We use the chunking dataset provided by the CoNLL-2000 shared task (Tjong Kim Sang and Buchholz, 2000).

we use 2K tokens' annotations as T 0 , 14K tokens as U 0 , and the benchmark testset (25K tokens) as T test .


The element-wise product ⊗ is applied between the token embedding Q tq and the last dimension of kernel weights U.

we create F convolutional kernels for each query token, where each kernel is "injected" with the embedding of that query token via element-wise product.


Given a query token t q and the j-th position in post p, we compute the interaction scores by taking the cosine similarity between the word embeddings of token t q and post tokens t p j:j+k−1 from position j to j + k − 1: where S j ∈ R k×1 and k is the width of the convolutional kernel we are learning.

for each token in the post within the window, we compute its cosine similarity with query token t q .


For example, NASH (Shen et al., 2018) passes the gradients through the Bernoulli stochastic layer using ST estimator.

they do: where the ST estimator lets the gradients of the stochastic layer dz dg φ (x) ≈ 1.


Text generation from LMs is commonly evaluated using probabilistic metrics.

given a test sequence of symbols (t 1 , .


Results ( §3) show that neural DMRS to English text generation can obtain up to 83.37 BLEU and 32% exact match, substantially higher than previous work.

we obtain an 11.6 BLEU improvement through semi-supervised training using the output of a grammar-based parser, compared to training on gold data only.


However, using this criterion is complicated by the fact that it is not trivial to map between the entities in the plan (that originate from the RDF triplets) and the entities in the text.

due to language variability, the same plan entity may appear in several forms in the textual sentences.


Splitting tendencies For each input size, we keep track of the possible ways in which the set of facts can be split to subsets of particular sizes.

we keep track of probabilities such as p s (s = [3, 2, 2] | 7) of realizing an input of 7 RDF triplets as three sentences, each realizing the corresponding number of facts.


Keeping the discriminator fixed, we then fine-tune the generator, using as reward the probability esti-mated by the discriminator that a generated question was in fact real.

the generator is rewarded for successfully fooling the discriminator.


The model fine tuned on a QA and LM objective is rated as significantly worse by human annotators, despite achieving higher scores in the automatic metrics.

the training objective given by these reward sources does not correspond to true question quality, despite them being intuitively good choices.


Since the discriminator is also able to consider a full context-question-answer triple as input (as opposed to a context-question pair for the QA task), we fuse this information in the output layers.

we apply max pooling over time to the output of the first two encoders, and we took the mean of the outputs of the third encoder that formed part of the answer span.


This additional modeling power allows the Graph Transformer to better articulate how a vertex should be updated given the content of its neighbors, as well as to learn global patterns of graph structure relevant to the model's objective.

v is embedded in a dense continuous space by the embedding process described at the end of this section, resulting in matrix which will serve as input to the graph transformer model shown in Figure 4.


In this paper, we consider the problem of dictionary learning in well-known datasets.

we consider the problem of dictionary learning, where the goal is to find a set of dictionary blocks that maximize the likelihood of a given set of dictionary blocks .


Tuple: pool, open, 6:00am daily As can be seen from these examples, harvesting facts from CQA data presents significant challenges.

the system must interpret information collectively between the questions and answers.


Motivated by these, NEURON allows constraints relevant to information extraction to be incorporated in the model.

we describe how the decoder can enforce vocabulary and structural constraints on the output.


2) that do not appear in the input (q, a) pair.

it computes a binary mask vector v, where |v| is vocabulary size and v i = 1 if and only if i-th token appears in q or a.


To compare the coverage of sequences extracted by NEURON against the baseline method, we compute relative coverage of NEURON as the fraction of all correct predictions that were generated exclusively by NEURON.

where TP denotes the correct predictions.


Since it is difficult to define the usefulness of the tuples without assuming a KB, we used relevance and concreteness as criteria to grade extracted tuples.

each worker was asked to choose one option from the three options: Not relevant or unclear (0), Relevant (1), Relevant and concrete (2).


2. we use predictions from a coreference resolution system to add mentions of elements in C q ∪ {s} beyond exact matching (including both noun phrases and anaphoric pronouns).

we use the end-to-end coreference resolution by Lee et al.


Each step of the algorithm (also referred to as a hop) updates all node representations in parallel.

a node is updated as a function of messages from its direct neighbours, and a message is possibly specific to a certain relation.


Keeping in mind we want an efficient model, we encode words in supporting documents and in the query using only a pre-trained model for contextualized word representations rather than training our own encoder.

we use ELMo 2 (Peters et al., 2018), a pre-trained bi-directional lan-guage model that relies on character-based input representation.


Since we do not have any component in our model that processes the documents, we expect a drop in performance.

in this ablation our model tries to answer questions without reading the context at all.


The second important observation is that if we then remove R-GCN (GloVe w/o R-GCN in Table 3), we lose 8.0 points.

the R-GCN component pushes the model to perform above Coref-GRU still without accessing context, but rather by updating mention representations based on their relation to other ones.


Secondly, we study how the model performance degrades when the input graph is large.

we observe a negative Pearson's correlation (-0.687) between accuracy and the number of candidate answers.


In Table 6, we report three samples from WIKI-HOP development set where out Entity-GCN fails.

we show two instances where our model presents high confidence on the answer, and one where is not.


In Figure 3, we show how the model performance goes when the input graph is large.

how Entity-GCN performs as the number of candidate answers or the number of nodes increases.


In this paper, we introduce a new training framework, called LaCE (Label Consistency Explorer), that leverages label consistency among paragraph summaries.

it encourages label consistency during end-to-end training of a neural model, allowing consistency bias to improve the model itself, rather than be enforced in a post-processing step, e.g., posterior regularization (Ganchev et al., 2010).


To examine whether this happens in practice, we compute and report the consistency score between paragraphs about the same topic (Table 5).

for an entity that appears in two paragraphs about the same topic, we compare whether the summaries of state change predictions for each match.


Since many of DROP's questions require similar discrete reasoning, it is appealing to port some of the successful work in semantic parsing to the DROP dataset.

we use the grammar-constrained semantic parsing model built by Krishnamurthy et al.


Count We model the capability of counting as a multi-class classification problem.

we consider ten numbers (0-9) in this preliminary model and the probabilities of choosing these numbers is computed based on the passage vector h P : Arithmetic expression Many questions in DROP require the model to locate multiple numbers in the passage and add or subtract them to get the final answer.


To prevent our model from degenerating into a partial word-based model, we assign word information to a single character and ensure that there are no shortcut paths between characters.

word information is assigned to its end character and start character in forward WC-LSTM and backward WC-LSTM respectively.


We carefully analyze the experimental results and find that there are a large number of informal texts in the MSRA test set.

the MSRA test set contains some very long sentences, in which there are a series of Chinese person name without delimiter.


In this work, we present a unified character level sequence-to-sequence deep learning model that recovers both types of diacritics without the use of explicit feature engineering.

we employ a standard neural machine translation setup on overlapping windows of words (broken down into characters), and then we use voting to select the most likely diacritized form of a word.


In this paper, we introduce a unified model for both diacritic types while improving upon the state-of-the-art.

we approached the task as a sequence-to-sequence (seq2seq) problem ; taking advantage of the recent advancements in Neural Machine Translation (NMT) (Britz et al., 2017;Kuchaiev et al., 2018) among other applications where seq2seq models made a breakthrough (Yu et al., 2016;Witten et al., 2016;Abadi et al., 2016).


We will analyze this fur-   To further demonstrate the effectiveness of our SC-LSTM model, we compared different variants with state-of-the-art approaches, that we classify into three broad categories: • Single sequence labeling where models are trained without the supervision of other tasks.

we compare our results to the LSTM-CRF model of Lample et al.


Despite those positive outcomes, several issues with multi-task learning for sequence labeling remain open.

we only considered 3 tasks here, and therefore plan to test our approach on more tasks, perhaps understanding better why some tasks are less useful to others.


Also, there are several ways we could have used our SC-LSTM cell into our models which we would like to investigate further.

in this work, we only used the specific-task hidden states in the last layer of the model, which can obviously be revisited.


Our basic type encoder uses trainable vectors as embeddings for each type and combines these with summing.


Each L ... is a sum of binary cross-entropy losses over all types in that category.

the typing problem is viewed as independent classification for each type.


It relies on PoS tags and dependency relations to identify all types of SVA errors.

our rule-based system operates as follows: (i) it identifies the candidate verbs based on PoS tags; 1 (ii) for a given verb, it uses the dependency relations to find its subject; 2 (iii) the PoS tag of the verb and its subject are used to check whether they agree in number and person.


We generate artificial subjectverb agreement errors from large amounts of data.

we use the British National Corpus (BNC, BNC-Consortium et al., 2007), a collection of British English sentences that includes samples from different media such as newspapers, journals, letters or essays.


In order to measure the real performance of a language model (LM) on the detection of SVA errors, we choose to use the BERT system (Devlin et al., 2018) to assign probabilities to different versions of the test sentences.

we use the pre-trained uncased BERT-Base model.


We analyze the effect of adding artificial errors to the training data.

we focus on the robustness of our models by looking at how sensitive they are to grammatical errors in the surrounding context; and by looking at how good the models are at predicting agreement relative to the distance between the subject and verb.


Using more data for training -i.e., our artificial training data which does not include context errors -generally boosts performance on data with and without grammatical errors in the context.

training with additional artificially generated errors seems, overall, to be making our model more robust.


To test our hypothesis, we train dependency parsers on an English corpus and evaluate their transfer performance on 30 other languages.

we compare encoders and decoders based on Recurrent Neural Networks (RNNs) and modified self-attentive architectures.


We look at finer-grained dependency types than the 37 universal dependency labels 2 in UD v2 by augmenting the dependency labels with the universal part-of-speech (POS) tags of the head and modifier 3 nodes.

we use triples "(ModifierPOS, HeadPOS, DependencyLabel)" as the augmented dependency types.


One of the most challenging parts of unsupervised cross-lingual parsing is modeling cross-lingually shareable and language-unspecific information.


In order to improve the quality and disentanglement of the learned representations, we incorporate simple additional losses in our training, which are designed to force the latent representations to capture different information.

our semantic multi-task losses make use of aligned paraphrase data, whereas our syntactic multi-task loss makes use of word-order information.


As shown in Figure 1, VGVAE assumes a sentence is generated by conditioning on two independent variables: semantic variable y and syntactic variable z.

our model gives rise to the following joint likelihood where x t is the tth word of x, T is the sentence length, and p(x t |y, z) is given by a softmax over a vocabulary of size V .


The generative model p θ (x|y, z) is a feedforward neural network g θ with the output being a bag of words.

the expected output log-probability (the first term in Eq.


The key assumption underlying the PRL is that for a paraphrase pair x 1 , x 2 , the semantic information is equivalent between the two sentences and only the syntactic information varies.

to impose such constraints, PRL is defined as we swap the semantic variables, keep the syntactic variables, and attempt to reconstruct the sentences (shown in Figure 3).


When applying DPL, we select negative samples based on maximizing cosine similarity to sentences from a subset of the data.

we accumulate k mini-batches during training, yielding a "mega-batch" S .


Our baselines are a simple word averaging (WORDAVG) model and bidirectional LSTM averaging (BLSTMAVG) model, both of which have been shown to be very competitive for modeling semantic similarity when trained on paraphrases .

wORDAVG takes the average over the word embeddings in the input sequence to obtain the sentence representation.


To investigate what the syntactic variable has captured, we propose several syntactic similarity tasks.

we consider using the syntactic latent variable in calculating nearest neighbors for a 1-nearest-neighbor syntactic parser or part-ofspeech tagger.


Such methods rely on one strong underlying assumption: it is necessary to train the document embedding to optimize the prediction of the target words in the document.

the objective requires the model to learn to predict the target words in surface text.


• We further propose to evaluate unsupervised document embedding models in weaklysupervised classification.

lots of unlabeled documents with only few labels attached to some of them, which is a realistic scenario that unsupervised embedding could be particularly useful.


In this paper, we are concerned with how to transfer classifiers trained on one domain to another domain.

we focus on the transfer of models across languages to alleviate the need for multilingual training data.


In this work, we propose an adversarial category alignment network (ACAN), which attempts to enhance category consistency between the source domain and the target domain.

we increase the discrepancy of two polarity classifiers to provide diverse views, locating ambiguous features near the decision boundaries.


Thus, we can utilize two classifiers to provide diverse views to detect points near the decision boundaries and train the generator to create more discriminative features into high-density region.

we first maximize the discrepancy of the outputs of two classifiers to locate the inconsistent polarity prediction points.


We adopt the Kullback−Leibler (KL) to estimate the distribution divergence: In this way, the latent network representations of two domains are encouraged to be similar.

the marginal distribution is forced to be aligned.


We define a discrepancy between probabilistic outputs of the two classifiers p 1 (y|x) and p 2 (y|x).

the formula is: where defines the average absolute difference for K classes, which is: (4) we first fix the generator G and train the classifiers F 1 ,F 2 to detect points near the decision boundary by maximizing their discrepancy.


Inspired by (Luo et al., 2017), we consider the connections between data points to regularize G(•) in the feature space.

the regularizer is formulated as follows: here l G is to approximate the semantic similarity of two feature embeddings.


In our implementation, the feature encoder G consists of three parts including a 300-dimensional word embedding layer using GloVe (Pennington et al., 2014), a one-layer CNN with ReLU activation function adopted in (Yu and Jiang, 2016;He et al., 2018) and a max-over-time pooling through which final sentence representation is obtained.

the convolution filter and the window size of this one-layer CNN are 300 and 3 separately.


PBLM (Ziser and Reichart, 2018): This is a representation learning model that exploits the structure of the input text.


CRF considers the correlations between tags in neighborhoods and score the whole sequence of tags.

we use a linear-chain CRF and score the tag sequence as conditional probability: where Y is the set of all possible tag sequences and s(r, y)) = n i (A y i−1 ,y i + P i,y i ) is the score function.


We collect data from Reddit, which is a discussion forum platform with a large number of subreddits on diverse topics and interests.

we crawl all the posts from 2013-Jan to 2018-Mar in the TIFU subreddit, where every post should strictly follow the posting rules, otherwise they are removed.


We use the same embedding matrix W emb for both source text and output sentences.

we represent a source text where x i is a one-hot vector for i-th word in the source text.


word-level, sentencelevel, paragraph-level and document-level).

we generate S sets of memory tensors, each of which associates each cell with different number of neighboring word embeddings based on the level of abstraction.


To validate the contribution of each component, we exclude one of key components from our model as follows: (i) -NoDilated with conventional convolutions instead, (ii) -NoMulti with no multi-level memory (iii) -NoNGTU with existing gated linear units (Gehring et al., 2017).

-NoDilated quantifies the improvement by the dilated convolution, -NoMulti assesses the effect of multi-level memory, and -NoNGTU validates the normalized gated tanh unit.


In addition to the word and sentence similarities, we investigate methods to model the content similarity between the candidate summary and the reference directly at the document level.

we use the following five approaches to construct vector representations of learner summaries: TF-IDF is a common method to construct document representations in information retrieval.


The example shows that detailed descriptions like the names of the ships "JING GANGSHA" and "HENG SHUI" are dropped by all models.

the Seq2Seq-gen model has the problem of generating words not present in the original text (e.g., Two Chinese war ships , " JING GANGSHA " and " HENG SHUI " arrived at the port of Trincomalee on 13 th January 2014 on a good will visit .


The example illustrates similar compression properties as seen in the in-domain settings.

the two models learned to drop appositions, optional modifiers, detailed clauses, etc.


Little is known on how this learning approach compares to RL not only regarding task success, but also in terms of the quality of the linguistic output, a gap we seek to fill in this paper.

our contributions are: 2 • The introduction of a single visuallygrounded dialogue state encoder jointly trained with the guesser and question generator modules to address a foundational question of how to integrate visual grounding with dialogue system components; this yields up to 9% improvement on task success.


As shown in Table 2, the linguistic output of SL & CL is closer to the language used by humans: Our agent is able to produce a much richer and less repetitive output than both BL and RL.

it learns to use a more diverse vocabulary, generates more unique questions, and repeats questions within the same dialogue at a much lower rate than the baseline and RL: 93.5% of the games played by BL contain at least one verbatim question repetition, for RL this happens in 96.47% of the cases, whereas for SL and CL this is for only 55.8% and 52.19% of the games, respectively.


We take m 1 = l(anguange) and m 2 = v(isual).

a-Bot encodes language inputs into vectors h l , and visual inputs into h v , respectively.


We adopt the idea of early-fusion (Srivastava and Salakhutdinov, 2012), which means the bimodal and trimodal interactions are captured by the concatenated features from different modalities.

we define our factorized model as: where ⊕ denotes vector concatenation for bimodal and trimodal features.


Section 6 (Related Work) further contrasts our task with summarization.

formally we define our task as follows: given an existing curated text s and a document d describing novel information relevant to that text, the system must produce a revised text s that incorporates the most salient information from d. We restrict our focus to the cases where the revised text s can be obtained by appending the new information from d to the original curated text s. 2 we assume we can transform the old curated text s into the new text s by appending one additional update sentence x to s. This paper operates in a conventional supervised learning setting.


The proposed content transfer task is clearly related to a long series of papers in summarization, including recent work with neural techniques (Rush et al., 2015;Nallapati et al., 2016).

one recent paper casts the the task of generating an entire Wikipedia article as a multidocument summarization problem (Liu et al., 2018).


Following the above definition, the sequence of the highlight embedding is of the same length as d. We replace d with 3.4 Self-Assessment (SA) While in previous work (Radford et al., 2018), the original GPT is directly fine-tuned on an MRC end task, we instead develop a fine-tuning approach inspired by the self-assessment reading strategy.

we propose a simple method to generate questions and their associated multiple spanbased answer options, which cover the content of multiple sentences from a reference document.


For human, writing down an equation that solves a math word problem requires the ability of reading comprehension, reasoning, and sometimes real world understanding.

to solve a math word problem, we first need to know the goal of the given problem, then understand the semantic meaning of each numerical number in the problem, perform reasoning based on the comprehension in the previous step, and finally decide what to write in the equation.


tures r sa and r opd are extracted to select a stack action (section 3.3.2) and an operand to push (section 3.3.3).

the features are the gated concatenation of following vectors: • h D t is the output of an LSTM, which encodes the history of applied actions: where res t−1 is the result from the previous stack action similar to the seq2seq model (Sutskever et al., 2014).


This is done using a coverage vector, v S i for each training instance that keeps track of the production rules triggered by x i , and gets updated whenever one of those desired productions is produced by the decoder.

v S i is a vector of 1s and 0s, with 1s indicating the triggered productions that are yet to be produced by the decoder.


Limited efforts have been devoted to leveraging the advantages of both types of models, such as utilizing word information and conducting exact inference, which are complementary characteristics.

the candidate word information for a character is beneficial to disambiguate word boundaries because a character in the sentence has multiple candidate words that contain the character.


Various neural network architectures have been explored for Chinese word segmentation to reduce the burden of manual feature engineering.

character-based neural models have been developed to model the task as a sequence labeling problem, starting with earlier work by (Zheng et al., 2013) and (Mansur et al., 2013), which applied feed-forward neural networks.


The BPE algorithm constructs a subword list from raw data and lattice LSTM introduces subwords into character LSTM representation.

our baseline is a BiLSTM-CRF segmentor (Chen et al., 2015b) and we replace LSTM with lattice LSTM using subwords to encode character composition information.


In this paper, we propose a semi-supervised word-based approach to improving cross-domain CWS given a baseline segmenter.

our model only deploys word embeddings trained on raw text in the target domain, discarding complex handcrafted features and domain-specific dictionaries.


Given a character c t+1 and a hypothesis h t , h t+1 can be generated in two ways, by either appending c t+1 to BU F t (t = n), or first moving the sequence in BU F t into SEG t as a new word, and then appending c t+1 to BU F t (t = n).

in the former case: in the latter case: when generating the hypothesis for c n+1 (the end of sentence marker), the sequence in BU F n has to be moved into SEG n , and c n+1 also needs to be moved into SEG n as a new word.


Results in Table 3 show that WEB-CWS outperforms these two state-of-the-art models with a large margin in terms of F-measure.

on the DM dataset, Liu and Zhang's model only achieves a relatively low F-score improvement rate (1.8%), which is likely to be caused by the large difference between the source and target domains.


To compare our model to state-of-the-art character-based POS taggers, we evaluate its accuracy on the English part of the Universal Dependencies (UD) v1.2 dataset (Nivre et al., 2015  Zeljko Agic, 2017), using the splits, training and evaluation rules from the CoNNL 2017 shared task (Zeman et al., 2017).

we calculate joint tokenization and UPOS (universal POS) F 1 scores.


We also show that removing syntactic information hurts word f-score.

performing syntactic parsing and disfluency detection as a multi-task training objective yields higher disfluency detection accuracy than performing disfluency detection in isolation.


We first study a prediction task that estimates whether the user engaged in a socialbot conversation would accept a suggested topic.

we use a corpus of human-socialbot conversations collected during the 2017 Alexa Prize competition (Ram et al., 2017) from the Sounding Board system (Fang et al., 2018;Fang, 2019).


The model achieves promising results on two representative tasks in dialogs: user topic decision prediction in human-socialbot conversations and dialog act classification in human-human conversations.

we demonstrate that the model can benefit from unlabelled data in the dialog act classification task, where we achieve the state-of-theart results.


employ the visualization tool (Lin et al., 2017) to highlight important words based on the attention output.

we run our method on IMDb dataset, wherein each instance is a movie story associated with relevant genres as labels.


However, distant supervision may suffer a wrong label problem.

the instances in one bag may not actually have the relation.


Text Annotation Generation Finally, for all the APT documents, we annotate the text with malware attribute labels.

for any word in the APT documents, if it is a keyword associated with K different labels, we annotate the word with K attribute labels.


In addition, to facilitate attribute classification, we want to capture the semantic relations between words and their labels.

we want the words and their attribute labels to be close to each other in the embedding space and the embeddings of different labels to be far away from each other.


To further strengthen their relations, the labels of the target word are also used to predict the target word.

given a sequence of T words (W 1 ,..,W t ,...,W T ) and their annotations ((A 1,1 ,...,A 1,M 1 ),...,(A T,1 ,...,A T,M T )), the objec- tive of the WAE model is to maximize the average log probability shown in Equation 1, where C is the size of the context window, W t is the target word, W t+j is a context word, M t is number of annotations W t has and A t,k is the k-th annotation Label-aware Negative Sampling: Negative sampling (Mikolov et al., 2013) was introduced as an approximation method in word2vec to improve the efficiency of model training.


Despite their success, PLLbased models still suffer from Confirmation Bias by taking its own prediction as optimization objective in the next step.

given an entity mention, if the typing system selected a wrong  Figure 2: The overall framework of CLSC.


Recall that a fact is likely to be true when all the facets exhibit strong support.

we can prune away the fact if one of the facets is low and stop calculating other facets.


In the iteration, the information from different perspectives is shared with each facet via the newly added facts.

after predicting the top-n h facts for each head entity, we select top-n f b (n f b < n h ) most probable facts according to the score of each triple and then feed them into the existing knowledge base for re-training the FFD model.


We tackle this issue with two strategies: 1) we use a novel importance module (explained later) to focus on important answer aspects, and 2) we only consider those context nodes that have overlap with the question.

for each context node (i.e., a sequence of words) of a candidate, we first compute the longest common subsequence between it and the question, we then encode it via a BiLSTM only if we get a non-stopwords substring.


We use a convolutional network (CNN) to encode Q into a d-dimensional vector e. For candidate topic entity C i , we encode three types of KB aspects, namely, the entity name, entity type and surrounding relations where both entity name and type are represented as a sequence of words while surrounding relations are represented as a bag of sequences of words.

we use three CNNs to encode them into three d-dimensional vectors, namely, C n i , C t i and C r 1 i .


In this paper we propose an alternative: we test models on their ability to answer naturally occurring yes/no questions.

questions that were authored by people who were not prompted to write particular kinds of questions, including even being required to write yes/no questions, and who did not know the answer to the question they were asking.


Therefore, we follow the recent trend in NLP of using transfer learning.

we experiment with pre-training models on related tasks that have larger datasets, and then fine-tuning them on our training data.


We also experiment with combining these approaches using a two-step pre-training regime.

we fine-tune the pre-trained BERT L on MultiNLI, and then fine-tune the resulting model again on the BoolQ train set.


To address the challenges, we propose a novel solution to make conventional KV-MemNNs feasible to open domain KB-QA.

we introduce a flexible KV-MemNN solution that can work in both the IR and semantic parsing style with large-scale memory.


Traditional KV-MemNNs first preselect a list of candidate KB facts to compose the key-value memory.

one can first detect entity mentions in the question, and include all KB facts that contains with one of those entities as subject into the memory.