As for the second challenge, a robust model could extract precise relation features even from low-quality sentences containing noisy words.

previous neural methods are always lacking in robustness because parameters are initialized randomly and hard to tune with noisy training data, resulting in the poor performance of extractors.


Dependency trees help relation extraction models capture long-range relations between words.

existing dependency-based models either neglect crucial information (e.g., negation) by pruning the dependency trees too aggressively, or are computationally inefficient because it is difficult to parallelize over different tree structures.


Traditional feature-based models are able to represent dependency information by featurizing dependency trees as overlapping paths along the trees (Kambhatla, 2004).

these models face the challenge of sparse feature spaces and are brittle to lexical variations.


Another popular approach, inspired by Bunescu and Mooney (2005), is to reduce the parse tree to the shortest dependency path between the entities (Xu et al., 2015a,b).

these models suffer from several drawbacks.


It is therefore desirable to combine our GCN models with tree pruning strategies to further improve performance.

pruning too aggressively (e.g., keeping only the dependency path) could lead to loss of crucial information and conversely hurt robustness.


Traditionally, evaluation on SemEval is conducted without entity mentions masked.

as we will discuss in Section 6.4, this method encourages models to overfit to these mentions and fails to test their actual ability to generalize.


While the unmasked model achieves a 83.6 F 1 on the original SemEval dev set, F 1 drops drastically to 62.4 if we replace dev set entity mentions with a special <UNK> token to simulate the presence of unseen entities.

the masked model is unaffected by unseen entity mentions and achieves a stable dev F 1 of 74.7.


Attention mechanisms are often used in deep neural networks for distantly supervised relation extraction (DS-RE) to distinguish valid from noisy instances.

traditional 1-D vector attention models are insufficient for the learning of different contexts in the selection of valid instances to predict the relationship for an entity pair.


The latter introduces a multilingual framework which employs a monolingual attention mechanism to utilize the information within monolingual texts, and further uses a cross-lingual attention mechanism to consider the information consistency and complementarity among cross-lingual texts.

extra resources are difficult to obtain in many practical scenarios.


set r L2 to 9, but there are only one and two instances for selection in tasks One and Two, so the 2-D matrix cannot demonstrate its full potential.

in All, many entity pairs contain multiple or more than 9 instances, so it can learn a better 2-D matrix to focus on different instances.


In this example, the comma implies a semantic relationship location/location/contains for the entity pair (Maryland, Baltimore).

biGRU+2ATT allocates quite a small probability to it; and (2) we can see that our model focuses on different words via different attention vectors (9 in total).


The bi-directional DAG LSTM model showed superior performance over several strong baselines, such as tree-structured LSTM (Miwa and Bansal, 2016), on a biomedical-domain benchmark.

the bidirectional DAG LSTM model suffers from several limitations.


On the other hand, EM-BED assigns each edge label to an embedding vector, but complicates the gated operations by changing the U s to be 3D tensors.

1 we take edge labels as part of the input to the gated network.


In particular, our model adopts the same methods for calculating input representation (as in Section 3.1) and performing classification as the baseline model.

different from the baseline bidirectional DAG LSTM model, we leverage a graph-structured LSTM to directly model the input graph, without splitting it into two DAGs.


By revisiting Table 2, we can see that the average number of tokens for the ternary-relation data is Train Decode Bidir DAG LSTM 281s 27.3s GS GLSTM 36.7s 2.7s 74, which means that the baseline model has to execute 74 recurrent transition steps for calculating a hidden state for each input word.

our model only performs 5 state transitions, and calculations between each pair of nodes for one transition are parallelizable.


The first case generally mentions that Gefitinib does not have an effect on T790M mutation on EGFR gene.

note that both "" and "was not" serve as indicators; thus incorporating them into the contextual vectors of these entity men- Single Cross  73.9 75.2 Miwa and Bansal (2016) 75.9 75.9 Peng et al.


Our dataset contains five classes as mentioned in Section 6.1.

previous work only investigates binary relation detection.


Many efforts have been devoted to identifying valid instances from noisy data.

most existing methods handle each relation in isolation, regardless of rich semantic correlations located in relation hierarchies.


These methods focus on adding an extra model to reduce the noisy label.

stacking extra model does not fundamentally solve the problem of inadequate supervision signals of distant supervision, and will introduce expensive training costs.


Distant supervision (Mintz et al., 2009) aimed to obtain large-scale training data automatically, which becomes the most versatile supervision method.

it suffers from the noisy label problem.


Such relations require the model to predict both relation types and the entity order correctly.

for undirected relations, such as Synonym-of in ScienceIE and Associated in Phenebank, both directions can be accepted.


Schutte et al., 2011;Meena et al., 2012;Zarrieß et al., 2016;Shore and Skantze, 2017).

discerning REs from non-referring language in dialog is not trivial.


Given a literary model of reference, they should have supplied exactly enough information to identify the referent and no more (the big pinkish asteroid to the left), adhering to Grice's (1975) maxim of quantity.

rL can undergo not only expansion but also replacement: For color, speaker B proposes both purple and pinkish, but only pinkish is then accepted by A.


However, RL can undergo not only expansion but also replacement: For color, speaker B proposes both purple and pinkish, but only pinkish is then accepted by A.

to a literary model of reference, a collaborative model represents reference resolution as a process of iteratively presenting RL to the other participant(s) in a dialog, which is then either accepted as being sufficient to identify a referent or rejected as insufficient (Clark and Wilkes-Gibbs, 1986, 9).


(2011) algorithmically extracted RL as utterance(s) preceding a discrete event in a shared environment within a certain timeframe.

one drawback to this method is that "the references must be contained in instructions that cause events involving the referents" and "it must be possible to automatically detect these events" (Schutte et al., 2011, 189).


Other approaches use language structure to infer RL, namely in parsing said language using a combination of statistical or rule-based methods.

both entail that a solution be specialized for language specific to a given domain, such as for route-following instructions (Meena et al., 2012) or for a specific instructor-manipulator pair task (Shore and Skantze, 2017): Meena et al.


Deepak (2016) presents MiXKmeans, a variation of k-means algorithm, suited for clustering threads present on forums and Community Question Answering websites.

most techniques use unsupervised clustering to group similar questions/queries, without modeling intents.


Finally, our work is related to a large body of research on dialog acts (Stolcke et al., 2000;Kim et al., 2010;Chen et al., 2018): our low-level intent labels (Table 1) can be seen as very finegrained dialog acts (Core and Allen, 1997;Bunt et al., 2010;Oraby et al., 2017).

our paper's objective is different as our goal is not to rigidly define intents and then exploit them to derive a semantic interpretation.


In fact, w learns to score the edges since the structural feature vector decomposes over the edges.

imposing a structure onto the output is supposed to produce a better w, which we test in our experiments described in Section 5.


Here, we apply the same LSSVM, LSP, and the SVM classifier models trained in the experiments of the previous paragraph.

we recompute all the four unsupervised clustering baselines supplying them with the new k -the number of gold intent-based annotated clusters.


A typical conversation involves recalling important points from this background knowledge and producing them appropriately in the context of the conversation.

most existing large scale datasets (Lowe et al., 2015b;Ritter et al., 2010;Serban et al., 2016) simply contain a sequence of utterances and responses without any explicit background knowledge associated with them.


There has been considerable work on incorporating background knowledge in the context of goal-oriented dialog datasets even before the advent of large-scale datasets for deep learning (Raux et al., 2005;Seneff et al., 1991) as well as in recent times (Rojas-Barahona et al., 2017;Williams et al., 2016; where datasets include small sized knowledge graphs as background knowledge.

the conversations in these datasets are very templated and nowhere close to open conversations in specific domains such as the ones contained in our dataset.


Additionally, existing large-scale datasets are noisy as they are extracted from online forums which are inherently noisy.

since we use crowdsourcing, the extent of noise is reduced since there are humans in the loop who were explicitly instructed to use only clean sentences from the external knowledge sources.


While playing the role of Speaker 1, the worker was not restricted to copy/modify sentences from the background resources but was given the freedom to create (write) original sentences.

when playing the role of Speaker 2, the worker was strictly instructed to copy/modify sentences from the shown resources such that they were relevant in the current context of the conversation.


We would expect a poor performance for BiDAF (ms) as the resource has more noise because of the sentences from irrelevant resources.

we speculate the model learns to regard irrelevant sentences as noise and learns to focus on sentences corresponding to the correct resource resulting in improved performance (however, this is only a hypothesis and it needs to be verified).


Thanks to the automatic generation process, these datasets can be very large in size, leading to significant research progresses.

compared to how humans would create cloze questions and evaluate reading comprehension ability, the automatic generation process bears some inevitable issues.


Large-scale automatically-generated cloze tests (Hermann et al., 2015;Hill et al., 2016;Onishi et al., 2016) lead to significant research advancements.

generated questions do not consider language phenomenon to be tested and are relatively easy to solve.


Grammar questions are easily differentiated from other two categories.

the teachers themselves cannot specify a clear distinction between reasoning questions and vocabulary questions since all questions require comprehending the words within the context and conducting some level of reasoning by recognizing incomplete information or conceptual overlap.


The same conclusion can also be drawn from the success of a concurrent work ELMo which uses LM representations as word vectors and achieves state-ofthe-art results on six language tasks (Peters et al., 2018).

if we increase the context length to three sentences, the accuracy of 1B-LM only has a marginal improvement.


Additionally, the 1B-LM is trained on the sentence level, which might also result in the inability to track paragraph level information.

to investigate the differences between training on sentence level and on paragraph level, a prohibitive amount of computational resource is required to train a large model on the 1 Billion Word Corpus.


Automatic question answering (QA) has made big strides with several open-domain and machine comprehension systems built using large-scale annotated datasets (Voorhees et al., 1999;Ferrucci et al., 2010;Rajpurkar et al., 2016;Joshi et al., 2017).

in the clinical domain this problem remains relatively unexplored.


Recent advances in crowd-sourcing and search engines have resulted in an explosion of large-scale (100K) MC datasets for factoid QA, having ample redundant evidence in text (Rajpurkar et al., 2016;Trischler et al., 2016;Joshi et al., 2017;Dhingra et al., 2017).

complex domainspecific MC datasets such as MCTest (Richardson et al., 2013), biological process modeling (Berant et al., 2014), BioASQ (Tsatsaronis et al., 2015), InsuranceQA (Feng et al., 2015), etc have been limited in scale (500-10K) because of the complexity of the task or the need for expert annotations that cannot be crowd-sourced or gathered from the web.


The performance of the proposed models is summarized in Table 5. emrQL results are not directly comparable with GeoQuery and ATIS because of the differences in the lexicon and tools available for the domains.

it helps us establish that QL learning in emrQA is non-trivial and supports significant future work.


When the "oracle" science fact f used by the question author is provided to the knowledge-enhanced reader, it improves over the knowledge-less models by about 5%.

there is still a large gap, showing that the core fact is insufficient to answer the question.


In the second-order false-belief task, Sally observes the new location of the milk; thus, she has a true belief about the milk's location.

anne's belief about Sally's mental state does not match reality because anne does not know that Sally has observed the change in the environment.


Note that the median accuracy (the dark blue line in box) is close to 1 for these questions.

the model often fails (median accuracy around 0.5) given the first-order question ("where will Sally look for the milk") and the false-belief and second-order false-belief tasks.


The MemN2N model succeeds at both bAbi task 1 and the reality question given the true-belief task.

the correct answer to the memory question ("where was the milk at the beginning?")


Second, the syntactic parsers are unreliable on account of the risk of erroneous syntactic input, which may lead to error propagation and an unsatisfactory SRL performance.

syntactic information is considered closely related to semantic relation and plays an essential role in SRL task (Punyakanok et al., 2008).


Following Marcheggiani and Titov (2017), we focus on argument labeling and formulate SRL as sequence labeling problem.

we differ by (1) leveraging enhanced word representation, (2) applying recent advances in recurrent neural networks (RNNs), such as highway connections (Srivastava et al., 2015), (3) using deep encoder with residual connections (He et al., 2016), (4) further extending Syntax Aware Long Short-Term Memory (SA-LSTM) (Qian et al., 2017) for SRL, and 5introducing the Tree-Structured Long Short-Term Memory (Tree-LSTM) (Tai et al., 2015) to model syntactic information for SRL.


More specifically, Tree-LSTM only considers information from arbitrary child units so that each node lacks of the information from parent.

our Syn-GCN and SA-LSTM combine bidirectional information, both head-to-dependent and dependent-to-head.


The DFS approach is an applicable linearization of trees since a recursive traversal, which starts at the root and explores all outgoing edges, is guaranteed to visit all of the graph's nodes.

dFS linearization is not directly applicable to SdP, as its graphs often consist of several non-connected components.


In addition, an ablation study shows that multi-tasking the PRIMARY tasks is beneficial over a single task setting, which in turn is outperformed by the inclusion of the AUXILIARY tasks.

simulating disjoint annotations -with sDP's complete overlap of annotated sentences, multi-task learning often deals with disjoint training data.


We can see the question is asking "What state is San Antonio located in?".

the natural language word order in Indone-  sian is different from English, where the phrase "berada di" that corresponds to m 2 (i.e., loc) appears between "San Antonio" (which corresponds to m 5 -san antonio ) and "what" (which corresponds to m 1 -answer).


Prior work, such as ✏-greedy exploration (Guu et al., 2017), has reduced the severity of this problem by introducing random noise in the search procedure to avoid saturating the search on high-scoring spurious programs.

random noise need not bias the search towards the correct program(s).


We therefore rewrite Equation 1as follows: As such we now define our model as: During training, the target length is set to T y = T y .

at test time, the target length generally varies according to the domain, genre, and language at hand.


In Section 3.1.1, we used the Euclidean distance of vector pairs to define M (G) and Sinkhorn distance d sh (G).

in our preliminary experiment, we found that Euclidean distance of unnormalized vectors gave poor performance.


When using our alignment process, our method is competitive with the TRANSLATE TRAIN baseline, suggesting that it might be possible to encode similarity between languages directly in the embedding spaces generated by the encoders.

these methods are still below the other machine translation baseline TRANSLATE TEST, which significantly outperforms the multilingual sentence encoder approach by up to 6% (Swahili).


A Burst Information Network (BINet) is a graphbased text stream representation and has proven effective for multiple text stream mining tasks (Ge et al., 2016a,b,c).

to many information networks (e.g., (Ji, 2009;Li et al., 2014)), BI-Nets are specially for text streams.


One reason is that the topic overlap of coordinated cross-lingual text streams is not so significant as the Wikipedia data used for their experiments, and the other reason is that their approaches focus on common fundamental words like "城市(city)" while our targets are OOVs like "东协(ASEAN)" which do not frequently appear in a corpus.

our approach is more practical: it not only works well in easily available and endless coordinated text streams without high content overlap requirement, but also can accurately mine translations of many OOVs which do not appear frequently and really need mining their translations.


In contrast to the word-level alignment methods, we attempt to mine burst-level alignment to largely narrow down candidates, and introduce powerful clues for improving accuracy and discovering various language knowledge.

to previous cross-lingual projection work like data transfer (Pado and Lapata, 2009) and model transfer (McDonald et al., 2011), we do not require any parallel data.


Therefore, the semantic meanings of words can reflect in the contexts according to these distributed representation models.

homographic puns always have multiple meanings.


In recent years, homographic puns have increasingly become a respectable research topic, which widely appears in rhetoric and literary criticism.

there were little related works in the fields of computational linguistics and natural language processing by (Miller, Tristan and Turković, Mladen, 2016).


The best perform of SemEval2017 task7 is Fermi.

fermi only evaluates on 675 of 2250 homographic contexts (Miller, Tristan and Hempelmann, Christian and Gurevych, Iryna, 2017) in SemEval2017 task7.


Chinese spelling check (CSC) is a challenging yet meaningful task, which not only serves as a preprocessing in many natural language processing (NLP) applications, but also facilitates reading and understanding of running texts in peoples' daily lives.

to utilize datadriven approaches for CSC, there is one major limitation that annotated corpora are not enough in applying algorithms and building models.


To build an annotated corpus of P-style errors, we follow the similar inspiration with those for V-style errors and OCR tools, and adopted a pipeline as shown in Figure 3.

given the availability of various speech recognition datasets, we employ a simpler approach.


considering the preceding word 政企 (translation: politics and industry) and the subsequent words 是一 种痼疾 (translation: a kind of chronic disease), we can see that 部分 does not fit the current context and should be corrected as 不分.


An possible reason is that with more instances containing different spelling error including in the training dataset, the number of unseen spelling error in the testing dataset is reduced, thus facilitating the model to detect more spelling errors.

the improvement of the precision is not so obvious as that of the recall.


Grammatical error correction (GEC) systems deployed in language learning environments are expected to accurately correct errors in learners' writing.

in practice, they often produce spurious corrections and fail to correct many errors, thereby misleading learners.


This level of performance is impressive since GEC is a difficult task given the diversity and complexity of language errors.

in real-world use cases such as language learning, erroneous feedback from automatic GEC systems can potentially mislead language learners.


They use a dataset of learner sentences manually annotated with subjective scores of grammaticality.

their method was to assess learner writing and not for system evaluation.


number of edits number of reference tokens In MT, the reference translations for HTER are targeted, i.e., they are created by post-editing system translated sentences.

in GEC, highquality datasets annotated by experts with minimal edits are available (Dahlmeier et al., 2013;Yannakoudakis et al., 2011) and GEC systems are typically trained to make minimal changes to input sentences.


Because most POS tagging methods are based on supervised models, they usually require a large amount of labeled data for training.

the existing labeled datasets for Twitter are much smaller than those for newswire text.


Hence, to help POS tagging for Twitter, most domain adaptation methods try to leverage newswire datasets by learning the shared features between the two domains.

from a linguistic perspective, Twitter users not only tend to mimic the formal expressions of traditional media, like news, but they also appear to be developing linguistically informal styles.


It is a dataset of POS-tagged tweets consisting almost entirely of tweets sampled from one particular day (October 27, 2010).

the test set was introduced in (Owoputi et al., 2013), and contains 574 tweets (DAILY547).


We can see that our method could significantly outperform the TweetNLP Tagger.

our method was worse than the ARK tagger.


's limited window-based representation (97.3%).

existing local models "regularly make egregious errors" (Manning, 2011), notably on imperative detection 1 .


The best performing tagger (Bohnet et al., 2018) was 3.9% above the next best model.


The improvements from predicted case results are interesting, since in nonneural parsers, predicted case usually harms accuracy (Tsarfaty et al., 2010).

we note that our taggers use gold POS, which might help.


For most other dependencies (and all dependencies in German), Lemma is the most important feature, suggesting a strong reliance on lexical semantics of nouns and verbs.

we also notice that the model sometimes attends to features like Aspect, Polarity, and Verb-Form-since these features are present only on verbs, we suspect that the model may simply use them as convenient signals that a word is verb, and thus a likely head for a given noun.


Research in affective computing has mainly focused on understanding affect (emotions and sentiment) in monologues.

with increasing interactions of humans with machines, researchers now aim at building agents that can seamlessly analyze affective content in conversations.


The gold annotations are available for every 0.2 seconds in each video (Nicolle et al., 2012).

to align with our problem statement, we approximate the utterance-level annotation as the mean of the continuous values within the spoken utterance.


This suggests that multi-hop is more crucial than the latter.

best performance is achieved by variant 6 which contains all the proposed modules in its pipeline.


3a), where all regions without a positive label can be used as negative examples.

query-Adaptive R-CNN is trained using the open-vocabulary phrases annotated to the regions (Fig.


Using hard negative examples has proven to be effective in the object detection to train a discriminative detector (Felzenszwalb et al., 2010;Shrivastava et al., 2016).

adding negative examples is usually not easy in the openvocabulary setting, because it is not guaranteed that a region without a positive label is negative.


1) The first approach uses the WordNet hierarchy: if two categories have parentchild relationships in WordNet (Miller, 1995), they are not mutually exclusive.

the converse is not necessarily true; e.g., man and skier are not mutually exclusive but do not have the parent-child relationship in the WordNet hierarchy.


Using a softmax layer augmented with hidden state vectors, they predict the verb and the nominal fillers of its roles.

to above works on ImSitu, we do not link the roles of a verb to their lexical fillers.


Using the triplet loss with memory modules led to greater performance when compared to the attn 3 model, but the performance sits around the use of either triplet only or memory only.

when we increase the N diag to 16 or 32, we find a jump in performance.


As in this earlier work, we are also interested in navigation with a topological map of the environment.


We also observe that inherent ambiguities in instruction following make exact goal identification difficult, as demonstrated by imperfect human performance.

the gap to human-level performance still remains large across both tasks.


On CHAI, CHAP-LOT18 and MISRA17 both fail to learn, while our approach shows an improvement on stop distance (SD).

all models perform poorly on CHAI, especially on manipulation (MA).


Furthermore, arbitrary IRF kernels are not guaranteed to afford analytical estimator functions or unique real-valued solutions.

recent advances in machine learning have led to libraries like Tensorflow (Abadi et al., 2015)which uses auto-differentiation to support optimization of arbitrary computation graphs -and Edward (Tran et al., 2016) -which enables black box variational inference (BBVI) on Tensorflow graphs.


They argue that this constitutes the first strong evidence of memory effects in broadcoverage sentence processing.

it turns out that when one baseline predictor -probabilistic context free grammar (PCFG) surprisal -is spilled over one position, the reported effects disappear: p = 0.816 for constituent wrap-up and p = 0.370 for dependency locality.


the use of the canonical HRF to convolve predictors in fMRI models prior to linear regression.

since DTSR is domain-general, it can be a valuable component in any analysis toolchain for time series.


(2016) has formalized the grammar of these queries and proposed semisupervised algorithms for the adaptation of parsers originally designed to parse according to the standard dependency grammar, so that they can account for the unique forest grammar of queries.

their algorithms rely on resources typically not available outside of big web corporates.


We observe a universal improvement across all POS tags for each of the three variations of the system compared to the baseline.

it is notable that the biggest gains in HDLAS are for open word classes: NOUNs, VERBs and ADJs.


These kinds of depth bounds in sentence processing have been used to explain the relative difficulty of center-embedded sentences compared to more right-branching paraphrases like It was awful for the plant's parts to fail.

depth-bounded grammar induction has never been compared against unbounded induction in the same system, in part because most previous depth-bounding models are built around sequence models, the complexity of which grows exponentially with the maximum allowed depth.


tion, we generate a series of tables of size O(|w| × |w|) with elements of size O(|Q P | × |Q P |).

since we use the sliding window technique, we only save O(1) of these tables, leading to an overall space complexity of O((|w||Q P |) 2 ).


They aim at maximizing the probability of generating a response given an input query, and generally use the maximum likelihood estimation (MLE) as their objective function.

various problems occur when Seq2Seq models are used for dialogue generation tasks.


It estimates the beliefs of possible user's goals at every dialogue turn.

for most current approaches, it's difficult to scale to large dialogue domains.


the slots and values are defined in advance, and can't change dynamically.

this is not flexible in practice (Xu and Hu, 2018).


For example, in Set an alarm at 8 am for Monday and Wednesday, 8 am needs to be associated with both Monday and Wednesday which would require graph-structured repre-sentations.

we found that just 0.3% of our dataset would require a more expressive representation to model adequately.


Many deep learning approaches have been explored to train a classifier with those datasets to develop an automatic abusive language detection system (Badjatiya et al., 2017;Park and Fung, 2017;Pavlopoulos et al., 2017).

these works do not explicitly address any model bias in their models.


This could be attributed to the difference in the source and target tasks (abusive & sexist).

the decrease was marginal (less than 4%), while the drop in bias was significant.


;Zhang et al., 2018) employ adversarial training methods to make the classifiers unbiased toward certain variables.

those works do not deal with natural language where features like gender and race are latent variables inside the language.


Talk page data already underpins research on social phenomena such as conversational behavior (Danescu-Niculescu-Mizil et al., 2012, 2013, disputes (Wang and Cardie, 2014b), antisocial behavior (Wulczyn et al., 2017;Zhang et al., 2018) and collaboration (Kittur et al., 2007;Halfaker et al., 2009).

the scope of such studies has so far been limited by a view of the conversation that is incomplete in two crucial ways: first, it only captures a subset of all discussions; and second, it only accounts for the final form of each conversation, which frequently differs from the interlocutors experience as the conversation develops.


One way to tag multiple datasets is to concatenate all the datasets with all the output labels and train a single BiLSTM-CRF model.

this assumes that each text snippet is completely annotated across the label sets, which is not true.


They generate large quantities of parse trees by parsing unlabeled data with two existing parsers and selecting only the sentences for which the two parsers produced the same trees.

the trees produced this way have noise 1 and tend to be short sentences, since it is easier for different parsers to get consistent results.


2018, we then combine the input vector ( .., L) by a Softmaxnormalized weight (W woc ) and a scalar parameter (γ): The parameters of word ordering model are fixed during the training of parsing model.

the weight and scalar parameters are tuned to better adapt to it.


This might be partly explained by the fact that learning morphological features at the word level is difficult due to data sparsity -indeed the rarest French words in our dataset are observed only 10 times in the training data.

additional experiments showed that our finding is consistent across different word frequency bins: that is, even the embeddings of frequent words do not encode morphological features better than the majority baseline.


This can be explained by the fact that number remains most often unchanged through translation, and is marked in all target languages -albeit to different extents.

tense is determined by the semantics but also by languagespecific usage, while gender has little semantic value and is mostly assigned to nouns arbitrarily.


Initially, we also experimented with only Levenshtein distance as loss, similar to previous work on character-level problems (Leblond et al., 2018;Bahdanau et al., 2017).

models did not learn much, which we attribute to sparse training signal as all action sequences producing the same y would incur the same sequence-level loss, including intuitively very wasteful ones, e.g.


In the image domain, these perturbations are often virtually indistinguishable to human perception, causing humans and state-of-the-art models to disagree.

in the natural language domain, small perturbations are clearly perceptible, and the replacement of a single word can drastically alter the semantics of the document.


In terms of training speed, we do not observe obvious decrease, which in turn demonstrates the advantage of our disagreement regularizations.

the combinations of different disagreement regularizations fail to further improve translation performance (Rows 5-7).


This is also consistent with the finding in (Tu et al., 2016), which indicates that neural networks can model linguistic information in their own way.

to attended positions, it seems that the multi-head attention prefer to encoding the differences among multiple heads in the learned representations.


Typical acquisition functions select examples for which the current predictor is most uncertain.

how precisely to quantify uncertainty, especially for neural networks, remains an open question.


For the input-to-hidden matrix W x the linear combination (W x x t ) is normally distributed.

sampling the same W x for all timesteps and sampling the same noise i for preactivations for all timesteps are not equivalent.


We work with z in the same way as with other weights W : we use a log-uniform prior and approximate the posterior with a fully-factorized normal distribution with trainable mean and variance.

since z is a one-dimensional vector, we can sample it individually for each object in a mini-batch to reduce the variance of the gradients.


framework in (Luong et al., 2015) for the P2C task.

to related work that simply extended the source side with different sized context window to improve of translation quality (Tiedemann and Scherrer, 2017), we add the entire input utterance according to IME user choice at previous time (referred to the context hereafter).


Reducing the size of the embedding m leads to a significant reduction in the number of parameters, proportional to |V |, and the acceleration of softmax computation.

the size of the additional matrix L is only n × m and contributes very little to the overall size of the model.


Moreover, an LSTM cell carries over information both through the hidden state and a memory state; the latter is affected by tying only indirectly (see Hochreiter and Schmidhuber (1997) for details on LSTM architectures).

our experiments show that, in practice, two-layer LSTM LMs are still affected by tying despite these caveats.


The skipgram algorithm shows a small degradation of performance for the tied+L architecture with respect to the non-tied one; note that, as explained in Section 2.3, tying makes the most sense for CBOW.

the fact that standard tying obtains much worse results (similarly to the results of Press and Wolf, 2017) shows that the linear mapping substantially relaxes the tying constraint.


Most of these methods use an additional encoder Wang et al., 2017) to extract contextual information from previous source-side sentences.

this requires additional parameters and it does not ex-ploit the representations already learned by the NMT encoder.


The Zh-En subtitles corpus is a compilation of TV subtitles designed for research on context (Wang et al., 2018).

to the other sets, it has three references to compare.


Neural machine translation (NMT) typically makes use of a recurrent neural network (RNN) -based encoder and decoder, along with an attention mechanism (Bahdanau et al., 2015;Cho et al., 2014;Kalchbrenner and Blunsom, 2013;Sutskever et al., 2014).

it has been shown that RNNs require some supervision to learn syntax (Bentivogli et al., 2016;Linzen et al., 2016;Shi et al., 2016).


(2016b), our networks operate at the subword level using byte pair encoding (BPE) with a shared vocabulary on the source and target sides.


The parse2seq and multi-source systems require parsed source data at inference time.

the parser may fail on an input sentence.


2 Attempts to measure the impact of translation divergences in MT have focused on the introduction of noise in sentence alignments (Goutte et al., 2012), showing that statistical MT is highly robust to noise, and that performance only degrades seriously at very high noise levels.

neural MTs seem to be more sensitive to noise (Chen et al., 2016), as they tend to assign high probabilities to rare events (Hassan et al., 2018).


Accuracies obtained for various combinations of negative examples, where we see that non-divergent words in parallel and unpaired sentences (columns P and U) are easy to spot, as long as the model has seen these types of examples in training.

the accuracy drops dramatically when the model is not trained with unpaired sentences (rows PR, PI and PRI).


This solution, both with ISF and CSLS criteria, is applied with a transformation W learned using the square loss.


Most of the Neural Machine Translation (NMT) models are based on the sequence-tosequence (Seq2Seq) model with an encoderdecoder framework equipped with the attention mechanism.

the conventional attention mechanism treats the decoding at each time step equally with the same matrix, which is problematic since the softness of the attention for different types of words (e.g.


Our work is most similar to the work of Zoph and Knight (2016) and Anastasopoulos and Chiang 2018, which share a decoder and two separate attention models to read from two different sources.

we share information at the level of reconstructed frames.


Saying "I am happy" in English, does not encode any additional knowledge of the speaker that uttered the sentence.

many other languages do have grammatical gender systems and so such knowledge would be encoded.


The increasing amount of work on automatic author classification (or 'author profiling') reaching relatively high accuracies on domain-specific data corroborates these findings (Rangel et al., 2013;Santosh et al., 2013).

determining the gender of an author based solely on text is not a solved issue.


1 Human translators rely on contextual information to infer the gender of the speaker in order to make the correct morphological agreement.

most current MT systems do not; they simply exploit statistical dependencies on the sentence level that have been learned from large amounts of parallel data.


we also encountered cases where the gender-informed system fails to produce the correct agreement, as in (4), where both the BASE and the TAG system produce a male form ('embarassé') instead of the correct female one ('embarassée' or 'gênée').


From the experiments, we see that informing the NMT system by providing tags indicating the gender of the speaker can indeed lead to significant improvements over state-of-the-art baseline systems, especially for those languages expressing grammatical gender agreement.

while analyzing the EN-FR translations, we observed that the improvements are not always consistent and that, apart from morphological agreement, the gender-aware NMT system differs from the baseline in terms of word choices.


This suggests that increasing the splitting factor k in Equation 1 might improve the model performance.

it also reduces the efficiency in terms of GPU memory usage.


We also observe gains of 0.7 and 1.0 BLEU for RNMT+ models, on En→De and Cs→En respectively, as indicated by Tables 3 and 4.

experiments comparing wide models against deeper ones are inconclusive.


In contexts that are not copy-prone, minimal copying occurs.

14 as they are placed in increasingly copyprone contexts, even these words that the system has learned it should translate are being copied.


Lowercase words are the least frequently copied (average copy rate of 40.2%), uppercase words are the most copied (94.4%), and the natural case falls in the middle (81.7%).

changing casing changes the BPE segmentation, and uppercase words tend to be split into more pieces: a mean of 4.4 segments, as compared to means of 3.1 (lowercase) and 2.9 (natural case).


The highlighted fragments of the source sentence and the matched TM source sentence are not actually the same in terms of their surface forms.

they are semantically close and can be translated into the same target translation.


We call this phenomenon the "beam search curse", which is listed as one of the six biggest challenges for NMT (Koehn and Knowles, 2017).

there has not been enough attention on this problem.


Therefore, their explanations are not satisfactory.

previous work adopts several heuristics to address this problem, but with various limitations.


By default, OpenNMT-py (Klein et al., 2017) stops when the topmost beam candidate stops, because there will not be any future candidates with higher model scores.

this is not the case for other rescoring methods; e.g., the score of length normalization (5) could still increase.


A general tendency is that ATT performs better compared to MAX, AVG and CON as the number of views increases (lines 30 and 34) and so the average number of views without information (i.e., missing views) for an entity increases.

to MAX, ATT can combine different views.


In contrast to MAX, ATT can combine different views.

to CON and AVG, ATT can ignore some of them based on low attention weights.


of learning a high-quality mapping from the utterance to its intent directly, so that such mapping can be further capitalized to measure the compatibility of an utterance with emerging intents.

the diverse semantic expressions may impede the learning of such mapping.


In the meanwhile, utterances of the same emerging intent but with nuances in expressions result in their proximity in the t-SNE space.

we do observe less satisfied cases where the model mistake an emerging intent DecreaseScreenBrightness (No.


All these methods heavily rely on numerous carefully hand-engineered features such as lexical (bag-of-words (BOW)), semantic (hypernyms, synonyms), structural (part of speech (POS) tags, lemmas, orthographic shapes, headings), statistical (statistical distributions of token types) and sequential (sentence position, surrounding features, predicted labels) features.

current emerging artificial neural network (ANN) based models have removed the need for manually selected features; instead, features are self-learned from the token and/or character embeddings.


This observation implies that the overall representation ability of TMN is enhanced as the increasing complexity of the model via combining more hops.

this enhancement will reach saturation when the hop number exceeds a threshold, which is 5 hops for most datasets in our experiment.


In our prior work (Rios and Kavuluru, 2018), we combine matching networks with a sophisticated thresholding strategy.

in Rios and Kavuluru (2018) we did not explore the few-and zeroshot settings.


We find that the word embedding derived label vectors work best for ESZSL on zero-shot labels.

this setup is outperformed by GRALS derived label vectors on the frequent and few-shot labels.


ZAGCNN outperforms ACNN by almost 5% and ZACNN by 1% in R@10 on few-shot classes.

aCNN still outperforms all other methods on frequent labels, but by only 0.3% when compared with ZaGCNN.


Both R@k and P@k give more weight to frequent labels, thus it is expected that ACNN outperforms ZAGCNN for frequent labels.

we also find that ACNN outperforms our methods with respect to Macro-F1.


In recent years, some neural models have made remarkable progress in this task.

they are all based on maximum likelihood estimation, which only learns common patterns of the corpus and results in lossevaluation mismatch.


Some neural models are proposed and achieve significant improvement.

existing models are all based on maximum likelihood estimation (MLE), which brings two substantial problems.


Poems generated by Base center in a few topics, which again demonstrates the claim: MLE-based models tend to remember the common patterns.

humanauthored poems spread on more topics.


The word 'fishing jetty' is confusing without any necessary explanation in the context.

poem (2) describes a clearer scene and expresses some emotion: a lonely man takes a boat from morning till night and then falls asleep solitarily.


Combining the virtues of probability graphic models and neural networks, Conditional Variational Auto-encoder (CVAE) has shown promising performance in many applications such as response generation.

existing CVAE-based models often generate responses from a single latent variable which may not be sufficient to model high variability in responses.


The CVAE based models incorporate stochastic latent variables into decoders in order to generate more relevant and diverse responses (Serban et al., 2017;Shen et al., 2017).

existing CVAE * Corresponding author based models normally rely on the unimodal distribution with a single latent variable to provide the global guidance to response generation, which is not sufficient to capture the complex semantics and high variability of responses.


2 Related Work As neural network based models dominate the research in natural language processing, Seq2Seq models have been widely used for response generation (Sordoni et al., 2015).

seq2seq models suffer from the problem of generating generic responses, such as I don't know (Li et al., 2016a).


VAE (Kingma and Welling, 2013) is one of the most successful models (Serban et al., 2017;Shen et al., 2017;Cao and Clark, 2017).

vAE-based models only use a single latent variable to encode the whole response sequence, thus suffering from the model collapse problem (Bowman et al., 2016).


we found that CVAE+BOW tends to copy the given queries (the first and fourth example in Table 4) and repeatedly generate redundant tokens (the second example).


Hybrid, DRESS, and DRESS-LS are good at generating shorter sentences, but they are not as good at choosing shorter words.

sBMT-sARI, DCss, and DMAss all generate shorter words.


Therefore, we believe that, by optimizing language model as a goal for the reinforcement learning, DRESS and DRESS-LS are tuned to simplify sentences by shortening the sentence lengths.

with the help of an integrated external knowledge base, SBMT-SARI and our models have more capability to generate shorter words in order to simplify sentences.


The neural text generation community has also recently been interested in "controllable" text generation (Hu et al., 2017), where various aspects of the text (often sentiment) are manipulated or transferred (Shen et al., 2017;Zhao et al., 2018;.

here we focus on controlling either the content of a generation or the way it is expressed by manipulating the (latent) template used in realizing the generation.


After training, we could simply condition on a new database and generate with beam search, as is standard with encoder-decoder models.

the structured approach we have developed allows us to generate in a more template-like way, giving us more interpretable and controllable generations.


Whereas there has been much recent interest in learning continuous latent variable representations for text (see Section 2), it has been somewhat unclear what the latent variables to be learned are intended to capture.

the latent, template-like structures we induce here represent a plausible, probabilistic latent variable story, and allow for a more controllable method of generation.


Neural text generation, including neural machine translation, image captioning, and summarization, has been quite successful recently.

during training time, typically only one reference is considered for each example, even though there are often multiple references available, e.g., 4 references in NIST MT evaluations, and 5 references in image captioning data.


There are many recent efforts in improving the generation accuracy, e.g., ConvS2S (Gehring et al., 2017) and Transformer (Vaswani et al., 2017).

all these efforts are limited to training with a single reference even when multiple references are available.


For example, the NIST Chinese-to-English and Arabic-to-English MT evaluation datasets (2003)(2004)(2005)(2006)(2007)(2008) have in total around 10,000 Chinese sentences and 10,000 Arabic sentences each with 4 different English translations.

for image captioning datasets, multiple references are more common not only for evaluation, but also for training, e.g., the MSCOCO (Lin et al., 2014) dataset provides 5 references per image and PASCAL-50S and ABSTRACT-50S (Vedantam et al., 2015) even provide 50 references per image.


These models embed entities and relations into latent vectors and complete KGs based on these vectors, such as TransE (Bordes et al., 2013), TransH (Wang et al., 2014) and TransR (Lin et al., 2015b).

most of the existing works simply embed relations into vectors.


Organizing scientific information into structured knowledge bases requires information extraction (IE) about scientific entities and their relationships.

the challenges associated with scientific IE are greater than for a general domain.


The prior knowledge based s 0 helps the agent narrow down the candidate space more quickly when the target object is a popular object.

it also becomes misleading when the target object is not popular and makes the agent even harder to correct the confidence of the target object.


However, it also becomes misleading when the target object is not popular and makes the agent even harder to correct the confidence of the target object.

the uniform distribution s 0 makes the agent keep track of the target object only based on the user's answers.


Besides, Zhao and Maxine (2016) also explores Q20 in their dialogue state tracking research.

they only use a small toy Q20 setting where the designed questions are about 6 person attributes in the Knowledge Base (KB).


Despite their simplicity, embedding-based models achieved state-of-the-art performance on KGQA (Das et al., 2018).

such models ignore the symbolic compositionality of KG relations, which limits their usage in more complex reasoning tasks.


In particular, we expect reward shaping to accelerate the convergence of RL (to a better performance level) as it propagates prior knowledge about the underlying KG to the agent.

a fair concern for action dropout is that it can be slower to train, as the agent is forced to explore a more diverse set of paths.


We observe that in general our proposed enhancements are effective in improving query-answering over both relation types (more effective for to-many relations).

adding the ConvE reward shaping module on WN18RR hurts the performance over both to-many and toone relations (more for to-one relations).


We define a learning algorithm as transductive 1 if its goal is to generalize from specific training examples to specific test examples (Vapnik, 1998).

in-ductive inference learns a general model that is independent of any test set.


Our example in §1 is that the model has difficulties producing initial letters not seen during training.

within each paradigm, forms are generally similar; thus, input subset sources contain valuable information about how to generate output subset targets.


On average, MED+PT clearly outperforms SIG17, the strongest baseline: by .0796 (.5808-.5012) on SET1, .0910 (.7486-.6576) on SET2, and .0747 (.8454-.7707) on SET3.

looking at each language individually (refer to Appendix A for those results), we find that MED+PT performs poorly for a few languages, namely Danish, English, and Norwegian (Bokmål & Nynorsk).


In contrast, the performance of MED, the neural model, is relatively independent of the choice of source; this is in line with earlier findings (Cotterell et al., 2016).

even for MED+PT, adding SHIP (i.e., MED+PT+SHIP) slightly increases accuracy by .0061 (.7547-.7486) on SET2, and .0029 (.8483-.8454) on SET3 (L53).


MED does not perform well for either SET1 or SET2.

on SET3 it even outperforms SIG17 for a few languages.


In contrast, on SET3 it even outperforms SIG17 for a few languages.

mED loses against mED+PT in all cases, highlighting the positive effect of paradigm transduction.


The correlation is not perfect because languages have different degrees of morphological regularity.

the overall trend is clearly recognizable.


Character-level features are currently used in different neural network-based natural language processing algorithms.

little is known about the character-level patterns those models learn.


Spanish While there is no single clear-cut rule for the Spanish gender, in general the suffix a denotes the feminine gender in adjectives.

there exist many nouns that are feminine but do not have the suffix a. Teschner and Russell (1984) identify d, and ión as typical endings of feminine nouns, which our models identified too as for example ad$ or ió/sió.


For Turkish, lemma performs better than lemma+morph, perhaps because the morphological analyzer outputs so many redundant properties which reduce the distance between words that are not particularly similar.

morph helps and lemma hurts in Hindi, perhaps because the morph analyzer outputs only a small number of highly informative properties, but is a poor general-purpose lemmatizer.


(2018) treated different argumentation formalisms as different tasks and combined respective extraction tasks and datasets in a MTL setting.

to these efforts that combine several AM subtasks or formalisms with joint optimization and MTL models, in this work we examine the dependencies between argumentative components and other rhetorical aspects of scientific writing.


2 The neural model described above generates an unlabeled temporal dependency tree, with each parent being the most salient reference time for the child.

it doesn't model the specific temporal relation (e.g.


(2017), both of which extract related spans of words (entity mentions for coreference resolution, and events or time expressions for temporal dependency parsing).

our temporal dependency parsing model differs from Lee et al's coreference model in that the ranking model for coreference only needs to output the best candidate for each individual pairing and cluster all pairs that are coreferent to each other.


", the emphasized text span is considered a causal explanation which indicates pessimistic personality -a negative event where the author believes the cause is pervasive.

in "My parser failed because I barely worked on the code.


Neural Network Model We used bidirectional LSTMs for causality classification and causal explanation identification since the discourse arguments for causal explanation can show up either before and after the effected events or results and we want our model to be optimized for both cases.

there is a risk of overfitting due to the dataset which is relatively small for the high complexity of the model, so we added a dropout layer (p=0.3) between the Word-level LSTM and the DA-level LSTM.


In other words, it's possible the linear model will not perform as well if the training size is increased substantially.

a linear model could still be used to do a first-pass, computationally efficient labeling, in order to shortlist social media posts for further labeling from an LSTM or more complex model.


Multimodal learning has shown promising performance in content-based recommendation due to the auxiliary user and item information of multiple modalities such as text and images.

the problem of incomplete and missing modality is rarely explored and most existing methods fail in learning a recommendation model with missing or corrupted modalities.


The authors did not set the problem in a supervised learning setup and instead find entities closest in terms of similarity of documents containing them.

in Section 3.2 we justify a supervised learning setup of the task.


The collection has a total of 1,003 books.

the dataset did not include book covers.


However, the visual features are not able to contribute much when combined with strong textual features that were already performing well.

for the MT setting, the performance decreases for most of the feature combinations with the addition of the visual modality.


After the subscription, users will receive a notification when a new comment arrives in that thread.

the speed of content generation in a well-known discussion forum is breakneck.


On one hand, the agent can choose the action with the highest estimated Q-value to exploit its current knowledge of the Q-function.

the agent can choose a non-greedy action to get more information about the Q-value of other actions.


The Q-value prediction of the permutation (1, 3, 2) almost triples that of the permutation (2, 3, 1).

any permutation of the comments does not change the Q-value prediction when we use our DRRN-Attention model.


We surmise that this information is related to the sentiment-bearing word "better" of the aspect "coffee", because a comparison using the word "better" implies the presence of a good ("coffee") and a bad ("cosi sandwiches") object.

this semantics is misconstrued by IAN, which leads to aspect misclassification.


We exploit this property to obtain state-of-the-art performance in aspect-based sentiment analysis in two distinct domains: restaurant and laptop.

there remains plenty of room for improvement in the memory network, e.g., for generation of better aspect-aware representations.


An intuitive way is to define the global error function for the network on the training set.

some important characteristics of relevant emotion ranking, such as ranking, not considering irrelevant emotions, are not considered in the classical back propagation algorithm (Rumelhart et al., 1988).


In addition, existing works train each instance separately.

we observe that the interactions among the aspects, which have the same context words, could bring extra useful information.


The main advantage of deep learning approach is that it is effective in exploring both linguistic and semantic relations between words, thus can overcomes the problems of lexicon-based approach.

current deep learning approach for sentiment analysis usually faces with the major shortcoming, i.e., being limited by the quantity of high quality labeled data.


For instance, the winning approach for SwissCheese (SemEval 2016) uses an ensemble of 6 CNN models along with a meta-classifier (random forest classifier).

our proposed model is a single neural model.


Traditionally, sentiment analysis Lee, 2005, 2008) has been applied to a wide variety of texts (Hu and Liu, 2004;Liu, 2012;Turney, 2002;Akhtar et al., 2016Akhtar et al., , 2017Mohammad et al., 2013).

multi-modal sentiment analysis has recently gained attention due to the tremendous growth of many social media platforms such as YouTube, Instagram, Twitter, Facebook Poria et al., 2016Poria et al., , 2017dZadeh et al., , 2016 etc.


We evaluate our proposed approach on two benchmark datasets, namely CMU Multi-modal Opinion-level Sentiment Intensity (CMU-MOSI) corpus (Zadeh et al., 2016)  Each utterance in CMU-MOSI dataset has been annotated as either positive or negative, whereas in CMU-MOSEI dataset labels are in the continuous range of -3 to +3.

in this work we project the instances of CMU-MOSEI in a two-class classification setup with values ≥ 0 signify positive sentiments and values < 0 signify negative sentiments.


For MO-SEI dataset we observe that the precision and recall for positive class (84% precision & 88% recall; are quite encouraging.

the same are comparatively on the lower side for the negative class (68% precision & 58% recall.


We count the total occurrences for each noun sense (synset) of the candidate and match the candidate to the most frequent synset.

such a method is not good enough for our problem, as shown in the results later.


L = [l 1 , l 2 , ..., l 25 ] are the 25 golden aspect terms, where L (h) = [l 5h−4 , ..., l 5h ] are from the h-th human annotator.

the hard accuracy is defined as: counting the number of exact matches makes the accuracy score discrete and coarse.


(2015b) used forensics features for detecting multimedia fabrication to verify rumors.

these features did not lead to noticeable improvement.


Previous rumor verification datasets are mainly monolingual, such as English (Derczynski et al., 2017) or Chinese (Wu et al., 2015).

textual information in the native language where the rumor happened can be more helpful when it comes to ver-ifying worldwide rumors.


Those missing features make TFB perform poorly in the task setting.

tFB performs better than two of the baselines in the event setting.


Explanatory models can potentially help human moderators make quicker and more consistent decisions about whether to remove comments (Lakkaraju et al., 2016).

we propose that truly minimal explanations are liable to give only a partial portrait of why a comment is objectionable, making it harder to render a fair holistic decision.


When working with a recurrent unit that has no internal bias term, this behavior is entirely determined by the bias term of the final sigmoid output layer, σ(wx + b), which with typical random initialization of b results in a default predicted value of roughly 0.5.

this 0.5 default value is not always optimal or semantically appropriate to the predictive task.


In S3, "Information" and "those services" are considered more vague by humans than "often depends."

if those terms are removed from the sentence, yielding "The [..] we obtain from [..] often depends on your settings or their privacy policies."


Note that for a left leaning article (see Figure 3a), the model focuses on sentences involving gun-control, feminists, and transgender.

a visualization of sentence attention scores for an article which the model predicted as "right-leaning" ((see Figure 3b)) reveals a focus on words like god, religion etc.


Most of these studies attempt to extract efficient features from text content (Liu and Hsieh, 2006;Lin et al., 2012;Aletras et al., 2016;Sulea et al., 2017) or case annotations (e.g., dates, terms, locations, and types) (Katz et al., 2017).

these conventional methods can only utilize shallow textual features and manually designed factors, both require massive human efforts and usually suffer from the generalization issue when applied to other scenarios.


In response to the growth in online hate, there has been a trend of developing automatic hate speech detection models to alleviate online harassment (Warner and Hirschberg, 2012;Waseem and Hovy, 2016).

a common problem with these methods is that they focus on coarse-grained 1 http://www.pewinternet.org/2017/07/11/onlineharassment-2017/ Figure 1: A portion of the hate group map published by the Southern Poverty Law Center (SPLC).


Our work is most closely related to (Van Hee et al., 2015), which focuses on fine-grained cyberbullying classification.

this study only focuses on seven categories of cyberbullying while our dataset consists of 40 classes.


Because of their predictive power, socio-demographics and other extralinguistic information can additionally be leveraged when building the model itself.

a central challenge in integrating community attributes is that they have very different properties than linguistic features and can be lost, in essence, like a needle in a haystack.


Authoritarian countries such as Russia and China have received a great deal of attention for trying to control and distort the spread of information through "fake news" and censorship.

authoritarian governments might also use subtle tactics of media manipulation that are much harder to detect, like flooding communication channels with irrelevant information or highlighting particular viewpoints of an event to distract public attention (Rozenas and Stukal, Forthcoming;Munger et al., 2018;King et al., 2017).


Censorship can be detected by checking what content is no longer available.

we have no systematic way of identifying more subtle forms of media manipulation.


About 11,900 articles are hand-annotated with frames: annotators highlight spans of text related to each frame in the codebook and assign a single "primary frame" to each document.

the MFC, like other prior framing analyses, relies heavily on labor-intensive manual annotations.


This indicates that MLE objective is helpful to stabilize the training and improve the model performance, as we expected.

further increasing α does not bring more gain.


(2013b) propose to merge them into a single token in a pre-processing step.

for that purpose, they use a scoring function based on their co-occurence frequency in the training corpus, with a discounting coefficient δ that penalizes rare words, and iteratively merge those above a threshold: we also need to learn representations for compositional n-grams in our scenario, as there is not always a 1:1 correspondence for n-grams across languages even for compositional phrases.


Nevertheless, these systems work in a word-by-word basis and have only been shown to work in limited settings, being often evaluated in word-level translation.

our method builds a fully featured phrasebased SMT system, and achieves competitive performance in a standard machine translation task.


(2018), who use a separate encoder for each language, sharing only a subset of their parameters, and incorporate two generative adversarial networks.

our results in Section 6.1 show that our SMT-based approach obtains substantially better results.


when this answer text is considered as a sequence, it is highly possible to be predicted as the category of positive or negative rather than conflict.


(2014) investigate the identification of topic-relevant claims, an approach that was later extended with evidence extraction to mine supporting statements for claims (Rinott et al., 2015).

both approaches are designed to mine arguments from Wikipedia articles; it is unclear whether their annotation scheme is applicable to other text types.


In general, we observe an overall lower score for trl models that use the DIP2016 corpus compared to those using the SemEval corpus.

to the mtl model, for trl models all parameters are transferred to the main task, not just parameters that represent shared knowledge.


Dynamic aspect extraction is advantageous since it assumes nothing more than a set of relevant reviews for a product and may discover unusual and interesting aspects (e.g., whether a plasma television has protective packaging).

it suffers from the fact that the identified aspects are finegrained, they have to be interpreted post-hoc, and manually mapped to coarse-grained ones.


Emotion Corpora: There are several affective datasets such as SemEval-2017 Task 4 (Rosenthal et al., 2017) and Olympic games dataset (Sintsova et al., 2013).


As shown in Table 4, their model performs as well as other traditional methods.

our model (CARER) significantly outperforms theirs (↑20%).


16 As shown in Figure 3, due to the limited coverage of the LIWC lexicon, such resources may not be feasible on evolving, large-scale datasets.

word2vec contains over 3 million unique word embeddings and has been proven effective for text classification.


(2016) partially motivate the ranking-based variant throught the importance sampling viewpoint of Bengio and Senécal (2008).

there are two critical differences: 1) the algorithm of Bengio and Senécal (2008) does not lead to the same objective L n R in the ranking-based variant of NCE; instead it uses importance sampling to derive an objective that is similar but not identical; 2) the importance sampling method leads to a biased estimate of the gradients of the log-likelihood function, with the bias going to zero only as K !


It refers to the critique (Ranzato et al., 2016) that MLE-trained models tend to have suboptimal performance as they are trained to maximize a convenient objective (i.e., maximum likelihood of word-level correct next step prediction) rather than a desirable sequence-level objective that correlates better with the common discrete evaluation metrics such as ROUGE (Lin and Och, 2004) for summarization, BLEU (Papineni et al., 2002) for translation, and word error rate for speech recognition, not loglikelihood.

training models that directly optimize for such discrete metrics as objective is hard due to non-differentiable nature of the corresponding loss functions (Rosti et al., 2011).


Towards sequence level optimization, previous works (Ranzato et al., 2016;Paulus et al., 2018) employ reinforcement learning (RL) with a policy-gradient approach which works around the difficulty of differentiating the reward function by using it as a weight.

rEINFOrCE is known to suffer from high sample variance and credit assignment problems which makes the training process difficult and unstable besides resulting in models that are hard to reproduce (Henderson et al., 2018).


Most neural sequence generation models are trained with the objective of maximizing the probability of the next correct word.

this results in a major discrepancy between training and test settings of these models because they are trained with cross-entropy loss at word-level, but evaluated based on sequence-level discrete metrics such as ROUGE (Lin and Och, 2004) or BLEU (Papineni et al., 2002).


Moreover, the words should match the leave-one-out method's selections, which closely align with human perception (Li et al., 2016b;Murdoch et al., 2018).

rather than providing explanations of the original prediction, our reduced examples more closely resemble adversarial examples.


Most reduced inputs are nonsensical to humans ( Figure 2) as they lack information for any reasonable human prediction.

models make confident predictions, at times even more confident than the original.


In this example, the leave-one-out method will not highlight "Broncos".

this is not a failure of the interpretation method but of the model itself.


This method adjusts the softmax function's temperature parameter using a small held-out dataset to align confidence with accuracy.

because the output is calibrated using the entire confidence distribution, not individual values, this does not reduce overconfidence on specific inputs, such as the reduced examples.


This approach has become popular because it avoids exposure bias, and directly optimizes a measure of summary quality.


(2018) recently proposed an extractive summarization approach based on deep Q learning, a type of reinforcement learning.

their approach is extremely computationally intensive (a minimum of 10 days before convergence), and was unable to achieve ROUGE scores better than the best maximum likelihood-based approach.


One potential cause of this high variance can be seen by inspecting (3), and noting that it basically acts to change the probability of a sampled index sequence to an extent determined by the reward R(i, a).

since ROUGE scores are always positive, the probability of every sampled index sequence is increased, whereas intuitively, we would prefer to decrease the probability of sequences that receive a comparatively low reward, even if it is positive.


Two of the most commonly used assumptions are that simple words are associated with shorter lengths and higher frequencies in a corpus.

these assumptions are not always accurate and are often the major source of errors in the simplification pipeline (Shardlow, 2014).


(2016) used a list of 3,000 most common English words; Lee and Yeung (2018) used an ensemble of vocabulary lists of different complexity levels.

to the best of our knowledge, there is no previous study on manually building a large word-complexity lexi-con with human judgments that has shown substantial improvements on automatic simplification systems.


In this situation, the original system should be extended to support new user actions based on user feedback.

adding new intents or slots will change the predefined ontology.


RL methods have shown great potential in building a robust dialog system automatically.

rL-based approaches are rarely used in real-world applications because of the maintainability problem (Paek and Pieraccini, 2008;Paek, 2006).


The first strategy requires a new interaction environment.

building a user simulator or hiring real users once the system needs to be extended is costly and impractical in real-world applications.


The user simulator can provide infinite simulated experiences without additional cost, and the trained system can be deployed and then fine-tuned through interactions with real users (Su et al., 2016;Zhao and Eskenazi, 2016;Williams et al., 2017;Dhingra et al., 2017;Liu and Lane, 2017;Peng et al., 2017b;Budzianowski et al., 2017;Peng et al., 2017a;Tang et al., 2018).

due to the complexity of real conversations and biases in the design of user simulators, there always exists the discrepancy between real users and simulated users.


Respectively, the policy model is trained via real experiences collected by interacting with real users (direct reinforcement learning), and simulated experiences collected by interacting with the learned world model (planning or indirect reinforcement learning).

the effectiveness of DDQ depends upon the quality of simulated experiences used in planning.


Theoretically, larger amounts of high-quality simulated experiences can boost the performance of the dialogue policy more quickly.

the world model by no means perfectly reflects real human behavior, and the generated experiences, if of low quality, can have negative impact on dialogue policy learning.


From the table, with fewer simulated experiences, the difference between DDQ and D3Q may not be significant, where DDQ agents achieve about 50%-60% success rate and D3Q agents achieve higher than 68% success rate after 100 epochs.

when the number of planning steps increases, more fake experiences significantly degrade the performance for DDQ agents, where DDQ(10, fixed θ G ) suffers from bad simulated experiences after 300 epochs and achieves 0% success rate.


Recent work ((Vinyals and Le, 2015;Bordes et al., 2016;Serban et al., 2016)) has shown that dialog models can be trained in an end-to-end manner with satisfactory results.

human dialog has some unique properties that many other learning tasks do not.


The system asks questions to fill the missing fields and eventually generate the correct corresponding API call.

the system asks for information in a deterministic order -Cuisine → Location → People → Price to complete the missing fields.


The system must propose options to users by listing the restaurant names sorted by their corresponding rating (from higher to lower) until users accept.


Synthesized data can also be an option to obtain a large dataset.

these are often built from templated responses which make it meaningless for dialogue models to learn.


Otherwise a status action of "no flight found" will be returned.

the task logic for customers with a goal of "change" would be slightly different.


During the self-play experiments we perform similar predictions on the dialogue states.

instead of asking the models to predict those states given ground truth history, we now ask the models to predict given the generated dialogues.


Paraphrase generation is an important task in NLP, which can be a key technology in many applications such as retrieval based question answering, semantic parsing, query reformulation in web search, data augmentation for dialogue system.

due to the complexity of natural language, automatically generating accurate and diverse paraphrases is still very challenging.


As a result, end-to-end neural text generation has drawn increasing attention from the natural language research community (Mei et al., 2016;Lebret et al., 2016;Wiseman et al., 2017;Kiddon et al., 2016).

a critical issue for neural text generation has been largely overlooked.


In fact, other value likes -2 or -3 is close to -1, and the word "edges" is also applicable to them.

directly establishing the lexical choices on various sparse numeric values is not easy (Reiter et al., 2005;Smiley et al., 2016;Zarrieß and Schlangen, 2016).


(2016) have built a biography generation dataset from Wikipedia.

a recent study by Perez-Beltrachini and Gardent (2017) shows that existing datasets have a few missing properties such as lacking syntactic and semantic diversity.


Table 2 shows that 11.7% of facts in the game summaries can be inferred based on the input data.

this dataset focuses on generating long text and 27.1% of facts are unsupported 2 , which brings difficulties to the analysis of fidelity for the generated text.


Some have focused on conditional language generation based on tables (Yang et al., 2017), short biographies generation from Wikipedia tables (Lebret et al., 2016;Chisholm et al., 2017) and comments generation based on stock prices (Murakami et al., 2017).

none of these methods consider incorporating the facts that can be inferred from the input data to guide the process of generation.


we keep all samples instead of only keeping the sentence-question pairs that have at least one non-stop-word in common (with 6.7% pairs dropped) as in (Du et al., 2017).


In order to test a model's real semantic parsing performance on unseen complex programs and its ability to generalize to new domains, an SP dataset that includes a large amount of complex programs and databases with multiple tables is a must.

compared to other large, realistic datasets such as ImageNet for object recognition (Deng et al., 2009) and SQuAD for reading comprehension (Rajpurkar et al., 2016), creating such SP dataset is even more time-consuming and challenging in some aspects due to the following reasons.


This is not a big problem if we use one single dataset because we have enough data domain specific examples to know which columns are default.

it would be a serious problem in cross domain tasks since the default return values differ cross domain and people.


The attention and copying mechanisms do not help much either.

sQLNet and TypesQL that utilize sQL structure information to guide the sQL generation process significantly outperform other seq2seq model.


Recent success in Deep Learning motivated researchers to use neural networks instead of human designed rules and templates to generate meaningful sentences from structured information.

these supervised models work well only when provided either massive amounts of labeled data or when restricted to a limited domain.


Nevertheless, there are two major differences between the training procedure of a DAE and an inference instance in NLG: First, we do not need to predict any content information in NLG as all of the content information is already provided by the structured data.

a DAE training instance can also remove content words from the sentence.


Recent neural networkbased approaches employ the sequence-tosequence model which takes an answer and its context as input and generates a relevant question as output.

we observe two major issues with these approaches: (1) The generated interrogative words (or question words) do not match the answer type.


In this framework, a good discriminator that can assign reasonable reward for the generator is a critical component.

directly applying a classifier as the discriminator like most existing GAN models (e.g., SeqGAN (Yu et al., 2017)) cannot achieve satisfactory performance.


Currently, a popular model for text generation is the sequence-to-sequence model (Sutskever et al., 2014;.

the sequenceto-sequence model tends to generate short, repetitive , and dull text (Luo et al., 2018).


In this paper, to handle this problem, we propose to use adversarial training (Goodfellow et al., 2014;Denton et al., 2015;Li et al., 2017), which has achieved success in image generation (Radford et al., 2015;Gulrajani et al., 2017;Berthelot et al., 2017).

training GAN is a non-trivial task and there are some previous researches that investigate methods to improve training performance, such as Wasserstein GAN (WGAN)  and Energybased GAN (EGAN) (Salimans et al., 2016;Gulrajani et al., 2017;Zhao et al., 2017;Berthelot et al., 2017).


It shows that the classifier still cannot tell the difference between the text with low novelty.

the analysis of experimental result shows that our proposed discriminator can better distinguish high-novelty text from lownovelty text without the saturation problem.


However, this could be computationally expensive because the time complexity is O(T 2 ).

our discriminator can calculate the reward of all words with the time complexity of O(T ), which is more computationally efficient.


As we can see, the reward distribution of SeqGAN saturates and cannot distinguish the novelty of the text accurately.

dP-GAN has a strong ability of resisting reward saturation and can give more precise reward for text in terms of novelty.


If the accuracy of classifier is too high, the classifier cannot give reasonable reward to the generator for generating real and diverse text .

the language-model based reward given by DP-GAN better reflect the novelty of the text.


The above example might give the impression that named entities are essential but other words are not.

this is misleading and may not always be the case.


Note that, an alternate way of collecting human judgments would have been to take the output of existing AQG systems and ask humans to assign answerability scores to these questions based on the presence/absence of the above mentioned relevant information.

when we asked human evaluators to analyze 200 questions generated by an existing AQG system, they reported that the quality was poor.


In our initial evaluations, we also tried showing the actual source (image or document) to the annotators.

we realized that this did not allow us to do an unbiased evaluation of the quality of the questions.


To accept two arguments, i.e., input sentence X and style id k, we can directly concatenate the one-hot representation of style id one hot(k) and the embedding vector h T obtained by bi-LSTM encoder and then feed the concatenated vector [one hot(k), h T ] instead of h T into the decoder model without changing anything else.

there is no theoretical guarantee that the output sentence generated by the decoder is strongly correlated to the style id input one hot(k).


By disentangling poems in different styles, our model can learn a more compact generation model for each style and write more fluent and meaningful poems.

by fixing the style when generating three lines in a quatrain, SPG is able to generate more coherent poems.


One aspect that is central to a question is the context that is relevant to generate it.


Advances in image captioning (Xu et al., 2015;Fang et al., 2015;Karpathy and Fei-Fei, 2015;Vinyals et al., 2015) are effective in generating sentencelevel descriptions.

sentences generated by these approaches are usually generic descriptions of the visual content and ignore background information.


We require images with well-aligned, news-style captions for training.

we want to test our model on real social media data and it is difficult to collect these informative captions for social media data.


Textsummarization model generates results from documents retrieved by hashtags, so it tends to include some long phrases common in those documents.

the templates generated by our model is based on the language model trained from the news captions, which has different style with Flickr captions.


not expected to describe all objects in the scene.

describing objects that are not present in the image has been shown to be less preferable to humans.


We use beam size of 5, and for all models trained with cross entropy, it outperforms lower beam sizes on CHAIR.

when training models with the self-critical loss, beam size sometimes leads to worse performance on CHAIR.


This is somewhat surprising, as a model which has access to image information at each time step should be less likely to "forget" image content and hallucinate objects.

it is possible that models which include image inputs at each time step with no access to spatial features overfit to the visual features.


All three metrics do penalize sentences for mentioning incorrect words, either via an F score (METEOR and SPICE) or cosine distance (CIDEr).

if a caption mentions enough words correctly, it can have a high METEOR, SPICE, or CIDEr score while still hallucinating specific objects.


Likewise, our work is related to other work which aims to build better evaluation tools , (Anderson et al., 2016), (Cui et al., 2018)).

we focus on carefully quantifying and characterizing one important type of error: object hallucination.


Recent neural sequence-to-sequence models have shown significant progress on short text summarization.

for document summarization, they fail to capture the longterm structure of both documents and multisentence summaries, resulting in information loss and repetitions.


The Mask Only model with increased supervision on the copy mechanism performs very similar to the Multi-Task model.

bottom-up attention leads to a major improvement across all three scores.


While we observed both cases quite frequently in generated summaries, the fraction of very long copied phrases decreases.

either with or without bottom-up attention, the distribution of the length of copied phrases is still quite different from the reference.


As for the other two models, they have better ROUGE score on Trunc version.

as the example shown in Table  4 6 , higher ROUGE scores do not necessarily mean 6 The entities in different color indicate two important roles in the text.


Most existing document summarisation techniques require access to reference summaries to train their systems.

obtaining reference summaries is very expensive: Lin (2004) reported that 3,000 hours of human effort were required for a simple evaluation of the summaries for the Document Understanding Conferences (DUC).


Simpson and Gurevych (2018) proposed to use an improved Gaussian process preference learning (Chu and Ghahramani, 2005) for learning to rank arguments in terms of convincingness from crowdsourced annotations.

such Bayesian methods can hardly scale and suffer from high computation time.


The neural encoder-decoder framework has recently been exploited to summarize single documents, but its success can in part be attributed to the availability of large parallel data automatically acquired from the Web.

parallel data for multi-document summarization are scarce and costly to obtain.


If a text piece has been attended to during summary generation, it is unlikely to be used again.

the attention value assigned to a similar text piece in a different position is not affected.


These issues may be alleviated by improving the encoder-decoder architecture and its attention mechanism (Cheng and Lapata, 2016;Tan et al., 2017).

in these cases the model has to be re-trained on large-scale MDS datasets that are not available at the current stage.


Precision and recall are commonly used to evaluate recommendation systems (Karypis, 2001) and information retrieval task (Zuva and Zuva, 2012).

we only care about whether the image appears in the reference image set.


our text input is long text, and it contains enough information for text generation.


One drawback of our method is that its expressive power is basically the same as that of each single model.

this alternatively means that the lower bound of the quality of each output is guaranteed with the worst case of the outputs of single models, while the current runtime-ensemble method can perform worse than each single model for the worst case input.


For example, Stahlberg and Byrne (2017) proposed a method of unfolding an ensemble of multiple translation models into a single large model once and shrinking it down to a small one.

all methods require extra implementation on a deep-learning framework, and it is not easy to apply them to other models.


Although, various evaluation measures have been proposed, ROUGE-N (Lin, 2004), Basic Elements (BE) (Hovy et al., 2006) remain the de facto standard measures since they strongly correlate with various manual evaluations and are easy to use.

the evaluation scores computed by these automatic measures are not so useful for improving system performance because they merely confirm if the summary contains small textual fragments and so they do not address semantic correctness.


Thus, PEAK regards subjectpredicate-object triples as alternatives to SCUs and constructs a pyramid by clustering semantically equivalent triples.

the performance of subject-predicate-object triples extraction is not satisfying for the practical demands and semantic similarity utilized for clustering the triples does not correlate well with human judgment (see Section 2).


Thus, to construct high quality pyramids, PEAK is required to not only accurately extract the triples but also measure the semantic similarity between them accurately.

in general, both extracting the triples and measuring the semantic similarity are still challenging NLP tasks.


(C)) due to the mismatch word distributions between the summaries of the source and target domains.

the discriminator still regularizes the generated word sequence.


(2017) proposed a multitask neural architecture where the three tasks are trained together with the same representation.

they do not model comment-comment interactions in the same question-comment thread nor do they train taskspecific embeddings, as we do.


Formally, The final task-specific embedding for subtask C is formed as One simple way to exploit the interdependencies between the subtask-specific embeddings ( is to precompute the predictions for some subtasks (A and B), and then to use the predictions as features for the other subtask (C).

as shown later in Section 6, such a pipeline approach propagates errors from one subtask to the subsequent ones.


Differences among the question styles: The biggest advantage to the answer extraction style is its ease in generating questions, which enables us to produce large-scale datasets.

a disadvantage to this style is that it rarely demands meta/whole and math/logic skills, which can require answers not contained in the context.


Much progress has been made in reasoning-based MRC-QA on the bAbI dataset (Weston et al., 2016), which contains questions that require the combination of multiple disjoint pieces of evidence in the context.

due to its synthetic nature, bAbI evidences have smaller lexicons and simpler passage structures when compared to humangenerated text.


More recent datasets such as QAngaroo (Welbl et al., 2018) have prompted a strong focus on multi-hop reasoning in very long texts.

qAngaroo is an extractive dataset where answers are guaranteed to be spans within the context; hence, this is more focused on fact finding and linking, and does not require models to synthesize and generate new information.


gN-EF with text improves over the KB-only approach in all settings.


Recently, there has been a surge of interest in reading comprehension-based (RC) question answering (QA).

current approaches suffer from an impractical assumption that every question has a valid answer in the associated passage.


Increasing Nil F1 scores also help to improve the overall F1 scores.

the overall F1 score degrades with increasing length of the associated passage.


Advanced neural machine translation (NMT) models generally implement encoder and decoder as multiple layers, which allows systems to model complex functions and capture complicated linguistic structures.

only the top layers of encoder and decoder are leveraged in the subsequent process, which misses the opportunity to exploit the useful information embedded in other layers.


In addition to two sub-layers in each decoder layer, the decoder inserts a third sublayer D l d to perform attention over the output of the encoder stack H L e : where Multi-layer network can be considered as a strong feature extractor with extended receptive fields capable of linking salient features from the entire sequence .

one potential problem about the vanilla Transformer, as shown in Figure 1a, is that both the encoder and decoder stack layers in sequence and only utilize the information in the top layer.


They conclude that CNNs are better than RNNs for sequence modeling.

their CNN models perform much worse than the state-of-art LSTM models on some sequence modeling tasks, as they themselves state in the appendix.


In ATR, the role of q t is similar to that ofh t in GRU (see Equation 4).

we completely wipe out the recurrent non-linearity Figure 2: Visualization of the difference between σ(x + y) (the input gate) and σ(x − y) (the forget gate).


This is reasonable since the Transformer can be trained in full parallelization.


(2015) (and many others) only used the top-30k frequent words as target vocabulary, and replaced others with UNK.

the final normalization operation still brought high computation complexity for forward calculations.


Translating characters instead of word fragments avoids these problems, and gives the system access to all available information about source and target sequences.

it presents significant modeling and computational challenges.


Strictly speaking, the story generation task requires systems to generate a story from scratch without any external materials.

for simplification, many existing story generation models rely on their given materials, such as short text descriptions (Harrison et al., 2017;Jain et al., 2017), visual images (Charles et al., 2001;Huang et al., 2016), and so on.


A good skeleton is expected to contain all key information and ignore other information.

the skeletons that contain too much detailed information or lack necessary information are considered as bad skeletons and should be punished.


When the input is short and the model often sees the input, the generated story tends to have high coherence.

when the length of input increases and the model is not familiar with the input, the coherence goes down.


Experimental results show that our model significantly improves the quality of generated stories, especially in coherence.

even with the best human evaluation results, the error analysis shows that there are still many challenges in narrative story generation, which we would like to explore in the future.


It has also been applied in dialogue generation (Serban et al., 2017;.

all the above methods stick to the MLE objective function and do not optimize with respect to the mutual information.


Coherence models that purely rely on the recurrent neural networks process words sequentially within a text.

in such models, long-distance dependencies between words cannot be captured effectively due to the limits of the memorization capability of recurrent networks.


However, Content Model requires manual feature engineering that costs great human efforts.

the self-attention mechanism used in ATTOrder-Net directly captures the global dependences for the whole text while requiring no linguistic knowledge anymore and enables ATTOrderNet to further improve tau score to 0.92 on the same dataset.


We present the weight distribution in four heads as an instance.

it is interesting to see that all of them showing significant higher attention weights on the true second sentence "text use changes ..." than other sentences in the text.


data (Dyen et al., 1992;Greenhill et al., 2008).

3 if a tree is given a priori, phylogenetic models can also be used to estimate the parameters of a TRM, which controls how languages change their feature values over time.


A popular approach (Greenhill et al., 2010;Dunn et al., 2011) is to construct a time-tree with absolute (calendar) dates, using binary-coded lexical cognate data, and then to fit each trait of interest independently on the time-tree.

4 cognate data are available only for a handful of language families such as Indo-European, Austronesian and Niger-Congo (or its mammoth Bantu branch).


We address these problems by including geographic information via retrofitting (Faruqui et al., 2015;Hovy and Fornaciari, 2018): we use administrative region boundaries to modify the city embeddings, and evaluate the resulting vectors in a clustering approach to discover larger dialect regions.

to most dialectometric approaches (Nerbonne et al., 1999;Prokić and Nerbonne, 2008), and in line with common NLP practice (Doyle, 2014;Grieve, 2016;Huang et al., 2016;Rahimi et al., 2017a), we also evaluate the clustered dialect areas quantitatively.


This heavily contributes to vulgar word volatility across different functions and higher ambiguity in context.

this type of usage can allow computational approaches that model the immediate context around a word to generalize across words to functions.


This highlights both the challenges in modeling vulgar word functions and the opportunity of using the function to improve practical applications.

table 5 shows the vulgar words which are most likely to be used with each of the six functions.


Activation functions have been characterized by a variety of properties deemed important for successful learning, such as ones relating to their derivatives, monotonicity, and whether their range is finite or not.


Initially, we do not consider the more popular LSTMs here for reasons indicated below.

we include a comparison after discussing the RNN performance.


Note that as LSTM-Shuttle proceeds not only forwards but also backwards, the output shuttle size is 2K + 1: K forward, K backward, and 1 for stopping.

to LSTM-Jump (Yu et al., 2017), when going back, our shuttle step is counted before reading sequentially.


We compute θ R via backpropagation directly by minimizing J 1 (θ R ), the cross entropy loss, which is differentiable over θ R and is the target objective function of the classification task.


For bi-directional LSTM-Jump, since it applies LSTM-Jump twice, it predicts better than the original.

we doubt whether it is worth sacrificing so much efficiency for such a small increase in prediction accuracy (+0.2%).


A smaller N means LSTM-Shuttle shuttles less often, so the model tends to read through as much as possible, making for a lower backward ratio.

lSTM-Shuttle can shuttle many times so it is willing to go back to correct misunderstandings.


Some fields such as Invoice Number are relatively easy to detect for a model that operates on serialized text, as discriminative keywords are commonly preceding the words to be extracted.