For example, although the Wikipedia article 2 "The geometry and topology of three-manifolds" is filed under the categories of interest "Hyperbolic geometry", "3-manifolds" and "Kleinian groups", the title as a whole does not represent a single concept.

the technical term "Riemannian manifold" would completely match the title of the Wikipedia 3 (or Encyclopedia of Math) article for the concept and would thus be identified as a type by our method.


On one hand, the performance of Types2X suggests that information coming from the types occurring in the queries alone may not be enough to produce significant improvements in retrieval efficiency.

our experiments have shown that it is only the combination of query expansion with type information (rather than with simple terms) that yields significant performance gains on these difficult queries.


The types "iwasawa decomposition" and "cartan decomposition" in the expanded set strongly relate to the types in the query (both are related to Levi decomposition of Lie algebras).

there are also some instances of weak association between query and expanded types.


To be successful, in addition to accuracy, real-world systems must be scalable and select the most accurate answer sentences in a short amount of time.

accuracy and speed/scalability are competing forces that often counteract each other.


For example, models based on neural networks have become very popular due to their strong accuracy for this task (Yu et al., 2014).

they are typically slower at training and test time as compared to simple models, which may limit their use on very large datasets (Joulin et al., 2016).


We used the dependency relations of words to alleviate this problem and thus improved the performance.

we still need to find more effective methods to overcome Korean linguistic barriers such as the free word order and the lack of language resources; and this problem will be studied in future research.


For example, 85% questions of WEBQUESTIONS can be directly answered via a single Freebase predicate.

all questions of QALD-6 involve at least one DBpedia predicate and one textual relation, thus can not be accurately answered using DBpedia only.


Only the words that have dependency relations with the center word are used as the context words, as illustrated in Figure 1.

syntactic parsing is a more difficult and time-consuming task than finding word embeddings.


Bamman and Smith (2015) report an accuracy of 75.4% on a balanced dataset, which is lower than our result.

they performed evaluation on a different set of data, thus the results are not directly comparable.


As shown in the figure, most "+"s are in the top-right area, and most "•"s are in the bottom-left area, which indicates that the accuracies of both models are reasonably high.

the samples are more scattered along the x-axis.


Our intuition was that, since our snippets usually include more than one sentence, resolving pronouns and coreferential expressions may make the content more explicit, thus enabling a better agreement detection.

results show that this information causes a slight performance drop.


Based on Freebase, two benchmark data-sets, WebQuestions (Berant et al., 2013) and SimpleQuestions (Bordes et al., 2015) are constructed and used in most of KBQA work (Berant and Liang, 2014;Bordes et al., 2014a;Fader et al., 2014;Yang et al., 2014;Bao et al., 2014;Reddy et al., 2014;Dong et al., 2015;Yih et al., 2015).

mark Rydell  about 85% of questions (Yao, 2015) of WebQuestions and all questions in SimpleQuestions are 'simple' questions, where a 'simple' question denotes that it can be answered based on a single KB relation.


We trained a sentence level classifier, on the training partition of the corpus, as defined in Section 4.1 -considering the question-level annotations as gold standard.

the latter choice introduces a lot of false positives as sentences in relevant questions may be unrelated.


Our experiments verify that the part-of-speech embeddings used by us contain rich semantic information.

our proposed Attention-CNN model can still yield higher F1 without prior NLP knowledge.


Multi-task training is performed via switching across multiple tasks in a block of training steps.

we perform switches between ER and RC subtasks based on the performance of each task on the common validation set and update learning rate only when task is switched from RC to ER (Figure 8).


A relation for a word pair is marked correct if the NE boundaries and relation type are correct.

in separate approach, a relation for a word pair is marked correct if the relation type is correct.


In contrast, the latter are relatively simple and scalable.

they often heavily rely on only one descriptor of the bag-of-words embeddings (e.g.


APE assumes the availability of source texts (S ip ), corresponding MT output (T mt ) and the human post-edited (T pe ) version of T mt , and APE systems can be modelled as an MT system between S ip T mt (i.e., a joint representation of S ip and T mt ) and T pe .

statistical APE (SAPE) systems can also be built without the availability of S ip using only sufficient amounts of parallel "target-side" T mt -T pe text within the statistical MT (SMT) framework.


This may be partly due to the fact that the human postedited reference translations are biased towards GT output.

manual analysis revealed that some of these 39 translations are indeed worse than the GT output.


The productivity changes vary from 46.6% to -40%, which indicates that the utility of SAPE also varies from person to person.

even taking into account the decrease in productivity of T4, average productivity increases 12.96% with SAPE.


Improving SMT for the dialogue acts under consideration resembles cross-lingual question answering (Tiedemann, 2009).

when considering finer dialogue act granularities, it may be profitable to exploit context information, which is not used in our current SMT setup.


We can see that our method can recognize support sentences well.

the thesis and main ideas are identified with moderate performance.


This proves that vocabulary gap exists between queries and factual story descriptions of anecdotes.

great improvements are obtained when measuring relatedness between anecdotes and queries with anecdote implications.


Studies in the past decade on discourse parsing, such as (Ji and Eisenstein, 2014), (Feng and Hirst, 2014), and (Joty et al., 2015), greatly improved the performance of discourse parsing in general.

it has been observed that the performance across the discourse relations varies significantly (Joty et al., 2015), and that poor performance may be linked to underfitting, i.e., a lack of training data (Feng and Hirst, 2014).


This method also reports a good overall performance with linear running time.

all these state-of-the-art discourse parsers still perform badly on infrequent relations due to insufficient training examples.


One solution is, of course, to create more labeled data, ideally for all the relations.

given the resources required for manually creating labeled data for discourse parsing, we explore in this paper a training data enrichment framework that relies on co-training of the CODRA parser and the SR-parser on unlabeled documents.


The construction of our discourse structure looks similar to the building of an RST tree (Duverle and Prendinger, 2009), and there are also prior efforts in combining the benefits of RST and PDTB (e.g., when building a Chinese discourse corpus (Li et al., 2014)).


The sentiment associated with any aspect can mostly be inferred by checking the terms associated with it (e.g., awesome -location).

this may not be very beneficial if the statement is ambiguous.


The grouping method discussed later in this paper is inspired by their work.

these works are still constrained to correlations between adjacent labels only.


L{x} : Undefined links, where a clear connection between two units may not exist.

sentiment flow (or transition) can still exist.


In this paper, we have focused on method to incorporate non-local context information into input representation of aspect units.

such method makes i.i.d assumption for output labels.


For CRF (Level-2), features f 1 to f 10 are used.

prediction is made over full sequence of output labels and features F(U 1 ) to F(U N ) are fed together.


This may seem counter-intuitive since a CRF should be able to model inter-label dependencies well.

we wish to emphasize that the experiment with CRF is not aimed at comparison against SVM, but to check the performance of available CRF tool on review data.


For crfsuite, the full sequence of aspect unit features is fed as input, and full sequence of output labels is predicted at once.

in the SVM (level-2) model we form new groups (or modify existing group information) as new labels are predicted.


Therefore, the source text and the response text are casted respectively as two views in a co-training algorithm to perform semi-supervised learning.

the success of co-training largely depends on two strong underlying assumptions, i.e., sufficiency and independence, of the two views (Blum and Mitchell, 1998), which are actually violated in reader emotion classification when the source text and response text are utilized as two views.


Even worse, as an extreme example, the source text (e.g., some newly posted news) sometimes has no response at all.

the response text normally depends on the source text, since both the response text and the source text talk about the same topics.


Generic sentiment classification is formulated as determining whether a piece of text is positive, negative, or neutral.

in SC, systems must detect favorability toward a given (pre-chosen) target of interest.


Our discriminative model works effectively for supervised stance classification tasks.

manual annotation requires painstaking work by researchers, which can be even more difficult for tasks such as sentiment annotation .


A baseline that completely ignores sentence structure, as well as words that have no intrinsic polarity, is shown in Figure 3(b): the only two words left are negative and, hence, the total polarity is negative.

the syntactic tree can be re-interpreted in the form of a 'circuit' where the 'signal' flows from one element (or subtree) to another, as shown in Figure 3(c).


Several recent methods are proposed to extend word representation to phrases (Yin and Schütze, 2014;Yu and Dredze, 2015;Passos et al., 2014).

they do not use structured knowledge to derive phrase representations.


The overcomplete word embeddings Z strongly differ from the word embeddings X; hence, the denoising is affected.

the performance of the vectors Z * still outperforms the original vectors X, A and B after the denoising process.


The accuracy of synonymy detection increases sharply from 63.2% to 88.6% according to the number of stages T from 0 to 3.

the denoising performance of vectors falls with the number of stages T > 3.


On the SemEval corpus, it achieves an accuracy of 74.8, outperforming all participants in the original shared task (Section 5).

these results are limited by the small size of both training sets.


Some such documents are written by professionals and contain well-formed, explicit arguments-i.e., propositions supported by evidence and connected through reasoning.

informal arguments in online argumentative discourses can exhibit different styles.


This extended approach employs conditional random fields (CRF) using dictionary-based features along with all the features from the original technique.

it resulted in lower accuracy than the SVMs.


CoVAR Sp : proposed technique similar to CoVAR Fro .

the distance between two matrices is determined using the spectral norm.


Hence the context model does not see enough data for learning and hence, if the learnt context model is fed directly to the hidden state of the target RNN, the improperly learnt context model can play a big role.

if a Concat kind of architecture is used, the linear plus softmax layer can decide on how much importance to give to the context model.


In fact, the winning architecture was Conditional-State with RNN-RNN combo, which did better in terms of test accuracy than the feature based models (Bowman et al., 2015) and one tree-based model (Mou et al., 2016).

it came close to the stateof-the-art attention based model (Parikh et al., 2016).


In other words, the above theorem implies that if we add the conditions defined in Equation (4)) to Ng's theorem, it is possible to obtain a succinct representation (Equation (5)) whose predictive performance is nearly as good as that of Ng's theorem, albeit with a probabilistic penalty f 1 (m) and an error penalty f 2 (m).

the penalties are negligible because f 1 (m) = o(1) and f 2 (m) = o(1).


Distributional methods applied to large-sized, often temporally stratified corpora have markedly enhanced the methodological repertoire of both synchronic and diachronic computational linguistics and are getting more and more popular in the Digital Humanities (see Section 2.2).

using such quantitative data as a basis for qualitative, empirically-grounded theories requires that measurements should not only be accurate, but also reliable.


Here, we extend their work into the multimodal domain by comparing the performance between visual and linguistic representations at encoding different types of attributes.


The attributes were learned from both, visual and textual input.

with Silberer and Lapata (2014), here we aim at spotting differences on the fine-grain semantic knowledge encoded by vision and language instead of building multimodal representations and to use them in a task.


The classification of temporal relations between events in text has been long studied and attacked from different perspectives in the NLP community.

existing approaches heavily rely on information overtly expressed in text, such as explicit temporal markers (e.g.


predicate-argument structure, as features for the classifiers (Llorens et al., 2010;Laokulrat et al., 2013;D'Souza and Ng, 2013).

the evaluation results of TempEval-3 (UzZaman et al., 2013) show that a system with basic morphosyntactic and lexical semantic features, such as ClearTK (Bethard, 2013), is hard to beat even if using more sophisticated semantic features.


These studies model a type of CSP: a subject or an object can be regarded as an additional context to restrict a set of possible fillers of a query predicate.

the context captured in our study is not a local context of a query predicate but that of a query argument, working as a validator of the narrative consistency between a query predicate and events in which a query argument participates (see Section 4).


In Section 6.2, we reported the results of a binary classification task in which negative instances were artificially generated.

this section describes a more realistic task setting: ranking coreference clusters in the OntoNotes corpus (Hovy et al., 2006).


Based on the growth rate of SP and SP-CW12, we conjectured that we needed 10 3 times more instances of TYPE A for SP to reach the same performance as that of CSP.

it is inefficient and unrealistic to increase the number of training instances of TYPE A in terms of the training time and availability of training data.


For example, for the test pronoun it in ⟨you, own, it⟩, the CSP model can rank the correct antecedent ⟨you, buy, something⟩ obj at the top by capturing the narrative consistency between buy X-own X.

the context is attached to a correct coreference cluster in only 31.5% of the degradations.


By iteratively clustering adjectives on the basis of co-occurring nouns and vice versa, the hidden attributes connecting both can be crystallized out.

bigger gold clusterings and more reliable evaluation measures are still missing.


The models of the previous sections provide a variety of options for representing the meaning of a verb from its arguments.

none of these constructions takes into account the distributional vector of the verb itself, which includes valuable information that could further help in entailment tasks.


Predicate argument relations are usually marked by case particles denoting grammatical cases in Japanese, therefore identifying dependencies marked by the major obligatory cases, ga (nominative), wo (accusative) and ni (dative) is the main task.

since ellipses are ubiquitous in Japanese texts, arguments might be identified beyond the sentence including the target predicates (inter-sentence arguments) as well as within the sentence (intra-sentence arguments).


Concerning the directly dependent ga argument cases, the FixRank shows the better recall value with the similar precision value.

in the cases of directly dependent 'other' case markers, the BiReg shows the better precision value.


the directly dependent "better negative examples", to obtain 428 candidates out of 1,288 candidates (= 184 texts × 7 annotators) that make 33% of the total training examples.

the number of the correct arguments that directly depend on the target predicate is 35 out of 184 examples, making 19% of the texts.


Among them, only the attention mechanism can make the two sentences contact with each other.

the word-by-word attention does not represent a better understanding of the sentences.


For instance, on the clean-text Microsoft Paraphrase benchmark database, the existing systems attain an accuracy as high as 0.8596.

existing systems for detecting paraphrases and semantic similarity on user-generated short-text content on microblogs such as Twitter, comprising of noisy and ad hoc short-text, needs significant research attention.


• The best known system, designed by (Ji and Eisenstein, 2013), currently delivers an F1-score of 0.8596 on the Microsoft Paraphrase dataset, while ours delivers 0.825, without any adaptation.

our system at an F1-score of 0.741, delivers a significantly better performance compared to their system with an F1-score of 0.641 on the Twitter data, also without any adaptation.


the parents "inherit" the alignment from their children.

this does not mean that the direction of entailment needs to be the same; in Figure 7, "Super PACs" entails "campaign funding" (t 1 s 1 t 2 ), while "X criticizes Y on Z" entails "X disagrees with Y over Z" (π(t 2 , s 2 ) s 2 π(t 1 , s 1 )).


Residual connection is added after every n layers.

for stacked LSTM, n > 3 is very expensive in terms of computation.


Figure 4 and the results from Table 5 suggest that perplexity is a good loss function for training paraphrase generation models.

a more ideal metric to fully encode the fundamental objective of paraphrasing should also reward novelty and penalize redundancy during paraphrase generation, which is a notable limitation of the existing paraphrase evaluation metrics.


A straightforward way to enrich Chinese KB is to directly translate English KB (source) to Chinese (target) based on the surface texts of a triple with existing machine translation system.

we find that they suffer from the problem of ambiguity.


(2013) compiled an initial analysis of the questions in these datasets, and identified 7 broad categories of knowledge and inference requirements.

this analysis forced a single knowledge type for each question, for example causality, and from our detailed analysis we find that many types of knowledge are necessary to arrive at the correct answer, e.g., causality, actions, and purposes.


One reason could be SDA and SDA-DSS separate domain-dependent and domain-independent features and keep all features in the learned representation, while our model suppresses domain-dependent feature.

in general, "domain-dependent" is a relative definition.


In order to determine the scope of the negative phrase, most computational approaches make use of a syntactic parser.

for languages suffering from a lack of such resources, such as Norwegian, this strategy would be too expensive.


This feature combination includes all three types of COTs.


Integrating such models with human interaction enables many new use cases.

adding human interaction to probabilistic models requires inference algorithms which are both fast and accurate.


We also note that variational inference is inappropriate for this model, as it only achieves good performance when used in conjunction with hyper-parameter optimization.

such optimization tends to undo the constraints, rendering the model useless (Hu et al., 2011).


These measures can be used by experts or individuals for diagnostic and management purposes, but also in aggregation, for large scale surveys.

the reliance on self-reporting required to obtain these measures is time consuming and expensive and can only produce sparse data on small populations.


Such approaches have been employed in various tasks including sentiment (Wang et al., 2014) and emotion analysis (Poria et al., 2015;Wimmer et al., 2008) and benefit from the ability of the learning model to capture the semantic relations between different modalities; however, the resulting features are treated in the same way by the learning model (Akbari et al., 2015).

in late fusion approaches different models are trained per modality and their outputs are combined at a later stage, usually by employing weighted sum (Dobrišek et al., 2013;Poria et al., 2016).


The lowest errors are observed with respect to the negative target (the comparison with the well-being case in terms of the error is not straight-forward, due to the larger scale that is used in WEMWBS).

r 2 for this task is considerably lower, pointing to the low variance in each model's prediction.


Most of these methods depend on sparse lexical features including bag-of-word (BoW) models and exquisitely designed patterns.

feature engineering is labor-intensive and the sparse and discrete features cannot effectively encode semantic and syntactic information of words.


More recently, Gong and Zhang (2016) propose an attention-based convolutional neural network, which incorporates a local attention channel and global channel for hashtag recommendation.

to the best of our knowledge, there is no work yet on employing both topic models and deep neural networks for this task.


The feature vector of a composed component such as a pair, pair(x1, x2) is usually described by the local features of x1, x2 and the relational features between them, such as the order of their position , etc.

the main label set for SRL is l = {l ispredicate , lisargument, largmenttype}.


The availability of manually annotated corpora of high quality (or, at least, reliability) is therefore key to the development of the field in any given language.

the creation of such resources is notoriously costly, especially when complex annotations, e.g.


The proposed tasks all relate to semantic disambiguation (noun vs verb, co-reference identification, named entity annotation, etc) and while some are relatively easy -like Play Twins (noun vs verb) or Play Names (named entity annotation) -most require some more advanced (at least school-level) knowledge.

the interface does not provide a training phase and the only help available is a short guide to the task.


(In the Czech Republic, the presentation of syntax in school is very close to dependency syntax.)

the authors report in (Hana and Hladká, 2012) that the accuracy of the annotations they obtained is significantly lower than that of their parser.


The other way is to train a suitable deep learning model (Collobert et al., 2011;Mikolov et al., 2013a) on a raw corpus in that language and then use the obtained embeddings of these in-language words as input to the sentiment classification model.

learning context-rich word embeddings in any language requires large datasets, generally of the order of billions of words, thereby eating up a lot of time as well as system resources.


Generally, a large annotated corpus is able to overcome this anomaly, as the network gets enough data to learn the correct linguistic patterns.

when the labeled dataset is small in size, as is often the case in most languages, this problem adversely affects the classification performance.


The success of NMT lies in its strong ability of composing the global context information.

as a newly approach, the NMT model has some flaws and limitations that may jeopardize its translation performance (Luong et al., 2014;Sennrich et al., 2015;.


This is partly because that the name entity usually occurs rarely in the training corpus and is often mapped to an unknown word by the RNN search model.

in character-aware NMT model, the name entity is split into a sequence of characters and each character can be found in the vocabulary.


(2015), where they treat word alignments as a constraint to the RAE model.

as discussed in Section 1, the composition criterion in RAE (i.e.


On the one hand, we simultaneously maximize the Score con ( * ) and minimize the Score inc ( * ) of SAC nodes.

we take an opposite approach to deal with non-SAC nodes.


In neural machine translation, the attention mechanism facilitates the translation process by producing a soft alignment between the source sentence and the target sentence.

without dedicated distortion and fertility models seen in traditional SMT systems, the learned alignment may not be accurate, which can lead to low translation quality.


In addition, the attention in NMT is learned in an unsupervised manner without explicit prior knowledge about alignment.

2 in conventional statistical machine translation (SMT), it is standard practice to learn reordering models in a supervised manner with the guidance from conventional alignment models (Xiong et al., 2006;Koehn et al., 2007;Bisazza and Federico, 2016).


Evaluating the quality of output from language processing systems such as machine translation or speech recognition is an essential step in ensuring that they are sufficient for practical use.

depending on the practical requirements, evaluation approaches can differ strongly.


It can be used to predict arbitrary quality metrics, provided that suitably labeled training data is available.

in practice automatic QE has been found difficult and not reliable enough when applied to new domains or unknown systems, e.g.


Because automatic scores are available for all segments, the sample size is much bigger.

the regressor used for prediction is often biased to predicting values p i close to those indicated by the training data, which may differ considerably from the true values for y i , especially for documents that are less similar to the training data.


Based on these definitions, we observe that Q (man) is subject to high variance, because the small sample size makes it sensitive to the randomness of the data.

its bias is zero, since averaging over all randomness would cancel out estimation errors exactly.


The reasonable size of the label set makes it possible to have only one model.

we have one model per word for fine-grained WSD.


The cross-lingual mate relation in Wikipedia is a strong indicator for parallelism.

wikipedia entries in different languages are not necessarily translations of each other, but can be edited independently.


For most document pairs, only a single sentence pair was extracted.

there were a few document pairs that yielded several hundred sentence pairs.


Preordering English sentences into Japanese word order thus only involves two simple steps: (1) Finding the parse tree of the English sentence (the authors used HPSG derivations) and (2) moving the head of each constituent to the initial position.

this approach does not seem to scale up easily because manually encoding reordering rules for all the world's language pairs would be a rather difficult and very slow process.


The second technique is to train a self-normalized NNJM to avoid computation of the softmax normalization factor (i.e., the denominator in Equation 2) in decoding.

selfnormalization does not solve the computational cost of training the model.


With the surge of word embeddings trained by neural networks, recent approaches that learn bilingual word representations from non-parallel data for bilingual lexicon induction have also shown promise (Mikolov et al., 2013b;Vulić and Moens, 2015).

none of the existing methods explicitly considers multiple alternative translation, i.e., the phenomenon that one source language word may have multiple possible translations in the target language.


Following its monolingual counterpart (Mikolov et al., 2013c, inter alia), bilingual word representation learning has attracted considerable attention.

most of the works require parallel data as the cross-lingual signal (Zou et al., 2013;Chandar A P et al., 2014;Hermann and Blunsom, 2014;Kočiský et al., 2014;Luong et al., 2015;Coulmance et al., 2015), making them unsuitable for bilingual lexicon induction.


When inspecting the output of both splitters, we noticed that the hybrid/linguistic splitter is more likely to produce a correct splitting.

the corpus-based splitter provides better results for NMT.


We found that sentences were often restructured causing the performance to drop.

an example of this restructuring may look as follows: Source Sentence the new rules put in place will undoubtedly make it more difficult to exercise the right to vote in 2012 .


For investors such as venture capitals, information content should reflect a firm's intrinsic value, or potential of future growth.

measuring the information content of text can be challenging due to uncertainties and subjectivity.


A typical way of modeling a sentence is to treat it as a sequence and input the sequence to a long short-term memory (LSTM;(Hochreiter and Schmidhuber, 1997) model, which is capable of learning semantic features automatically.

information may present different impact to individual firms and therefore, we need a way to represent information conveyed in a news sentence depending on a specific firm.


In the previous example, a bidirectional model would output only one representation for the sentence no matter what the target firm is.

the proposed tree transformation algorithm outputs different binarized trees when different targets are given.


Over the last few decades, technology has provided us many powerful tools that have completely changed our daily routines.

one crucial area where technology has yet to have the significant impact suggested by its true promise is in education.


As can be seen, the student fails to address primary MMP 2 from the reference answer.

the student successfully understood the concepts expressed in MMPs 1 and 3.


Although much of the work done by c-rater has been automated in the past years (Sukkarieh and Stoyanchev, 2009), it still requires an appropriate set of responses that have  already been holistically scored by trained raters.

1 our approach is fully automated and can be used in a dynamic setting to recognize the focused relationships between a specific reference answer proposition and the student's response.


The UMASS system represents the documents as TF-IDF weighted vectors.

in order to compare documents in terms of their relations we need a relation-oriented representation.


(3) A much larger synthetic dataset (D3) of 106, 000 documents from Yahoo News is used in order to evaluate the scalability of our method.

this dataset is not annotated and thus is not used for evaluating the accuracy.


The improvement in Det min score is statistically significant for the dataset (D1) at the p < 0.05 level using a paired t-test.

for dataset (D2) all methods perform close to the best method (LSH-RelEntFSD) and so the results are not statistically significant.


This value can be estimated empirically by a pilot study before the item is used in an actual test.

reliable estimates of item difficulty require a substantial amount of test taker responses.


Therefore we initially hypothesized that the text complexity of a common listening passage would be a strong predictor of the item difficulty.

the empirical results did not support this.


As discussed earlier, the former are grouped into sets and are not sequenced according to difficulty of individual items.

even for simpler items the item sequence was not a strong predictor of item difficulty.


We also found that the most highly ranked features were related to item vocabulary, such as lexical frequency of the words as well as the level of concreteness.

the system based on vocabulary features performed worse than the system based on all features.


The appropriate use of collocations is a challenge for second language acquisition.

high quality and easily accessible Chinese collocation resources are not available for both teachers and students.


Thus a better strategy should incorporate richer annotation of corpus and take more consideration of Chinese grammatical features.

to definitions of collocation in the previous studies, We introduce function words into our collocation study, and identify four characteristics of collocations: (1) can be word combinations with more than two words; (2) can contain both content words and function words; (3) collocated words can be either adjacent or non-adjacent; (4) collocated words must hold syntactic or semantic relations.


We count the collocations of the 19 entries in these resources and find OCCA is much higher than D1 and D2 in collocation quantities.

by analysing the collocation data, we also find that OCCA does not contain some collocations in D1 and D2, e.g. "


外交 (waijiao/diplomatic) 问题 (wenti/problems)", and "农村 (nongcun/village) 发展 (fazhan/develop)", mainly because of the domain and size limit of the CTC corpus.

the WIKI database could serve as a good supplement because it covers nearly 63% of the 278 collocations that cannot be retrieved in CTC.


To offer a better solution to event detection without the above limitations, we propose to use a novel text stream representation: Burst Information Networks (BINets) (Ge et al., 2016a;Ge et al., 2016b).

to the keyword graph which is based on word co-occurrence, a BINet is constructed based on burst co-occurrence.


So it is easy to enlarge the size of the corpora by mixing the data of the same emotion category from two corpora.

corpus fusion for emotion classification is challenging due to the following two factors: First, the emotion taxonomies are often different between two emotion corpora because of the lack of an accepted standard.


The other nodes correspond to ADUs, and the edges to SUPPORT or ATTACK relations, where the latter distinguish between REBUT (challenging the validity of the assertion in the ADU) and UNDERCUT (challenging not the validity of individual ADUs, but the supposition of a SUPPORT relation between two).

to an RST analysis, there is no constraint that only adjacent segments may be conjoined by a relation.


by using around thousands of dependencies.

if more than ten thousands of dependencies are added to the vocabulary, the performance tends to slightly drop.


Moreover, since consumption level is an important factor of economic status (Stutzer, 2004), effective prediction of consumption level will facilitate social economic researches on social media.

unlike the gender attribute that is displayed on one's page, or political orientation that is unequivocally stated in one's tweets, consumption behavior related attributes are hard to acquire automatically from microblogging service.


Intuitively, it is straightforward to generate two rankings of users, either by average spending or by topic preference.

it is noted that ρ is sensitive to small value differences of both measures, and it will be difficult to obtain robust correlation values in this case.


Thus, it is highly desirable to rapidly generate meaningful labels for a topic word distribution while assign labels for new emerging topics as correctly as possible.

to the best of our knowledge, no existing method has been proposed to satisfy the demand, except that the simplest method which uses top-n words in the distribution to interpret a topic.


In most existing research effects on statistical topic modeling, people generally either select top words in the distribution as primitive labels (Blei et al., 2003;Ramage et al., 2009;Ramage et al., 2011), or generate more meaningful labels manually in a subjective manner (Mei et al., 2006;Mei and Zhai, 2005).

extracting top terms is not very useful to interpret the coherent meaning of a topic (Mei et al., 2007).


Since the time for each iteration grows with the number of documents, the total time for BLLDA grows fast.

oLLDA only processes the new coming document and update parameters.


For seq2BF without content introducing, we obtain low scores, showing that artificially splitting a sequence into two parts itself is not a fancy way of modeling natural language sentences.

given a keyword predicted by PMI statistics, the backward and forward sequence generation can significantly improve the dialogue system in comparison with pure seq2seq.


The coupling formulation is inspired by the Multimodal RNN that generates textual description from visual data .

unlike Karpathy's approach, we do not allow the feature that is generated by the CNN component to diminish between distant timesteps (i.e.


Our framework produces comparative results in metadata extraction tasks.

it significantly outperforms state-of-the-art systems in structural information extraction tasks.


The results of our initial analyses indicate that the flexibility offered by our tool is important for gaining important insights into the data: higher level analyses and visualisation can reveal patterns not visible in a simpler point visualisation (as in the case of theštošta variable).

point visualisation can serve as a good tool for manual checkups of the reliability of both the extracted data and higherlevel analyses.


Some patterns of such relations would certainly be relevant for an analysis of argumentation strategies, like the rebuttal of a common ground that seems to counter the author's stance.

similar to the low number of attack relations in persuasive essays (Stab and Gurevych, 2014), we found few insightful patterns of this kind in editorials.


The first unit could be interpreted as an implicit assumption about the reviews in the second unit, say, that the review is corrupt or hard to understand.

it could also simply be seen as an interjection not belonging to any argument.


so far, previous work has not investigated the overall correctness and consistency of the frame lexicon.


An English example for the verb turn is to turn something for TURN.01 and to turn into something for TURN.02.

we ask curators to create this form for TL verbs 2 .


And if a larger filter is used, the convolution can detector more features, and the performance may be improved, too.

the networks will take up more storage space, and consume more time.


We note that the dominance of the MFS in the training instances now drops to 30-40%.

this also results in very impressive performances on the LFS instances, between 33.33% and 47.62%.


Table 2 shows that, indeed, monolingual semantic evaluation performance is consistently positively correlated with (semantic) language similarity for BBA.

for CCA, correlation is positive in eight cases and negative in six cases; moreover, coefficients are significant in only two cases.


This idea also underlies very well-known lexical semantic resources such as the paraphrase database (PPDB) (Bannard and Callison-Burch, 2005;Ganitkevitch et al., 2013); see also Eger and Sejane (2010).

we directly use bilingual embeddings for this similarity measurement by jointly embedding p and , which are arguably best suited for this task.


This has the disadvantage that translation pairs need to be known, which typically requires large amounts of parallel text.

bilingual word embeddings, which form the basis of our experiments, can be generated from as few as ten translation pairs, as demonstrated in Zhang et al.


The performance of word embeddings can be drastically affected by their parameters (Levy et al., 2015;Lai et al., 2015), which prompts parameter searching for different tasks.

accuracy of solving word analogies also varies immensely for different linguistic relations (Gladkova et al., 2016).


The results of GloVe and Skip-Gram improve with LRCos as compared to 3CosAdd, but the simple average 3CosAvg works even slightly better for them.

sVD gets an over 15% boost from LRCos, but not from 3CosAvg.


Unfortunately, this means that word analogies fail to provide sufficient "context" to words: ideally, king:queen :: man:woman and king:kings :: queen:queens should profile sufficiently different aspects of the king vector to avoid the nearest-neighbor trap.


LRCos offers a significant boost in accuracy for detecting analogical relations over the most widely-used 3CosAdd method, including derivational relations where the latter does not perform well.

lRCos is by no means perfect, and there is room for further improvement, especially with respect to lexicographic relations.


we found GRUs to yield comparatively better validation data performance on semtags.


The convolutions used in this manner cover a few neighbouring letters at a time, as well as the entire character vector dimension (d c ).

to dos Santos and Zadrozny (2014), we treat a word analogously to an image.


The residual bypass effectively helps improve the performance of the basic CNN.

the tagging accuracy of the CNN falls below baselines.


(2016) system on semtags, we are substantially outperformed on UD 1.2 and 1.3 in this setup.

adding an auxiliary loss based on our semtags markedly increases performance on POS tagging.


Numerous studies address FrameNet's lack of lexical coverage (Pennacchiotti et al., 2008;Das and Smith, 2012;Pavlick et al., 2015).

little work has been done on extending frame relations except by Ovchinnikova et al.


We chose OPTED because is a public and free-access dictionary, based on Webster's Unabridged Dictionary, and an important and recognized dictionary.

we chose DRAE because it is the most authoritative dictionary of the Spanish language.


For example, a person or a bird is animate because they move or communicate under their own power.

a chair or a book is inanimate because they do not perform any kind of independent action.


For example, horse is normally animate, but a dead horse is obviously inanimate.

tree is an inanimate word but a talking tree is definitely an animate thing.


Our word animacy model achieved an F 1 of 0.98, whereas the prior state of the art achieved F 1 of 0.99 for marking inanimacy.

for marking animacy our model achieved F 1 of 0.90 where the state of the art achieved F 1 of 0.93.


(2017a) present the ZP-centered-LSTM architecture that learns to encode zero pronouns by their text words.

it could bring with some defects: their model regards all the words in the sentence equally, thus fails in capturing informative parts of the sentence.


On the base of Zhao and Ng (2007), Chen and Ng (2013) further investigate their model, introducing two extensions to the resolver, namely, novel features and zero pronoun links.

these work deeply rely on annotation dataset.


All entities of Our 4 can be identified from the context of this dialogue so it is annotated with the known entities Jack and Judy.

only one of you 7 can be identified in this context so it is annotated with the known entity Ross and also OTHER, implying that it refers to some other entity that can be identified in a separate dialogue.


The entity-mention models try constructing representations of discourse entities, and associating different mentions with the entity representations (Luo et al., 2004).

none of these model types consider more than two mentions together at the low level.


In most cases, we have dozens of mentions in an article, which is not an issue.

some long articles have hundreds of mentions, so generating all triads is unpractical and unnecessary.


For testing, we consider the mentions with distances up to 40.

this does not mean the long-distance coreference can never be detected.


In principle, we would like distance metrics to have 0 as the minimum, which can be achieved by subtracting 1.

for the purpose of clustering, it is not necessary.


Our system performs by far the best with the MUC evaluation metric, and is also the best with B 3 metric, measured with F1 score.

the performance is quite low from the CEAF φ4 metric.


From a text, system cr1 identifies two entities {the American administration, it 1 , it 2 , it 3 } and {they 1 , they 2 , them, their}.

it misses the fact that the two are actually the same entity.


Our intuition is cr1 does a better job, because it resolves much more coreference relations.


Paradigms can extend the attested morphological forms from few but high frequency words to low frequency words, likely the majority, for which there is little data.

high quality paradigms may prove effective at detecting spurious morphological relations between words that have plagued many previous models.


This is due to fact that the transformation rules can well capture the morphology of English and thus significantly increase the true positive segmentations.

the transformation rules increase the over-segmentation problem for Turkish and Finnish as indicated by the decreased precisions.


Indigenous languages of the American continent are highly diverse.

they have received little attention from the technological perspective.


Regarding parallel corpora, the bible is a common source that contains translations to many of these languages, although it is not always straightforward to extract the content in a digital format.

there are some projects that offer parallel content through a web search interface, e.g., Axolotl (Spanish-Nahuatl parallel corpus) that was mainly gathered from non-digital sources (books from several domains), the documents have dialectal, diachronic and orthographic variation (Gutierrez-Vasques et al., 2016).


In NLP, lemmatization and stemming methods are used to reduce the morphological variation by converting words forms to a standard form, i.e., a lemma or a stem.

most of these technologies are focused in a reduced set of languages.


One way to overcome this is to include linguistic information, e.g., morphology, syntax.

this kind of knowledge or linguistic tools is not always available, especially for low-resource languages.


Keyword heuristics have also been used to overcome language and domain barriers using bilingual dictionaries (Szarvas, 2008;Tran et al., 2013).

a weak bilingual dictionary could result in low coverage with this method.


This causes the model to be able to learn more unique contexts for SF types, thereby increasing recall.

the annotated dataset in the target language ( A ), being made from the output of the NN model on a separate dataset, will mostly help the model to do better recognition of false positives, thereby improving precision.


The additional English data apparently allowed the NN model to find a strong correlation between the crime violence class and the terrorism class, which is consistent with our intuition.

the NN model fine-tuned on the Tigrinya annotations apparently found the crime violence and terrorism classes tend to occur alone (last column of Figure 3c).


(2015) constructed a subject-shared predicate network with an accurate recognizer of subject-sharing relations and deterministically propagated the predicted subjects to the other predicates in the graph.

this method is applied only to subject sharing, so it cannot take into account the relationships among multiple argument labels.


(2017) used Grid RNN to incorporate intermediate representations of the prediction for one predicate generated by an RNN layer into the inputs of the RNN layer for another predicate.

in this model, since the information of multiple predicates also propagates through the RNNs, the integration of the prediction information is influenced by word order and distance, which is not necessarily important for aspects of syntactic and semantic relations.


The model is, however, not applicable to tasks with structured output such as trees or graphs.

to POS tagging or NER where we try to detect annotation errors in the predicted labels for individual tokens, when looking for errors in parse trees we have to deal with directed, labelled relations between nodes, and changing the relation or label between two nodes in the tree usually requires the adjustment of other attachment and labelling decisions in the same tree.


Generating trees Once we are done with error correction, we want to output the trees.

what we get from the variational model are local decisions for individual labels and edges, and it is not straight-forward to generate connected trees based on these decisions.


Please note that we did not optimise the parsers on the data and thus the comparison should be taken with a grain of salt.

it is interesting to see that there is not one best parser but that the different parsers (excluding the 'vintage' Malt system) all yield results in the same range and the best performing parser varies depending on the dataset.


The authors report a high precision for automatic error correction.

their method only addresses attachment errors but does not handle label errors.


Ideally, our model assigns a high probability to this span.

the span [0, 3] is not a valid constituent span and our model labels it with 0.


The use of RNN for the evaluation of grammaticality can already be found in Tomida and Utsumi (2013) (a reply to a non-NN learning model in Pearl and Sprouse 2013).

their RNN (a Jordan RNN, Jordan 1997) worked on abstract data (it was trained on preassigned constituent labels and had to generate other label sequences), a choice that in our opinion requires too many underlying assumptions.


The fact that the ungrammatical cases were graded, with pronouns next best after gaps (see Table 3) shows that the network wasn't just using the simple rule "If it starts with Wh*, pick the version with fewer words, if not, don't".

the overwhelming effect of processing factors like the level of embedding (Figure 2), and the fact that the apparent success of the NN in the island task in not based on the island extraction effect itself cast doubts on the idea that the NN is using an abstract dimension of 'grammaticality'.


As is well-known to anyone who has practiced a musical instrument, pronounced tongue-twisters or read centrally-embedded sentences, processing difficulties improve a lot with practice.

multiple repetitions of Who did John see Mary?


The states that improve the most over baseline, with 15% improvement or more using only textual features are Oregon, Oklahoma, Tennessee, D.C., South Carolina, Louisiana (lower), Georgia (lower), and Alabama (lower).

text is least predictive in Connecticut, Wyoming, Idaho, New Jersey, Utah (upper), New Hampshire (upper), North Dakota (upper), all underperforming the baseline.


Because our goal is to investigate whether a model can learn to search through a document, it is important that a non-negligible fraction of the questions require navigation through the document.

in Wikipedia each document starts with a preface that summarizes the document, and thus often contains the answer.


We observe, compared to TRIV-IAQA, that the first occurrence of an answer is much more spread out across the document, and that the median increases from 3 to 14, which will require more navigation from the agent.

even in TRIVIAQA-NOP answers tend to appear at the beginning of the document, because document content is usually organized by importance.


Another thrust has focused on skimming text in a sequential manner (Yu et al., 2017), or designing recurrent architectures that can consume text quickly (Bradbury et al., 2017;Seo et al., 2018;Campos et al., 2018;Yu et al., 2018).

to the best of our knowledge no work has previously applied these methods to long documents such as Wikipedia pages.


Co-occurrence probabilities of such events play important roles for predicting Conjunction.

there are few connectives can be used for pursuing the co-occurred events or they are extraordinarily general.


A connective like "than" is effective to signal a pair of Contrastive events.

due to the limitation of literal expression, in general, such event mentions aren't directly connected by "than", but instead the concrete elements in the events are.


Note that when α = 0, the regularizer is disabled and the output probability distribution is very noisy and neighboring values have large variance.

when the regularizer is properly enabled (α = 0.16), observe how the output probability distribution is much smoother and neighboring probability values are more similar.


On short documents NEO is easily misled due to lack of effective sample size.

observe that as the length of the document increases NEO's error reduces significantly (note Table 3 that for 2000 word documents on COHA-FICTION the mean absolute error is now 24.80).


We can see that an equation system includes one or more equations and it is a solution for a specific math word problems.

an equation template can correspond to several math problems.


The output equation structure is correct by the basic model.


(2017) apply a standard seq2seq model to generate equations under the constraint of one variable.

the model is prone to generate numbers that do not exist in the problem or in wrong positions.


This algorithm applies a personalized complex word identification (CWI) model in two steps in the LS pipeline: • CWI for detection: Most current approaches deploy a user-independent CWI model as the first step in their pipeline to detect words that should be simplified (Section 2.1).

we train a personalized CWI model for this purpose, such that the choice of target words can vary from one user to another.


This configuration also yielded the highest precision (18.01%), outperforming the baseline by almost 10%.

the highest accuracy (50.91%), similar to the first experiment, was obtained by using automatic CWI for ranking only (detect=nil, rank=auto).


Adding POS information benefits the SimVerb and SimLex verb performance, which can be attributed to the coarse disambiguation of verb-noun and verb-adjective homonyms.

the type.POS targets show a considerable performance drop on SimVerb and SimLex verbs.


We have introduced word class suggestion as an evaluation benchmark for word embeddings.

the WCS output might be used in vocabulary-based application scenarios, e.g.


Paetzold and Specia (2016a) claim that their approach outputs customized simplifications depending on the user's profile, and evolves as users provide feedback on the output produced.

they provide no details of the approach they use to do so, nor do they present any results showcasing its effectiveness.


Our ETD framework admits any means of identifying the keyphrases, so any keyphrase generation algorithm can be employed (e.g., TextRank).

since our task is slightly different in that we want to generate keyphrases for overall trend detection of an area (as opposed to the more typical characterization of a single publication), we need to introduce several refinements.


We can see that in this example the premise entails the hypothesis, and in order to correctly identify this relation, one has to know that the word kettle entails the word pot.

if we train a neural network model on a set of labeled sentence pairs, and if the training dataset does not contain the word pair kettle and pot anywhere, it would be hard for the learned model to know that kettle entails pot and subsequently predict the relation between the premise and the hypothesis to be entailment.


However, if we train a neural network model on a set of labeled sentence pairs, and if the training dataset does not contain the word pair kettle and pot anywhere, it would be hard for the learned model to know that kettle entails pot and subsequently predict the relation between the premise and the hypothesis to be entailment.

from WordNet we can easily find out that pot is a direct hypernym of kettle and therefore kettle should entail pot.


Given the advantages of deep learning approaches to RTE, it is therefore desirable to incorporate WordNet knowledge into deep learning based solutions to RTE.

it is not immediately clear how WordNet knowledge could be easily brought into neural network models.


Here we use two different ways to perform the comparison, depending on whether the entailment vectors are learned using the standard neural network model or the set-theoretic model.

if the standard neural network model is used to learn the entailment vectors, we simply concatenate all the vectors to perform comparison as follows: if the entailment vectors are learned using the set-theoretic model, because of the special properties of the entailment vectors as explained in Section 2.1, we perform comparison as follows: where ⊙ is element-wise multiplication of two vectors.


This demonstrates that without observing the necessary word pairs from the training set, it is hard to make the right predictions on the test set.

using entailment vectors performs better in the non-overlap split setting.


Text representation learned with relevance information captures relevance rather than term proximity, which clearly accounts better for IR requirements .

supervised signals such as click-through data are often limited outside of large industrial research labs, probably due to user privacy concerns.


They show, that the typical aggregation of word frequencies across documents are less informative than richer representations including frequency standard deviations.

to English, research on readability assessment for other languages, such as German, is more limited.


Grounding our work on those previous findings, we go even further and hypothesize that the background knowledge required during text reading, and which impacts the conceptual complexity of the text, can be estimated by analyzing such networked knowledge resources.

news articles are well known for their abundance in named entities, therefore semantic networks based on lexical resources cannot cover much of the required background knowledge for news understanding.


The problem has been extensively studied and a wide range of features has been explored (Stamatatos, 2013;Schwartz et al., 2013;Seroussi et al., 2013;Hürlimann et al., 2015).

there has been a lack of analysis of the behavior of features across multiple datasets or using a range of classifiers.


Notice that the lack of available models has been explicitly mentioned, in a recent work, as the cause for the missing comparison of this technique with other competitors (Raganato et al., 2017b, footnote 10).

we present other experiments to shed more light on the value of this and similar methods.


This system uses an SVM to train classifiers for each lemma using only annotated data as training evidence.

graph-based WSD systems do not use (un)annotated data but rely on the synset relations.


As might be expected, a bigger corpus leads to more meaningful context vectors and therefore higher performance on WSD.

the amount of data needed for 1% of improvement in F 1 grows exponentially fast (notice that the horizontal axis is in log scale).


As pointed out at the beginning of Section 5, our biggest models take months to train, making training multiple versions of them impractical.

we trained our smallest model (h = 100, p = 10) ten times and our second smallest model (h = 256, p = 64) five times and observed that as the number of parameters increased, the standard deviation of F 1 decreased from 0.008 to 0.003.


Let us assume that for a B i that is classified as relevant in the first step, the source page pointed to by B i has m references R 1 , R 2 , ..., R m .

all of these m references might not be appropriate to inherit.


We use 70% of the Wikipedia pages for training and rest 30% of them for testing using the gold standard dataset.

we observe from the gold standard dataset that on average only 10% wikilinks are suitable for reference inheritance while the rest 90% are irrelevant.


This means that P@10 would never become 1, and also explains why R@10 is greater than P@10.

some users have more than 10 books in their test set, making the R@10 very low even for an ideal system.


Table 1: Examples of simplification in PorSimples.

we know that even complex texts have simple sentences, what makes it difficult to identify precisely where complexity lies.


The corpus PorSimples contain a lot of explanation relating to difficult words (this is a simplification strategy to deal with lexical complexity).

once explained, the difficult words are repeated along the text.


In this way, it can extract the dependency relations in the passages and compare semantics in different granularities.

they adopt the tree-LSTM to bottomup generate the intermediate representations, and then directly compare each unit with the merged representation of the other passage.


On one hand, the decent precision shows potentials of the KG-Net to capture patterns with entity-sequence and KG information.

we suggest that the weakness of the KG-Net might be caused by the sparsity of entity-sequence space (the dataset scales down after word-to-entity mapping), and we can enhance it by exploring other information such as relational paths (Lin et al., 2015;Zeng et al., 2017); (2) Corpus-Net achieves comparable results with CNN+ATT and PCNN+ATT, which reveals the effectiveness of Corpus-Net that could be the backbone of the CORD framework; (3) The CORD outperforms other methods on most recall area, demonstrating the effectiveness of our methods.


We evaluate P@N of the KG-Net without rules p(S e ) and compare the rule-projected p (S e ) on two kind of sentence amounts setup in  From Table 1, we can observe that the KG-Net gets lower precisions (average reduces 0.9%) on the whole test data comparing with the filtered data.

the KG-Net with rules gets higher precisions on the whole test data because it can deal with noisy instances effectively in sentence-level and hence be more robust on long tail situation.


Among them, distant supervision is popular as it is efficient to obtain large-scale training data automatically.

it suffers from noisy labeling problem which severely degrades its performance.


Recently, deep neural networks (Zeng et al., 2015;dos Santos et al., 2015) and attention mechanism (Wang et al., 2016) show the effectiveness in relation classification.

the training of neural network relies on large-scale labeled instances.


Different from answer retrieval from knowledge bases, this type of studies used vector to represent QA pairs and compare the distance in vector space to match answer text (Tan et al., 2015;Feng et al., 2015).

the performance of these neural models greatly depends on a large amount of labeled data.


A recent hot comprehensive QA task is the SQuAD challenge (Rajpurkar et al., 2016) which aims to find a text span from given paragraph to answer the question.


Sentence simplification aims to improve readability and understandability, based on several operations such as splitting, deletion, and paraphrasing.

a valid simplified sentence should also be logically entailed by its input sentence.


Evaluation Metrics Following previous work (Zhang and Lapata, 2017), we report all the standard evaluation metrics: SARI (Xu et al., 2016), FKGL (Kincaid et al., 1975), and BLEU (Papineni et al., 2002).

several studies have shown that BLEU is poorly correlated w.r.t.


Next, we find that our multi-task model also has low match-with-input scores (2% exact match, 9% BLEU, 38% ROUGE), similar to the behavior of the ground-truth references.

dRESS-LS (and pointer baseline) model is generating output sentences which are substantially closer to the input and hence is not making enough changes (14% exact match, 43% BLEU, 68% ROUGE) as compared to the references (which explains their higher adequacy but lower simplicity scores).


We train models from scratch for Newsela and WikiSmall (using Adam (Kingma and Ba, 2014) optimizer with learning rate of 0.002 and 0.0015, respectively).

because of the large size and computation overhead for WikiLarge, we first pre-train both main and auxiliary models on their own domain until they reach 90%


departure city, arrival date, are annotated to the expressions in the user utterances.

the utterances in real dialogues include information that does not always directly correspond to database fields but provides useful information for constructing database queries.


Table 4 suggests that the performance of the evidence span identification has a significant impact on that of the DB field classification in the RCNN-model.

table 4 also shows that there are cases where the RCNN-model can correctly classify the DB fields even though it fails to identify the correct evidence spans.


The model seems to have learnt this collocation.

there are only 40.4% of test cases where the word just after a question mark belongs to the evidence span.


(2017) propose an attention-based neural network for charge prediction by incorporating the relevant law articles.

charge prediction is still confronted with two major challenges which make it non-trivial: On June 24, 2015, the defendant pry open a company employee dormitory door into room, stole 2 mobile phone, a wallet and a tablet computer.


There are several existing QA systems that answer factual questions with short answers (Iyyer et al., 2014;Bian et al., 2008;Ng and Kan, 2015).

systems which attempt to answer questions that have long answers with several well-formed sentences, are rare in practice.


Most of the existing works either focus on better representations for questions or linguistic information associated with the questions.

the model proposed in this paper is a hybrid model.


This was built by analyzing the TREC questions.

to Li and Roth (2002), along with TREC questions we also make a thorough analysis of the most recent question answering dataset (SQuAD) which has a collection of more diversified questions.


The accuracy of the final SQL statements is very low compared to the sketch accuracy, which means that the main difficulty with the WikiSQL dataset is correctly instantiating the SQL sketch with the appropriate columns names and constants thus resembling a slot filling task.

the sketch accuracy for SENLIDB is significantly lower compared to WikiSQL.


As WikiSQL contains data from a simplistic database schema with artificial constraints for the SQL statements, it resembles a slot filling task.

the main challenge for solving the more complex queries in SENLIDB is the generation of the correct SQL sketch corresponding to a query.


A discourse unit or a paragraph could contain a larger number of words, and it will lead to generating an enormous matching matrix.

the number of training samples that can be used in our model is relatively small, which results in great difficulty with training the parameter .


Especially through many neural network methods used for this task such as convolutional neural network (CNN) (Qin et al., 2016b), recursive neural network (Ji and Eisenstein, 2015), embedding improvement , attention mechanism (Liu and Li, 2016), gate mechanism , multi-task method (Lan et al., 2017), the performance of this task has improved a lot since it was first introduced.

this task is still very challenging with the highest reported accuracy still lower than 50% due to the hardness for the machines to understand the text meaning and the relatively small task corpus.


Character-level embeddings have been used widely in lots of works and its effectiveness is verified for out-of-vocabulary (OOV) or rare word representation.

character is not a natural minimal unit for there exists word internal structure, we thus introduce a subword-level embedding instead.


The standard approach (referred to as Most-used Split)   is to use sections 2-21 for the training set, section 22 for the development set and section 23 for the test set (Lin et al., 2009;Rutherford et al., 2017).

shi and Demberg (2017) argued that the standard test set was too small for a reliable evaluation especially when second-level classification was employed.


They warned that the gap could be an accidental feature of the PDTB annotation.

our results lend support to the hypothesis that the gap reflects an intrinsic feature of the discourse relations, or at least that of the PDTB's task specifications.


In the literature, a variety of models have been proposed to capture these dependencies in the context of SMT, such as cache-based language and translation models (Tiedemann, 2010;Gong et al., 2011), topic-based coherence model  and lexical cohesion model .

integrating inter-sentence information into an NMT system is still an open problem.


One might use the concatenation of two neighboring source sentences as input of RNNSearch to explore the information of the preceding sentence.

this will degenerate translation quality as shown in Table 2.


Setting the c a t to a random vector, the information from the pseudo preceding sentence becomes meaningless, and even has a bad or uncorrelated impact on the translation of the current sentence.

the drop of the performance is not as big as that of N M T ISG (+z=0).


One significant reason is that the embedding layer of subwords with large merge operations is not trained well, as described in Section 4.2.

our proposed model can make use of both large and small features for correct translations of such rare words.


Other works show that bidirectional long short-term memory (LSTM) neural networks and the encoder-decoder architecture (Sutskever et al., 2014) achieve comparable results with WFST based n-gram models that are considered state-of-theart (Rao et al., 2015;Yao and Zweig, 2015).

these works only studied grapheme-to-phoneme (G2P) conversion from English to standard English pronunciation sets, such as the CMU pronunciation dictionary (Weide, 2014).


The recent Tensor2Tensor Transformer architecture outperforms the WFST approach and the Seq2Seq approach on every language.

it is worth noting that the training time using Tensor2Tensor is between 5-8 hours using an AWS p3.2xlarge 11 instance type, which has a Tesla V100 GPU.


The Transformer architecture works by relying on a self-attention (intra-attention) mechanism, removing all the recurrent operations that are found in the previous approach.In other words, the attention mechanism is repurposed to compute the latent space representation of both the encoder and the decoder sides.

with the absence of recurrence, positional-encoding is added to the input and output embeddings.


In addition to reducing training and maintenance complexity of several single language pair systems, the two main advantages of multilingual NMT is the performance gain for low-resource languages, and the possibility to perform a zero-shot translation.

the translations generated by multilingual and zero-shot systems have not been investigated in detail yet.


In particular, for the multilingual and the zero-shot models, the gain is statistically significant.

the mTER and lmmTER scores are better for the Recurrent architecture; in this case, the outcome is misleading since the nine post-edits include those generated by correcting the outputs of the three Recurrent systems.


Overall, across all experiments, we see slight changes in the distribution of errors types.

increases or drops of specific error types with respect to the bilingual reference model show sharper differences across the different conditions.


For experiments in section 6, where we report the BLEU score for a vanilla model and several adversarially-trained models against different attackers, we use a decoder with a beam width of 4.

our white-box attacker uses a model with greedy decoding to compute gradients.


As expected, adversarially-trained models usually perform best on the type of noise they have seen during training.

we can notice that our FIDS-W model performs best on the Nat noise amongst models which have not been trained on this type of noise.


Each entity exists only once in the physical world.

this is different in our communication where: 1. certain surface forms are very prominent and others occur only rarely; 2. certain instances are very prominent and others are mentioned incidentally.


denoting the instance United States 462 times is reflected in both the form and the instance distributions.

these two distributions are only identical if the ambiguity and variance are both 1.


Entity linking (EL), mapping entity mentions in texts to a given knowledge base (KB), serves as a fundamental role in many fields, such as question answering (Zhang et al., 2016), semantic search (Blanco et al., 2015), and information extraction (Ji et al., 2015;Ji et al., 2016).

this task is non-trivial because entity mentions are usually ambiguous.


The second drawback of the global approach has been alleviated through approximate optimization techniques, such as PageRank/random walks (Pershina et al., 2015), graph pruning (Hoffart et al., 2011), ranking SVMs (Ratinov et al., 2011), or loopy belief propagation (LBP) (Globerson et al., 2016;Ganea and Hofmann, 2017).

these methods are not differentiable and thus difficult to be integrated into neural network models (the solution for the first limitation).


Learning a reliable model/program would require higher manual effort in STG and more training data in CRF.

lUSTRE, with small human input, achieves high precision and recall for both in-domain and out-of-domain data.


ERLearn-CIKM already identifies a significant subset of the true links in such a sparse space that it is non-trivial for ERLearn-LUSTRE to have found additional 45 true links.

the ER task is more challenging for the Crystal scenario, with more matching functions (158 vs. 68) (Qian et al., 2017).


As shown in Figure 4, the percentage consumption of the sentence pool to reach up to 0.95 F-Score follows almost a linear growth.

this cost grows exponentially if we want to reach 1.0 F-Score (needing on average around 33% more sentences).


(2008) propose AL-based annotation systems.

their work differs from ours in the following ways: (1) we propose a more accurate online evaluation method than theirs, (2) we use ESE to bootstrap the learning framework collaboratively with a user-in-the-loop, and finally, (3) we provide auto-annotation modes which reduces the number of sentences to be considered for annotation and so allows for better usability of the framework.


(2011) proposed a NER algorithm to recognize ten categories of entities from Twitter text.

in FG-NER, there are hundreds of NE categories, which are fine-grained classification of coarse-grained categories.


When the training data size is small, CRF+SVM+Dict outperforms LSTM+CRF+Dict+Cate by a wide margin (F-score of 60.60%, compared to 45.43%).

when the training data size is increased to 100%, LSTM+CRF+Dict+Cate is the best method (it achieves an F-score of 75.18%, compared to 73.30% of CRF+SVM+Dict).


We found that the state-of-the-art method for English NER, which is based on neural network architecture, also works well with English FG-NER.

for Japanese FG-NER, it does not achieve state-of-the-art performance.


Their model also involves an external layer to extract some character level features.

it is not explicit how to model the dependencies of more tags or use the dependency information in these lines of work.


The contributions of this work are as follows: • We extend the LSTM model to higher order models.

the performance of the high order models which are supposed to capture longer tag dependencies is getting worse when increasing the order.


The reason is that BiLSTM model predicts the tag independently, and it predicts "O" as the tag of "of" regardless of the neighboring tags.

mO-BiLSTm takes account of the neighboring tags, and works well in this case.


We introduce a single order model, which is supposed to capture more tag dependencies.

the performance of the single order model is getting worse when increasing the order.


MDS-ACS outperformed the ID5 system according to ROUGE-1, ROUGE-2, and ROUGE-L metrics.

there was no significant difference between MDS-ACS and ID5 according to ROUGE-SU4.


RuSentiLex, the largest sentiment lexicon for Russian 3 (Loukachevitch and Levchik, 2016), currently contains 16,057 words, which exceeds the size of such manually constructed English resources as, for example, SentiStrength (Thelwall and Buckley, 2013).

there is nothing like SentiWordNet (Baccianella et al., 2010), SentiWords, (Gatti et al., 2016), or SenticNet (Cambria et al., 2018) for Russian.


Figure 3 shows that the distribution of positive posts in the pre-selected sample turned out to be similar to the original one.

the classifier was successful in reducing the number of skipped and speech act posts, and the ratio of negative posts increased.


Our results suggest that regularized softmax models perform competitively as long as we are only interested in low test time complexity.

when train time is also a factor, NCE has a notable advantage.


As could be expected, the larger the value of α is, the better the self-normalization becomes, reaching very good self-normalization for α = 10.0.

the improvement in self-normalization seems to occur at the expense of perplexity.


DEV-LM and NCE-R-LM perform very similar in all respects.

we note that NCE-R-LM's advantage is that during training, it performs sparse computations of the costly normalization term and therefore its training time depends much less on the size of the vocabulary.


Both industry and academia have realized the importance of the relationship between aspect term and sentence, and made attempts to model the relationship by designing a series of attention models.

most existing methods usually neglect the fact that the position information is also crucial for identifying the sentiment polarity of the aspect term.


Traditional approaches have defined rich features about content and syntactic structures so as to capture the sentiment polarity (Jiang et al., 2011).

this kind of feature-based method is labor-intensive and highly depends on the quality of the features.


Then it regards the average value of all hidden states as the representation of sentence, and puts it into softmax layer to predict the probability of each sentiment polarity.

it can not capture any information of aspect term in sentence .


(2007) explored to use structural clues that could extract polar sentences from HTML documents, and built lexicon from the extracted polar sentences.

these methods are labor-intensive, and usually results in high dimensional and high sparse phenomenon for the text representation.


Neural sequence-to-sequence models have been successfully extended for summary generation.

existing frameworks generate a single summary for a given input and do not tune the summaries towards any additional constraints/preferences.


Our current work is limited to replacing words with better synonyms.

introduction of new words can benefit tuning the generation towards a specific aspect or tone.


While pivot-based domain adaptation methods are well-motivated, they are often outperformed by autoencoder methods.

both approaches to domain adaptation effectively lead to a loss of information, as they must reduce the effect of discriminant features which are domain-dependent.


Domain transfer for sentiment analysis has been widely studied on the Amazon sentiment domain corpus.

we hypothesize that progress previous approaches have made on this particular corpus may not hold when tested on more divergent domains.


For example, ∼17% of the 'active' vocabulary (i.e., frequency >30) of English in PE EN is also contained in PE DE .

only 6% of the active vocabulary of German in PE DE occurs also in PE EN .


This is unsurprising since CRC is considerably smaller in size than PE.

we observe that the cross-language drop is much larger than it is for the PE DE↔EN setting.


(Chung et al., 2016) utilized BPE to build a decoder.

these non-word-level models could sharpen the long-term dependencies issue (Hochreiter and Schmidhuber, 1997) and lost much semantic and syntactic information.


Results with External Data As shown in Table 4, incorporation of the RACE dataset for semisupervised learning improves accuracy from 65.3% to 70.4%.

mPNet and SemimPNet still underperform a pretrained state-of-the-art neural language model One-billion-word-Lm (Jozefowicz et al., 2016).


Attention mechanisms have been leveraged for sentiment classification tasks because not all words have the same importance.

most existing attention models did not take full advantage of sentiment lexicons, which provide rich sentiment information and play a critical role in sentiment analysis.


is taken from a one-star review in Yelp 2013, which meets our expectation.

in other domains or contexts, long is likely to be positive.


Lexical databases such as WordNet (Miller et al., 1990), FrameNet (Baker et al., 1998) and PropBank (Palmer et al., 2005) can be viewed as a superset of events, and their subtaxonomies seem to provide an extensional definition of events.

these databases have a narrow coverage of events because they are generally expected to cover basic terminology due to their dictionary nature.


Others explore self-training (Liao and Grishman, 2011), event vector representation (Peng et al., 2016), tensorbased composition , and distant supervision (Chen et al., 2017).

their models focus on predicate-argument structures and are validated in a few domains, mostly in ACE.


Semantic parsers critically rely on accurate and high-coverage lexicons.

traditional semantic parsers usually utilize annotated logical forms to learn the lexicon, which often suffer from the lexicon coverage problem.


(2013) learns lexicons by aligning Freebase 2 predicates with relations from ClueWeb 3 , and then the alignments are used as lexicons.

the lexicon coverage of these alignment-based methods highly depends on entity co-occurrences, and they mostly can only learn predicates which indicating relations between entities.


Krishnamurthy (2016) also learned a lexicon for semantic parsing.

they aim to extend the predicate side as they think the predicates have limited coverage for new sentences.


The UAM Spanish Treebank and the SFU Review SP -NEG corpora take only into account syntactic negation, and the IULA Spanish Clinical Record corpus also considered it along with lexical negation.

in general, it is not specified whether the complexity of negation has been taken into account during the annotation process.


The analysis of these aspect ratings could not only benefit mining interested aspects for users, but also help companies better understand the major pros and cons of the product.

compared with the overall rating, users are less motivated to give aspect ratings.


Many other studies (Titov and McDonald, 2008;Wang et al., 2010;Wang et al., 2011;Diao et al., 2014;Pappas and Popescu-Belis, 2014;Pontiki et al., 2016;Toh and Su, 2016) solve multiaspect sentiment classification as a subproblem by utilizing heuristic based methods or topic models.

these approaches often rely on strict assumptions about words and sentences, for example, word syntax has been used to distinguish aspect word or sentiment word, or appending an specific aspect to a sentence.


(2016) employ attention-based LSTM and deep memory network for aspect-level sentiment classification, respectively.


Document-level sentiment classification (Li and Zong, 2008;Li et al., 2010;Li et al., 2013;Xia et al., 2015; is also a related research field because we can treat single aspect sentiment classification as an individual document classification task.

they did not consider multiple aspects in a document.


Given the embedding of the center word w, the Skip-gram model for instance uses a single output matrix to predict every contextual word.

the structured Skip-gram adapts the model to the positioning of the surrounding words.


Incorporating distributed word embeddings as features has proven effective in a variety of natural language processing tasks, including parsing (Socher et al., 2013), language modeling (Bengio et al., 2003;Mnih and Hinton, 2008) and sentiment analysis (Socher et al., 2011;Labutov and Lipson, 2013;Tang et al., 2014;Tang et al., 2016).

the effectiveness of generic word embeddings has been shown to be heavily task-dependent (Labutov and Lipson, 2013;Bansal et al., 2014).


All the above-mentioned approaches of learning task-specific affective embeddings (Tang et al., 2014;Tang et al., 2016;Felbo et al., 2017) rely on tweets data obtained from Twitter, automatically labeled using emoticons.

tweets data do not generalize well to texts from other domains such as blogs, narratives, etc.


A corpus analysis by Gkatzia and Mahamood (2015) shows that intrinsic human-based measures are used to a much greater degree compared to extrinsic human-based evaluations, although increasingly more effort regarding the latter evaluation can be observed in the past few years (Gkatzia et al., 2017;Goldstein et al., 2017;Ramos-Soto et al., 2017, for instance) These extrinsic task-focused evaluations have traditionally been regarded as the type of evaluation that provides the most meaningful results.

it can be an expensive and timely undertaking to execute such an evaluation (Reiter and Belz, 2009).


They did not find significant behavioral differences between people that received a tailored vs. non-tailored letter.

it might be argued that such behavioral changes are difficult to achieve with only a letter.


The results showed that the language use of the human-written texts was found to be more fluent and easy to read, as well as more clear and understandable.

the computer-generated texts gave a better overview of the match-data it was based on.


Another reason to think that the PASS-generated texts are effective in their goal is the fact that the intended tailoring was correctly identified by participants during evaluations, to a similar degree as the tailoring in human-texts.

the second study did not show any effects of tailoring on perceived text quality.


This model facilitates the development of machine learning approaches and the automatic evaluation of performances.

as criticized by (Jia and Liang, 2017), a machine can succeed in a task of this kind by remembering and recalling linguistic patterns that are prominent in a target dataset.


Their approach can be contrasted with ours, in that they modify text passages, whereas we propose to incorporate NAQs.

both share the same objective of extending the conventional MRC framework.


Machine understanding of language is difficult to define and accomplish.

we must approach this issue by developing a computational mechanism, along with the relevant resources.


During the forward pass the GRL computes the identity function and feeds its input to a shallow Multi-Layer Perceptron (MLP) style classifier.

during back-propagation the gradient of the style classifier is flipped in sign.


For the sequence-to-sequence models, → is the primary indicator of successful obfuscation-by-transfer.

← gives some indication how much the output is still related to the original.


Adversary Impact (∆ acc) We compare the accuracy of the adversary on the generated sentence to that of the original to assess obfuscation strength.

as our goal is to keep the adversary's performance level close to chance, we define ∆ accuracy = accuracy − p where p is majority baseline.


Looking at the target BLEU and METEOR, the sequence-to-sequence model without the target token generates sentences that are closer to source than they are to the target; and achieves low scores overall, with the sentences being quite far off based on WMD.

note that this is many-to-many translation without any signal regarding the target, given languages with largely the same vocabulary.


Recent work also explores how to perform style transfer without parallel data.

this line of work considers transformations that alter the original meaning (e.g., changes in sentiment or topic), while we view style transfer as meaning-preserving.


Other work has focused on specific realizations of stylistic variations, such as T-V pronoun selection for translation into German (Sennrich et al., 2016a) or controlling voice (Yamagishi et al., 2016).

we adopt the broader range of style variations considered in our prior work, which introduced the FSMT task : in FSMT, the MT system takes a desired formality level as an additional input, to represent the target audience of a translation, which human translators implicitly take into account.


It is a compromise to an alternative behavior of restarting over again from the root, which causes many non-termination problems and unpredictable behavior.

two alternatives still being considered as options at the command line are: to resume from the leftmost child of the node matching the primary node specifier (which usually is the primary placeholder, though not always); and also resuming from the node which would be the next, in a preorder left-to-right traversal, after the whole subtree substituted at primary placeholder was visited.


Of course there are other ways of doing this.

this greatly simplifies the transductions that have to eliminate intermediate nodes.


We are not going to exhaust these aspects here.

it is important to notice that the definition and use of placeholders interact very dangerously with the logical operators.


Pichotta and Mooney (2016a) showed that the LSTM-based event sequence model outperformed previous co-occurrence-based methods for event prediction.

this line of work build their models based on discrete verbs and tokens, which is far from being a complete sentence or a story.


(2017) also developed an end-to-end model for future subevent prediction.

they use large-scale news corpus as training data, which is quite noisy and far from being reasonable stories.


Because this task essentially has no gold-standard answers, and any reasonable story ending can be the right one.

purely MLE trained Seq2Seq model tends to generate frequent words or phrases in the test stage, which is a well known intractable obstacle.


(2017) applied adversarial networks on this task, which is most similar to our work.

all these studies put their focuses on choosing the correct story ending through discriminative approaches.


Generating texts conditioned on the records is a non-trivial problem because a sentence could likely contain several records.

there have been few works explicitly addressing this issue due to the lack of challenging datasets.


The special placeholder could be "<data>".

in this specific task, we make a distinction between entities and numeric values, and define two placeholders, "<entity>" and "<number>".


There are more than 7 repetitions in this short text.

in our model, the delayed copy network avoided such problem since it is actually not based on language models.


These constraints make delete-based sentence compression a relatively easier task.

in spite of the strong ability of deleting undesired words, delete-based models are not able to rephrase the words, which is far from human sentence compression.


Due to the difficulty of abstractive sentence compression, there was only a limited number of work on the task (Cohn and Lapata, 2008;Cohn and Lapata, 2013;Galanis and Androutsopoulos, 2011;Coster and Kauchak, 2011a).

with the recent success of the sequence-to-sequence (Seq2Seq) model, the task of abstractive sentence compression has become viable.


These abstractive models (which will be termed generate-based models hereafter) have the ability to reorder words or rephrase.

none of these models consider explicit word deletion.


(2002), and is usually used for automatic evaluation of statistical machine translation systems.

it can also be used for evaluating sentence compression task (Napoles et al., 2011).


(2016) further improved the model with Recurrent Neural Networks.

both works used vocabularies of fixed size for target sentence generation.


It is beneficial for models to focus on nearby context words.

as high level semantic concepts of terms, aspects usually have more generalizable representations.


For example, the term "sandwich" is surely about the aspect "food".

existing neural network based methods fail to utilize the relevance between the aspects and the terms as very few datasets are annotated with both aspects and terms.


10  Table 3: Examples of improved classification upon the addition of author profiling features (AUTH).

a number of abusive tweets still remain misclassified despite the addition of author profiling features.


Review text has been widely studied in traditional tasks such as sentiment analysis and aspect extraction.

to date, no work is towards the end-to-end abstractive review summarization that is essential for business organizations and individual consumers to make informed decisions.


(ii) According to what we observe, summary styles and words in different categories can significantly vary.

existing methods apply a uniform model to generate text summaries for the source documents in different categories, which easily miss or under represent salient aspects of the documents.


the maximum likelihood estimation (MLE) method suffers from two main issues.


In that case, the computed attention weights rely entirely on the semantic associations between context words and the target.

this may not be sufficient for differentiating opinions words for different targets.


As shown in example 1), the sentence holds a positive sentiment on atmosphere, but expresses no specific opinion on drinks.

affected by the word perfect, the predicted sentiment towards drinks is positive.


The main finding we see is that SSWE by themselves are not as informative as W2V vectors which is different to the findings of .

we agree that combining the two vectors is beneficial and that the rank of methods is the same in our observations.


A collapsed Generator is not capable of providing meaningful rationales since it extracts the same rationales repeatedly regardless of the context.

as shown in Figure 1, a collapsed Generator does not always lead to poor classification performance when compared to a finely converged Generator.


The objective of GAN is to jointly train a Generator and Discriminator, where the loss for Discriminator and the reward for Generator comes from failed attempts to determine ground-truths from the emulated results created by Generator.

the adversary process in GAN is the major source of mode collapse, hence further extensions of GAN aimed towards addressing this issue.


(2016) first attempt to adopt multi-lingual transfer learning for RE.

both of these works learn predictive models on a new language for existing KBs, without fully leveraging semantic information in text.


Second, we want to explore the effectiveness of text descriptions in different relation classification frameworks.

it should be noted that the BRCNN-based encoder cannot be used in the description representation learning.


"Pointer-Generator-all" has 99.6% of unigrams, 95.2% bigrams, and 87.2% trigrams contained in the source documents (DUC-04).

the ratios for human summaries are 85.2%, 41.6% and 17.1%, and for "AMRSumm-Clst" the ratios are 84.6%, 31.3% and 8.4% respectively.


The information overlap between the documents from the same topic makes the multi-document summarization more challenging than the task of summarizing single documents.

in case of multi-document summarization where source documents usually contain similar information, the extractive methods would produce redundant summary or biased towards specific source document (Nayeem and Chali, 2017a).


MSC is a text-to-text generation process in which a novel sentence is produced as a result of summarizing a set of similar sentences originally called sentence fusion (Barzilay and McKeown, 2005).

lexical paraphrasing aims at replacing some selected words with other similar words while preserving the meaning of the original text.


(Boudin and Morin, 2013) improved Filippova's approach by re-ranking the fusion candidate paths according to keyphrases to generate more informative sentences.

grammaticality is sacrificed to improve informativity in these works (Nayeem and Chali, 2017b).


(Yasunaga et al., 2017) is limited to extractive summmarization.

(Li et al., 2017a) is limited to compressive summary generation using an ILP based model, and there is no explicit redundancy control in the summary side.


Dataset Models R-1 R-2 R-WE-1 R-WE-2 LexRank (Erkan and Radev, 2004) 35.95 7.47 36.91 7.91 Submodular (Lin and Bilmes, 2011) 39.18 9.35 40.03 9.92 RegSum  38.57 9.75 39.12 10.33 ILPSumm (Banerjee et al., 2015) 39.24 11.99 40.21 12.08 PDG* (Yasunaga et al., 2017) 38 TextRank (Mihalcea and Tarau, 2004) 27.56 6.12 28.20 6.45 Opinosis (Ganesan et al., 2010) 32.35 9.13 33.54 9.41 Biclique (Muhammad et al., 2016) 33.03 8.96 33.91 9.25 ParaFuse doc (ours) 33.86 9.74 34.46 10.09 Table 3: Results on DUC 2004 (Task-2) and Opinosis 1.0 Evaluation Metric: We evaluate our summarization system using ROUGE 11 (Lin, 2004) on DUC 2004 (Task-2, Length limit (L) = 100 Words) and Opinosis 1.0 (L = 15 Words).

rOUGE scores are unfairly biased towards lexical overlap at surface level.


Both decision makers have access to the image and implicitly to the linguistic dialogue history: DM1 exploits the dialogue encoding learned by QGen's LSTM, which is trained to record information relevant for generating a followup question.

dM2 leverages the dialogue encoding learned by the Guesser's LSTM, which is trained to capture the properties of the linguistic input that are relevant to make a guess.


While DM1 tends to make a decision to stop asking questions and guess in easier games, surprisingly DM2 is more likely to make a guessing decision when the image complexity is higher (note the contrasting tendency of the coefficients in the last column of Table 2 for DM2).

similarly to DM1, once DM2 decides to guess (decided games in Table 2), the simpler the image the more likely the model is to succeed in picking up the right target object.


In this paper, we study the problem of data augmentation for language understanding in taskoriented dialogue system.

to previous work which augments an utterance without considering its relation with other utterances, we propose a sequence-to-sequence generation based data augmentation framework that leverages one utterance's same semantic alternatives in the training data.


Success has been achieved with data augmentation on a wide range of problems including computer vision (Krizhevsky et al., 2012), speech recognition (Hannun et al., 2014), text classification (Zhang et al., 2015), and question answering (Fader et al., 2013).

its application in the task-oriented dialogue system is less studied.