11: until convergence  We performed experiments on synthetically generated dataset since it gives us a better control over the distribution of the data.

we compared the gains obtained using our approach versus the variance of the distribution.


Then, if the ratio for a word lies close to 1.0, word frequencies in the target Twitter counties are replaced with their corresponding means from the source data.

we compute Wherew F j is the mean of word frequency j for Facebook users andw T j is the mean of word j for Twitter counties.


We see that our predictions mostly accord with the personality literature, and that domain adaptation often strengthens the correlations that we expected and weakens the correlations that had been predicted from language, but were not expected.

in the original data, there is a predicted strong correlation between agreeableness and education.


Our two-path bootstrapping architecture consists of two learning components, an explicit slur term learner and a neural net classifier (LSTMs (Hochreiter and Schmidhuber, 1997)), that can capture both explicit and implicit phrasings of online hate speech.

our bootstrapping system starts with automatically labeled online hateful content that are identified by matching a large collection of unlabeled online content with several hateful slur terms.


3 The Two-path Bootstrapping System for Online Hate Speech Detection 3.1 Overview  Figure 1 illustrates that our weakly supervised hate speech detection system starts with a few preidentified slur terms as seeds and a large collection of unlabeled data instances.

we experiment with identifying hate speech from tweets.


Then the slur term learner recognizes a unigram with a score higher than a certain threshold as a new slur.

we use the threshold score of 100 in identifying individual word slur terms.


We aim to recognize implicit hate speech expressions and capture composite meanings of tweets using a sequence neural net classifier.

our LSTM classifier has a single layer of LSTM units.


Table 3 shows the number of hateful tweets our bootstrapping system identified in each iteration during training.

the columns Slur Match and LSTMs show the number of hateful tweets identified by the slur learning component and the weakly supervised LSTM classifier respectively.


The error of our model comes from semantic drift in bootstrapping learning, which partially results from the complexity and dynamics of language.

we found dynamic word sense of slurs and natural drifting of word semantic.


In this paper, we address the problem of how to leverage both review text and implicit feedback simultaneously in order to provide richer user experiences and more meaningful recommendations.

we focus on estimating latent user preferences through implicit feedback while simultaneously predicting the contents of the reviews themselves.


Only reviews in the training set are used (since reviews are not available at test time).

we first train a word2vec model on all reviews in the training set; then we aggregate reviews written by u or about i, and take an average over the representations.


In one ('subset') we consider a subset of implicit feedback instances in R such that each is associated with exactly one review in T ; in the second ('wholeset') we consider all implicit feedback instances in R, but only a subset of the reviews.

the two settings have the same subset of reviews, but differ in the amount of implicit feedback used.


Figure 3 gives a representative example of a generated review, for a (user,item) pair that is not present in the training set.

only the user and item IDs are provided to the model, from which it must synthesize a plausible review.


In this work, we focus on a CQA site and examine the characteristics of the question summarization task with a CQA dataset as a case study.

we first examine CQA data consisting of pairs of a question text and its title, which we refer to as "question-title" pairs.


We next analyzed question-summary pairs in terms of whether the summaries can be generated by an extractive method or an abstractive method is required.

we randomly selected pairs whose question text had 3-5 sentences and manually classified them into one of the following 3 categories: 1) The pair cannot be regarded as a question-summary pair, 2) An extractive method can generate a summary that is equivalent to the title, and 3) Others (i.e., an abstractive method might be needed to generate a summary that is equivalent to the title).


We adopted encoder-decoder based methods for abstractive approaches.

we trained three models: a vanilla encoder-decoder model, an encoder-decoder model with an attention mechanism and an encoder-decoder model with a copying mechanism.


A selected subset of partial trees are linearized using the transition based syntactic linearization model detailed in Section 3.3 to generate cluster representative sentence.

we evaluate our method using the test sets of DUC 2004, DUC-2007and TAC-2011 the set of attributes of the summary including content coverage of summaries and linguistic quality and factuality of newly generated summary sentences are evaluated.


The connections at each decoding time step are dynamically changing according to the decision of the selector.

at the decoding time step i, firstly the selector on the top decides to use which type of decoder D t i ∈ {D P , D A } 3 , and then the decoder D t i decodes the token y i which is naturally given the label t i .


Richer datasets are becoming available that provide more complete event orderings which include logically implied relations that are less evident from local text features.

the TimeBank-Dense corpus provides a significantly more dense and complete set of annotations, allowing for the evaluation of methods that make use of broad contextual information across many event pairs .


We compute additional features from annotations generated using the mate-tools SRL system (Björkelund et al., 2009).

for a given pair, we compute features representing SRL predicates of the events as well as their conjunction.


Instead of relying on patterns to capture important triples, we use an importance-based strategy by exploring the entire dependency tree structure to automatically measure the connection strength of candidate argument pairs.

we assume that a relational triple is important if there is a relatively short random walk-based distance between two relatively important arguments, measured against the entire dependency tree of a given sentence.


This is because, in contrast to simple events, complex events consist of more than two elements or another event.

the task of extracting binding events is usually treated as finding ternary relations, where a system is supposed to recognize two biological entities together with a particular site where their binding takes place.


As a modification to the sequence-to-sequence model, we suggest that the model uses a static RNN decoder, which does not use a previous token vector (t i−1 ).

the model used randomly normalized token vectors.


In our paper, we propose a simple PMI-based approach, which utilizes local mutual frequencies in the course corpus, to calculate the phraseness score for each candidate.

for a k- Then, the phraseness score ph(c) is defined as follows.


Following the above assumption, the confidence score of a candidate course concept is evidenced by its semantic relations with other course concepts.

high-confidence course concepts in CCG could propagate their confidence scores to their neighbor nodes that have high semantic relatedness with them, to discover other potential course concepts.


Therefore, we propose an iterative graph-based algorithm, namely course concept propagation (CCP), which first assigns each vertex of CCG an initial confidence score, and iteratively updates the score for each vertex through propagation.

the initial confidence score of each candidate course concept is determined by a seed set S, which contains a list of known course concepts in D. let us denote the confidence score of the vertex c in the k-th iteration as conf k (c), and the initial confidence score of c as conf 0 (c).


These video titles serve as an outline of a course and become a potential good resource for providing high-quality course concepts.

for each course C i , we first extract candidates from video titles of C i , following the same pattern-based procedure in the candidate extraction step.


From the table, we find that the proposed methods outperform all baselines on all datasets, which indicates the robustness and effectiveness of CCP 7 .


For this classification task, we use only the data from the portrayed identities.

we use the responses from the 16 Y/O male, 65 Y/O male, 16 Y/O female, and 65 Y/O female identities.


Surprisingly, the analysis of age deceivers in Table 5.3 shows a strong correlation trend between the different age-based identities.

the correlation between Real Old and Young suggests high language similarity between the two groups.


Through several experiments, we showed that we can build accurate identity detectors.

we focused on the prediction of gender and age impersonators.


In addition to the above similarities, words in the current sentence are encoded into the state vector using a Long Short Term Memory (LSTM) network and mean pooling.

we take the sequence of words in the current sentence as input, pass their one-hot vector embeddings to the LSTM cells, and output a corresponding vector representation, which combined with the similarities (described above) produces a state vector to serve as the input for the DQN module.


The advent of microblogging services has impacted the way people think, communicate, behave, learn, and conduct their daily activities.

the lack of regulation has made social media an attractive tool for people to express online their thoughts, beliefs, emotions and opinions.


Similarly, (Warner and Hirschberg, 2012) propose a template-based strategy that exclusively focuses on the detection of anti-semitic posts from Yahoo!.

some efforts were made to propose new feature sets, but with unsuccessful results.


and xenophobic urls identified by the American Jewish Congress.

they use a template-based strategy to generate features from the part-of-speech tagged corpus augmented with Brown clusters.


(2016) present a dictionary-based approach to racism detection in Dutch social media comments following the findings of Burnap and Williams (2016).

they broaden the coverage of the categories in their dictionaries by performing a dictionary expansion strategy that uses a word embedding (skip-gram model) obtained from a general-purpose 3.9 billion words Dutch corpus.


(2015), who propose to build a specific paragraph embeddings for hateful speech.


From this set, potentially racist tweets are collected by selecting all those that contain at least one racist candidate keyword from (1) a collection of 3000 plus racial and ethnic slurs or (2) a set of racially motivated phrases.

the first list of keywords is compiled from the Wikipedia list of ethnic slurs 4 and the racial slur database.


As a consequence, we propose different text processing methodologies to compute the three demographic variables: age, gender and location.

age and gender are predicted using the text-based models described in Sap et al.


Each sub-collection consists of 1000 tweets that are uploaded to the crowdsourcing service of the CrowdFlower platform 8 for annotation.

each tweet is represented by its text, location information, user's age and gender, and a multiple choice question is asked to the annotators to decide whether the tweet has indeed a racist intent or not.


We are aware that different tweet representations could have been tested, namely those based on neural sequence modeling.

future work will aim to adapt models such as para-graph2vec (Le and Mikolov, 2014) or LSTM (Tai et al., 2015) to demographic embeddings.


The demographic word embeddings were computed based on the implementation provided by Bamman et al.

(2014) 11 with k = 100 for the low representation dimension, while demographic-agnostic models were built using the word2vec 12 implementation for the same size of k. we built 8 different embeddings i.e.


As for evaluation, we proposed to split the goldstandard data set of 7358 tweets described in section 3.2 into a training data set of 5149 tweets (70%) and a test data set of 2209 tweets (30%).

the test data set is equally distributed among male, female, the two age groups, and the top 20 countries with most tweets in our collected unlabeled data set.


Indeed, all demographic-aware embeddings improve over the demographic-agnostic embeddings, even boosted by discrete demographic variables, to the only exception of the gender-aware model (row 10 in Table 4).

a maximum difference is obtained by the demographic word embedding that counts with Age, Gender and Location variables at levels of 9.5% of F 1 score, when compared to demographic-agnostic word embeddings.


Indeed, at each inclusion it slightly improves results, while Location is the most productive variable, as hypothesized by Chaudhry (2015).

note that the demographic-aware continuous model with the single Gender variable is outperformed by the model built on general word embeddings increased by discrete context variable (row 8 in Table 4).


Note that all improvements in Table 4 and Table 5 are statistically significant.

we adopted a bootstrap-sampling test similarity to Hovy (2015) with a standard cutoff of p < 0.05.


Results show that our proposed framework performs consistently better than its counterpart.

the results show that with 3k training examples better results can be obtained by our approach than relying on 5.1k training examples for the state-of-the-art supervised machine learning approach (Baseline) based on common word embeddings for racist tweet detection.


Recall the assumption that a document can be a revision of at most one document.


Word2vec produces semantic embeddings for words using a two-layer neural network.

word2vec relies on a skip-gram model that uses the current word to predict context words in a surrounding window to maximize the average log probability.


when the corpus size increases, the performances of WMD, VSM, PV-DTW and PV-TED drop faster than wDTW and wTED.


Note that the X t is an attention vector generated based on the current state and the memory of the passage: (Shen et al., 2016).

the attention score a t,i on a memory vector where λ is set to 10 in our experiments and the projection matrices w 1 and w 2 map the memory vector and state into the same space, they are learned during training.


The confidence factor p(d|D(S −w )) is assumed to be proportional to a document score ds(d, S −w ) and we only consider top-k search results.

letting d i denote the document ranked i-th in the search results according to ds(d, S −w ), we assume where is the normalization factor.


Pattern 1: In human-to-human communication, people are inclined to both answer the questions and then initiate a new question to let the dialog last.

a speaker can change from information-provider to information-seeker in a single speaker turn.


The network takes as input a document x i and outputs a document vector u i .

it has two levels of abstraction, word vs. sentence.


In Table 3, we observe that when we replace the DENSE encoder layers with GRU or biGRU layers, the improvement from the multilingual training is still present.

the multilingual models with shared atten-   tion are superior to alternatives, regardless of the employed encoders.


Intuitively, this measure is computed as the change in word frequency distribution associated with an editor's edits between the article revision prior to discussion and the article revision when the discussion ends.

this score is the proportion of an editor's edits-words deleted and words added-that remain 1 day after the discussion ends.


We are interested in linguistic moves that characterize editors in conversation, and so we extract features that represent conversational acts.

we extract dialogue act features from the model of Jo et al.


This model assumes that the whole is not greater than its parts.

it assumes that the sum total of positive impact the features can achieve on performance is just through their inclusion as separate features.


This process is identical to the training process, except that there is only the role assignment step without the weight adjustment step.

the first role is assigned to the target editor as in training, and the other roles are assigned according to the original model.


Inspired by the algorithm, we incorporate CNNs that imitate the CKY table into the attention mechanism of ANMT.

the proposed attention constructs CNNs in the same order as the calculation procedures in the CKY table, and then ANMT decodes a target sentence by referring to each state of the hidden layers of CNNs, which corresponds to each cell in the CKY table.


In Figure 2, the size of filters and the number of output channels for each CNN are shown in a parenthesis.

the filter sizes of CNN1, CNN2, CNN3, and CNN4, are 1 × 1, 1 × 2, 1 × 1, and 1 × 2, and their channel numbers , and d, respectively.


In the CKY-CNN, the state of a cell is induced from multiple candidates of outputs from DUs, similar to the CKY algorithm.

the state of a cell is set to the output vector with the highest sum of values of all dimensions as follows:    Figure 3 shows an example of convolutions in the CKY-CNN, highlighting the process of generating the state of the yellow cell.


In short, contributions of this paper can be summarized as follows; We introduce a weighting algorithm to improve annotation projection based on cues obtained from syntactic and translation information.

instead of utilizing manually-defined rules to filter projections, we define and use a customized cost function to train over noisy projected instances.


Our SRL system is formed as a pipeline of classifiers consisting of a predicate identification and disambiguation module, an argument identification module, and an argument classification module.

we use our re-implementation of the greedy (local) model of Björkelund et al.


Data-dependent cost-sensitive training In our baseline approach, we use the standard perceptron training.

whenever the algorithm sees a training instance x i with its corresponding label y i , it updates the weight vector θ for iteration t based on the difference between the feature vector φ(x i , y i ) of the gold label and the feature vector In Eq.


Our analysis on VERB+A0 instances shows that source-target dependency match percentage also increases during iterations leading to increase the recall.

weighting projection instances based on dependency match helps classifier label some of the instances which were dismissed during projection, thereby will increase the recall.


Word vector space models can only capture differences in meaning (Sahlgren, 2006).

one can infer the meaning of a word by looking at its neighbors.


The euclidean distance between any two words from the set {like, love, hate, rate} is √ 2.

they form a regular tetrahedron with side length = √ 2.


Table 1 shows that 1-LAYER-BISEQ2SEQ obtains better results than the strong BASELINE.

1-LAYER-BISEQ2SEQ improves the baselines with 0.3+ BLEU in both cases of ATT and NON-ATT, indicating the usefulness of using bidirectional architecture.


In this paper, we address improving the performance on the language modeling task.

we focus on boosting the quality of existing Recurrent Neural Network (RNN) language models.


These tables, which contain both the scores reported in the previous studies and those obtained by our reproduced models, indicate that IOG reduced the perplexity.

iOG boosted the performance of the baseline models.


In the ensemble setting, we applied only one IOG to the multiple models.

we used the same IOG for computing the probability distributions of each language model, namely, computing the Equation (8).


A basic example of a CRNN is shown in Figure 1.

we designed the CRNN to use the representations learned from an automated task as a concatenative input (Ghosh et al., 2016;Lipton et al., 2015) for the primary task.


In this work we propose a model for text processing using string kernels, capable of flexibly representing non-contiguous sequences.

we derive a vectorised version of the string kernel algorithm and their gradients, allowing efficient hyperparameter optimisation as part of a Gaussian Process framework.


We incorporate tf-issf into the structured perceptron as the initial feature weight for unigrams instead of initializing the weight vector to 0.

5 we use log(tf-issf(y i ) + 1) for the initial weights to avoid overemphasizing the tf-issf value (Kaji and Kitsuregawa, 2011).


Finally, the output gate regulates the output from the memory.

let x t be the input at time t, and i t , f t , o t correspond to input, forget, and output gates, respectively.


, ω |w| ] to this BiLSTM is a vector of embeddings: we define where are the characters in word w i .

we run an LSTM over the character stream and concatenate it with a word embedding for each type.


As a baseline method, we use word-level neural CRF proposed by (Ma and Hovy, 2016) since their method achieves state-of-the-art performance on NER.

they propose Bi-directional LSTM-CNN CRF (BLSTM-CNN-CRF) for sequential tagging.


This paper examines the usefulness of semantic features based on word alignments for estimating the quality of text simplification.

we introduce seven types of alignment-based features computed on the basis of word embeddings and paraphrase lexicons.


We bring a total of seven types of features that are proven useful for the similar task, i.e., finding corresponding sentence pairs within English Wikipedia and Simple English Wikipedia (Hwang et al., 2015;Kajiwara and Komachi, 2016).

we assume that some of these features are useful to capture inequivalency between original sentence and its simplified version introduced during simplification, such as lexical paraphrases and deletion of words and phrases.


We evaluated our proposed features in the supervised classification fashion as previous work.

we compared three types of supervised classifiers that had been also used in the above baseline systems: SVM, MLP, and Rand-Forest.


The difficulty of finding an argument tends to differ depending on the relative position of the argument filler and the predicate.

if the argument is omitted and the corresponding filler appears outside the sentence, the task is much more difficult because we cannot use the syntactic relationship between the predicate and the filler in a naive way.


Finally, we also plan to apply the ideas showed here to other NLP problems with similar settings.

we believe the proposed approach can be useful in any setting where (noisy) embeddings should be mapped to manually provided scores.


The architecture of our model is shown in Figure 1.

a word embedding layer transforms indexed words to real valued vectors x i with a pre-trained word embedding matrix (Mikolov et al., 2013;Pennington et al., 2014).


Counting by relation, 52% of the relations have signals only in the second discourse unit (49% of which at the boundary), 20% have signals only in the first discourse unit, and 28% have signals in both.

in 69% of the cases, all signals for the DR are covered after reading the relation boundary.


Considering that the semantic labels may provide rich cues, the middle supervision signal is utilized as intermediate guidance for the sentence encoding module in order to guide them to project from input utterances to a more meaningful feature space.

for each utterance, we compute the cross entropy loss between the encoder outputs and corresponding intent-attributes shown in Figure 2.


The whole architecture of MMR-BS is illustrated in Figure 1.

the decoding process maintains a subsequence list S; the number of subsequences in S is N .


The scoring function of MMR con-Algorithm 1: MMR-BS Decoding Input: the user-issued query q, the max length of reply l, λ in MMR function Output: generated reply set R R = ∅; S 0 = ∅; for t = 1; t ≤ l; do // obtain the subsequence set at time i S t = Decoding(q, S t−1 ); // coarse-grained selection siders the quality of a candidate as well as its diversity against previously selected ones.

we have two metrics: sim qua (s i , q) measures the similarity of a candidate subsequence s i with respect to the query q, indicating the quality of s i .


We present in this paper a statistical framework that generates accurate and fluent product description from product attributes.

after extracting templates and learning writing knowledge from attribute-description parallel data, we use the learned knowledge to decide what to say and how to say for product description generation.


We then adopt SVM-rank (Joachims, 2002) with linear kernel to rank the candidate descriptions, and treat the top candidate as the answer.

we use BLEU score between refer-ence description and generated description as label score and use the features shown in Table 2.


For example, a product in our dataset has 21.2 attributes on average, while only 7.6 attributes are mentioned in a reference description on average.

we extract attributes which mentioned in a reference description, and compare them with those in its corresponding generated descriptions.


This is because BLEU only takes word overlap into consideration but not attributes.

descriptions that share same words obtain high BLUE scores, although they are talking about different attributes.


Then in the second stage, we apply one heuristic rule to extract extra words from the dependency tree.

if a word is in a particular dependency relation 2 , aux, auxpass or cop, with a word that is already in the chain after the first stage, then we include this word in the chain as well.


Statistical significance testing shows that the improvements are significant at the p < 0.01 level (t-test).

for on-going and future events, the dependency chain based LSTM model improves the temporal status classification F-scores by 4 and 10 percentages respectively.


We can clearly see that the dependency chain input effectively retains contexts that are relevant for predicting event temporal status.

as shown in Figure 2(b), the context event "launch" that indirectly governs the target event "protest" in the dependency chain together with the auxiliary verb "will" have received the highest weights and are most useful in predicting the correct temporal status.


Recent breakthrough in neural network (NN) led to increasing amount of work that apply NN on various text-mining tasks.

convolutional neural networks (CNN) (Lecun et al., 1998) have been most commonly adapted for PPI.


The input gate and forget gate jointly control the content of the memory c t , and the output gate regulates output from c t .

let x t be the input at time t, and i t , f t , o t correspond to input, forget, and output gates, respectively.


However, by taking advantage of pre-trained embeddings, RNN further advances Fscores by 7% and 13% on AIMed and BioInfer, respectively.

pre-training contribute to relative improvements of 10% and 18%.


The basic LSTM model shown as Base-LSTM in Table  3 performs less than CNN-WangP and similar to CNN-Wang without profile information.

lSTM has no obvious model advantage in this set of training data.


During this process, there is no distinction between training instances.

all the instances are regarded as equal and used to train the NMT model equally.


(2016) train a parent model on a (possibly unrelated) high-resource language pair, then use this model to initialize a child model which is further trained on a low-resource language pair.

they showed that a French-English model could be used to improve translation on a wide range of low-resource language pairs such as Hausa-, Turkish-, and Uzbek-English.


Because the parent and child source languages have different vocabularies, this amounts to randomly assigning parent source word embeddings to child source words.

even if a word exists in both the parent and child vocabularies, it's extremely unlikely that it will be assigned the same embedding in both models.


We rely on the article's MeSH terms only to select RCTs.

only the articles with the MeSH term D016449, which corresponds to an RCT, are included in our dataset.


Frequency is a good estimator for difficultyrelated properties, namely familiarity and age of acquisition.

familiarity was previously reported to be one (Tanaka-Ishii and Terada, 2011).


However, since this is very difficult in practice, we intend to test this indirectly by analyzing the results of Google Scholars 3 .

we analyzed the Google Scholars' search results and compare the pairs with high scores to those with low scores to verify the validity of the proposed methodology.


CCA is a statistical method used to investigate the relationship between two words, and it is a technique that simultaneously examines the correlation between variables in a set and variables in another set.

it is a useful tool to grasp the correlation of variable group (X, Y) and to grasp the relationship between two features (Jang et al., 2013).


Table 3 shows how well each algorithm discriminates between good markers and bad mark-   ers.

the larger the difference between the values of BEST and WORST, the more favorable it is.


This provides a direction for new clinical studies.

among the markers with a high degree of similarity, the markers that are searched with a low frequency in the Google Scholar will be subject to various clinical studies in the future.


In this study we find that a simple retrieval model based on latent semantic analysis (Deerwester et al., 1990, LSA) works relatively well.

we get semantic vector representations for each document and the given topical word by performing singular value decomposition on the term-document matrix, where each element corresponds to the tf-idf value for a particular word in the document.


Therefore we utilize DivRank (Mei et al., 2010), a well-known diversified ranking algorithm, for calculating the ranking scores for different chains.

we utilize the pointwise variant of Di-vRank.


In total, there are 9,734 unique photographs in our training set, and 1,071 photographs and 5,371 sketches in the test set (that is, for most images there are 5 different sketches).

there are 5371 sketch/photograph ensembles in our image retrieval evaluations.


We trained classifiers for words or (normalised) attribute types which occurred more than 10 times in the corpus.

given an image description D : where w a i indicates an attribute word, and w c indicates a category word, we compute a score for a given photograph P and using the word/image classifiers s w (•) as follows: (attribute contributions are combined additively and then multiplicatively with the category.


We showed that a model of grounded word meaning trained on these data can be combined with an existing model of sketch/image relation, where the combination improves retrieval performance relative to the separate models.

the model profited even from small amounts of iconic information (sketches reduced to 30% of their strokes).


The principle of compositionality suggests that "a positive solution of x 2 = 1" has "1" as its denotation, but "x 2 = 1 has 1" does not make sense.

we must not supply the "omitted" argument to "a solution" in this sentence to derive a correct semantic representation, e.g., ∃x(x 2 = 1 ∧ x > 0).


For a referring expression such as tsuginoshiki/joken "the following formula/condition", the referents are searched from the succeeding context.

the sentence or math expression just after the sentence including "the following condition" is identified as the referent and the referring expression is rewritten as follows: Tsugi-no houteishiki-o mitasu △ABC-o kangaeru : sin A = sin B Consider △ABC which satisfies the following equation : sin A = sin B After rewriting sin A = sin B-o mitasu △ABC-o kangaeru.


Here, we focus on the bilingual word embedding (BWE) approach.

we are interested in leveraging recent advances in learning BWE from comparable corpora (Hermann and  Figure 1: Our multi-task framework, which trains bilingual word embeddings from comparable corpora while optimizing an NER objective on the high-resource language.


In this paper, focusing on chat-oriented dialogue systems, we investigate the effects of conveying understanding results of user utterances.

we investigate the types of results that can be conveyed to users without lowering user satisfaction.


We summarise the generative process of contraLDA as follows: The contraLDA model can be trained flexibly depending on the type of supervision information available.

if there are only labelled features available (e.g., sentiment lexicon, or topic seed words), our model will incorporate the labelled features to constrain the Dirichlet prior of topic-word distributions, which essentially plays a role in governing the model inference.


The SemEval dataset (Nakov et al., 2016) was created from questions posted on the Qatar Living forum and has a different distribution and structure.

a question has two fields: a body containing the actual question and a subject.


The main idea of this experiment is to compare the results obtained when an original English question triggers the QA system with the results obtained when the questions are the result of the (intermediary) translation by the MT engine from a different language.

when English is acting as the pivot language for accessing the information encoded in the QA repository.


When the reference answer is presented, different results are obtained compared to the previous table.

the evaluations are more homogeneous among all the languages and among the three different options.


In this paper, we present three different strategies to utilize Babel-Net resources in SMT systems, namely using direct training data, domain adaptation and OOV post-processing approaches.

the first strategy very straightforwardly appends the bilingual dictionary extracted from Ba-belNet to the initial training data, and then verifies the impact on translation performance; the second uses domainadaptation methods to select in-domain entries from the extracted bilingual dictionary, which are then added to the initial training data in a similar manner; finally we directly call the BabelNet API to post-process OOVs contained in the translation of the source sentence.


Different from the methods above, we utilize an extra semantic resource to handle the problem of OOVs.

we use BabelNet (i) as direct parallel data, or (ii) to retrieve the translations of OOVs via an API call.


We use the BabelNet API to call the precompiled index bundle (version: 2.5.1) 2 to retrieve the translation for an OOV.

an OOV in the source sentence is automatically marked in the output of the SMT decoder, and then we recognize this OOV and retrieve its 1-best candidate by calling the BabelNet API.


Finally, we obtain 5, 493, 323 Chinese-English pairs from the original 5, 975, 619 pairs.

far less data is filtered out than for EN-PL; for ZH-EN, just 8% of the original data is discarded, indicating that on the whole, the ZH-EN data is of superior quality than the EN-PL entries.


Explicit evaluation involves automatic and human evaluation of the translation output.


However, the Edit Distance score by itself does not take into account the actual time spent on a given translation unit and provides an absolute score, whereby the maximum number of edits possible per segment would be the total number of characters of the reference translation.

if the "gold standard" reference translation has 25 characters, the Edit Distance Score can be 25 at most.


In fact, as sentences can be clearly related to polarity, a classifier can always be trained in such spaces and used to transfer sentiment information from sentences to words.

we train a polarity classifier by observing sentences expressing some polarity and we adopt it to classify words in order to populate a polarity lexicon.


Their contribution is measured against different SA tasks in the two languages.

our evaluation is based on Twitter Sentiment Analysis, as recently it has been the focus of highly participated challenges, as the recent SemEval (Nakov et al., 2013;Rosenthal et al., 2014) and Evalita (Basile et al., 2014) tasks demonstrate.


In (Saif et al., 2014) word contexts are adopted to generate sentiment orientation for words.

the sentiment of context words, available in an already built lexicon, is shown to contribute in deriving the sentiment orientation of a target word.


The generation process relies on an embedded representation of lexical items, that is derived by approaches in the family of the Distributional Model (DM) of Lexical Semantics; in these models, semantics relationships between words and sentences can be exploited at the same time.

the approach is based on the possibility to acquire comparable representations (e.g.


Word spaces are derived according to a specific Distributional Model, that is the Skip-gram model defined in (Mikolov et al., 2013a) with the word2vec tool 7 .

we derived 250-dimensional word vectors, by analyzing these corpora made of tweets.


The specific scores assigned within the polarity lexicon, as the ones reported in  representation can be used to enrich a pure semantically oriented representation of words (from the original DMs) in order to inject sentiment related information in the embedding.

we can combine the original word space vectors and the DPL vectors in a way such that the resulting representation can still represent the semantics of the words being at the same time more suitable for sentiment analysis related tasks.


In other words, we can combine the original word space vectors and the DPL vectors in a way such that the resulting representation can still represent the semantics of the words being at the same time more suitable for sentiment analysis related tasks.

for each word in the vocabulary, we derived such representation by juxtaposing 9 the original 250-dimensional word space vector, which is derived through word2vec, and a DPL vector: the result of this operation is a new 253-dimensional representation.


In this Section, an indirect evaluation of the polarity lexicons is provided.

all the released lexicons are used in Sentiment Analysis tasks in Twitter with respect to English and Italian.


The core affect map in Figure 1 will be used to identify the emotional state or sentiment label of a given entity (message, topic or user) according to its valence (X-axis) and arousal (Y-axis) values.

given an entity e with valence and arousal values (v e , a e ), we used the Euclidean distance for identifying the closest distance sentiment to e. For instance, an entity e with valence and arousal values (v e , a e ) = (7.26, 3.56) falls within region R2 and its associated sentiment label would be S e = {serene}.


This process learns the probability of a topic z ij in document j being assigned a word w i , given all other topic assignments to all other words.

learning the assignment of words to topics z given the observed words w by repeatedly sampling the latent variables z ij regarding the other values of z.


X mean value of valence (Y , mean value of arousal) N total number of emotional words within the message φi,t topic distribution for word i in the t th topic µ word's mean value of valence (equivalently for arousal) σ word's standard deviation of valence (equivalently for arousal) (2) calculates the sentiment score of a message by estimating the overall mean value of all emotional words within the message (see Table 2).

x (respectively, Y ) is the overall mean value of the valence (respectively, arousal) considering the N emotional words within the message.


The purpose of the experimental evaluation is to identify the behaviour of users' sentiments over a conversation.

we are interested in analyzing the impact of topic distribution of a general conversation in the users' sentiments.


An additional use of advanced machine learning algorithms and contemporary feature selection methods may further improve the ER performance.

we consider using the deep learning concept to perform ER (Kim et al., 2013), and the multi-objective genetic algorithm-based feature selection (Sidorov et al., 2015) and state-of-the-art iVector-based SI procedure to further enhance the performance of the ER systems.


To address these questions, we conduct a discriminative or predictive analysis on the typological database.

we use a machine-learning classifier to estimate the value of each feature of each language using the values of the other features, under different choices of training data: all the other languages, or all the other languages except for the ones having the same origin or area with the target language.


For example, it is widely known that, if VSO is the dominant order of a language, then the language has prepositions (Universal 3 by Greenberg (1963)).

the values of features provide clues to the value of another feature.


we use a machine-learning classifier to estimate the value of each feature of each language using the values of the other features in WALS, under different choices of training data: all the other languages, or all the other languages except for the ones having the same origin or area with the target language.


For this purpose, we employ leave-one-out within the languages, for which the value of the target feature is recorded in WALS.

we (i) regard one such language as a test instance and the remaining languages as training instances, (ii) represent both the training and test instances with the features other than the target feature, (iii) see whether the value of the target feature in the test instance is correctly estimated or not, (iv) iterate this process for all those languages to calculate the estimation accuracy.


Since the typological study concerns only the language type 6 and the language universals, the effect from the shared origin and the language contact needs to be eliminated.

we are going to answer the second question in Introduction: (ii) to what extent can we attribute this predictive power to genealogical or areal factors, as opposed to being provided by tendencies or implicational universals?


The goal of this paper is firstly to present a thorough analysis of the adequacy of currently used taggers for historical Dutch and secondly to explore methods for generating higher accuracy tags.

we will asses the effect of different methods for preprocessing (spelling normalisation, as well as word-for-word translation of the text) on the accuracy of tags generated with a tagger trained on contemporary Dutch and we will explore whether making adaptations in the tagger based on knowledge extracted from a diachronic parallel corpus can improve tagging results.


Inspecting the statistical distribution reported in Table 5, it can be noted that in both years (Column Total %) orthographic and grammatical errors are the most frequent ones (46.55% and 47.33% respectively) while the lexical errors are far less (about 6%).

the most frequent errors are the orthographic not-classified (Other) ones (22.32%) followed by the erroneous use of verb tenses (11.26%), the grammatical not-classified errors (6.37%) and the erroneous use of prepositions (6.6%).


In an attempt to improve the performance of the speech-based system, we employed a late fusion scheme.

the posterior probabilities of the two systems were combined in an algebraic scheme, i.e., the mean.


The anger annotations were used both for training and evaluation purposes.

the friendly and neutral labels were mapped to the "not angry" class, while the slightly angry, angry and very angry labels were mapped to the "angry" class.


Moreover, the MLP model has been re-trained using 70 hours of multi-lingual telephone speech data corresponding to a sub-set of the SpeechDat datasets.

the training dataset is composed of speech utterances from the Portuguese version of SpeechDat(II), and additional speech from the English, Spanish, Italian, French and German versions of SpeechDat(M).


Notice that in both datasets not only most of the turns are extremely short, but there is also a significant number of turns without speech content.

around 12% of the Let's Go turns do not contain useful speech, which affects negatively the performance of the classifiers.


1 PentoRef presents a rich resource for investigating human conversational strategies for referring to objects, on different levels of linguistic realization (including speech and timing/turn-taking) and in different yet consistently represented interactive and visual contexts.

pentoRef is useful for developing automatic systems for, and studying the human mechanisms for, two concrete tasks, namely reference resolution (RR) and referring expression generation (REG).


The toolkit allows multi-coding layers annotation at the same time and the graphical interface (Figure 3) introduces a multiple  In general, the annotation scheme in WikiCoref mainly follows the OntoNotes scheme (Pradhan et al., 2007).

only noun phrases are eligible to be mentions and only non-singleton coreference sets are kept in the version distributed.


We observe that half of the chains have only two mentions, and that roughly 5.7% of the chains gather 10 mentions or more.

the concept described in each Wikipedia article has an average of 68 mentions per document, which represents 25% of the WikiCoref mentions.


The basic assumption underlying this approach is that the semantic similarity of two words is a function of their contexts.

in other words, the meaning of words can be inferred from the frequencies of the words they immediately occur with and this can happen in such a way that the degree of similarity of those meanings can be measured.


10 In order to exclude these factors and in order to exploit the full potential of OCE co-reference information shall be integrated directly into the training process of a standard count model.

we will retreat from first integrating co-reference information into raw text and then performing training.


Further, a qualitative analysis of the resulting vector spaces has to be carried out in order to explain how the different performances caused by OCE come about.

11 also the question whether we can regard OCE as yielding just more data of the same kind as the linear distribution of words or whether we may gather new-otherwise possibly rare-kinds of information.


In above example (12), mention आईसी -८०३ IC−803 is not at all a reference of िवमान सं ख्या Flight_number but they shared some attributes.

one is the function and other describes the value that the function can take.


Even without the coreference resolution component, Odin is capable of recognizing some relations involving coreference, because of their grammatical regularity.

it can recognize relations through relative pronouns such as which, as exemplified in 3.


Unlike open-domain coreference resolution, which aims to resolve all pronominal and nominal mentions in text, here we consider as anaphors only entity and event mentions that participate in fragments of protein signaling pathways.

we only consider entity mentions that are arguments of relevant biochemical interactions previously extracted by Odin (e.g., phophorylations, ubiquitinations, bindings, translocations), and nominal event mentions, e.g., "this binding".


To simplify the problem, we convert a sentence with n errors into n sentences, each of which with only one error.

the following sentence, which contains three errors, ○ ○ E1 ○ ○ ○ ○ ○ ○ E2 ○ ○ ○ ○ E3 ○ will be converted to three sentences like: Chinese, a sentence is usually composed of several segments separated by comma "，".


This study examines how to use a specialised lexicon such as FLELex in order to automatically identify those words in a text that are be difficult for and unknown to a non-native speaker and which might subsequently serve as potential candidates for lexical substitution.

we investigate two possibilities of using the FLELex resource.


In order to solve this classification problem, we need to explore other possible rules for transforming the FLELex distributions into a single difficulty level.


However, in order for FLELex to be successfully integrated in a system that analyses whether and how a text's vocabulary should be simplified for a non-native reader, we need to continue improving our method.

we need to establish a better rule to transform the word's FLELex distributions into a single CEFR level, distinguishing a level's core vocabulary from its peripheral vocabulary.


It will be convenient to set and define our metrics in terms of the four quantities C A , C R , F A , F R , which total to unity.

we consider precision: Kanters et al., 2009 Table 2: Systems from (Kanters et al., 2009) and (Rayner et al., 2015) + baseline "always accept" systems, with values for different metrics.


In the case of interactive system adaptation, the user feedback is specifically about the errors of the system on the initial input, possibly constrained in its format, and it is used to improve the overall system performance.

it is more the adaptation than the system which is interactive.


As per (Brill and Wu, 1998), this is the maximum improvement that one candidate labeling can offer over another.

comp(A, B) indicates the maximum proportional error reduction that B offers over A. complementary rate is agnostic to the set of labels involved, and so can also be applied beyond binary classification.


WordNet WordNet 2 (Fellbaum, 1998) is a large lexical database of English nouns, verbs, adjectives, and adverbs grouped into sets of cognitive synonyms called synsets, where each synset expresses a distinct concept.

each synset represents a list of synonyms, intended as words that denote the same concept and that are interchangeable in many contexts.


SenticNet SenticNet 3 (Cambria et al., 2014) is a publicly available resource for opinion mining, which exploits both artificial intelligence and semantic Web techniques to infer the polarities associated with common-sense concepts and to represent them in a semantic-aware format.

senticNet uses dimensionality reduction to calculate the affective valence of a set of Open Mind 4 concepts and it represents them in a machine accessible and processable format.


In order to represent SenticNet in a machine-accessible and processable way, information about each concept is encoded as a set of RDF triples using an XML syntax.

concepts are identified using the Con-ceptNet Web API and statements, which have the form "concept-hasPolarity-polarityValue", are encoded in the RDF/XML format on the basis of the human emotion ontology (HEO) (Grassi, 2009), a high-level ontology of human emotions, which supplies the most significant concepts and properties which constitute the centerpiece for the description of every human emotion.


Consequently, the introduction of further download options figure prominently among the user's wishes (characterized as "relevant" or "very relevant" by over 70% of participants who saw a need for improvement).

praat and EXMARaLDA were mentioned several times as formats suitable for further analysis of downloaded data, as were MS Office or MS of the DGD corpus collection, also leads to a more precise Excel or, simply, a pDF version of transcripts for printout.


At a high level, the design of this rule language follows the simplicity principles promoted by other natural language processing (NLP) toolkits, such as Stanford's CoreNLP, which aim to "avoid over-design", "do one thing well", and have a user "up and running in ten minutes or less" (Manning et al., 2014).

we aimed for the following desirable characteristics: Simplicity: The language extends familiar concepts from regular expressions and context free grammars.


Before developing the system, it first necessary to define or design the structure of temporal information.

it is necessary to design a language which defines how to annotate the temporal information.


The reason for this limitation is the inconsistent usage of the attribute freq.

the freq is used to annotate not only a periodic frequency (e.g., 'twice'), but also a periodic duration (e.g., 'one day').


From a practical point of view, this inconsistent functionality of event tags may cause difficulty in parsing of the annotated event tags, which would result in the inefficiency of further applications.

it would be necessary to implement a program to recognize the role of the event tag, which would slow down the applcations.


A tuple T (with reference to Table 1, where we show a real tuple from our dataset) which in wath follow is our data structure which store an hypernymy relation, is defined as (t where: • l t , t, r t are the elements of the hyponym component of the "isa" relation.

l t is the hyponym premodifier, t is the hyponym head noun and r t is the hyponym post-modifier, e.g.


l t ="second", t ="law" and r t ="of thermodynamics"; • l h , h, r h are the elements of the hypernym component of the "isa" relation.

l h is the hypernymy premodifier, h is the hypernymy head noun and r h is the T uple ID = 12, 265, 628 "This would seem to involve violations of certain basic laws of physics such as conservation of energy, the second law of thermodynamics, and statistical laws of quantum mechanics."


Frequent lexico-syntactic patterns are easily recognizable and indisputably indicate the lexical relation of interest.

we focused on 59 patterns that we collected from the past literature.


The #Likelihood and #Cosine similarity methods calculate the similarity between two terms by always using values between 0 and +1: when the value is 0 the two are completely different, while when it is 1 the words are identical; and a value around 0,8 indicates a high degree of similarity.

the #Likelihood method (329 occurrences) measures the associative strength of terms thus enhancing the exclusive links with other terms, while the #Cosine similarity method (934 occurrences) analyses the similarity on the basis of the occurrences they have in common.


Detection and correction of spelling errors have been studied over the last few decades.

the emergence of the web has created a large amount of consumer-generated texts, which include countless numbers of spelling errors.


Gamification allows us to obtain resources without paying users by carefully designing a game suitable for information extraction.

rodrigues and rytting (2012) proposed to extract English spelling errors by using a word-typing game.


5 A comparison of available Vietnamese tokenization tools is reported in (Dinh et al., 2008).

the CRFbased tool of the group (Tu et al., 2006), the PVnSeg tool (unknown source) and the hybrid method of (Hong Phuong et al., 2008) are compared using a test corpus containing 1,264 articles from Politics-Society section of a Vietnamese online newspaper "Tuoi Tre", where words have been manually segmented by linguists.


To this end, we used QUANDHO to retrain It-MSRP on the historical domain by conducting a set of 3-fold cross validation experiments.


Common approaches to modelling grammatical information tend to make grammatical features subsidiary to particular POS tags.

each POS tag is further subclassified based on the morpho-syntactic distinctions that it bears in the language inspiring that classification.


In order for this to become maximally useful data, though, the URI of the translated sense is preferable, so as to avoid having to disambiguate in the case of polysemous target entries.

a link between two sense level concepts is better than a link among sense level and word level concepts.


Since the lexical entry is generated from headword label, language, and lexical category, the structure would benefit greatly from having translations of all the lexical categories to be able to make an IRI in the language.

the aim is to replace ur noun with a more readable and consistent as shown in Figure 4.


Language resources (LR) are indispensable for the development of tools for machine translation (MT) or various kinds of computer-assisted translation (CAT).

language corpora, both parallel and monolingual are considered most important for instance for MT, not only SMT but also hybrid MT.


In order to facilitate the discovery of the data needed for the project, the criteria should be as specific as possible and should also cover all aspects of the data use.

all the aspects that can ease the clearance of IPR issues and the rights negotiation with the owners and/or the rightholders.


Language Resource citation is nowadays considered as an important issue.

having a persistent identifier for each existing LR can be considered as an important parameter to compute a "Language Resource Impact Factor" (LRIF).


A first jet of interviews and information collection is currently being documented in the action's deliverables.

the analysis has looked so far at the needs and requirements of institutions such as MT@EC and Europeana, DSIs with very challenging multilingual needs involving the area of Machine Translation (MT).


The idea that the durations of gaze fixations could be used as a proxy for measuring cognitive load dates back to the strong eye-mind hypothesis by Just and Carpenter (1980), according to which, "there is no appreciable lag between what is fixated and what is processed" (Just and Carpenter, 1980).

when a subject looks at something, he/she also processes it cognitively.


We observe that multimodal features consistently outperform unimodal feature sets.

complex behavioral assessments such as the overall performance and confidence of the speaker benefit from features of multiple modalities.


The resulting videos are being annotated and processed, and will be compared with our data of native speakers.

we will investigate whether nonnative participants benefit as much from the training as native participants.


This algorithm aims at measuring the similarity between two objects having multiple comparison criteria.

the overall similarity score between two objects is better when those objects have reasonable similarity scores for all criteria than when they have a very good similarity score for one criterion, and less good or no value for the others.


As such, it can be exploited by intelligent systems to extract information from textual resources.

an intelligent system that anchors entities from text to existing entities in a knowledge base, namely an entity linking system, could benefit from the structured semantic knowledge describing those entities in the knowledge base.


According to these observations, the research community has reached the following assumptions for computing sentence-level alignments (Tiedemann, 2011) : • Each side of an alignment link is a consecutive group of sentences, or is empty.

if E i and E i+2 are both inside a link, then so must be E i+1 .


The reference alignment of these 220 sentence pairs contains 2,720 S one-to-one links, among which only 37 are not included in any of our bi-segments.

our bi-segmentations contain a large majority of the S one-to-one links of Och and Ney (2000).


8 We plan to use these annotations to evaluate results delivered by standard sentence alignment and word alignment tools.

we are interested in using these data to evaluate confidence estimation measures, e.g.


Whilst goldstandard test sets exist for the coreference resolution task, they are not suitable for assessing machine translation.

monolingual gold-standard test sets lack reference translations, and there exist neither monolingual nor multi-lingual test sets that provide the additional pronoun type-specific features used to define the fine-grained categories for the test suite pronouns.


We briefly show the application of our test suite to the results of the DiscoMT 2015 shared task on pronoun translation (Hardmeier et al., 2015), an MT task evaluated with special attention to pronouns.

the focus of the task was on the translation of subject-position instances of "it" and "they" in English-to-French translation.


Among which we chose the word2vec tool (Mikolov et al., 2013), 2 because of its efficiency and wide uses in nat-ural language processing.

we used the skipgram, which inputs each current word to a log-linear classifier with a continuous projection layer, and predicts its context words within a certain window.


In this paper, we focus on dependency parsing of noisy text.

we are interested in how much parse quality can be gained by text normalization.


It has two advantages compared to the traditional linear model (Nivre, 2008), namely fast speed and automatic feature combination.

the speed of a neural dependency parser can be 1000 sentences per second on a single CPU, which is significant faster than traditional parsers.


The tokenizer simply splits on white-space and punctuation.

the POS tagger and lemmatizer components were developed in a separate project (Eiselen and Puttkammer,5 the FoLiA XML as used by the Frog parser; see http://ilk.uvt.nl/folia.


For example, if we apply the rule from the introduction to the symbolic sentence above, the result is the following symbolic sentence: The example rule can only affect readings of word2 that have a "verb" tag, so we create only two new variables w 4 and w 5 for the result, and reuse the other variables.

we must also add the following constraint for w 4 : after applying the rule, the reading "verb sg" (represented by the variable w 4 ) can only be in the resulting sentence exactly when (1) "verb sg" was a reading of the input sentence (so w 4 is true) and (2) the rule has not been triggered (the rule triggers when v 1 is true and at least one of the non-verb readings w 1 .


How do the differences in the global scores relate to differences in the analyses of particular grammatical functions?

: is one of the parsers better in every respect, or are the improvements limited to a better performance with particular functions (and therefore dependent on the frequency of these)?


Values, produced by variables (activity, activity2 and date), are employed to generate a SPARQL query, able to retrieve surname of such persons which perform a specific activity/job/profession in a determinate interval.

the previous automaton may process a query as the following one: (1) Tutti gli archeologi che sono stati anche scrittori nati nel '900 (All the archaeologists who have been also writers and were born in 19th century) 3 .


We have described how we prepared the corpus data to take up these challenges.

we have presented corpus analyses and speech data preparation steps.


The transcription includes punctuation and is casesensitive.

the transcription indicates disfluencies such as the word "euh" in French.


This paper compares three different methods to elicit multiple utterances for given semantics via crowd sourcing, namely with pictures, with text and with semantic entities.

we compare the methods with regard to the number of valid data and linguistic variance, whereby a quantitative and qualitative approach is proposed.


The participants were asked 1 http://www.apple.com/ios/siri/ to perform tasks such as entering the address into the navigation application.

the elicitation methods differed in the way how the tasks were presented to the participants, by means of: we compare the methods with regard to the number of valid data and linguistic variance, whereby a quantitative and qualitative analysis approach is proposed.


Since the elicitation and analyzing methods they used, show some limitations to elicit data for in-car SDS, we argue for methods that satisfy the requirements that are mentioned above.

our methods are applicable to different kinds of commands: Commands with one slot like "Call John Smith", commands with more than one slot like "Navigate to 11 Main Street in Springfield" and also simple commands like "Next station".


1 Table 2: Example of sentences in the BulPhonC corpus First, from the whole set of 21838 sentences we automatically selected 422 which contain at least three times almost each phoneme bigram possible for Bulgarian language.

in the 422 sentences from all 806 possible bigrams only 58 have less than 3 occurrences.


This database will serve as a useful data source for pathological speech research and is expected to facilitate the development of various assistive systems incorporating an ASR component.

its use in the CHASING project will stimulate additional dysarthric speech data collection through an ASR-based serious game that has been developed to conduct research experiments and to offer alternative, sustainable speech therapy beyond the project lifecycle.


In this paper we report our effort to construct the first ever Indonesian corpora for chat summarization.

we utilized documents of multi-participant chat from a well known online instant messaging application, WhatsApp.


However, the log contains complex structure of the dialog since it talks about question and answering of technical problem like Stack-Overflow 8 .

it's conversation is not as natural as our chat corpus that mostly talks about daily conversation.


Thus, it is important to study the validity of existing methodologies applied to the evaluation of news article summarization for this task.

we raise the important question of how effective is ROUGE, as an evaluation metric for scientific summarization?


SERA is defined as follows: To also account for the ranked position differences, we modify this score to discount rewards based on rank differences.

in ideal score, we want search results from candidate summary (R C ) to be the same as results for gold-standard summaries (R G ) and the rankings of the results also be the same.


This paper presents a novel method for the automatic enrichment of WordNet with disambiguated semantics of ConceptNet 5.

the proposed technique is able to disambiguate common-sense instances by linking them to WordNet synsets.


Since there were no obvious patterns of disagreement in the early phase of the annotation process, we suspect that this is a result of differences in our data set rather than an indicator of low quality.

the lemmas with systematic disagreement are indeed an artifact of their low polysemy and lower applicable occurrence counts.


First, the proportion of substitute tokens not in GermaNet is slightly lower than the proportion of substitute types not in GermaNet (16% vs. 24%).

of all substitute types in the data set, the annotators were more likely to apply those in GermaNet.


Finally, we note that the majority of substitutes cannot be reached by following semantic relations of a single type.

some 60% of all substitutes exist as synonyms somewhere in the GermaNet graph, but are reachable from the target synset only by following semantic relations of at least two different types.


However, because the majority of our substitutes are found outside this neighbourhood, we now perform an investigation which includes these more distant relatives.

we are interested in determining the similarity of parasets representing the same word sense.


While the other three terms are plausible synonyms for this broad sense of Athlet, they would not necessarily fit every context.

olympionike ("olympian") suggests that one of the annotators has exploited his or her real-world knowledge of the context's subject (in this case, Hollywood actor and competitive swimmer Johnny Weissmuller).


This paper presents a study on German infinitive-verb compounds with an inflected head verb and an infinitive modifier.

we consider infinitive-verb compounds with the head verbs bleiben and lassen which allow for an idiomatic, i.e., conventionalized figurative interpretation in addition to a literal interpretation.


We include in the corpus not only the adjucated files, but also the diverging annotations.

we consider not all disagreement to be noise, but rather to contain valuable linguistic information that can help us improve our annotation schemes and our learning algorithms.


To these aims, we take a new approach to human-annotated corpus resources by double annotating a much larger part of the corpus than what is normally seen 1 , and by including in the corpus not only the adjucated files, but also the diverging annotations.

we do not consider all disagreement to be noise, but rather to contain valuable linguistic information that we can use to improve our annotation schemes as well as in our learning algorithms.


The better the annotators agree on a particular pattern -concordance match, the smaller the range of their judgments is.

2 Median and Range translate as "Goodness of match between a pattern and a concordance" and "Agreement on this goodness of match", respectively.


To accommodate for multiple word senses, the MSSG model works by predicting the sense of the current word given its context, and then updating the current word sense vector and context vector.

each word w is associated with two types of vector representations, one being the output embedding v(w), and the other being the context v(w).


Given a query that consists of a string and a document ID, the task is to determine the KB entity to which that document's string refers to (or to establish that the entity is not present in the reference KB).

for each query, a system must return a KB ID (or NIL when there is no corresponding KB-instance).


Currently, the Apertium RDF Graph includes data from 22 Apertium bilingual dictionaries and constitutes a large unified array of linked lexical entries and translations that are available and accessible on the Web (http://linguistic.linkeddata.es/apertium/).

its graph structure allows for interesting exploitation opportunities, some of which are addressed in this paper.


Its graph structure allows for interesting exploitation opportunities, some of them addressed in this paper.

we propose a method to discover candidate translations for a given lexical entry and an algorithm to compute the confidence degree for such translations, based on the density exploration of the graph.


Thanks to the mappings of different predicate models, such as SemLink (Palmer, 2009) and the Predicate Matrix (Lacalle et al., 2014), and to their integration in Semantic Role Labeling (SRL) tools, they have become central to a number of tasks such as information extraction, question answering and natural language generation.

due to their laying at the syntactic-semantics interface, predicate models are increasingly used within the Semantic Web (SW) community, for knowledge extraction in tools such as NewsReader (Rospocher et al., 2016) and PIKES (Corcoglioniti et al., 2016), or as the starting point for deriving general-domain ontologies grounded in natural language, such as FrameBase (Rouces et al., 2015) and the ESO ontology (Segers et al., 2015) derived from FN.


pmo:SemanticClass homogeneously represents the semantic classes from the various predicate models.

individuals of this class correspond to rolesets in PB and NB (e.g., pm:nb10-seller.01 and pm:pb17-sell.01 10 in Figure 2), verb classes in VN (e.g., pm:vn32-give-13.1-1), and frames in FN (e.g., pm:fn15-commerce sell).


By extending lemon, PreMOn endorses the latter's goal to provide proper linguistic grounding for ontologies.

preMOn enables to ground event and frame ontologies, such as FrameBase and ESO, on a proper representation of the linguistic information for predicates, thus supporting the development of a comprehensive catalogue, from the linguistic to the knowledge level, of event and event participant types.


Table 1 summarizes the number of monosemous (a single sense) nominalizations for verbs and nouns, described in NomLex-PT and in the morpholinks of PWN.

from 4,238 nominalizations in NomLex-PT, we have only 315 nominalizations where both the verb and the noun are monosemous, less than 10% of the listed pairs.


As a result, the current landscape of morphology consists of isolated but extensive non-RDF resources on the one side, and interoperable Linked Data vocabularies which are insufficiently expressive to model morphology, on the other side.

the fact that concrete segmented morpheme data could not be identified in RDF resources reduces the applicability of the mentioned models, dictionary resources, and tools on languagespecific textual datasets or corpora.


In order to bridge the gaps that currently separate the various existing morphological data resources and models described above, we developed the MMoOn Core model 23 .

it focuses on the description of the necessary concepts and their relations involved in the domain of morphology.


(2012) has been built by applying the LexIt framework to the 'La Repubblica' (Baroni et al., 2004)  When compared with other state-of-the-art extraction models, the LexIt framework has a series of advantages, among which those that are crucial for our purposes are: • the fact that the most salient frames are identified in a unsupervised manner.

it is not based on a pre-compiled list of valid SCFs, as it is the case for the VALEX model; • the fact that this methodology can be applied to different parts of speech; • the fact that it is an open and parameterizable framework, easily adaptable to novel languages or domains.


Some reflections are offered as to the status of roots with respect to previously suggested representations.

roots are, in our opinion are to be not treated as lexical entries, but modeled as lexical metadata for classifying and identifying lexical entries.


This component is used to link a monolingual lexicon (such as ours) to an external resource that may be also in a different language 5 .

two Monolingual External Refs are created, one to link the first 4 The class u/u implies that the perfective scheme is R1aR2uR3a and the imperfective scheme is yaR1R2uR3u.


Finally, for each complex word and the contexts where it appears, we select as simpler synonyms the three most frequent ones that belong to the sense that appears most frequently for the n-gram corresponding to that (word,context) pair.

to disambiguate the sense, CASSA uses the context where the target word appears in the n-grams corpus.


In this paper, we report on our recently completed extension of a German connective lexicon, which has grown both in the number of entries and in the amount of information per connective.

we focus here on the difficult problem of selecting the coherence relations that are to be associated with lexical entries.


The first principle is to retain a primary division of lexical entries into syntactic frames.

when a connective has more than one syntactic category, each of them is stored in a separate <syn> section, where the appropriate syntactic features are collected.


Figure 1 shows the possible marker linkings, where the correct linkings are in bold line, and the other is in dotted line.

two word-pair markers and one single-word marker remain.


The number of punctuation features are 6 (23).

a binary feature for a punctuation mark in each discourse unit.


The first step in this approach includes binarizing the RST trees.

we transform all multi-nuclear relations into nested binary relations with the left-most EDU being the head.


In our proposed annotation scheme, we mark not only the speakers and listeners but also their coreference chains.

we link character mentions to their antecedents using the annotation guidelines for MUC-7 (Hirschman and Chinchor, 1997).


Given the agglutinative properties of the language, it is pedagogically interesting to know which morphemes contribute most to the Esperanto lexicon, in order to teach them first.

on top of basic word lists and word frequencies (Quasthoff et al.


Most often, if W 1 can be motivated with respect to W 2 , then we can also motivate W 2 with respect to W 1 .

most lexemes are connected to each other in both directions.


ognizable expressions of sentiments, application aspects and links between both.

many reviews of the two categories focus on the accuracy (e.g.


(2007)  ) and served as words for which we used to determine whether the participant was fluent in Spanish and/or was taking the task seriously.

if a participant were to indicate ratings for these words that were highly discrepant to those reported by Redondo et al.


Even though our study is based on a substantial amount of work, more things still need to be done.

we are currently testing our annotation guidelines on a new group of undergraduate students to see whether the provided descriptions generalize to other coders as well.


For example, there are 34 positive and 39 negative phrases where the first word is a positive adjective and the second word is a negative noun.

a positive adjective and a negative noun can form either a positive phrase (e.g., happy accident) or a negative phrase (e.g., great loss).


These properties constitute one of the appraisal criteria in the Appraisal Theory Models.

we considered the emotions fear, anger, joy and disgust.


The learning algorithm which we put forward here is a step towards the implementation of this kind of reasoning.

we learn from ConceptNet (Singh et al., 2002) properties of objects, actions and phenomena which cause certain emotions.


According to these guidelines, take (place) in Example 1 above would not be annotated, since it merely helps constituting the description of the single 'real' event in this sentence expressed by demonstrations.

only one event in this sentence should be put on a timeline.


An important property of our annotation scheme is that it is multilayered.

we have defined separate layers for different semantic and pragmatic phenomena involved in the expression of perspectives, and these layers are annotated in a logical order.


13 However, the uncertaint y expressed by suspected is not so much directed towards the proposition as a whole as it is towards a specific aspect of the event: the two Libyans being the ones who did it.

both the AUTHOR and the GEN AUTHOR do commit to the factual status of the Pan Am jumbo jet being blown up over Scotland in 1988, but they are not fully certain about who was responsible (see Figure 3).


Not only do we want to test whether the crowd can achieve the same quality of annotation for such a complex task, but also whether the crowd annotation preserves the compositionality of the perspective for complex constructions, possibly including double negations or nested perspectives.

does the crowd read the text in the same way as experts do?


Secondly, we computed statistics on the number of elongations in the corpus (see Phase 2 of Figure 1).

we searched for the elongation of a specific size of 3, 4, and more than 4 for 5 vowels (a, e, i, o, u), 5 consonants (g, r, t, c, d), and exclamation marks.


In literature, there is a large bundle of studies for injecting some level of word order information into word embeddings (Qiu et al., 2014;Johnson and Zhang, 2015;Lai et al., 2015;Ling et al., 2015;Trask et al., 2015) or into general word-space models (Sahlgren et al., 2008;De Vine and Bruza, 2010;Basile et al., 2011).

we refer to the work of (Trask et al., 2015) that extends the CBOW model proposing to split the context into two partitions, the first containing the sum of all the context word embeddings before the target word and the other containing the sum of all the word embeddings after.


In this paper, we follow the evaluation procedure used to evaluate lexical similarity to sentiment analysis.

we propose to evaluate our word vector spaces against opinion lexicon and opinion thesaurus.


In (Turney, 2012), the compositionality task is investigated in the space of similarities instead of features.

two distinct models are built, namely, domain and function spaces.


Each activation area is formulated as a vectorial representation enabling the construction of VSM on top of which well-established similarity metrics, e.g., cosine of vectors, can be applied.

the words of the entire vocabulary V are used for defining a |V |dimensional vector for a target word w i .


14 http://scikit-learn.org/ zh-to-en translation processes are necessary.

the asterisk mark attached to a language code indicates that the sentence in the language has to be translated into the P L. the task en/ja * -zh * requires both of the target sentences (in ja and zh respectively) to be translated into the P L en.


In the tables: • It is clearly indicated that employing more similarity features can lead to better results (ML+ approach).

the alignment score, that alone could achieved comparable performances in monolingual tasks, should also be incorporated as one of the similarity features into the machine learning process.


Extended syntactic and semantic information was added to the lexicon, in order to formalize and process them.

(i) 33 support verbs were identified as such in the lemma dictionary with the attribute +SUP, (ii) nominalizations that are predicative nouns were assigned the attributes +Npred+Nom, and were formalized by adding derivation codes to 4,968 verbal entries, which allow the generation of the corresponding predicative noun, and (iii) the attribute +VSUP is assigned the support verb that occurs with the predicative noun in the SVC.


Syntactic Features that are composed of dependency syntactic features.

the mean depth and width of the dependency trees in a post and the frequencies of individual relations in the dependency trees of the sentences in a post.


We combine a variety of classifiers whose features encode the words and technical terms in the description of funded grants as well as Latent Dirichlet Allocation (LDA) topics.

for each research topic we train five classification models based on Support Vector Machines (SVMs) that employ as features different combinations of words, techincal terms, LDA topics (Blei et al., 2003) and link LDA topics (Erosheva et al., 2004).


To investigate this further, we computed separate average scores for research topics where the aggregate methods perform well or not so well.

as shown in table 8 we computed average scores for research topics for which the 5 agreements method achieved an F-score higher than 75%, i.e.


The annotation task is framed as a multilabel classification.

each edit will be assigned to one or more edit categories.


In this part, as an example of utilizing the corpus, we designed a machine-learning model to automatically identify the edit categories associated with edits.

the goal was to classify an edit into one or more of the edit categories based on characteristics of the text changed, the comments editors used to describe their edits, and characteristics of the edit.


Given that the selection criterion for lemmas in DeriNet 0.9 was based on their frequency in the Czech National Corpus, most irregularly derived words were probably captured already in DeriNet 0.9, and the complement lemmas can be relatively reliably detected by automatic rules.

we expected that the less frequent a lemma is, the more regularly it is derived from the viewpoint of derivational patterns.


To preserve the integrity of the transcript, we audited the corpus using a combination of automated and manual methods to retain only the sentences for which the speaker read the sentence as prompted.

automatic speech recognition techniques were used to identify potential reading errors, with those sentences audited through human listening.


1 Inter-rater agreement (per session and rater gender) was computed using the Intraclass correlation (ICC; (Shrout and Fleiss, 1979)).

we computed ICC(2, k) as described by Shrout and Fleiss (1979), which estimates the absolute agreement between subjects for k ratings.


A Cohen Kappa test indicates an agreement level of 83% between the three annotators, which belong to the same company: this shows that the task is not so straightforward.

some terms are judged to have a low level of fuzziness may not be tagged.


Next, we collected mention-to-entity mappings from Wikipedia.

each mention candidate m is mapped to a set of potential entities we used the following resources.


Ambient Assisted Living aims at enhancing the quality of life of older and disabled people at home thanks to Smart Homes.

regarding elderly living alone at home, the detection of distress situation after a fall is very important to reassure this kind of population.


We could even add that, they also assess whether this fall is acceptable, and conform to pre-defined scripts.

the non-detection (nonrecognition) of a fall by CIRDO may mean that: (1) The individual has poorly conducted the choreography of the falling; or (2) the movement cannot be categorized as a risk movement.


Right-handedness predominates in the study; only two subjects are left-handed.

the proportion of left-handed people in the study is 4 % and thereby at the lower bound of, but still in accordance with, what would be expected according to geographical variation (Llaurens et al., 2009, p. 882).


the annotation of start and end locations of gestures within the gesture space (Start.Pos, End.Pos) turned out to be difficult, with percentage agreement values ranging from about 60 % to 80 %.


The animated videos as demonstrated in Figure 2 illustrated on the IMAGACT platform exhibit some ambiguity with reference to different types of actions.

one action shows more than two types of representative verbs and hence causes disambiguation problems.


On one hand, there is no necessity for the existence of the object for ghumnA (to move) as it is an intransitive verb, whereas, on the other hand, it is likely for the action ghumAnA (cause to move) to take at least one object.

in the causation of the action ghumAnA (cause to move) the action has to take the agent as the causal factor for the accomplishment of the action.


We build a term relation knowledge base from term bank in computer science areas.

we cluster different kind of relations between terms into groups.


These three types of relations are used as prior knowledge instances to supervise domain ontology learning process in this paper.

we increase each distributional similarity by different weight according to its original value and implemented relation instance type between terms.


See (Eger et al., 2015) who evaluate an approach to incorporating lexical knowledge into taggers without native lexicon  In order to get more insights into tagging accuracy, we investigated the contingency of feature selection and PoS tagging, which may hint at the features' impact on predicting PoS.

we calculated the χ 2 coefficient of representational features on the one hand and parts of speech on the other.


The proposed method and the previous one are complementary.

by both method, we can expect to achieve wider coverage of error correction.


Linguistically annotated data sets are of the utmost importance for the study and computational processing of natural languages.

they enable the training and evaluation of data-driven NLP tools, such as taggers, chunkers and parsers.


The detection of dependency relations of subclauses is a challenging task as there is no relevant prior information in the EDT annotation.

it is not a conversion task but rather an annotation task.


According to the UD guidelines, clausal complement of a verb is a dependent clause which is a core argument.

it functions like an object of the verb.


NDT implements a somewhat broader definition of prepositions than the UD scheme and includes several cases which are counted as adverbs in the universal schemes.

this is the case for demonstrative adverbs such as her 'here' and der 'there'.


We realize now that some of the questions in both surveys lacked context and were interpreted in different ways than expected (and probably differently by respondents).

it seems that our question on whistle blowing ("Do you know of any way to blow the whistle (issue an alert) in case of a major ethical infringement in your domain?")


Notice that the labeling of each area has been done by using the more frequent and representative word of the cloud area itself.

in the upper left side of the cloud, that labeled as family, we can see mariage (marriage), enfants (children), parents (parents) and pére (father); loi (law),égalité (equality) and droits (rights) feature instead the lower left areas, which has been labeled as legal aspects.


This is a multiclass categorization problem in which instances are categorized into one of several classes.

we aim at learning a predictor h : X → Y, where X is the set of messages and Y is a finite set of categories.


In the current study, we analyzed CS data from Twitter for pragmatic or functional aspects (Barredo, 1997;Sanchez, 1983;Maschler, 1991;Blom and Gumprez, 1972).

we are interested in asking the following question: In the context of a particular tweet, is there a pragmatic motivation or functional reason that made the user to switch the code?


The noun(common.adjectival) words are similarly tagged with NOUN or ADJ depending on context.

the stems of the nominal adjectives, e.g., jiyuu "free," are tagged ADJ as heads when used as adjectives, and tagged NOUN when used as nouns, as shown in 3 Suffix UniDic has the suffix (and prefix) categories as independent word classes.


We use name for names constituted of multiple proper nouns, leftheaded.

name would be used between the words of Hillary Rodham Clinton but not to replace the usual relations in a phrasal or clausal name like The Lord of the Rings.


We explore the issue in the context of the Italian Treebank, considering both the efforts involved, how to avoid losing linguistically relevant information and the loss of accuracy in the process.

we compare different strategies for parsing and discuss the implications of simplifying the pipeline when detailed part-of-speech and morphological annotations are not available, as it is the case for less resourceful languages.


Given that specific guidelines are being developed for any level of analysis (data format, sentence splitting, tokenization, POS tagging, morphology, parsing), full support of the new standard has an impact on the whole chain of linguistic tools and training resources, as well as their use in a cascade.

for parsing we need to carefully evaluate two alternative strategies: using native parsers and converters or using parsers directly trained on the UD resources.


Universal Dependencies are an attempt at standardization of syntactic annotations that provide detailed guidelines for all levels of linguistic analysis 1 .

:  data encoding: UTF-8;  data format: an extension of the tab separated CoNLL-X format;  sentence splitting: only one root per sentence;  tokenization: one syntactic word per token; multiword expressions span several tokens; no empty tokens or traces are allowed;  PoS tags and morphological features are to be taken from a fixed inventory (Zeman, 2008); a column with language specific tags is also allowed.


In order to support automatic conversion into UD, the MIDT Treebank was extended with extra information (we call the enhanced version MIDT+).

: additional categories for vocative, discourse, and appositions (appos) had to be manually introduced 2 ; similarly, open clausal complements had to be annotated manually in order to account for the subtle xcomp/ccomp distinction.


Topics have been annotated on top of the syntactic dependency structures, which means that they can easily be queried and analyzed in their syntactic context, bearing in mind that their selection was constrained by the syntactic considerations (which is in contrast to our own project).

(Buráňová et al., 2000) proposed explicit rules for identifying topic and focus on dependency trees.


For computational linguistics, this is clearly relevant for dialogue processing, but also -as argued in this paper -for understanding monolgue text.

topic-related questions have been discussed, modeled and partially implemented under the heading of "entity-based coherence".


We investigate the utility of automatic argumentative and rhetorical annotation of documents for this purpose.

we integrate automatic Core Scientific Concepts (CoreSC) classification into a prototype context-based citation recommendation system and investigate its usefulness to the task.


We index all sentences of the same class into the same field for each document.

we index a bag-of-words for each class (Hypothesis, Background, Method, etc.


Therefore, what we attempt to identify are the fixed and variable components of the query context, which we find more informative about the issuer's goals than the information that can be retrieved via sole pronominalization.

we focus on the discourse entities that remain constant across discourse transitions, which we consider the Topic of the issuer's interest.


This is because query issuers tend to formulate their questions in a way that minimizes usage of complex linguistic phenomena, while maximizing redundant language which may be unnatural in human-to-human conversation.

pronominalization and anaphoric relations as well as sluicing and other types of ellipsis are less evident than expected in natural human-to-human conversation.


The final pilot corpus we used in this study consists of 200 query sessions with the following distribution: • 100 randomly sampled query sessions from the first naturally occurring dataset • 100 randomly sampled query sessions from the second semi-synthetic dataset We have made several observations throughout our corpus analysis, which we will be presenting in this section.

personal pronouns were found to play a vital role in TEL, because they represent a natural way of keeping and exploring the same discourse entity (i.e.


In order to measure the degree of agreement between annotators, we used one of the most commonly applied metrics in NLP research (Carletta, 1996): Kappa.

fleiss's kappa coefficient κ (fleiss, 1971), since there are three annotators for each annotation task as described above.


The cooperative (rather than competitive or argumentative) interaction between the interlocutors may restrict the type of disfluencies, exclamations and laughter exhibited.

in the powerasymmetric Border Control task, we note the prevalence of meta-situational laughter about the strangeness of the task, rather than laughter deriving from the situation of the characters the participants were role-playing.


It is still an open issue which vectorspace projections yield the semantically best clusters (Hill, Cho & Korhonen, 2016) and further experiments are needed.

for storyline (Rissen et al., 2013) clustering the signals for the stories belonging to the same storyline might be not so much the semantic similarity of the articles (they might report various developments of the storyline from differing viewpoints), but rather the matching time and location as well as same organizations and people being involved -the information typically supplied by Named Entity Linking (NEL) tools.


In the case of irony, the tweet text itself is sometimes not sufficient to notice that the author is being ironic.

only the presence of a hashtag indicates that irony is  being used.


The precision in state of the art text classification is approaching the level where a handful of individual classification errors in the evaluation corpus can threaten to dominate over the error rate of the filter in a technology evaluation.

the result from an experiment with an ideal 100% accurate classifier would be reported as something like 99.5% because of classification errors in the corpus -thus, it is not implausible to think that a less accurate classifier could win an evaluation.


The relative frequency of duplicates strongly correlates with the size of the collection from that location.

the bigger a location's collection, the higher the relative amount of published duplicates.


This sentence is coded POS, EXT, and NEG, under Sent_Tone, Attr and Attr_Tone, respectively.

pOS refers to the increase in the profit margin while NEG refers to the increase in raw material prices that negatively affects profit margins (as margins are defined as the difference between sales and direct costs, divided by sales).


Since the hierarchy of types was designed to cover messages posted in newsgroups, forums and issue trackers, the OSSMETER Threaded Corpus contains representative messages from these sources.

the first part of the corpus consists of 345 Bugzilla threads, randomly chosen from the Bugzilla server for eclipse (bugs.


Classification features include the position of messages in threads and the heuristics that were shown to correlate with requests or non-requests in our previous work (Korkontzelos and Ananiadou, 2014).

the heuristics are the following three observation and their combinations 3 : -the prefix "RE:" or "Re:" in the message subject -the occurrence of one or more question marks in the message body -the occurrence of question words, such as what, when and where in the message body Nouns, Adjectives, Verbs 68.0% Table 6: Evaluation results for the single-label setting.


The remaining 8,584 microblogs form the NoisyData dataset.

a further 17.78% (8584/48,305) data are screened out.


If an utterance x only contains language independent tokens, its code-mixing is zero; for other utterances, the level of mixing depends on the fraction of language dependent tokens that belong to the matrix language (the most frequent language in the utterance) and on N , the number of tokens in x except the language independent ones (i.e., all tokens that belong to any language This initial measure has several short-comings.

it does not reflect what fraction of a corpus' utterances contain code-switching, nor take into account the number 1 Note that the formula in Equation 1 differs from, but is equivalent to, the one given in Gambäck and Das (2014).


Certainly, though, no such measure will ever be able to capture all types of differences between corpora.

the ways corpora were collected and annotated, and their intended usage also need to be taken into account.


Similar results are reported in Table 3 where the computed F-scores are averaged over the set of abstracts in the pilot annotation task.

the F-Scores are first computed per abstract file and then they are averaged to report the overall performance.


In many cases, authors shorten terms and use just the head of the nominal group instead of the whole term.

this is often the case in anaphoric contexts (e.g., as in ".


It was decided to perform single annotation to save resources.

one annotator per language pair performed the error classification task to identify the most salient problems of the systems.


In this work, we have shortly reported on the different aspects of evaluation and on the main tools we used.

we have reported on automatic measures and n-gram information provided by the MT-COMPAREVAL tool to obtain quick insights into system quality during development.