the synset "a#00117872 {angered, enraged, furious, infuriated, maddened}" is labeled with three different categories: anger, fury and infuriation).

after removing these synsets and those labeled with an emotional category not included in the 16-categories subset used in this work, the affective lexicon presents 798 synsets.


The disambiguation is carried out only over the words belonging to the grammatical categories noun, verb, adjective and adverb, as only these categories can present an emotional meaning.

we get the stem and sense in WordNet of each word, and this information is used to retrieve its synset.


In this step, this information is used to translate the sentence into a Vector of Emotional Occurrences (VEO), which will be the input to the machine learning classification algorithm.

each sentence is represented as a vector of 16 values, each of one representing an emotional category.


The evaluation is accomplishes using 10-fold cross validation.

100 instances of each dataset are held back for testing in each fold, and the additional 900 instances are used for training.


As described in section 4, we use WordNet to identify the subset of types that can actually produce the given words.

similar to the way Andrzejewski and Zhu (2009) handled a priori knowledge of topics, we will define an indicator variable δ ij that is 1 iff the WordNet information allows word i to be produced from entity set j and 0 otherwise.


From the attribute, each modifier word is drawn independently.

given an attribute b and a set of modifiers d: where s d is number of times modifier d is produced by attribute b.


2 Note the K-SVM can also be trained this way, by including every attribute in every sub-vector, as described earlier.

returning to our preposition selection example, an obvious attribute partition for the CS-SVM is to include as attributes for predicting preposition r only those counts for patterns filled with preposition r. x in will only include counts for context patterns filled with in andx bef ore will only include counts for context patterns filled with before.


We argue that for such tasks, a finer-grained set of cat-egories might be more appropriate than the coarsegrained PoS categories.

we propose a number of tasks for which we compare the performance based on various category sets, including those induced by our model.


For the first two tasks (word prediction and semantic inference), we do not use the content feature in labeling the test set, since the assumption underlying both tasks is that we do not have access to the form of the target word.

we do not measure the performance of these tasks on the Words category set.


Our results imply that in many language tasks, a fine-grained set of categories which are formed in response to the properties of the input are more appropriate than the coarser-grained part of speech categories.

the ubiquitous approach of using PoS categories as the gold standard in evaluating unsupervised category induction models needs to be reevaluated.


In the pseudo-code it is the function: relation-classifier (production, sentence, range1, range2, sub-card-pyramid1, subcard-pyramid2), where range1 and range2 are the ranges of the word indices of the two entities and sub-card-pyramid1 and sub-card-pyramid2 are the sub-card-pyramids rooted at its two children.

along with the two entities and the words in the sentence, information from these subcard-pyramids is also used in deciding the relation at a node.


On the other hand our pipelined approach never outputs incoherent relations because the grammar of relation productions enforce that the relations are always between entities of the required types.

the improvement obtained by our joint extraction method over our pipelined approach is always non-trivial.


If the relations are between relations instead of between entities, then card-pyramid parsing can handle it by considering the labels of the immediate children as RHS nonterminals instead of the labels of the left-most and the right-most leaves beneath it.

it would be interesting to apply card-pyramid parsing to extract higher-order relations (such as causal or temporal relations).


An asynchronous framework may use multiple processors more efficiently and minimize idle time (Nedic et al., 2001;Langford et al., 2009).

in this setting, the master sends θ and a mini-batch M k to each slave k. Once slave k finishes processing its mini-batch and returns g M k (θ), the master immediately updates θ and sends a new mini-batch and the new θ to the now-available slave k. slaves stay occupied and never need to wait on others to finish.


We show that if feature selection does not sensibly reduces w , the margin associated with w does not sensibly decrease as well.

the theoretical upperbound to the probability error does not sensibly increases; (iii) a proof that the convolutive nature of TK allows for filtering out an exponential number of features with a small w decrease.


The lemma suggests that, even in case of very aggressive feature selection, if a small percentage ρ of || w|| is lost, the margin reduction is small.

through Theorem 2, we can conclude that the accuracy of the model is by and large preserved.


all the subtrees whose size ranges from η to ν.

according to Lemma 1 and Theorem 2, we can discard an exponential number of features with a limited loss in accuracy.


During preliminary experiments, we observed an uneven distribution of examples in the traditional training/test split (the same used in P&M).

we used a random selection to generate an unbiased split, with 5,468 sentences for training and 485 for testing.


In P&M, the algorithm would only consider expansions of a fragment f where at most m nodes are expanded.

the set of mined fragments may include some small structures which can be less relevant than larger ones.


Concerning efficiency, the complexity of both mining algorithms is proportional to the number of fragments that they generate.

we can conclude that the new implementation is more efficient by considering that we can achieve the same accuracy with less fragments.


Dependency Parsing, the task of inferring a dependency structure over an input sentence, has gained a lot of research attention in the last couple of years, due in part to to the two CoNLL shared tasks Buchholz and Marsi, 2006) in which various dependency parsing algorithms were compared on various data sets.

of this research effort, we have a choice of several robust, efficient and accurate parsing algorithms.


references) should be avoided (see 3.2 for details).

the community-edited encyclopedia, Wikipedia became one of the subjects of the shared task as well.


On the other hand, the list of such ignorable phenomena is arguable, especially across domains.

we considered the strict boundary matching to be a straightforward and unambiguous evaluation criterion.


Lexical categories contain more information than PoS tags (mainly on subcategorization) and they are more numerous, thereby making their assignment a relatively difficult task.

the parse selection module takes into account multiple predictions per token which allows recovery from supertagging errors while still reducing the ambiguity propagated.


If a cue is a present, any other (potentially "unhedging") token becomes irrelevant to the task.

we cast the task as a binary token classification problem and determine the sentence label from the token-level decisions.


This agrees with the minimal strategy for marking cues stated in the corpus guidelines (Szarvas et al., 2008) and it simplifies scope detection.

during training multi-token cues are resolved to their syntactic head according to the dependency output, e.g.


"indicate that" is the only consistent multi-token cue pattern in the training data.

when our system identifies as a cue a token with the lemma "indicate", if this token is followed by "that", "that" is added to the cue.


The possible reason is our careless work when dealing with the trained model file.

we retrain a model for Wikipedia and the performance is listed on the bottom line (Wikipedia * ) in Table 2.


2 Architecture and Set-Up To ease integration of annotations across system components, we converted the XML training data to plain-text files, with stand-off annotation linked to the raw data by virtue of character start and end positions (dubbed characterization in the following).

hedge cues, scope boundaries, tokenization, Part-of-Speech (PoS) assignments, etc.


We divide entity identification into three token classification tasks, one for each entity type.

we use the original corpus to train three classifiers.


Similarly, for the whole candidate, we have three more fragments: one containing all the tokens before the candidate, another containing all the candidate tokens, and the third one containing all the tokens after the candidate.

there are 12 fragments for each candidate, three for each entity plus three for the whole candidate.


It is also important to note that hedging in scientific writing is a core aspect of the genre (Hyland, 1998), while it is judged to be a flaw which has to be eradicated in Wikipedia articles.

hedge detection in these genres serves different purposes: explicitly encoding the factuality of a scientific claim (doubtful, probable, etc.)


Huang and Low (2007) notes that structure information stored in parse trees helps identifying the scope of negative hedge cues, and Szarvas (2008) points out that the scope of a keyword can be determined on the basic of syntax.

we believe that a highly accurate extraction of syntactic structure would be beneficial for this task.


In our experiment we find that our dependency analysis did not distinguish between the various syntactic roles of quantified NPs.

we marked several sentences with a quantifier as a "weasel" sentence, even where the quantified NP was in a non-subject role -leading to overly many weasel sentences.


As it mentioned in section 2, our system divides scope finding into two sub-tasks:  The first one is the same as hedge detection task in strategy, but quite distinct in target of feature set, because hedge detection is a task of sentence classification while the first step of scope finding aims at high accuracy of labeling hedge cues.

three independent procedures of feature selection are conducted for BioScope corpus dataset.


Although most cue words appear just once, the adjective usually happens to be the same, and we call them core words.

the hedge cue dictionary (HCDic) can be simplified with the core words.


The strings ahead of scope head and tail not listed are also counted, but they do not give such valuable information as the two items listed in Table 4.

when the CRF model gives low confidence, we just set the most probable positions of scope head and tail.


The sentences that contain at least one hedge cue are considered as uncertain, while sentences without cues are considered as factual.

the task of uncertain information detection can be converted into the task of hedge cue identification.


However, sometimes their number predicted by classifier is not same.

we need to process the output of the classifier to get the complete sequence of the scope.


Hedge detection is largely based on the identification of lexical items like suggest and might which indicate sentence level uncertainty.

reasonable hedge detection in English can be accomplished by collecting a list of all lexical items that convey hedging.


The depth of a CCG derivation tree is proportional to the number of word tokens in the sentence.

the processing of a whole derivation tree by the classifier is highly demanding and many subtrees are not relevant for the classification of speculative/non-speculative sentences, in particular when the scope of the speculation is a small proportion of a sentence.


Since these texts are abundant with hypotheses postulated by researchers, one hurdle that an information extraction system must overcome is to be able to determine whether or not the information is part of a hypothesis or a factual statement.

detecting hedge cues that indicate the uncertainty of the statement is an important subtask of information extraction (IE).


Reevaluation was conducted on the labeled test set released by the shared task organizers after our system's output had been initially evaluated.

these figures are comparable to the sentence classification results reported in Farkas et al.


Merging train and trial sets was also motivated by the sparsity of data and the massively different train/trial ratio observed for the two types of biomedical texts (Table 1).

building separate models for abstracts and full texts may   The statistical model considers a sentence uncertain, if it contains at least one cue from a validated set of cue phrases.


Although the recall ratio increases, the precision ratio decreases at the same time.

the ensemble classifier approach is a trade-off between precision and recall.


Naively sampling the lexicons L is computationally infeasible since their sizes are unbounded.

we employ an approximation which turns is similar to performing inference with a Dirichlet Process segmentation model.


Raw word streams, such as utterances transcribed by speech recognizers, are often difficult even for humans (Kim and Woodland, 2002).

one would expect grammar inducers to exploit any available linguistic meta-data.


Infants appear to use the ends of utterances to aid segmentation, and as early at 7.5 months old they are able to recognize novel words in fluent speech if the novel words are presented at the ends of an utterance and not utterance medially (Seidl and Johnson, 2006).

the reliable boundaries presented by the edge of an utterance should be treated as informative for a learner.


Any token in a sentence can be a trigger for one of the nine event types and, depending on their associated event types, triggers are assigned appropriate arguments.

the task can be viewed as a structured prediction problem in which the output for a given instance is a (possibly disconnected) directed acyclic graph (not necessarily a tree) in which vertices correspond to triggers or protein arguments, and edges represent relations between them.


Similarly, the action of labeling "production" as Neg reg in this iteration would incur a cost of 6, as the learned hypothesis would assign a Theme incorrectly, thus resulting in 3 false negative and 3 false positive events.

the learned hypothesis will be informed that assigning the wrong event type to "production" is worse than not predicting a trigger.


We suspect that this is because the predictions are probabilities, not binary indicators of the existence of a location.

the richer real-valued feature space led to greater information for the final model despite any predictive errors in this base model.


As will be seen in the following experiment (Experiment-II) using the Blog dataset as well, the performance of PCFG models is very close to that of unigram language models.

one might wonder whether PCFG models are learning any useful syntactic pattern beyond terminal productions that can help discriminating gender-specific styles in language.


Furthermore, the authors are chosen so that the main research topics are substantially different from each other.

token-based language models are likely to learn topical words and phrases, and suffer when the topics change dramatically between training and testing.


The quality of these classifiers is important, because we will exploit them for contextual opinion analysis.

we evaluate them by 10-fold crossvalidation.


In the case of Turkish, the learner cannot attain the 0.99 level of performance for the 8-form sublanguage at all, though it does so for the 68-form language; this is caused by limitations in learner comprehension as well as the differing frequencies of forms.

the 8-form languages are neither uniformly easier nor uniformly harder than their 68-form counterparts.


In the case of the 8-form languagues, there are 10 denoting utterances, of which 8 refer to one object and 2 refer to two objects.

in situations of this kind (which are 5/6 of the total), utterances referring to two objects are 4 times more likely in the case of 68-form languages than in the case of 8form languages.


With feature feedback, we hope to filter out some of this noise and improve the model.

the amount of irrelevant information in the instance should play an important role in determining the benefit from feature feedback.


It is not clear how their results can be extended to alternate approaches for seeking and incorporating feature feedback.

in this work we analyze the maximum benefit a given problem can get from feature feedback independent of the feedback solicitation and incorporation approach.


On the other hand, with very large training set, the benefit from feature feedback can be expected to be small and not significant for all problems and all problems will look similar.

we evaluate the benefit from feature feedback at different amount of labeled data.


In our analysis, we focus on feedback on features in context of the instance they occur in, i.e., indirect feature feedback through rationales or direct feedback on features that occur in the instance being labeled.


The feature complexity measures require an 'oracle', simulated using a large amount of labeled data, which is often not available.

we need measures that do not require an oracle.


As discussed in Section 3.1, instances with feature feedback take on average twice as much time to annotate as instances without feature feedback.

we measure the labeled data in terms of the number of annotation cost units which may mean different number of labeled instances based on the annotation strategy.


Improvement in learning rate for problems with different granularity was statistically significant but improvement in performance was not significant.

both metrics should be used when evaluating the benefit from feature feedback.


The Shift Reduce parser constructs a dependency tree by performing a sequence of parser actions or transitions through a greedy parsing strategy.

of this parsing procedure, a Shift Reduce parser creates shorter arcs before longer arcs.


The sequences of POS tags that are more frequent in target corpus receive higher scores after proper regularization is applied to prevent potential biases.

the final score assigned to a constituency tree results from a combination of the scores of its extracted sequences of POSs.


For example, word-type features form the backbone of most POS-tagging systems, but types like "gene" and "pathway" show up frequently in biomedical literature, and rarely in newswire text.

a classifier trained on newswire data and tested on biomedical data will have seen few training examples related to sentences with features "gene" and "pathway" (Ben-David et al., 2009;Blitzer et al., 2006).


We can see that, the essential challenge of keyphrase extraction is the vocabulary gap between documents and keyphrases.

the task of keyphrase extraction is how to capture the semantic relations between the words in documents and in keyphrases so as to bridge the vocabulary gap.


In contrast to these baselines, WAM aims to bridge the vocabulary gap on word level, which avoids topic drift effectively.

our method can better solve the problem of vocabulary gap in keyphrase extraction.


The Metathesaurus assigns the synonymous concepts from the various sources a Concept Unique Identifiers (CUIs).

both the Autonomic nerve concepts in SNOMED CT and FMA are assigned the same CUI (C0206250).


However, it has been observed that a sparse W not only save time needed for classification, but also results in better classification accuracy 1 than the full similarity matrix W (Zhu, 2008).

we are concerned with how to sparsify W to obtain a sparse W ; i.e., the strategy of zeroing out some elements of W .


AfterP is determined, we let P ij = max(P ij ,P ji ).

p is a symmetric matrix, i.e., p ij = p ji for all i and j, whilep may  Table 1: Classification accuracy of vertices around hubs in a k-NN graph, before ("original") and after ("hubremoved") hubs are removed.


Further, the focus of this paper is on comparing the performance of our methods to that of the baseline methods.

we leave experiments on other feature types for future work (Section 5).


For each Gibbs sample, the algorithm iterates through all the tokens in the corpus, and for each token it iterates through all the topics.

the time complexity of learning the model is O(SW T ).


Once the model is learned, inferring the topic distribution of a test document of length N takes O(SN T ).

the time it takes to classify a document when using LDAH-S is O(SN T +AT ), where A is the number of authors, and O(T ) is the time complexity of calculating the Hellinger distance between two T -dimensional distributions.


Interestingly, running KOP for 5 iterations on the larger subsets decreased performance compared to running it for 1 iteration.

on the larger subsets, the most accurate KOP results took less time to obtain than those of our LDA+Hellinger variants.


However, our analysis reveals that we do not find such high entropy among the relatives of a polysemous noun that the semantic clustering effect (which is necessary for the success of the disambiguation algorithms described above in Section 3.3) is diminished.

to construct our network, we apply the disambiguation algorithms described above, with the following modification: when confronted with a pair of semantically related polysemous nouns, we apply the disambiguation mechanism described above in both directions, and then fuse the results together.


SSL assumes that, although unlabeled data hold no information about classes (e.g., "opinion" or "non-opinion"), they do contain information about joint distribution over classification features.

when a limited set of labeled data is available in the target domain, using SSL with unlabeled data is expected to achieve an improvement over supervised learning.


Since a document is normally a mixture of facts and opinions (Wiebe et al., 2001), sub-document level opinion detection is more useful and meaningful than document-level opinion detection.

we conduct all experiments on the sentence level.


It is expected to increase the recognition accuracy on domain words, particularly those appearing on the slides.

model 2 is likely to particularly increase the correct matching between bullets and transcript.


However, optimizing both language-model-based probabilities and parser-based probabilities is intractable.

we explore a randomized local search approach that makes greedy revisions using both language models and parsers.


In practice, term vectors are very sparse and their means are close to 0.

the correlation matrix is in fact close to the covariance matrix.


Based on the results on the development set, we found that PCA performs better than OPCA and CCA.

we initialized the models of S2Net and HDLR using the PCA matrix.


Pre-modifier acronyms can be coreferenced unless they refer to a nationality.

in the examples below, FBI can be coreferenced to other mentions, but U.S. cannot.


Another 1.5 points separates them from the fourth best score of 54.53 (nugues).

the performance differences between the better-scoring systems were not large, with only about three points separating the top four systems.


Since this was a logical extension of the task and since the data was available to the participants for the development set, a few of the sites did run experiments of this type.

we decided to provide the gold mentions data to a few sites who had reported these scores, so that we could compute the performance on the test set.


However, OntoNotes does have manual syntactic analysis in the form of the Treebank.

we decided to approximate the minimum spans by using the head words of the mentions using the gold standard syntax tree.


In our particular setup, the recall of the mention detection component is more important than its precision, because any missed mentions are guaranteed to affect the final score, but spurious mentions may not impact the overall score if they are left as singletons, which are discarded by our post-processing step.

our mention detection algorithm focuses on attaining high recall rather than high precision.


This approach combines classification and clustering in one step.

decisions are taken considering the entire set of mentions, which ensures consistency and avoids local classification decisions.


After some experiments over development data, the value of d was assigned to 5.

the negative examples were discarded when they have more than five attribute values different than any positive example.


For instance, the NP "an Internet business" is added as a mention, but also "Internet" itself is added in the case that the word is found once again in the document.

taking into account that just exact boundary matching is accepted, the mention detection achieves an acceptable recall, higher than 90%, but a low precision (see Table 1).


The measure which remains more stable on its scores is MUC given that it is link-based and not takes singletons into account anyway.

it is the only one comparable with the state of the art right now.


However, this model is not appropriate for the OntoNotes-4.0 data set, in which (in contrast to the ACE 2004 corpus) singleton mentions are not annotated: a specific noun phrase (NP) may correspond to a mention in one document but will not be a mention in another document.

we designed a high recall (∼ 90%) and low precision (∼ 35%) rule-based mention detection system that includes all phrases recognized as Named Entities (NE's) and all phrases tagged as NPs in the syntactic parse of the text.


As a result of the singleton mentions not being annotated, there is an inconsistency in the sample distributions in the training and inference phases.

we apply the mention detector to the training set, and train the classifier using the union set of gold and predicted mentions.


In the current version, our system does not consider verbs when creating candidate mentions.

the system does not resolve coreferences involving events.


Our current mention detector does not extract verb phrases.

it misses all the Event mentions in the OntoNotes corpus.


Besides the fact that the current COPA is not resolving any event coreferences, our in-house mention detector performs weakly in extracting date mentions too.

the system outputs several spurious coreference sets, for instance a set containing the September from the mention 15th September.


The first solution that intuitively comes to mind is binary classification of markable pairs (links).

at the heart of most existing approaches there is a binary classifier that classifies links to coreferent/disreferent.


Using an oracle to exclude all spurious mentions at evaluation time yielded improvements ranging from five to thirty percent across the various metrics used in this task.

we decided to focus our efforts on methods for detecting and filtering spurious mentions.


For the CONLL-2011 shared task, we investigated the importance of various additional features that can be included in the feature set used by the memory-based classifier.

we conducted experiments with a base set and an extended feature set, which makes use of lexical semantic features.


During our experiments, we observed that the distance between an entity and its closest antecedent could be as far as 23 sentences.

in the classification process, we empirically set N as 23.


In Broadcast Conversation or Web Log data, a speaker or author would most likely use "I" to refer to himself/herself.

we used it as the other rule to cluster all "I" pronouns and the same speaker information together.


is a PERSON type name entity while "Ye" is a ORG type name entity.

the system erroneously considers them to be a conflict and rejects them.


We use a subset of 750 randomly selected documents for training, since training on the entire set required too much memory.

2 we had ample validation data for tuning thresholds, etc.


While it is still broadly true that want and wollen both have verbal elements located to their right, it is less clear how to design features that can still take advantage of this while working around the differences we have described.

a gap between the per-formance of our features on English and the performance of our projected features, as is observed in Table 2, is to be expected in the absence of a more complete model of syntactic divergence.


Our new formulation, as explained above, relaxes this hard constraint and learns different mapping functions (U i and U j ) and hence our approach can potentially map both the distinct IPA sequences into a single point.

our approach successfully handles the language specific phonemic variation.


Nevertheless, all these approaches require either bilingual name pairs or phoneme sequences to learn to transliterate between two languages.

if we want to build a transliteration system between every pair of languages in a given set of languages then these approaches need resources between every pair of languages which can be prohibitive.


As we increase the residual parameter, it forces the residual vectors (R i ) to be smaller and thus the subspaces identified for each language are closely tied together.

it models the commonalities across languages and also the language specific variability.


Bilingual lexicons are important resources for bilingual tasks such as machine translation (MT) and cross-language information retrieval (CLIR).

the automatic building of bilingual lexicons from corpora is one of the issues that have attracted many researchers.


The same thing happens with the English word "piranha".

the pair of " (piranha)" and "anaconda" could be wrongly identified as a translation pair.


In each propagation, seeds are probabilistically propagated from linked vertices under the condition that larger edge weights allow seeds to travel through easier.

the closer vertices are, the more likely they have similar seed distributions.


However, a preliminary examination showed that the NTCIR parallel data covers less than 3% of all words because there are a number of technical terms and neologisms.

the patent translation task is a task that requires bilingual lexicon extraction from non-parallel data.


We selected documents belonging to the physics domain from each monolingual corpus based on International Patent Classification (IPC) code 6 , and then used them as a comparable corpus in our experiments.

we used 1,479,831 Japanese documents and 438,227 English documents.


Next, function words were removed since function words with little semantic information spuriously co-occurred with many words.

the number of distinct words in Japanese corpus and English corpus amounted to 1,111,302 and 4,099,825 8 , respectively.


For example, " (psychosis)" cannot be used in Andrade(S) because Lex S does not have the seed.

it is more difficult for Andrade(S) to find correct translation pairs.


On the other hand, the proposed methods are able to utilize all the seeds in order to find equivalents for words such as these.

the proposed methods work well even if the coverage of a seed lexicon is low.


In the proposed methods, edges linking rare words are sometimes generated based on accidental co-occurrences, and then unrelated seed information is transferred through the edges.

even our label propagation based methods, especially for Cooc, could not identify the correct translation equivalents for rare words.


For example, the loan word for the English word "user" is often written as " ", but it is sometimes written as " ", with an additional prolonged sound mark.

sim is particularly effective for the experiment data.


Experiments showed that bigram performed better than random, and the other two performed below random.

in the experiments we report later we train one model for patients and one for controls and use the perplexity of a given text according to the bigram language models on word and POS as features in prediction.


Representative applications include studying attention focus through personal pronouns, studying honesty and deception by emotion words and exclusive words and identifying thinking styles (Tausczik and Pennebaker, 2010).

it is reasonable to expect that LIWC derived features would be helpful in identifying schizophrenia patients.


In order to do this, we give labels to the stories told by each subject.

we could use our model to identify the status of the person who told each individual story, the task is to answer the question "Was the subject who told this story a patient or control?".


Doing this with our current dataset is not feasible because there are only about 40 transcripts per emotion.

we use our data to identify significant features that distinguish patients from controls only on narratives from a particular emotion.


2 For convenience let us assume thatf (c i ) ∈ N d .

where |v| 1 = i |v i | is the L1-norm of vector v, let z t be the normalizing constant at t: Now define f j (C), the j-th entry of f (C), as: f (C) represents the global relative frequency of each local, count-based feature.


In this paper we demonstrate a fully generative, latent variable probability model exhibiting both of these properties.

far, consistent simulators have been partially deterministic and have required some handengineering.


We can then define a distribution over utterances as: The simplest variant of g has string values for each of the slots the user is required to provide in order for the dialog to succeed.

we may have: as presented in Schatzmann et al.


Since Baseline 1 barges-in on users after an information peak, when ID may still be high, it continuously receives a negative reward reflecting the user preference for late barge-ins.

of this continuous negative reward, the agent will then learn to avoid barge-ins altogether, which may in turn lead to less efficient interactions because low confidence ASR scores are clarified only late in the interaction.


The discourse structure is a directed graph, where nodes correspond to segments of a document (which we will refer to as "blocks" of text), and the edges define the dependencies between them.

a text block is a set of tokens, while a document consists of the discourse graph and all blocks associated with it.


In recent years, both of the above two EL directions have shown their effectiveness to some extent, and obviously they are complementary to each other.

we believe that bring the above two directions together will enhance the EL performance.


Most sports names are in fact less ambiguous still.

taking a corpus of unlabeled sports articles, we use the information in the database to provide (uncertain) labels, and then train a log-linear model from this probabilisticallylabeled data.


For all remaining tests, we used µ exact for computing features, and µ partial for computing φ(m, DB).

far, all of our features work on relational data, not Wikipedia.


We also test a hypothetical system, Oracle Wikifier, which is given no information about entities in IMDB, but is assumed to be able to correctly resolve any mention that refers to an entity found in Wikipedia.

this system has perfect accuracy on mentions that can be found in Wikipedia, and accuracy similar to a baseline that predicts randomly on all mentions that fall outside of Wikipedia 1 .


Notice that rule R 7 used to define the view Per-sonInvalid illustrates another join predicate of AQL called Overlaps, which returns true if its two argument spans overlap in the input text.

at a high level, rule R 8 removes person candidates that overlap with an Organization span.


This is illustrated in the following CR rule: This is posited to be the way the CR rule learner operates -it tries to learn conjunctions of weak and strong clues so as to filter one from the other.

setting a precision threshold too high limited the number of such weak clues and the ability of the CR rule learner to find such rules.


While our complexity measure informs the biases in our system and leads to simpler, smaller extractors, it captures extractor interpretability only to a certain extent.

we believe more work is required to devise a more comprehensive quantitative measure for interpretability, and refine our techniques in order to increase the interpretability of induced rules.


Traditionally, uncertainty has been popularly used as a basic measurement in active learning (Lewis and Gale, 2004).

how to select most informative MI samples for manual annotation without violating the basic uncertainty requirement in active learning is challenging in imbalanced sentiment classification.


Since the information provided by the base predictors can be contradicting, we use ideas from ensemble learning (Dietterichl, 2002) to learn the most confident indicators and to exploit domain-dependent information revealed by document ratings.

instead of averaging base predictors, MEM integrates their features along with the available coarse-grained training data into a unified probabilistic model.


The score of subsequence u is defined as φ u (s 1 )φ u (s 2 ).

two shared subsequences have high similarity if they share significant words and few gaps.


Two words are considered the same when their synsets overlap.

for example, "writer" has the same representation as "author".


Overall, we found that MEM-Fine is the method of choice.

our multi-expert model can indeed balance the strength of the individual experts to obtain better estimation accuracy.


For a sentence s containing such a collocation c, since the in-sentence features are always ambiguous, it is difficult to disambiguate the polarity of c by using them.

some previous work turned to investigate its surrounding contexts' polarities (such as the sentences before or after s), and then assigned the majority polarity to the collocation c (Hatzivassiloglou and McKeown, 1997;Hu and Liu, 2004;Kanayama and Nasukawa, 2006).


Fortunately, most collocations may appear multiple times, in different forms, both within the same review and within topically-related reviews.

for a collocation, we can collect large amounts of contexts from other reviews to improve its polarity disambiguation.


In addition, as the new query is short, in many retrieved snippets, there also exist no modifying relations between the polarity words and targets.

if we just use this query strategy, the expanded pseudo contexts are limited and cannot yield ideal performance.


As a result, if we just use this query strategy, the expanded pseudo contexts are limited and cannot yield ideal performance.

we need to design some effective query expansion strategies to ensure that (1) the polarity words do modify the targets in the retrieved web snippets, and (2) the snippets are more enough.


The other is the "attribute-head" relation, such as the relationship between them in the sentence "This camera has long battery life".

three heuristic query expansion strategies are adopted to construct efficient queries for searching.


That's because the current review containing c can be considered as only one snippet in pseudo context acquisition.

we can just carry out the two steps in (2) of Figure 2 to obtain the original contexts.


Analyzing either the pseudo contexts or the original contexts, we can find that not all of them are useful contexts.

we will simply filter the noisy ones by context sentiment computation, and choose the contexts showing sentiment orientations as the useful contexts.


Even for the multiple collocations, they averagely appear less than 4 times.

for a collocation, if we only consider its original contexts alone or the expanded pseudo contexts from the domainrelated review set alone, the contexts are obviously limited and unreliable.


Furthermore, the set of variables that may be resampled is fixed given this ordering.

the probability of sampling any possible bispan in the sentence pair is still uniform (ensuring detailed balance), while our sampler remains fast.


In contrast, the models built from the output of the Gibbs sampler for the PYP-SCFG model achieve BLEU scores comparable to those of the MODEL 4 BASELINE.

the sampler has moved a good distance from its initialisation, and done so in a direction that results in better synchronous derivations.


These algorithms formulate the problem as integer linear programming (ILP) to optimize the score: however, as ILP is non-deterministic polynomialtime hard (NP-hard), the time complexity is very large.

we need some more efficient algorithm for calculations.


It is extremely difficult to solve this search problem because the final result of evaluation given by the given score function is not available until it finishes, and we therefore need to try all combinations of textual units.

the score function, which denotes some criterion for the quality of a summary, tends to be determined so that the function can be decomposed to components and it is solved with global inference algorithms, such as ILP.


The agent, however, should search as many possible states as it can.

the generalization function of the feature representation is of utmost importance.


This is achieved by obtaining the optimal state value function, V * (s), because if we obtain this, the greedy policy is optimal, which determines the action so as to maximize the state value after the transition occurred.

our aim is equivalent to finding V * (s).


Such sub-optimality averagely balanced by the feature representation raises the possibility of achieving states that have a highquality summary with a low score, since we do not have a genuine score function.

the most important thing in our method is to intentionally design the features of states and the score function, so that the agent can generalize states, while taking into consideration truly-essential features for the required summarization.


The number of lists reduces from |V | 2 to 2|V | because words in a bigram only have one neighbor.

for any word W in a bigram, we need only 2|V | lists ("words to the right of W" and "words to the left of W") instead of |V | 2 lists ("pairs of words that surround W").


Unfortunately, we do not have parallel Malay-Indonesian text, which complicates the process of learning when to apply these operations.

below we restrict our attention to the simplest and most common operation of word substitution only, leaving the other two 3 operations for future work.


Since the set of Indonesian options for a Malay word in pivoting is restricted to the Indonesian vocabulary of the small Indonesian-English bi-text, this is a severe limitation of pivoting.

assuming a large monolingual Indonesian text, we first build a lexicon of the words in the text.


However, the sampler converges to similar results for both the initializations.

we conclude that the choice of the initialization mechanism is not so important here and we choose the first one due to its simplicity.


The four hyperparameters α e (extended fertility model), β (edge model), γ (distance model), and δ (subtree model), were set by a grid search algorithm, 12 which found the following optimal values: α e = 0.01, β = 1, γ = 1.5, δ = 1 In informal experiments, parameters were tuned also for other treebanks and we found out that they vary across languages.

adjusting the hyperparameters on another language would probably change the scores significantly.


• Some previous papers report results on up-to-10-words sentences only.

we extract such sentences from the test data and evaluate on this subsets as well.


The exceptions are Czech, where both transitions are beneficial, and Swedish, where both are harmful.

when using projective buffer transitions in practice, the language and annotation scheme should be taken into account (or tests should be made) to decide which one to use.


Furthermore, dynamic programming in the style of Huang and Sagae (2010) can be done by merging k-best items that are equivalent in scoring.

our method is an application of integrated decoding with a language model in MT (Chiang, 2007) to dependency parsing, which has previously been applied to constituent parsing (Huang, 2008).


In our formulation, only the topranked labelled arcs would survive in each cell.

label features can be scored without combinatorial explosion.


The root of a sentence is a designated special symbol which all words in the sentence directly or indirectly modify.

the dependency graph for a sentence is constrained to be a directed tree.


The cardinality of the cross product is |x| n • |x| n .

the complexity of the algorithm augmented by n variables is O(|x| 3 • |x| 2n ) = O(|x| 3+2n ), where n ≥ 0.


As we mention below, we also consider conjunctions of all features.

a language family feature can combine with a phonetic context feature to represent a family specific phonotactic constraint.


Since consonants are produced as combinations of the three articulatory dimensions, the greatest number of consonants (for a given set of utilized feature values) will be produced when the distributions are close to uniform.

we can measure how economical each feature dimension is by computing the entropy of its distribution over consonants.


The descendants of this initial name are other names that subsequently evolved for that entity.

the child subtrees of ♦ give a partition of the name types into entities.


Our model says that new tokens are derived from previously generated tokens.

-other things equal-Barack Obama is more plausibly a variant of Barack Obama, Jr. than of Barack Obama, Sr. (which has fewer tokens).


A named entity has the form y = (e, w) where w is a string being used to refer to entity e. A single entity e may be referred to on different occasions by different name strings w. We suppose that this is the result of copying the entity with occasional mutation of its name (as in asexual reproduction).

we assume the following simple generative process that produces an ordered sequence of tokens y 1 , y 2 , .


Again, the approximate posterior probability of a given type parentage tree is edge-factored-it is the product of individual edge weights defined by δ.

we are again eligible to use the spanning tree algorithms in §5.3 below.


However, this approach would be insensitive to the phonological, syllabic and phrasal context that a word occurs in, which can have a large effect on word duration.

we use word duration estimates from the state-of-the-art open-source text-to-speech system MARY (Schröder et al., 2008, version 4.3.1), with the default voice package included in this version (cmu-slt-hsmm).


both A1-1 and A1-2 would contain sentiment expressions with the same polarity as that of Q1.

it is difficult to expect that the sentiment orientation alone will work well for recognizing A1-1 as a correct answer to Q1.


In the second setting, we used the same 10 subsamples of Set1 as in CV(Set1) and exploited Set2 (composed of 7,000 question-answer pairs) as additional training data for 10-fold cross validation.

in each fold 16,000 question-answer pairs (9,000 from Set1 and 7,000 from Set2) were used as training data for re-rankers, and all systems were evaluated on the remaining 1,000 questionanswer pair subsample from Set1.


We believe that this is mainly because SA@W and SWC are based on semantic and sentiment information at the word level, and these often capture a similar type of information.

for instance, disease names that are grouped together into one class in SWC are typically classified as negative in SA@W. the similarity in the information provided by SA@W and SWC causes a classifier trained with both of these features to obtain only a minor improvement over a classifier using only one of the features.


We did not assign subject/object roles in triploids and q-units because a natural language relation phrase might express the inverse of a semantic relation, e.g., the natural language expression 'directed by' and the relation isDirectorOf with respect to the movies domain are inverses of each other.

we check which assignment of arg1 and arg2 is compatible with the semantic relation.


For example, the aspect "size" is often modified by the sentiment terms such as "large", but seldom by the terms such as "expensive."

there are some associations between the aspects and sentiment terms.


Figure 1(a) shows such an example, in which a development set contains two sentences f 1 and f 2 with translations e and feature vectors h. When we tune examples in Figure 1(a) by MERT, it can be regarded as a nonlinearly separable classification problem illustrated in Figure 1(b).

there exists no single weight W which simultaneously obtains e 11 and e 21 as translation for f 1 and f 2 via Equation (1).


MBUU is a batch update mode which updates the weight with all training examples, but MIRA is an online one which updates with each example (Watanabe et al., 2007) or part of examples (Chiang et al., 2008).


3: 10: 13: 14: 16: vided by the source classifier.

the performance of transformation training is correspondingly determined by the accuracy of the source classifier.


An instruction generally refers to a subgraph of this large graph.

there are a combinatorial number of possible meanings to which a given sentence can refer.


The intuition behind utilizing an LHG is that the MR for each lexeme constitutes a semantic concept that corresponds to some naturallanguage (NL) word or phrase.

the LHG represents how complex semantic concepts are composed of simpler semantic concepts and ultimately connected to NL words and phrases.


This is not tractable for more complex MRs.

our extension incorporates a learned lexicon to constrain the space of productions, thereby making the size of the PCFG tractable for complex MRs, and even giving it the ability to handle infinite MR languages.


The vector z i contains the latent mention-level classifications for the ith entity pair, while y i represents the corresponding set of gold-standard labels (that is, y Using these notations, the log-likelihood of the data is given by: The joint probability in the inner summation can be broken up into simpler parts: where the last step follows from conditional independence.

the log-likelihood for this problem is not convex (it includes a sum of products).


Inference during training: During the inference process in the E-step, the algorithm incrementally "flips" mention labels based on equation 2, for each group of mentions M i .

z i changes as the algorithm progresses, which may impact the label assigned to the remaining mentions in that group.


(2010) observes that evaluating on this corpus underestimates true extraction accuracy because Freebase is incomplete.

some relations extracted during testing will be incorrectly marked as wrong, simply because Freebase has no information on them.


A subsequent qualitative analysis revealed that most of the heterogeneity was coming from textbooks following the new didactic approach recommended by the CEFR: the task-oriented approach, which focuses more on the task than the text when labelling the overall reading activity.

we decided to remove those type of textbooks from our corpus, which amounted to 5 books and 249 texts.


Like Exp1, it also contains four predictors, but they are all lexical or syntactic features.

this model does not maximize the type of information.


However, the assumption of upward movement in the phrase structure tree should rule out movement into the projection interval of a non-ancestor.

under these syntactic assumptions, we would expect at most one child to inherit a parent's gap.


7 The terminology is a nod to the London Underground but imagines parents admonishing children to mind the gap.

(b) If the interval does not contain n, all edges incoming to that interval come from n. (a) If the interval contains p, then the only edge incoming is from p's parent to p (b) If the interval does not contain p, then all edges incoming to that interval come from p. of the above, Proof.


No child can have descendants both to the left of x r and to the right of x r , because otherwise that child and x would be ill-nested.

we can split up the interval at x r to have two gap-minding trees, both rooted at p. The score of T is then the sum of the scores of the best subtree rooted at p over [i, k] (C[i, k, p]) and the score of the best subtree rooted at p over The above cases cover all non-empty gapminding trees, so the maximum will be found.


A joint model is expected to make more optimal decisions than a pipeline approach; however, such a model will be too complex and it is difficult to estimate model parameters.

we do not perform joint inference for training.


A problem arises when combining the two models and using it in joint segmentation and parsing, since the linear chain used in the character-based model is incompatible with CYK parsing model and the word-based model due to the transition informa- tion.

we slightly modify the linear chain CRFs by fixing the weights of transition features during training and testing.


We used the same preprocessing step as (Harper and Huang, 2009), collapsing all the allowed nonterminal-yield unary chains to single unary rules.

all spans in the binarized trees contain no more than one unary rules.


By Bayes' rule, where the prior probability P (S) can be viewed as uniform, and the P (Q) is constant across all sentences.

selecting a sentence to maximize P (S|Q) is equivalent to selecting a sentence that maximizes P (Q|S).


As pointed out by the previous work (Berger and Lafferty, 1999;Gao et al., 2010;Zhou et al., 2011), sentence-query selection requires a distillation of the sentence, while selection of natural language tolerates little being thrown away.

we restrict our attention to those key sentence words that form the distillation of S, do not consider the unaligned words in S, and assume that Q is transfered only from the key sentence words.


We identify and eliminate unimportant words, somewhat similar to Q&A retrieval (Lee et al., 2008;Zhou et al., 2011).

the average number of words (the total word number in Q and S) in cross-lingual sentence selection model would be minimized naturally, and the efficiency of cross-lingual data selection would be improved.


Perplexity is a metric of LM performance, and the lower perplexity value indicates the better performance.

we estimate the perplexity of adapted LMs according to English reference translations.


The observed improvement suggests that better initial translations which have less noisy data  lead to better adapted LMs, and thereby better second iteration translations.

it is advisable to use cross-lingual data selection for LM adaptation in SMT, which can address the problem of noisy proliferation.


In this paper, we are interested in developing a new topic model which can take into account the structural topic dependency by following the higher level document subject structure, but we hope to retain the general flavour of topic models, where components (e.g., sentences) can be a mixture of topics.

we need to depart from the earlier HMM style models, see, e.g., (Blei and Moreno, 2001;Gruber et al., 2007).


If it contributes to t i,j,k , then it recurses up to contribute a data count to the PDP for document i segment j − 1.

it also needs a table indicator at that node.


Table 2 shows the 11 contexts (l, v) that appear most frequently in T .

each target word's vector is defined component-wise as: for j = 1, ..., D. This approach is used when a fixed vector dimensionality is necessary.


Included features are in parentheses: "con" is sentence vector concatenation, "sub" is sentence vector subtraction, "other" stands for 4 other features (see Section 4) each sentence and how often, we define a vector wdCount i for sentence Sen i and enumerate all words occuring in the MSRPC: giving the word count vectors n MSRPC dimensions.

the k-th component of wdCount i is the frequency with which the word w (19) for k = 1, ..., n MSRPC .


It also suffers from the paucity of error-annotated training data for grammar correction.

applying a standard SMT system to error correction does not produce good results, as we show in this work.


As the search space grows exponentially, it is infeasible to perform exhaustive search.

we prune the search space by only accepting the most promising hypotheses to the pool of hypotheses for future consideration.


Instead, it has to have seen the exact correction rule, e.g., house → the house, in the training data.


As pointed out in Section 3.5, the majority of sentences require zero or few corrections.

the depth of the search tree is typically small.


The selected POS pairs were transformed into compound tags.

we generated three sets of POS units by this process: the original POS set without the compound unit (Base), the original set and an additional 50 compound units (Base+mi50), and the original set and an additional 110 units (Base+mi110   For each ngram, three sets of VSMs were built using three sets of tags as terms, yielding a total of nine VSMs.


Given the pinyin input A, the task is to find the best output hanzi sequence W*: In traditional conversion systems which do not consider spelling errors, P(A|W) is usually set to 1 if the word is found in a dictionary of wordpronunciation pairs, which also defines GEN(A).

the ranking of the candidates relies exclusively on the language model probability P(W).


A unique challenge of the current task is that the parallel data for A and W do not exist directly.

we generated the translation phrase table offline by merging the 8 No detail of this data is available in Chen and Lee (2000).


(2011) uses the existence of shared words as a feature to determine whether the given pair of expressions are paraphrases, and thereby extracts many pairs sharing the same words.

their output has a high potential to be used as an alternative seed for our method.


When sampling in a topic modeling setting, documents are unique and are never seen again.

the topics associated with old documents get stale and need to be periodically rejuvenated (i.e.


Indeed, using the approach described in (Ponomareva and Thelwall, 2012) we can choose the most appropriate data for training our classifier rather than averaging the results given by all data sets.

instead of average accuracies, the best accuracies with respect to the same target domain should be compared.


Then, we can compute the posterior which is essentially a weighted linear combination of a set of features.

to instantiate the MRF model, one needs to define a graph structure and a set of potential functions.


The potential functions over these cliques are defined as 7where the feature is defined as the log probability of generating an expansion bigram given Unlike the language models used for document ranking (e.g., Zhai and Lafferty 2001), we cannot compute the bigram probability by simply counting the relative frequency of in because the query is usually very short and the bigram is unlikely to occur.

we approximate the bigram probability by assuming that the words in are independent with each other.


2010), we assume that all features within the same feature class are weighted by the same tied parameter .

the number of free parameters of the MRF model is significantly reduced.


Both TC and SMT, considered as state-of-theart QE methods, have been frequently used for comparison in related studies.

we also used them as baselines in our experiments.


However, we found that the bilingual topic model often selects a different set of expansion terms and is complementary to the other two lexicon models.

unlike the case of combining the word model and triplet model features, incorporating the bilingual topic model features in the ranker leads to some visible improvement in NDCG at all positions (Row 11 vs. Row 10).


This study is part of a bigger, ongoing project, aiming to develop a real-time QE system for Web search, where simplicity is the key to the success.

what we learned from this study is particularly encouraging.


In the nonadjacent case (adj = F), these will be different words and their classes will, in general, not be the same.

2 non-adjacent stopping decisions will be made independently of a head word's identity.


2 Thus, non-adjacent stopping decisions will be made independently of a head word's identity.

all word classes will be equally likely to continue to grow or not, for a specific proposed fringe boundary.


However, sentence lengths -for which stopping probabilities are responsible -and distributions of root words may be different.

an additional comp parameter is added to the context of two relevant types of factors: For example, the new stopping factors could capture the fact that incomplete fragments -such as the noun-phrases George Morton, headlines Energy and Odds and Ends, a line item c -Domestic car, dollar quantity Revenue: $3.57 billion, the time 1:11am, and the like -tend to be much shorter than complete sentences.


include parenthesized expressions that are marked as noun-phrases, such as (See related story: "Fed Ready to Inject Big Funds": WSJ Oct. 16, 1989); false negatives can be headlines having a main verb, e.g., Population Drain Ends For Midwestern States.

our proxy is not perfect but seems to be tolerable in practice.


To initialize this model, we combined the final instance of DBM-2 with uniform multinomials for punctuation-crossing attachment probabilities (see §2.3).

average performance improved to 42.2% (see Table 8).


However, the scale and complexity of such collections frequently make it difficult to quickly grasp the important details and the relationships between them.

automatic interfaces for data navigation, exploration, aggregation, and analysis are becoming increasingly valuable.


To understand why this is the case, note that determinants are closely related to volumes; in particular, det(L Y ) is proportional to the volume spanned by the vectors q(y i )φ(y i ) for y i ∈ Y .

sets with high-quality, diverse items have the highest probability; see Figure 3 for an illustration.


If multiple domains are provided, we gather all the domain-specific ensembles into a grand ensem- Generally, the feature distributions are different in different domains.

effective domain adaptation requires using some knowledge of crossdomain similarity.


Nevertheless, it is difficult to judge the quality of the ensemble members in advance.

this validates the importance of choosing an active set of ensemble members in decision inference.


This pruning step is required because the common noun pattern generates a large number of lexical categories, the majority of which are incorrect.

we eliminate all common noun categories instantiated by fewer than  Table 1: Dependency parse patterns used to instantiate lexical categories for the semantic parser lexicon Λ.


Without sufficient text data for training, it is very difficult to build a practical and usable statistical language model.

most of the advances have been reported in so called resource-rich language such as English, Mandarin and Japanese, after creating linguistic resources of these languages at considerable cost.


Thirdly, the number of reordering permutations or paths are formidable when the reordering distance L is long as suggested by Table 1.

we apply histogram pruning to reordering paths, which only maintains top N most likely ones.


Given that John and Jim are the subject and object of kill, we can extract their semantic roles directly from the frame, which are killer and victim, respectively.

6 we replace the two names with their extracted semantic roles, and generate the search queries from the resulting sentence in the same way as before.


A possible generative process for a document m can be: It starts from A1, and chooses node A17 at level 2, and then chooses A18, A20 and A25 in the following levels.

we obtain a path: c m = {A1, A17, A18, A20, A25}.


We build the mod-els based on the train set and compute the preplexity of the test set to evaluate the models.

our goal is to achieve lower perplexity score on a held-out test set.


• Let {D-CONNECT(i, j, k) : 0 ≤ i < j ≤ n; 0 ≤ k ≤ n} be O(n 3 ) factors coordinating any number of dependency syntax Link(i, j) variables with representational variables on the end task, multiplying in 1 to the model score unless all variables are on, in which case it multiplies a connective potential φ derived from its features.

it functions logically as a soft NAND factor.


Likewise, there will never be a situation where the tag dictionary rules out all possible tag transitions between two adjacent tokens in training or testing.

no smoothing is needed in this scenario.


Because many D k achieve a particular reordering F , many reorderings F are able to minimize the loss L(F k |F k , A k ).

it is necessary to choose a single oracle derivation to treat as the target out of many equally good reorderings.


In order to keep track of these values, it is necessary to augment each node in the tree to be d = s, l, c, c + 1, r, tl, tr with two additional values tl and tr that indicate the position of the leftmost and rightmost words after reordering.

a straight nonterminal parent d with children d l = s l , l, lc, lc+  1, c, tl, tlr and d r = s r , c+1, rc, rc+1, r, trl, tr will have loss as follows with a similar calculation being possible for inverted non-terminals.


As mentioned in Section 1, monolingual syntactic structure is not necessarily suitable for translation, and sometimes the bilingual information in word alignment may help the parser find out the correct structure.

it is desirable if there is a way to re-train a parser with bilingual information.


Our primary goal in adapting parse trees is to improve their agreement with a set of external word alignments.

our first step is to define an agreement score metric to operationalize this concept.


For our task, we have already defined the evaluation metric (Section 2) and the initial state annotator will either be the gold Treebank trees or a Treebanktrained PCFG parser.

to fully describe our system, it only remains to define the set of possible tree transformations.


In addition to missing fluents for an entity, some temporal bounds may be missing from the database; missing bounds are unfortunately indistinguishable from unbounded ranges.

we can only trust concrete temporal boundaries in the database.


Instead, we only have positive examples of consistent timelines from the database.

we must treat predicting consistency as a density estimation rather than a classification problem.


There is nothing which requires that the fluents in question come from a single entity.

we can trivially ask questions about two entities which are linked by a fluent.


All dates need to be given at day resolution.

for gold fluents with only year-or month-level resolution, we treat them as their earliest (for starts) or latest (for ends) possible day.


We use four window sizes (1, 3, 6, and 12 months) and induce different features for each window size.


This differs from our semantic typing of unlinked entities because our approach assumes access to corpora-level relationships between a large set of linked entities (with semantic types) and the unlinked entities.

we are able to propagate 1,339 Freebase semantic types from the linked entities to the unlinked entities, which is substantially more types than fine-grained NER.


Thus, a token near the beginning or end of a sentence is unlikely to be the end of a segment.

for each token we include its relative position in the sentence and distances to the beginning and end as features.


While the MSR corpus is larger than our collection, the wording variations in its paraphrase pairs are usually lower than for our examples.

the final numbers of previous approaches might be vaguely comparable with our results: Das and Smith (2009) present two systems reaching f-scores of 0.82 and 0.83, with a precision of 0.75 and 0.80.


Since the linear order is exactly what we aim to produce, this deterministic conversion cannot be applied before linearization.

we use a statistical classifier as our initial lifting component.


In particular, we wanted to check whether the lifting-based linearizer produces more natural word orders for sentences that had a non-projective tree in the corpus, and maybe less natural word orders on originally projective sentences.

we divided the evaluated items into originally projective and non-projective sentences.


However there is as yet no satisfactory computational model that can match human performance.

identify-ing the best set of features and best learning algorithms for syntactic category acquisition is still an open problem.


The completeness errors are not surprising given that the words that have been split are not generally substitutable with the other members of their Penn Treebank category.

it can be argued that metrics that emphasize homogeneity such as MTO are more appropriate in this context than metrics that average homogeneity and completeness such as VM as long as the number of clusters is controlled.


We suspect that this was not an issue in previous studies with the coherence measures as LDA prefers to form topics from words that co-occur frequently, whereas NMF and SVD have no such preferences and often create low quality topics from completely unrelated words.


In fact, the lexical entry for "in" has more than 10 good translations in Portuguese.

the lexical translation entry for Word-based models splits the probabilistic mass between different translations, leaving the choice based on context to the language model.


However, if we add another word such as "Portugal" forming "in Portugal", it is more likely that "in" is translated to "em".

we define the joint event of "in" translating to "em" (A 1 ) and "Portugal" to "Portugal" (B 1 ), denoted as A 1 ∩ B 1 , in the phrase pair p(in P ortugal, em P ortugal).


Thus, we assume that all phrase pairs have the same probability regardless of their context in a sentence.

our pruning algorithm takes a threshold δ and prunes all phrase pairs that fail to meet the following criteria: The main components of this function is the ratio between the emission from the pruned model and unpruned models given by Pp(t|s) P (t|s) , and the weight given to each s, t pair given by P (s, t).


On the other hand, on the pruned model where p(s, t) does not exist, only S 1 , S 2 and S 3 can be used.

given a s, t pair one of three situations may occur.


Another problem with this algorithm is that the decision to prune each phrase pair is made assuming that all other phrase pairs will remain in the model.

there is a chance a phrase pair p 1 is pruned because of a derivation using p 2 and p 3 that leads to the same translation.


We believe that this is related by the fact the EUROPARL corpora is generated from automatically aligning documents, which means that there are misaligned sentence pairs.

many spurious phrase pairs are extracted.


For instance, "八(8) 三(3) 八(8)" is translated to "eight three eight" (Ex: "room eight three eight").

phrase pairs for number sequences can be removed, since those numbers can be translated one by one.


Note that here the pruning criterion only considers redundancy of the phrases, not the quality.

we are not saying that the government of France is a better translation than the French government, only that it is less redundant.


We can then choose a pruning threshold τ E and prune those phrase pairs with a contribution to the relative entropy below that threshold.

we prune a phrase pair (f ,ẽ), if We now address how to assign the probability p (ẽ|f ) under the pruned model.


We perform experiments on the publicly available WMT shared translation task for the following four language pairs:  • French-English For each pair, we train two separate system, one for each direction.

it can happen that a phrase is pruned for X-to-Y, but not for Y-to-X.


1 There exists a oneto-one correspondence between substitution edits and word alignments.

this example state transition sequence correctly generates an alignment for the word 43 and people.


In preliminary experiments, we found that most of the gain came from unigrams and bigrams, with little to no additional gains from trigrams.

we limited our experiments to bigram pFSM and pPDA models, and pruned the paraphrase table adopted from TERplus 5 to unigrams and bigrams, resulting in 2.5 million paraphrase pairs.


from the distribution of instances the system will encounter in practical use.

we also wish to compute how calibration degrades as the domain of the test set changes.


For example, "会见" is composed of two characters: " 会" and " 见" which have their own semantics and the semantics of " 会见" comes from that of its component characters "会" and "见".

it is natural to infer unknown triggers by employing compositional semantics inside Chinese triggers.


Finally, note that we train a separate PRA model for each relation r. Path Discovery: Given a graph and a target relation r, the total number of path types is an exponential function of the maximum path length and considering all possible paths would be computationally very expensive.

b is constructed using only path types that satisfy the following two constraints: 1. the path type is active for more than K training query nodes, and 2. the probability of reaching any correct target node t is larger than a threshold α on average for the training query nodes s. We will discuss how K, α and the training queries are chosen in Section 5.


A key observation is that the different stages of the PRA algorithm are based on independent computations involving individual queries.

we can use the MapReduce framework to distribute the computation (Dean and Ghemawat, 2008).


"President Obama") edges in text; otherwise these features are inactive.

these features are active only for specific subsets of queries.


The challenge for this phase is to cluster a very large set of Type A relations, on which it is infeasible to run a clustering algorithm that does pairwise all pair comparison.

we designed an evidence-based partition-then-cluster algorithm.


Furthermore, both phases can be run in parallel in a distributed computing environment because data is partitioned.

it is scalable and efficient for clustering a very large number of relation instances from a large-scale corpus like the web.


We should note that similar data selection methods can be applied to each of the baseline systems and the comparison to them would be fair with conducting that.

we consider our results without data selection to be final.


UPDATE: For each new item "x" with count c, the sketch is updated as: QUERY: Since multiple items can be hashed to the same index for each row of the array, hence the stored frequency in each row is guaranteed to overestimate the true count, which makes it a biased estimator.

to answer the point query (QUERY (x)), CM returns the minimum over all the d positions indexed by the hash functions.


However, FLAG applied FAST-PLEB (approximate search) to find these neighbors.

many of these edges can be noisy for our applications.


The results for LLR in Table 3 show similar trends with CM-CU having best results on small size sketches.

cM-cU scales using 10M counters (using fixed memory of 40 MB versus 4.6 GB to store exact counts).


This approach starts with an empty set of samples, S = {}, and iteratively samples factors without replacement to add to S, until the confidence interval around the estimated mean falls below a user specified maximum interval width threshold i.

for proposals that contain high-variance factors, this strategy examines a large number of factors, while proposals that involve similar factors will result in fewer samples.


Our proposal function selects a random mention, and moves it to a random entity, changing all the pairwise variables with mentions in its old entity, and the pairwise variables with mentions in its new entity.

evaluation of such a proposal function requires scoring a number of factors linear in the size of the entities, which, for large datasets, can be a significant bottleneck.


Although most of these methods apply MCMC directly, the rate of convergence of MCMC has become a concern as larger and more denselyfactored models are being considered, motivating the need for more efficient sampling that uses parallelism (Singh et al., 2011;Gonzalez et al., 2011) and domain knowledge for blocking (Singh et al., 2010).

we feel providing a method to speed up MCMC inference can have a significant impact.


Now, computing the ratio p(x 1 )/q (0) (x 1 ) gives a result much below 1, because from the viewpoint of the full model p, the trigram (the two dog) is very unlikely; in other words the ratio w 3 (dog|the two)/w 1 (dog) (and, in fact, already the ratio w 2 (dog|two)/w 1 (dog)) is very low.


However, no attempt is made to organize these patterns into synonymous patterns, let alone into a taxonomy.

the patterns themselves do not exhibit semantics.


More importantly, all methods in the TextRunner/ReVerb family are blind to the ontological dimension of the entities in the patterns.

there is no notion of semantic typing for relation phrases as in PATTY.


Textual patterns are tied to the particular surface form of the text.

we transform the textual patterns into a new type of patterns, called syntacticontologic-lexical patterns (SOL patterns).


This means that the pattern is semantically meaningless.


However, if a support set S has only few elements due to sparsity, it may become a subset of another support set B, even if the two patterns are semantically different.

one has to take into account also the support, i.e., the size of the set S. Traditionally, this is done through a weighted trade-off between confidence and support.


In addition, modulo backoff or smoothing, that unstructured symbol would often have rewrite parameters entirely distinct from, say, the indefinite but otherwise similar variant of the symbol (Klein and Manning, 2003).

when designing a grammar, one would have to carefully weigh new contextual annotations.


If we compile all annotation components into unstructured annotations, we can end up with a total grammar size of O(A M G 0 ), and so in general parsing time scales exponentially with the number of annotation components.

if we use latent annotations and the hierarchical splitting approach of Petrov et al.


(2006), then the grammar has size O(8 S G 0 ), where S is the number of times the grammar was split in two.

the size of annotated grammars can reach intractable levels very quickly, particularly in the case of latent annotations, where all combinations of annotations are possible.


(We describe this procedure more precisely in Section 5.4.)

each term f m is approximated in the context of the terms that come before it.


This function can be easily integrated into the CKY algorithm for a single annotated grammar by simply multiplying in the value of q whenever computing the score of the respective production over some span.

the modified inside recurrence takes the form: parsing with a pointwise product of an anchored grammar and an annotated grammar has no increased combinatorial cost over parsing with just the annotated grammar.


As expected, ADF does not perform as well as EP.

we exclude it from our subsequent experiments, focusing exclusively on EP.


Every discourse has a purpose: explaining a concept, narrating an event, critiquing an idea and so on.

each sentence in the article has a communicative goal and the sequence of goals helps the author achieve the discourse purpose.


Probability of this shorter match is based on how often the n-gram is seen in different contexts.

these shorter n-gram probabilities are not representative of cases where context is short simply because additional context is unknown at the time of scoring.


Feature functions have the possibility to store their own state information along with the document state to make sure the required information is available.

the framework makes it possible to exploit decomposability for efficient scoring without imposing any particular decomposition on the features as beam search does.


(2007) is the introduction of the possibility of handling document-level models by lifting the assumption of sentence independence.

enumerating the entire neighbourhood becomes too expensive, which is why we resort to a "first-choice" strategy that non-deterministically generates states and accepts the first one encountered that meets the acceptance criterion.


We use viable prefixes (Aho and Johnson, 1974) to indicate all possible target strings the translations of each node should starts with.

given a tree node to expand, our algorithm can drop immediately to target terminals no matter whether there is a gap or not.


As the translations of node IP should begin with either "the result" or "was released at night", the first rule must be either r 2 or r 5 .

r 1 will never be used as the first rule in any derivation.


Hence, the output of these 6 examples has exactly one of the truth representations, making it recursively compatible with further combinations of operators.

we can combine these operators to construct any propositional logic function of any number of inputs (including xor).


Recall that the cosine similarity between word-vectors in the original matrix W are preserved in the subspace representation of words.

if we construct the original matrix so that the columns representing antonyms will tend to have negative cosine similarities while columns representing synonyms will tend to have positive similarities, we will achieve the desired behavior.


Additive and multiplicative functions might also be expected to handle subsective modification with some success because these operations provide a natural account for how polysemy is resolved in meaning composition.

the vector that results from adding or multiplying the vector for white with that for dress should differ in crucial features from the one that results from combining the same vector for white with that for wine.


We noticed that Wikipedia categories have a simple structure of either <noun-phrase> or <noun-phrase><relation-token><noun-phrase>, where in the second case the category information is always on the left.

we first remove the text succeeding a set of carefully chosen relation tokens 5 .


(Rahman and Ng, 2011) used YAGO for similar purposes, but noticed that knowledge injection is often noisy.

they used YAGO only for mention pairs where one mention was an NE of type PER/LOC/ORG and the other was a common noun.


These features are also used in conjunction with string edit distance.

if our system sees two named entities which share the same finegrained type but have a large string edit distance, it will label the pair as non-coref.


The reason is that to avoid redundancy, humans refer to the same entity differently within the sentence, preferring titles, nicknames and pronouns.

when a similar-looking named entities appear in the same sentence, they are actually likely to refer to different entities.


3, our goal is to link vessel to Kursk and assign it the Russian/Soviet nationality prior to applying the pairwise co-reference classifier on (vessel, Norwegian ship).

our goal is to apply the pairwise classifier on pairs in prescribed order and to propagate the knowledge across mentions.


This prevented us from applying our approach to the Ontonotes dataset where singleton clusters and co-referent nested mentions are removed.

the gold annotation for training several sieves of our scheme is missing (e.g.


Because these classification decisions are local, they do not guarantee that candidate mentions are partitioned into clusters.

a mention clustering step is needed to resolve conflicts and generate the final mention clusters.


Best-First constraint: Here we assume that coref(j,i) returns true if candidate mentions j and i are coreferential and false otherwise.

for each candidate mention i, we should only select at most one candidate mention j to return true for the predicate coref(j,i) from all its preceding candidate mentions.


Algorithm: Automatic Taxonomy Optimization.

we propose to minimize the sum of semantic distances along a root-to-leaf path.


To collect a set of labels with many buzzes, we focused on the 1186 labels with more than four distinct questions.

we shuffled the labels into a canonical order shown to all users (e.g., everyone saw a question on "Jonathan Swift" and then a question on "William Jennings Bryan", but because these labels have many questions the specific questions were different for each user).


Cases where the opponent buzzes first but is wrong are equivalent to rapacious classification, as there is no longer any incentive to answer early.

we exclude such situations (Outcomes 3, 5, 6 in Table 2) from the dataset to focus on the challenge of processing clues incrementally.


We called this strategy the oracle strategy; it was able to consistently win against its opponents.

while the content model was able to come up with correct answers often enough to on average win against opponents (even the best human players), we were unable to consistently learn winning policies.


We used the maximal seating assignment (Wallach, 2008) for propagating counts through the Dirichlet hierarchy.

if the word v appeared B l,u,v times in label l following a preceding word u, S l,v times in label l, T c,v times in category c, and G v times in total, we estimate the probability of a word v appearing in label k, category t, and after word u as p where we use • to represent marginalization, e.g.


a learning algorithm cannot generalize the behavior of features from one domain to another.


Since we are doing POS tagging, we want the distribution to learn the information encoded in the original POS labels "JJ NNS RB VBP DT NNS".

by calculating the entropy between the hidden sequence and real labels, we can identify a subset of desired distributions that have low entropy, shown in the dotted oval.


Note thatP q depends on q, but unlike the original P from Equation 6, they do not depend on the data completions y.

we can computeP q once for each q t , and then substitute it forP for all values of Y in the computation of E Y∼q φ entropy (Y, z), making this computation tractable.


In our work (Huang and Yates, 2009), we used a plain HMM for domain adaptation tasks in which there is labeled source data and unlabeled source and target data, but no labeled target data for training.

here, we use the HMM technique as a baseline, and build on it by including biases.


CRFs, however, do not readily model potentially useful segment-level information like syntactic constituent structure.

we propose a semi-CRF-based approach to the task that can perform sequence labeling at the segment level.


Specifically, possible segmentations are generated by taking into account likely syntactic structure during learning and inference.

arbitrarily long expressions can be modeled and their boundaries can be influenced by probable syntactic structure.


To extract opinion targets, many studies regarded opinion words as strong indicators Popescu et al., 2005;Liu et al., 2005;Qiu et al., 2011;Zhang et al., 2010), which is based on the observation that opinion words are usually located around opinion targets, and there are associations between them.

most pervious methods iteratively extracted opinion targets depending upon the associations between opinion words and opinion targets (Qiu et al., 2011;Zhang et al., 2010).


mining associations between opinion targets and opinion words is a key for opinion target extraction .


In addition, by using WTM, our method can capture the "one-to-many" or "many-to-one" relations ("one-to-many" means that, in a sentence one opinion word modifies several opinion targets, and "many-to-one" means several opinion words modify one opinion target).

it's reasonable to expect that WTM is likely to yield better performance than traditional methods for mining associations between opinion targets and opinion words.


Compared with most traditional methods Liu et al., 2005;Qiu et al., 2011), we don't extract opinion targets iteratively based on the bootstrapping strategy, such as Double Propagation (Qiu et al., 2011), instead all candidates are dynamically ranked in a global process.

error propagation can be effectively avoided and the performance can be improved.


If we directly use this alignment model to our task, a noun/noun phrase may align with the irrelevant words other than adjectives, like prepositions or conjunctions and so on.

in the alignment procedure, we introduce some constrains: 1) nouns/noun phrases (adjectives) must be aligned with adjectives (nouns/noun phrases) or null words; 2) other words can only align with themselves.


Similarly, we can obtain translation probability ( | ) AN p w w .

similar to , the association between a noun/noun phrase and an adjective is estimated as follows.


On the other side, Ours uses WTM other than parsing to identify opinion relations between words, and the noises made by inaccurate parsing can be avoided.


We believe the reason is that the size of Customer Review Datasets is too small.

wTM may suffer from data sparseness for association estimation.


We consider a large set of weaklyrelated products instead of a homogeneous selection of a few products, and the reviews in our dataset are not product-centered but rather describe the overall experience of visiting a restaurant.

menu items are not always mentioned in reviews and rarely appear with their exact names.


To visualize the top feature weights learned by the model, we have to map the four weight vectors learned by the model back to the underlying twodimensional sentiment/price space.

we compute the following values: We then select for display the features which are the furthest from the origin (max w 2 $ + w 2 ) and represent the selected n-grams as points in the sentiment/price space to obtain Figure 3.


There is strong evidence that projecting from single languages can lead to good parsing performance .

our strategy is to choose a single source language for comparison.


For example, the roles of some English words in Table 1, such as "professional" and "high", are different with their original ones.

the POS tagging for mixed texts cannot be settled with simple methods, such as looking up in a dictionary.


Given a sample x, we define the feature Φ(x, y).

we can label x with a score function, where w is the parameter of function F (•).


The approximate algorithm is thus faster than the Brute-Force way by input : character sequence ; end Algorithm 1: Tagging Algorithm with Dynamic Features using word-level information.

given an example (x, y),ŷ are denoted as the incorrect labels with the highest scorê The margin γ(w; (x, y)) is defined as we calculate the hinge loss ℓ(w; (x, y), (abbreviated as ℓ w ) by  In round k, the new weight vector w k+1 is calculated by where ξ is a non-negative slack variable, and C is a positive parameter which controls the influence of the slack term on the objective function.


Historically, English became established as the language used and understood by the scientific community and, consequently, most resources were developed for it, including large-scale computational lexicons like WordNet (Fellbaum, 1998) and sensetagged corpora like SemCor (Miller et al., 1993).

wSD in other languages was hindered by a lack of resources, which in turn led to poor results or low involvement on the part of the research community (Magnini et al., 2004;Màrquez et al., 2004;Orhan et al., 2007;Okumura et al., 2010).


The key idea of our joint approach is that sense evidence from different translations provides complementary views for the senses of a target word in context.

combining such evidence should produce more accurate sense predictions.


This is to say, in order to level off the effects of unbalanced sense coverage we assume that, all things being equal, the more senses a translation covers, the stronger the disambiguation evidence it provides in context for specific senses.

the contributions of each translation are weighted differently and we are thus able to dampen the effects of a highly skewed distribution like, for instance, that  We evaluate our approach in two different settings, namely a monolingual all-words WSD task in Section 4.1, as well as two different cross-lingual disambiguation gold standards in Section 4.2.


Similarly to the case of monolingual WSD, manual inspection of the output reveals that translations help us rule out incorrect senses and let the disambiguation algorithm focus on the more coherent set of senses for the input context in a way similar to the one highlighted by the example in Figure 1.

of this we are able to improve the performance of both monolingual Degree and PLength, and compete with the state of the art on all disambiguation tasks.


• In turn, we use the acquired domain glossaries as a sense inventory for domain WSD.

we redefine the domain WSD task as one of picking out the most appropriate gloss (finegrained setting) or domain (coarse-grained setting) from a multi-domain glossary.


For each such text snippet s, we perform five substeps: a) extraction of the term/gloss separator: we  start from t i and move right until we extract the longest sequence p M of HTML tags and non-alphanumeric characters, which we call the term/gloss separator, between t i and the glossary definition (e.g., "</b> --" between "firewall" and "a" in the above example); b) gloss extraction: we expand the snippet s to the right of h i in search of the entire gloss of t i , i.e., until we reach a non-formatting tag element (e.g., <span>, <p>, <div>), while ignoring formatting elements such as <b>, <i> and <a> which are typically included within a definition sentence.

we obtain the sequence is our gloss for seed term t i in snippet s (which includes h i by construction) and p R is the nonformatting HTML tag element to the right of the extracted gloss.


A dataset for 30 domains We used the Gigaword corpus (Graff and Cieri, 2003) to extract a 6paragraph text snippet for each of the 30 domains.

we obtained a domain dataset made up of 180 paragraphs to which we applied tokenization, lemmatization and compounding, totaling 1432 domain content words overall (47.7 content words per domain on average).


For example, in dependency parsing the resulting structure must be a tree.

if every node in the MRF corresponds to a word in a document and its label corresponds to the index of its head word, the resulting dependency structure for each sentence must be acyclic.


From the example in Figure 2, it is clear that output of unified parser contains Chinese word segmentation information.

we can get results of word segmentation for each sentence in the test sets,  Table 9: Word segmentation results of our parser and the best performance reported in literature on the same dataset.


One final thing to note about the inference algorithm is that the notion of permissibility for a transition t out of a configuration c can be used to capture not only formal constraints on transitions -such as the fact that it is impossible to perform a SHIFT p transition with an empty buffer or illegal to perform a LEFT-ARC d transition with the special root node on top of the stack -but also to filter out unlike-ly dependency labels or tags.

in the experiments later on, we will typically constrain the parser so that SHIFT p is permissible only if p is one of the k best part-of-speech tags with a score no more than α below the score of the 1-best tag, as determined by a preprocessing tagger.


The price to pay for these speedups is that there may be collisions, so that different features are mapped to the same index, but this is often compensated by the fact that the lower time and memory requirements of the hash kernel enables the use of negative features, that is, features that are never seen in the training set but occur in erroneous hypotheses at training time and can therefore be helpful also at inference time.

the hash kernel often improves accuracy as well as efficiency compared to traditional techniques that only make use of features that occur in gold standard parses (Bohnet, 2010).


In order to investigate the relationship between the social environment of a person, her gender and the language, we split the users from the test set into two groups: those whose profile gender matched the gender propagated from the vicinity and those for whom there was a mismatch.

table 4 presents the same results as  Enhanced training set.


Note that datasets with different vocabulary size can lead to different entropy: the entropy of picking a word from the vocabulary uniformly at random would have been different.

we sample each dataset at different rates, and plot the (conditional) entropy in the sample against the corresponding vocabulary size.


the Theatre Royal, Hobart in Australia is predicted to be in London's theater district, and the Embassy of Australia, Paris is predicted to be in the capital city of Australia.

our model may be inadvertently capturing what Clements et al.


To make it even worse, merging and splitting errors are quite common in misspelling.

the assumption of one-toone mapping does not hold in practice.


In spelling correction, usually only one correction is valid for an input query.

we apply the 0-1 loss to our model: Given this loss function, the loss augmented inference problem can be solved easily with an algorithm similar to Algorithm 1.


We point out that authorship attribution is fundamentally different from text categorization in that it is often practically impossible to collect more than several documents for each author.

it is desirable that the attribution algorithms to detect the authors based on very small samples.


For example, adjectival forms of GPEs such as Chinese in "the Chinese leader", would not be linked.

we could coreference United States in "the United States policy" with another referent, but not American in "the American policy."


Definite NPs include NPs with a definite marker (the) as well as NPs with a possessive adjective (his).

the first element is the head in all of the following cases: (31) The chairman, the man who never gives up (32) The sheriff, his friend (33) His friend, the sheriff In the specificity scale, specific names of diseases and technologies are classified as proper names, whether they are capitalized or not.


More so than previous CoNLL shared tasks, coreference predictions depend on world knowledge, and many state-of-the-art systems use information from external resources such as WordNet, which provides a layer of information that could help a system recognize semantic connections between the various lexicalized mentions in the text.

in the case of English, similar to the previous year's task, we allowed the use of WordNet in the closed track.