Note that Turns-aware Aggregation and Attentive Turns Aggregation can be seen as two stages of interaction across the utterances (we call all these two process as "Context Fusion" henceforth).

the former is simply a combination after the Utterance Representation for richer turns-aware information while the latter is to aggregate matching states of previous turns after attention learning against each utterance itself and the response.


Response candidates on the test set are collected by a standard search engine Apache Lucene 4 , other than negative sampling without human judgment on Ubuntu Dialogue Corpus.

the last turn of each Douban dialogue with additional keywords extracted from the context on the test set is used as query to retrieve 10 response candidates from the Lucene index set.


In this paper, we study the role of rhetorical strategies when synthesizing argumentation.

we consider monological argumentative texts where an author seeks to persuade target readers of his or her stance towards a given topic, such as news editorials and persuasive essays.


To our knowledge, only we have explicitly worked towards a computational analysis of such strategies so far.

we presented a corpus with 300 news editorials whose units are labeled with their roles in the argumentation, such as "testimony" and "common ground" .


To reduce the bias of the expert's personal opinions in the study, we use only theses for one stance towards the respective topic.

we kept only those with the majority stance on the topic, resulting in at least four theses for each of the 10 topics.


Similar holds for the phrasing of argumentative texts, or at least for the given task of adding discourse markers and connectives.

table 7 presents the proportion of the most frequent categories of these.


To facilitate the development of such code-mixed conversation models, we build a goal-oriented dialog dataset containing code-mixed conversations.

we take the text from the DSTC2 restaurant reservation dataset and create code-mixed versions of it in Hindi-English, Bengali-English, Gujarati-English and Tamil-English.


Most of the datasets and state of the art systems mentioned above are monolingual.

all the utterances and responses in the conversations are in one language (typically, English) and there are no multilingual and/or code-mixed utterances/responses.


With the above motivation, we build a dataset containing code-mixed goal-oriented conversations for four Indian languages.

we take every utterance from the DSTC2 restaurant reservation dataset and ask a mix of in-house and crowdsourced workers to create a corresponding code-mixed utterance involving their native language and English.


Apart from reporting various statistics about this data (such as CM-index ) and I-index (Guzmán et al., 2016)), we also report some initial baselines by evaluating some state of the art approaches on the proposed dataset.

we evaluate a standard sequence-to-sequence model with an attention mechanism (Bahdanau et al., 2015) and a hierarchical recurrent encoder-decoder model (Serban et al., 2016).


To the best of our knowledge, this is the first conversation dataset containing codemixed conversations and will hopefully enable further research in this area.

since the data is 5-way parallel (English, Bengali, Hindi, Tamil, Gujarati) it would be useful for building jointly trained code-mixed models.


It is surprising that of the 63 conversation datasets developed in the past (Serban et al., 2015), none contain multilingual conversations.

none of them contain code-mixed conversations from multilingual regions of the world.


This dataset contains conversations between crowdsourced workers and existing dialog systems (bots).

the workers were asked to book a table at a restaurant with the help of a bot.


In this section, we describe the process used for creating a new dataset containing code-mixed conversations.

we describe (i) the process used for extracting unique utterance templates from the original DSTC2 dataset, (ii) the process of creating code-mixed translations of these utterances with the help of in-house and crowdsourced workers and (iii) the process used for evaluating the collected conversations.


In order to measure the qualities of partitions obtained using different clustering techniques for these web search data sets, we have used three cluster quality measures, F b 3 measure (Amigó et al., 2009), F 1 measure (Crabtree et al., 2005) and Rand Index (Hubert and Arabie, 1985).

f b 3 has been defined to evaluate completeness, cluster homogeneity, size-vs-quantity and rag-bag constraints.


The main aim of this paper is to identify those aspects of UD which do preserve AAD and to propose a modified version of the current standard which consistently eschews this ill-defined dichotomy.

we demonstrate that the presence of some vestiges of AAD in UD results in serious inconsistencies in the current classification of dependents as core or non-core -a dichotomy which is fundamental for UD.


Current guidelines do not make this clear, saying the following about ccomp: "A clausal complement of a verb or adjective is a dependent clause which is a core argument.

it functions like an object of the verb, or adjective" 2 (emphasis ours).


On the first -broader -view, complement clauses are treated as core, but they are not necessarily assumed to be objects.

it is possible for a single head to have an obj dependent and a ccomp dependent.


As core grammatical functions are different from non-core grammatical functions, this means that core dependents cannot normally be coordinated with non-core dependents.

coordinating a core dependent and a non-core dependent would result in the coordination of different grammatical functions, contrary to the overwhelming generalisation that only the same grammatical functions may be coordinated.


First, our classification targets lemmas not word senses.

we assume that the complex lemmas are monosemous or have a clearly dominant sense in our data.


However, the low result is very likely also heavily driven by the fact that the instances of könig make up close to 40% of our whole dataset.

when testing on könig, we use only 60% of the dataset to train on.


3 Representative Models for Sentence Pair Modeling Table 1 gives a summary of typical models for sentence pair modeling in recent years.

we investigate five models in depth: two are representative of the sentence encoding type of model, and three are representative of the interaction-aggregation type of model.


Similar to a significance test, given a user-specified probability threshold t we can define a critical term proportion value under Γ w A word w is thus considered too specific to an author a if a's usage is too unlikely to occur according to Γ w .

this occurs when the frequency f w,a is larger than the cutoff frequency f * w defined in Eq.


The first claims that all print-to-sound conversion is realised through a lexical route.

oral reading involves processes that all operate on a lexical database so that words and nonwords can be produced by the same mechanism.


First, we use 'full' pattern matching between input letter string and dictionary entries, as opposed to the 'partial' matching of D&N.

rather than starting with the two strings leftaligned, we start with the initial letter of the input string Z aligned with the last letter of the dictionary entry YV.


Distance can be negative, meaning that w2 occurred be/ore wl and for any size window (i.e., 2 to n).

we postulate that it might be possible to deduce part-ofspeech membership and, indeed, identify a set of part-of-speech classes, using the joint probability of words themselves.


It has been used in its most classi-cal setting (Duda and Hart, 1973).

assuming the independence of features, it classifies a new example by assigning the class that maximizes the conditional probability of the class given the observed sequence of features of that example.


The main motivation for modeling p(tls ) in terms of p(wlhi , s) is that it simplifies the "decoding" problem of finding the most likely target text.

if hi is known, finding the best word at the current position requires only a straightforward search through the target 1This ignores the issue of normalization over target texts of all possible lengths, which can be easily enforced when desired by using a stop token or a prior distribution over lengths.


Another important application is in the field known as augmentative and alternative communication (AAC).

people who have lost the ability to speak sometimes use a text-to-speech generator as a prosthetic device.


It follows that the probability that any single parameter is expressed unambiguously is also e ~ and the probability that all of the expressed, but as yet unset parameters are expressed unambiguously is e ~w.

the probability that an input is effectively unambiguous and hence usable for learning is equal to e ~w.


Let Di(x) denote the probability distribution of expression of the input sample.

the probability that an arbitrarily chosen sentence from the input sample I expresses x parameters.


Often, the sets of instances covered by rules overlap.

seen from the instance perspective, a single instance can match more than one rule.


1 we assume to have local signals that indicate the state.

we are given classifiers with states as their outcomes.


For the easier NP task, the HMM model is competitive with the others when the classifiers used are NB or SNOW.

a significant improvement in both probabilistic methods is achieved when their input is given by SNOW.


SVMs take a simple strategy that finds the separating hyperplane which maximizes its margin.

two dashed lines and margin (d) can be written as: SVMs can be regarded as an optimization problem; finding w and b which minimize [[w[[ under the constraints: yi Furthermore, SVMs have potential to cope with the linearly unseparable training data.


Basically, SVMs are binary classifiers, thus we must extend SVMs to multi-class classifiers in order to classify these 22 types of chunks.

it is 1the number of combination becomes 23.


There are a number of clear-cut ways in which we could attempt to improve our performance using this approach.

we would expect to obtain better results if we did not obliterate the part-of-speech of a lexeme in the process of tagging it with a chunk label.


When we transform the representational theory of the boolean concept learning framework to a substructural logic, we do the following: • eliminate the structural rules from the calculus of first order propositional logic When we want to translate the learnability result of k-CNF expressible boolean concepts we need to do the same with the formal learning framework and the strategy (algorithm).

: • the learning framework will contain concepts that are sensitive to the features which were before abstracted by the structural rules ('position' and 'arity' ) • the learning algorithm from above is no longer allowed to use the structural rules in its inductive steps.


A concept class F is PAC learnable if for all targets f E F and all probability distributions P on the sample space U* the learning algorithm A outputs a concept g E F such that with probability (1-5) it holds that we have a chance on an error with P(f Ag) _< e (where fag = (f -g)U(g-f)) We are especially interested in concept classes that are defined by some formalism (language).

a language can describe come collection of concepts.


At the end of training, the units form a map of the input space that reflects how the input space was sampled in the training data.

areas of input space in which there were a lot of inputs will be mapped in finer detail, using more units than areas where the inputs were sparsely distributed.


We also hypothesize that the semantic class membership of an English verb may inuence its word-level translation into Chinese.

the sublexical component| the precise morphemic constitution of the translated Chinese verb|may re ect properties of the class of the English verb.


We conclude that combining multilingual data has a signi cant performance bene t over monolingual data from either language.

in augmenting English-only data with Chinese data, we a c hieve higher accuracies than that using either the English HK-Laws subcorpus or the much larger WSJ corpus alone.


Ide, 2000 andResnik andYarowsky, 2000 made use of parallel corpora for word sense disambiguation.

a parallel Englishnon-English corpus was used as a source for lexicalizing some ne-grained English senses.


This paper investigates the question of whether work on shallow parsing is worthwhile.

we attempt to evaluate quantitatively the intuitions described above -that learning to perform shallow parsing could be more accurate and more robust than learning to generate full parses.


Pierre Vinken, 61 years, the board, a nonexecutive director and Nov. 29 are atomic phrases while other higher-level phrases are not.

an atomic phrase denotes a tightly coupled message unit which is just above the level of single words.


However, although these approaches are very simple to implement, tuning them most effectively is not an easy task.

it is unclear whether oversampling is more effective than undersampling and which oversampling or undersampling rate should be used.


Unfortunately, several characteristics specific to text data make its classification a difficult problem to handle.

the data is typically highly dimensional and it presents a large class imbalance, i.e., there, typically, are very few documents on the topic of interest while texts on unrelated subjects abound.


For each concept, we created a training set containing 240 positive and 6000 negative examples.

we 2 Throughout this work, we consider a fixed imbalance ratio, a fixed number of training examples and a fixed degree of concept complexity.


3 DNF expressions were specifically chosen because of their simplicity as well as their similarity to text data whose classification accuracy we are ultimately interested in improving.

like in the case of text-classification, DNF concepts of interest are, generally, represented by much fewer examples than there are counter-examples of these concepts, especially when 1) the concept at hand is fairly specific; 2) the number of disjuncts and literals per disjunct grows larger; and 3) the values assumed by the literals are drawn from a large alphabet.


5 In order to find out what happens when different sampling rates are used, we continued using the imbalanced data sets of the previous section, but rather than simply oversampling and undersampling them by equalizing the size of the positive and the negative set, we oversampled and undersampled them at different rates.

we divided the difference between the size of the positive and negative training sets by 10 and used this value as an increment in our oversampling and undersampling experiments.


These results suggest that different sampling rates have different effects on the accuracy of C5.0 on imbalanced data sets for both the oversampling and the undersampling method.

the following observation can be made: Oversampling or undersampling until a cardinal balance of the two classes is reached is not necessarily the best strategy: best accuracies are reached before the two sets are cardinally balanced.


This suggests a 3-level hierarchical combination approach consisting of the output level, which combines the results of the oversampling and undersampling experts located at the expert level, which themselves each combine the results of 10 classifiers located at the classifier level and trained on data sets sampled at different rates.

the 10 oversampling classifiers oversample the data at rates 10%, 20%, ... 100% (the positive class is oversampled until the two classes are of the same size) and the 10 undersampling classifiers undersample the negative class at rate 0%, 10%, ..., 90% (the negative class is undersampled until the two classes are of the same size).


The number of positive classifications each classifier makes on the unlabeled data set is recorded and averaged and this average is taken as the threshold that none of the expert's classifiers must cross.

any classifier that classifies more unlabeled data points as positive than the threshold established for the expert to which this classifier belongs needs to be discarded.


Second, the documents were transformed into feature vectors in several steps.

all the punctuation and numbers were removed and the documents were filtered through a stop word list 13 .


The generally better performance displayed by our proposed system when evaluated using the G Q P -measure and its generally worse performance when evaluated using the -measure are not surprising, since we biased our system so that it classifies more data points as positive.

it is expected that our system will correctly discover new positive examples that were not discovered by Ada-Boost, but will incorrectly label as positive examples that are not positive.


Furthermore, it is important to note that this method is not particularly computationally intensive.

its computation costs are comparable to those of commonly used combination methods, such as AdaBoost.


This paper presented an approach for dealing with the class-imbalance problem that consisted of combining different expressions of re-sampling based classifiers in an informed fashion.

our combination system was built so as to bias the classifiers towards the positive set so as counteract the negative bias typically developed by classifiers facing a higher proportion of negative than positive examples.


The greediness of the algorithm causes many problems.

it makes the algorithm very sensitive to the order in which the rules are acquired.


Frequency alone does not distinguish these two sequences, but Criterion 1 points to a distributional fact which does.


There are difficulties in modeling such a class, mainly stemming from the differences between this class and the constituent classes.

this class will not be distributionally cohesive.


A possible criticism of this work is that it relies on part-of-speech tagged data as input.

while there has been work on acquiring parts-ofspeech distributionally (Finch et al., 1995;Schütze, 1995), it is clear that manually constructed tag sets and taggings embody linguistic facts which are not generally detected by a distributional learner.


Psychology and psycholinguistics provide us with a significant amount of data from which we may derive a fairly good picture of how the problem is defined for humans.

we will concentrate on a child's acquisition of their first language and how this relates to a computational model, as this seems to be the point at which human language acquisition is at its most efficient.


Any linguistic annotation required during the extraction process, therefore, is produced through automatic means, and it is only for reasons of accessibility and comparability with other research that we choose to work over the Wall Street Journal section of the Penn Treebank (Marcus et al., 1993).

other than in establishing upper bounds on the performance of the different extraction methods, we use only the raw text component of the treebank.


It is important to realise that compatibility with the particle alternation is a sufficient but not necessary condition on verb-particlehood.

a small number of VPCs do not readily occur in the split configuration, including carry out (a threat) (cf.


In the evaluation of the various extraction techniques below, recall is determined relative to this limited set of 62 VPCs attested in the WSJ.

recall is an indication of the proportion of the 62 VPCs contained within the set of extracted VPCs.


Over the WSJ, the Brill tagger achieves a modest tag recall of 0.103 for particles, and tag precision of 0.838.

it is highly conservative in allocating particle tags, to the extent that it recognises only two particle types for the whole of the WSJ: out and down.


In the latter case, we use an unsupervised attachment disambiguation method, based on the log-likelihood ratio ("LLR", Dunning (1993)).

we use the chunker output to enumerate all the verb-preposition, prepositionnoun and verb-noun bigrams in the WSJ data, based on chunk heads rather than strict word bigrams.


They test their method over three different corpora, with results ranging from a precision of 0.21 and recall of 0.86 (F-score=0.34) for the smallest corpus, to a precision of 0.03 and recall of 0.85 (F-score=0.06) for the largest corpus.

high levels of noise are evident in the system output, and the F-score values are well below those achieved by our method for English VPCs.


In this paper we observe that another useful measure of (minimal) supervision is the additional cost of obtaining a desired functionality from existing commonly available knowledge sources.

we note that for a remarkably wide range of languages, academic libraries, many booksellers and websites offer a foundation of linguistic wisdom in reference grammars and dictionaries.


Given the early stage of this line of research, there is plenty of room for future work.

further research is needed to provide theoretic grounding for the CD clustering formulations and to specify their properties.


Empirical work is needed to explore the potential of the proposed paradigm for other textual domains as well as for related applications.

we have recently presented a similar framework for template induction in information extraction (crosscomponent clustering, , which should be studied in relation to the CD algorithm presented here.


Therefore, once a segment has been determined to modify another segment, such information is kept in both of the segments and is added to them as a new feature.

we take the following three types of dynamic features in our experiments.


Variable metric or quasi-Newton methods avoid explicit evaluation of the Hessian by building up an approximation of it using successive evaluations of the gradient.

we replace H −1 (θ (k) ) in (5) with a local approximation of the inverse Hessian B (k) : with B (k) a symmatric, positive definite matrix which satisfies the equation: where Variable metric methods also show excellent convergence properties and can be much more efficient than using true Newton updates, but for large scale problems with hundreds of thousands of parameters, even storing the approximate Hessian is prohibitively expensive.


For these experiments, we judged that convergence was reached when the relative change in the loglikelihood between iterations fell below a predetermined threshold.

each run was stopped when: where the relative tolerance ε = 10 −7 .


The performance of the model is roughly the same for both test samples of each language, though the performance differs somewhat between the two languages.

the performance on F l 8 4 G Ë entities is quite a bit better for Dutch than it is for Spanish, and the performance on 3 6 5 6 7 entities is quite a bit better for Spanish than it is for Dutch.


We estimate the probability of a derivation tree as P (t) = n∈t P (expansion(n)|history(n)).

the probability of the derivation tree is the product of the probabilities of the expansion of each node given its history of available features.


This section reviews the existing models for syntactic disambiguation from the viewpoint of representing parsing results of lexicalized grammars.

we discuss how the models incorporate syntactic/semantic preferences for syntactic disambiguation.


However, the models are based on the lexical dependencies of elementary trees, which is a simple extension of the LPCFG.

the models are still based on decomposition into primitive lexical dependencies.


However, the other two problems remain unsolved in this model.

when we apply Feature-Based LTAG (FB-LTAG), the above probability is no longer consistent because of the non-local constraints caused by feature unification (Abney, 1997).


Each slice contains two pairs of parses from each sentence.

slice i contains positive samples ((p k , p ki ), +1) and negative samples ((p ki ,p k ), −1), wherep k is the best parse for sentence k, p ki is the parse with the ith highest loglikelihood in all the parses for sentence k and it is not the best parse.


However, in the specialized case of personal names, there is more precise information available.

information extraction techniques can add high precision, categorical information such as approximate age/date-of-birth, nationality and occupation.


We adapted the information extraction pattern generation techniques described above to multiple languages.

the methodology proposed by Ravichandran and Hovy (2002) requires no parsing or other language specific resources, so is an ideal candidate for multilingual use.


In this paper we have presented a set of algorithms for finding the real referents for ambiguous personal names in text using unsupervised clustering and feature extraction methods.

we have shown how to learn and use automatically extracted biographic information to improve clustering results, and have demonstrated this improvement by evaluating on pseudonames.


First, the agreement-based method dynamically selects the best sample at each stage, which may not be the whole cache.

when the agreement rate cannot be improved upon, the selected sample can be rejected.


However, we try to use only some fundamental properties of the algorithm so that our methods are readily applicable to other systems with similar properties.

we use the sentence boundary detection problem to perform experiments since not only is it relatively simple and well-understood, but it also provides the basis for other more advanced linguistic problems.


However, typically they only considered very specific statistical models where the idea is to fit certain transformation parameters.

they did not consider the main issues investigated in this paper as well as generally applicable supervised adaptation methodologies such as what we propose.


We use an encoding scheme similar to that of (Zhang et al., 2002 token before the period token after the period character to the right type of character to the right character to the left type of character to the left character to the right of blank after word type of character to the right of blank after word character left of first character of word type of character left of first character of word first character of the preceding word type of first character of the preceding word length of preceding word distance to previous period The features presented here may not be optimal.

unlike (Zhang et al., 2002), we do not use higher order features (for example, combinations of the above features).


This idea is very similar to certain methods used in active learning.

a confidence-based sample selection scheme was proposed in (Lewis and Catlett, 1994).


we will compare the following methods in this study:  is the output of the first level classifier).


This provides useful insights into the potential value of various ideas.

we have made the following observations: 1.


Thus, we decided to replace missing values ourselves before presenting the datasets to our learners.

there are two types of missing values: genuine missing values (no value was provided; we will refer to them as missing values) and undefined values.


The next goal of our study was to see if we can characterize the performance of our predictors on various classes of instances defined by our exceptionality criteria.

we wanted to try to answer questions like: is IB1-IG better at predicting exceptional instances than Ripper?


We can observe that accuracy of both predictors increases with typicality.

the more typical the instance, the more reliable the prediction; the more exceptional the instance, the more unreliable the prediction.


We computed what could be the reduction in error rate if we were to employ both predictors and decide between them based on the instance exceptionality measure.

ripper prediction was used for exceptional instances and for the left-hand side boundary instances (CPS less than 0.5; typicality less than 1; local typicality less than 1); otherwise IB1-IG prediction was used.


Future work will address cross-dataset experiments.

we will see whether the word and hypernym associations learned over FrameNet can be carried over into Treebank, given a mapping of the fine-grained FrameNet roles into the coarsegrained Treebank ones.


We define "context" to be the syntactic structure of the sentence in which the word of interest occurs.

we define this notion at three abstraction levels.


The highest level is the abstract syntactic skeleton of the sentence.

it is in the form of the different combinations of six syntactic components.


The purpose of this work is to investigate the use of machine learning approaches for confidence estimation within a statistical machine translation application.

we attempt to learn probabilities of correctness for various model predictions, based on the native probabilites (i.e.


We describe an approach to tagging a monolingual dictionary with linguistic features.

we annotate the dictionary entries with parts of speech, number, and tense information.


In order to abstract away from pos tagging errors, the algorithm is only evaluated on those words that were assigned the appropriate pos for a given word.

if the test set contains a singular noun, it is looked up in the automatically produced target language dictionary.


This can be explained by the fact that French tags number in verbs differently in different tenses.

the algorithm is faced with more inflectional paradigms, which are harder to learn because the data is fragmented into different patterns of plural markings.


This paper demonstrates that mathematical models for measuring semantic similarity between concepts can be used to improve the learning of hyponymy relationships between concepts from free text.

we show that latent semantic analysis can be used to filter results, giving an increase in precision, and that neighbors in a graph built from coordination information can be used to improve recall.


After discussion between the authors on disputed annotations to create "gold standard" annotations, we found that 40 of the 100 relations in our random sample were correct according to this criterion.

40% of the relations extracted were exactly correct or would be correct with the use of minor post-processing consisting of lemmatization and removal of common types of qualifying words.


The LSA filtering had removed 154 incorrect relationships and only 45 correct ones, reducing the overall error rate by 33%.

this technique removed all but one of the spurious religious service hyponyms which were obtained through inappropriate similarities with mass in the example in Section 4, though it was much less effective in filtering the neighbors of nitrogen and dill, as might be expected.


Our classification method is variable-wise.

to classify elements in each ½ Ò , we classified the elements in each in order.


If revision results in any event having no nodes, the proposed event addition fails.

the new event is inconsistent with the rest of the correlation.


In the first layer, a set of NE candidates for a sentence is identified, out of all the potential phrases in N E. To do so, we apply learning at word level in order to perform a Begin-Inside classification.

we assume a function h B (w) which decides whether a word w begins a NE phrase or not, and a function h I (w) which decides whether a word is inside a NE phrase or not.


We assume a number of scoring functions, which given a NE phrase produce a real-valued score indicating the plausibility of the phrase.

for each category k ∈ K we assume a function score k which produces a positive score if the phrase is likely to belong to category k, and a negative score otherwise.


In particular, for each category k ∈ K we assume a function score k which produces a positive score if the phrase is likely to belong to category k, and a negative score otherwise.

given this, the NE-Chunker is a function which searches a NE chunking for a sentence x according to the following optimality criterion: among the considered chunkings of the sentence, the optimal one is defined to be the one whose NE phrases maximize the summation of phrase scores.


The key point of the voted version is that, while training, it stores information in order to make better predictions on test data.

all the prediction vectorsw j generated after every mistake are stored, together with a weight c j , which corresponds to the number of decisions the vectorw j survives until the next mistake.


Furthermore, we represent the (s, e) phrase by evaluating the predicates without capturing the relative position in the features.

for the words within (s, e) we evaluate the form, affixes and type patterns of sizes 2, 3 and 4.


The RRM method is showing its combining power when additional information sources are used.

the system was fed additional feature streams from a list of gazetteers and the output of two other named entity systems trained on 1.7M words annotated with 32 name categories.


The final system also contained some simple error-driven postprocessing.

repeated sub-elements (usually last names) of multi-word person names were given type PERSON, and a crude heuristic restoration of B-prefixes was performed.


The category probability distributions may be seen as meta-data that is used to train a meta-learner.

given n features A 1 , A 2 , ..., A n and m training instances I 1 , I 2 , ..., I m ,we may then randomly form l different training subsets S 1 , S 2 , ..., S l , with each S i containing a random subset of both attributes and training instances.


It performs well on a number of natural language processing problems, including text categorization (Schapire and Singer, 2000) and word sense disambiguation (Escudero et al., 2000).

it has also been demonstrated that boosting can be used to build language-independent NER models that perform exceptionally well (Wu et al., 2002;Carreras et al., 2002).


The main focus of this paper is to investigate the impact of some local features.

we show that the system performance can be enhanced significantly with some relatively simple token-based features.


The basic linguistic features considered here are all aligned with the tokens.

we will consider features listed in Table 1.


we consider a set V which consists of two types of variables The second set of variables R = {R ij } {1≤i,j≤n;i =j} is viewed as binary relations over E. for each pair of entities E i and E j , i = j, we use R ij and R ji to denote the (binary) relations (E i , E j ) and (E j , E i ) respectively.


We use N 1 and N 2 to denote the entity variables of a relation R ij .

in addition, we define a set of constraints on the outcomes of the variables in constraint values of the first argument of a relation.


Rather than doing that, for computational reasons, we first use a fairly standard transformation of MRF to a discrete optimization problem (see (Kleinberg and Tardos, 1999) for details).

under weak assumptions we can view the inference problem as the following optimization problem, which aims to minimize the objective function that is the sum of the following two cost functions.


The constraint cost indicates whether the labels are consistent with the constraints.

we use: Similarly, we use d 2 to force the consistency of the second argument of a relation.


Figure 1: Integer Linear Programming Formulation To solve an ILP problem, a natural idea is to relax the integral constraints.

replacing (6), (7), and (8) with: If LPR returns an integer solution, then it is also the optimal solution to the ILP problem.


When the coefficient matrix of a given linear program in its standard form is unimodular, it can be shown that the optimal solution to the linear program is in fact integral (Schrijver, 1986).

lPR is guaranteed to produce an integer solution.


Basic, only tests our entity and relation classifiers, which are trained independently using only local features.

the relation classifier does not know the labels of its entity arguments, and the entity classifier does not know the labels of relations in the sentence either.


We use softmax (Bishop, 1995) over the raw activation values as conditional probabilities.

suppose the number of classes is n, and the raw activation values of class i is act i .


Note that LP may change the predictions for either entity labels or relation labels, while pipeline fully trusts the labels of entity classifier, and only the relation predictions may be different from the basic relation classifier.

lP is able to enhance the performance of entity classification, which is impossible for pipeline.


Our theoretical basis is partly derived from this analysis.

we replace the abstract notion of 'optimum subspace' with a precise definition of a subspace useful for our task.


An example of ¤ might be the distribution observed in all the newspaper articles in 1980's, or the distribution observed in biomedical articles.


All the methods (except Spectral) exhibit the same tendency.

performance on random seeds is lower than that on high-frequency seeds, and the degradation is larger when the number of seeds is small.


The main drawback is structures may not be properly represented by flat features as: (1) these latter may not be able to capture the required properties or (2) the feature designer may not know what structure properties enable the processing of semantic information.

these problems arise for semantic information represented via predicate argument structures de-fined on syntactic parse trees.


The kernel functions can be used to measure similarities between two objects without explicitly evaluating the object features.

we do not need to understand which syntactic feature may be suited for representing semantic data.


Finally, there is a considerable work in Natural Language Processing oriented kernel (Collins and Duffy, 2002;Lodhi et al., 2000;Gärtner, 2003;Cumby and Roth, 2003;Zelenko et al., 2003) about string, parse-tree, graph, and relational kernels but, to our knowledge, none of them was used to derive semantic information on the form of predicate argument structures.

(Cristianini et al., 2001;Kandola et al., 2003) address the problem of semantic similarity between two terms by using, respectively, document sets as term context and the latent semantic indexing.


In addition, when looking at a more detailed breakdown of the 73 and 36 words included in these samples respectively, a considerable number of those words experience improved accuracy using part of speech features.

we observed that while verbs and adjec-tives are disambiguated best by part of speech of words one or two positions on their right (P 1 , P 2 ), nouns in general are aided by the part of speech of immediately adjacent words on either side (P −1 , P 1 ).


Different types of features are expected to be redundant to a certain extent.

the features will individually classify an identical subset of the instances correctly.


We show that even though lexical and syntactic features are redundant to a certain extent, there is a significant amount of complementarity.

we showed that simple lexical features (unigrams and bigrams) used in conjunction with part of speech features have the potential to achieve state of the art results.


The new weight in the graph is computed by averaging the weight on the edge between nodes I and Q and that on the edge between J and Q.

: In vector space, average link starts by assigning each vector to a single cluster.


However, a recent study (Zhao and Karypis, 2002) has suggested that these conclusions are based on experiments conducted with smaller data sets, and that with larger data sets partitional algorithms are not only faster but lead to better results.

zhao and Karypis recommend a hybrid approach known as Repeated Bisections.


These observations correspond with the hypothesis drawn from the SENSEVAL-2 results.

a large amount of training data will lead to a larger feature space and hence there is a greater chance of matching more features directly in the context of the test instances.


The present paper provides one solution to this issue.

we proposed one type of TOP kernel, because it has been reported that TOP kernels perform better than Fisher kernels in terms of categorization accuracy.


Although many other models can be used, we propose a model based on the separating hyperplanes in the original feature space.

a one-dimensional Gaussian-type function normal to a hyperplane corresponds to a category.


The TOP kernel consists of features which can minimize D(fθ).

we would like to have feature vectors fθ that satisfy the following: for certain values of w and b.


This form suggests that two vectors are considered to be more similar, if they have similar distributions over categories.

an occurrence of a word can have different contribution to the classification result, depending on the context (i.e., the other words in the document).


As can be seen, the grater the number of categories, the more likely it is that a test sample has been incorrectly classified.

the performance of flat model using 10 categories was 0.720 F-score and that of 50 categories was 0.695.


Given a sentence, the task consists of analyzing the propositions expressed by some target verbs of the sentence.

for each target verb all the constituents in the sentence which fill a semantic role of the verb have to be extracted (see Figure 1 for a detailed example).


Participant systems are described and compared in section 4.

information about learning techniques, SRL strategies, and feature development is provided, together with performance results on the development and test sets.


The CoNLL-2004 board for fruitful discussions and suggestions.

erik Tjong Kim Sang for useful comments from his valuable experience, and for making the baseline SRL processor available.


(1999) to have our examples represent relations between pairs of verbs and chunks.

we transform the semantic role labeling task to a classification task in which we decide for all pairs of verbs and chunks whether they stand in a semantic role relation.


Let y * be the gold set of arguments for a sentence x, andŷ those predicted by the SRL function.

let goldS(x i , k) and goldE(x i , k) be, respectively, the perfect indicator functions for start and end boundaries of arguments of type k. they return 1 if word x i starts/ends some k-argument in y * and -1 otherwise.


By taking two phase model based on SVMs, we can alleviate the unbalanced class distribution problem.

since we find only the boundary of an argument in the identification phase, the number of classes is decreased into two (ARG, NON-ARG) or three (B-ARG, I-ARG, O).


It incorporates several improvements over the basic Winnow update rule.

a regularization term is added, which has the affect of trying to separate the data with a think separator (Grove and Roth, 2001;Hang et al., 2002).


Therefore, the goal of the first phase is to output a superset of the correct phrases by filtering out unlikely candidates.

we learn two classifiers, one to detect beginning phrase locations and a second to detect end phrase locations.


• Clause position feature is the position of the target word relative to the predicate in the semi-parsed tree containing only clauses.

there are four configurations-target word and predicate share same parent, parent of target word is ancestor of predicate, parent of predicate is ancestor of target word, or otherwise.


If we assume that the classifier returns a score, score(S i = s i ), corresponding to the likelihood of seeing label s i for phrase S i , then, given a sentence, the unaltered inference task that is solved by our system maximizes the score of the phrase, score(S 1:M = s 1:M ), (1) The second step for phrase identification is eliminating labelings using global constraints derived from linguistic information and structural considerations.

we limit the solution space through the used of a filter function, F , that eliminates many phrase labelings from consideration.


Notice that unlike the templates in the first stage, these templates make no reference to lexical information.

no rule takes advantage of PropBank data in its tagging process 3 .


using latent semantic indexing, word co-occurrences or synonym relations using a word ontology have been shown not very effective.

when to extend the similarity function external prior knowledge is used, e.g.


The ideal tree should (a) link the two senses/nodes s 1 and s 2 with the minimal number of edges (isa-relations) and (b) maintain the same branching factor (bf ) observed ins.

this tree provides the minimal number of nodes (and isa-relations) sufficient to connect s 1 and s 2 according to the topological structure ofs.


The matching allowed between nounverb pairs still captures semantic information which is useful for topic detection.

during the similarity estimation, each word activates 60.05 pairs on average.


Harris articulated the expectation that words with similar meanings would be used in similar contexts (Harris, 1968), and recent empirical work involving large corpora has borne this out.

by associating each word with a distribution over the words observed in its context, we can distinguish synonyms from non-synonyms with fair reliability.


In this study, features are just unnormalized tokens, possibly augmented with direction and distance information.

we do not investigate the utility of stemming.


We also avail ourselves of measures suggested by probability theory.

for , we have the so-called H -divergences (Zhu and Rohwer, 1998):  .


Our model assumes that the term occurs at some low underlying base rate 1/λ 1 but, after the term has occurred, then the probability of it occurring soon afterwards is increased to some higher rate 1/λ 2 .

the rate of re-occurrence is modeled by a mixture of two exponential distributions.


In this paper we propose and evaluate a technique to perform semi-supervised learning for Text Categorization.

we defined a kernel function, namely the Domain Kernel, that allowed us to plug "external knowledge" into the supervised learning process.


The use of the Domain Kernel got a significant improvement in the learning curves of both tasks.

there is a notable increment of the recall, especially with few learning examples.


An extensive evaluation on two standard benchmarks shows that the Domain Kernel allows us to reduce drastically the amount of training data required for learning.

the recall increases sensibly, while preserving a very good accuracy.


This paper proposes a new clustering framework that addresses all the problems discussed above.

we define clustering as a learning task: in the training stage, a partition function, parameterized by a distance metric, is trained with respect to a specific clustering algorithm, with supervision.


The distance between the two elements is defined by converting the prediction confidence of the pairwise classifier, and clustering is then performed based on this distance function.

(Li et al., 2004) applied this approach to measuring name similarity in the entity identification problem, where a pairwise classifier (LMR) is trained using the SNoW learning architecture (Roth, 1998) based on variations of Perceptron and Winnow, and using a collection of relational features between a pair of names.


In order to make the comparison as informative as possible, the same settings are adopted in our experiments as those used by (LP ) 2 , which previously reported the highest results on this dataset.

the results are obtained by averaging the performance in ten runs, using a random half of the corpus for training and the rest for testing.


EM's role is to induce a probability distribution over candidates to maximize the likelihood of the (p, k) pairs observed in our training set: We can rewrite Pr(p, k) so that it uses a hidden candidate (or antecedent) variable c that influences the observed p and k: To improve our ability to generalize to future cases, we use a naïve Bayes assumption to state that the choices of pronoun and context are conditionally independent, given an antecedent.

once we select the word the pronoun represents, the pronoun and its context are no longer coupled: We can split each candidate c into its lexical component l and its jump value j.


We can see that all methods except for the previous noun heuristic score noticeably better when ignoring those sentences that contain quotation marks.

the difference between our three unsupervised solutions ((2), (3) and (4)) are more pronounced.


letter in the alphabet of the PCFG) is further modeled as a simple semantic frame that contains roles for an agent, an object, an action, and multiple optional modifier roles (see inset figure 1).

while in this initial work productions are created by hand (a task made feasible by the 2 high-level actions (e.g.


Looking closer, we find a strong negative correlation (r=-0.81) between the understanding performance using only channel probabilities (α=1) and the improvement obtained by including the intentional context.

the better one does without context, the less context improves performance.


Rather than directly encoding positional information, the second model class we investigate (the bigram model) models statistical dependencies between adjacent phonemes and adjacent syllables.

each onset or coda expands directly into one or more terminal phonemes, thus capturing the ordering dependencies between consonants in a cluster.


However, a more important result of parsing the corpus using the categorical parser is that its output can be used to define a model class (i.e., a set of PCFG rules) from which a model can be learned using EM.

our model class contains the set of rules that were proposed at least once by the categorical parser in its analysis of the training corpus; in the EM experiments described below, the rule probabilities are initialized to their frequency in the categorical parser's output.


Our work suggests that using models with higher bias but lower variance may lead to much more successful results.

we used universal phonological principles to induce a set of rules within a carefully chosen grammatical framework.


if t is a solution of x : y :: z :?, then t contains, in the same relative order, all the symbols in y and z that are not in x.


In such cases, all analyses are retained, and the system's analysis is given the most optimistic score.

if a CELEX analysis is found which matches the system's analysis, it is judged to be correct.


Cluster 180 contains words like "make", "pay", and "keep", while Cluster 185 contains "made", "paid", and "kept".

once a strong correspondence is found between two clusters, we can search for an alignment which covers the orphans in the respective clusters.


In this section, we will report on the evaluation of various subsets of the doubly annotated portion of the ABC corpus comprising 1000 sentences, which we sample according to a sample selection metric.

examples are added to the subsets according to the sample selection metric, selecting those with higher disagreement first.


The main effect we observed is the same as was suggested by the f-score inter-annotator agreement errors in section 2.1.

looking at the full random set of 1000 sentences, almost all errors (Where * is any entity phrase type, * /O + O/ * errors all errors = 95.43%) are due to problems with phrase boundaries.


Given a sentence, the task consists of analyzing the propositions expressed by some target verbs of the sentence.

for each target verb all the constituents in the sentence which fill a semantic role of the verb have to be recognized.


Probably, this is due to the versatility of the approaches and the availability of very good software toolkits.

8 teams used the Maximum Entropy (ME) statistical framework (Che et al., 2005;Haghighi et al., 2005;Park and Rim, 2005;Tjong Kim Sang et al., 2005;Sutton and McCallum, 2005;Tsai et al., 2005;Yi and Palmer, 2005;Venkatapathy et al., 2005).


We describe the classification strategy and the features used by the classifier.

the contribution of six parse tree path features is investigated.


Our SRL system consists of four stages: pruning, argument identification, argument classification, and inference.

the goal of pruning and argument identification is to identify argument candidates for a given verb predicate.


If we assume that the argument classifier returns an estimated conditional probability distribution, P rob(S i = c i ), then, given a sentence, the inference procedure seeks an global assignment that maximizes the following objective function, subject to linguistic and structural constraints.

this objective function reflects the expected number of correct argument predictions, subject to the constraints.


The inference process allows a natural way to combine the outputs from multiple argument classifiers.

given k argument classifiers which perform classification on k argument sets, {S 1 , .


For the systems that do not output for any candidate, we assign the probability with a prior to this phantom candidate.

the probability of the NULL class is set to be 0.6 based on empirical tests, and the probabilities of the other classes are set proportionally to their occurrence frequencies in the training data.


The phantom candidates are created for a 2 , a 3 , and b 4 of which probability is set to the prior.

for this implementation, we first train two SRL systems that use Collins' parser and Charniak's parser respectively.


As our system does not build an overall sentence contextual representation, it systematically produced errors such as embedded role labeling.

since no embedding is observed for the semantic arguments of predicates, in case of (multiple) embeddings the classifier output was automatically post-processed to retain only the largest embedding constituent.


In each fold, separate SVM classifiers were trained for the Collins and Charniak parses using 75% of the training data.

one system assigned role labels to the nodes in Charniak based trees and a separate system assigned roles to nodes in Collins based trees.


(2000) showed that performing parsing and information extraction in a joint model improves performance on both tasks.

one suspects that attachment decisions, which are both notoriously hard and extremely important for semantic analysis, could benefit greatly from input from higher-level semantic analysis.


But we are not actually interested in selecting a good parse tree; all we want is a good semantic frame.

this means that we should select the semantic frame   that maximizes the posterior probability: we should be summing over the parse trees instead of maximizing over them.


In previous research, we have found that memorybased learning is rather sensitive to the chosen features.

irrelevant or redundant features may lead to reduced performance.


We propose to exploit this property to transfer information from a large corpus to a statis-tical parser for a different domain.

we propose to train a statistical parser on data including the large corpus, and to derive the kernel from this trained model.


strong and that weakening them results in better models of language (Johnson, 1998;Gildea, 2001;Klein and Manning, 2003).

certain grammar productions often cooccur with other productions, which may be either near or distant in the parse tree.


However, for a mixture of grammars we need to sum over the individual grammars: Because of the outer sum, this expression unfortunately does not decompose into a product over scores of subparts.

a tree which maximizes the sum need not be a top tree for any single component.


Note that, unlike during parsing, there is no uncertainty over the actual rules used, so the E-step does not require summing over possible trees.

for the variable G we have For the hierarchy level L we can write , where we slightly abuse notation since the rule X → α can occur multiple times in a tree T. In the M-step, we find the maximum-likelihood model parameters given these posterior assignments; i.e., we find the best grammars given the way the training data's rules are distributed between individual and shared grammars.


A dependency tree specifies which words in a sentence are directly related.

the dependency structure of a sentence is a directed tree where the nodes are the words in the sentence and links represent the direct dependency relationships between the words; see Figure 1.


Given an input sentence ¢ , we would like to be able to compute the best parse; that is, a projective tree, 6 5 1 2 ¤ 3 ¢ 4 , that obtains the highest "score".

we follow (Eisner, 1996;Eisner and Satta, 1999;McDonald et al., 2005) and assume that the score of a complete spanning tree for a given sentence, whether probabilistically motivated or not, can be decomposed as a sum of local scores for each link (a word pair).


Given a training set of sentences annotated with their correct dependency parses, , the goal of learning is to estimate the parameters of the parsing model, w .

we seek values for the parameters that can accurately reconstruct the training parses, but more importantly, are also able to accurately predict the dependency parse structure on future test sentences.


Then, instead of just solving the constraint system (7) we can also ensure that similar links take on similar parameter values by introducing a penalty on their deviations that is weighted by their similarity value.

we use have a high similarity value, their parameters will be encouraged to take on similar values.


Although the constraints are more refined, they are fewer in number than (4).

we now only have a polynomial number of constraints corresponding to each word pair in (5), rather than the exponential number over every possible parse tree in (4).


Finally, given that the current approach is dependent on the availability of a large annotated corpus, an important question is if and how it can be extended to work with unlabeled data.

can we transform the push-npull algorithm to perform the unsupervised learning of STSGs?


This paper investigates an isolated setting of the lexical substitution task of replacing words with their synonyms.

we examine this problem in the setting of subtitle generation and evaluate state of the art scoring methods that predict the validity of a given substitution.


Context independent models score the general likelihood that the original word is "replaceable" with the candidate synonym, in an arbitrary context.

trying to filter relatively bizarre synonyms, often of rare senses, which are abundant in WordNet but are unlikely to yield valid substitutions.


Our findings suggest the dominance, in this setting, of generic contextindependent scoring.

considering distributional similarity amongst WordNet synonyms seems effective for identifying candidate substitutions that are indeed likely to be applicable in actual texts.


Overall, the distributional similarity based method (sim) performs much better than the other methods.

lin's similarity also performs better than semcor, the other context-independent model.


Yet, there is still a substantial room for improvement in the ranking quality of this model, as measured by av-erage precision, and it is possible that a smart combination with a high-quality contextual model would yield better performance.

we would expect that a good contextual model will identify the cases in which for potentially good synonyms pair, the source word appears in a sense that is not substitutable with the target, such as in examples 1, 4 and 5 in Table 1.


SVMs) can select the most relevant features from a high dimensional space.

we can use a tree kernel function to estimate the similarity between two AST s (see Sec- The second problem can be approached in two ways: (1) We can increase the recall of T BC to enlarge the set of candidate arguments.


The experiments were carried out within the setting defined in the CoNLL-2005 Shared Task (Carreras and Màrquez, 2005).

we adopted the Charniak parse trees available at www.lsi.upc.edu/∼srlconll/ along with the official performance evaluator.


The second step, which couples all the predictors, can be done by setting the rows of ¢ to the most significant left singular vectors of the predictor (weight) matrix Í Ù ½ Ù Ñ ℄, and setting Ú ¢Ù .

the structure matrix ¢ is computed so that the projection of the predictor matrix Í onto the subspace spanned by ¢'s rows gives the best approximation (in the least squares sense) of Í for the given row-dimension of ¢.


Using this model, ASO computes the structure matrix ¢ for each feature group separately.

sVD is applied to the sub-matrix of the predictor (weight) matrix corresponding to each feature group , which results in more focused dimension reduction of the predictor matrix.


However, with respect to the position-sensitive local context features, these two disambiguation problems may not have much in common since, for instance, we sometimes say "the bank announced", but we rarely say "the save announced".

whether problems share predictive structure may depend on feature types, and in that case, seeking predictive structure for each feature group separately may be more effective.


By contrast, global context features (GC) can be more effectively exploited when ASO is applied to all the disambigua-3 Our precision and recall are always the same since our systems assign exactly one sense to each instance.

our F-measure is the same as 'micro-averaged recall' or 'accuracy' used in some of previous studies we will compare with.


The probability of a parse tree is computed by summing up the probabilities of all derivations producing it, while the probability of a derivation is computed by multiplying the (smoothed) relative frequencies of its subtrees.

the probability of a subtree t is taken as the number of occurrences of t in the binary tree-set, | t |, divided by the total number of occurrences of all subtrees t' with the same root label as t. Let r(t) return the root label of t: The subtree probabilities are smoothed by applying simple Good-Turing to the subtree distribution (see Bod 1998: 85-87).


We provide empirical evidence that our approach is robust, by showing good performance on three different data sets.

substantial improvement is obtained over previously published results of worddistribution based systems when evaluation is done on a corpus of recorded and transcribed multi-party dialogs.


In this manner, we will have u i,j > 0 if and only if at least two occurrences of the j-th term occur within (2 • winSize) utterances on opposite sides of a boundary candidate.

each u i,j is capturing how many word cooccurrences appear across the candidate utterance in an interval (of (2•winSize) utterances) centered in the boundary candidate utterance.


In con-   trast to precision and recall, these metrics allow for a slight vagueness in where the hypothesized thematic boundaries are placed and capture "the notion of nearness in a principled way, gently penalizing algorithms that hypothesize boundaries that aren't quite right, and scaling down with the algorithm's degradation" (Beeferman et al., 1999).

computing both P k and WindowDiff metrics involves the use of a fixed-size (i.e.


We assume that the sentences with little or no perspective information, i.e., S = s 0 , are generated independently of the perspective of a document.

sentences that are presenting common background information or introducing an issue and that do not strongly convey any perspective should look similar whether they are in Palestinian or Israeli documents.


For raw text, the only boundaries that are explicitly marked are at the start and end of sentences, and it is this information we use to decide when to attach contexts to expressions 4 .

if a context is likely to start a sentence, we assume it is also likely to start a constituent at other positions within a sentence.


In this paper we describe and analyse these effects and suggest ways in which they may be overcome.

we show that by quarantining gazetteer features and training them in a separate model, then decoding using a logarithmic opinion pool (Smith et al., 2005), we may achieve much higher accuracy.


These cover names of people, places and organisations.

we have gazetteers containing surnames (88 © 799 entries), female first names (4 © 275 entries), male first names (1 © 219 entries), names of places (27 © 635 entries), names of companies (20 © 638 and 279 © 195 entries) and names of other organisations (425 entries).


The highest number of head error occurred at the CPOS tags PRP with 193 and V with 176.

just four prepositions (em, de, a, para) accounted for 120 head errors.


According to these two principles, we propose an improved pipeline framework for multi-lingual dependency parsing that aims at addressing the limitations of the pipeline processing.

(1) we use local search, a look ahead policy, to improve the accuracy of the predicted actions, and (2) we argue that the parsing algorithm we used minimizes the number of actions (Chang et al., 2006).


Analysis of the baseline parser's errors suggested the need for additional feature types to improve the identification of the root of the sentence.

the parser was frequently making errors in identifying the root of periphrastic constructions involving an auxiliary verb or modal and a participle.


(3) We haven't explored syntactic structures in depth for multi-lingual parsing and more structural features need to be used in the Check procedure.

we make error analysis to Chinese and Turkish.


Finally, we add constraints for Chinese, Dutch, Japanese and Slovene.

arity constraints to Chinese and Slovene, coordination and arity constraints to Dutch, arity and selective projectivity constraints for Japanese 2 .


In Turkish, we also find an interesting result where the recall rate of the distance=2 parsing (56.87) is lower than distance=3-6, and >7 (62.65, 57.83).

for Turkish, our parser failed to recognize the distance=2 dependency relations.


Due to its discriminative nature, the model is able to use a large feature set and to optimise compression accuracy directly.

mcDonald's model has a head start against our own model which does not utilise a parallel corpus and has only a few constraints.


We remedy this shortcoming by treating the semantic role assignment as a global optimization problem.

we model the interaction between all pairwise labeling decisions as a minimum weight bipartite edge cover problem (Eiter and Mannila, 1997;Cormen et al., 1990).


The reformulated sentences are submitted as queries to an IR engine for retrieving sentences with relevant answers.

we use the Lemur Toolkit 3 , a state-of-the-art language model-driven search engine.


We also add to Lemur's output gold standard sentences, which contain and support an answer for each question.

documents relevant for each question are retrieved from the AQUAINT Corpus 4 according to TREC supplied judgments.


Our evaluation was motivated by the following questions: (1) How does the incompleteness of FrameNet impact QA performance on the TREC data sets?

we wanted to examine whether there are questions for which in principle no answer can be found due to missing frame entries or missing annotated sentences.


(2) Are all questions and their corresponding answers amenable to a FrameNetstyle analysis?

we wanted to assess whether questions and answers often evoke the same or related frames (with similar roles).


To group the web search results into clusters, we assume that the candidate answer in each Google snippet can represent the "signature" of its cluster.

the Google snippets containing the same candidate are regarded as aligned snippets, and thus belong to the same cluster.


While both of the above papers focus on improving word alignment quality, we demonstrate that our approach can yield improvements in translation performance.

we aim to improve an Arabic-to-English (Ar-En) system using multiparallel data from Spanish (Es), French (Fr), Russian (Ru) and Chinese (Zh).


(2006) to improve BLEU and NIST scores, here the WSD training and predictions operate on full multi-word phrasal units, resulting in significantly more reliable and consistent gains as evaluted by many other translation accuracy metrics as well.

: • Instead of using a Senseval system, we redefine the WSD task to be exactly the same as lexical choice task faced by the multi-word phrasal translation disambiguation task faced by the phrase-based SMT system.


Our simplest syntactic feature is the identity of a synchronous rule.

we record its source tree, its target tree and their combination.


In this paper we address the task of inserting new information into existing texts.

we wish to determine the best location in a text for a given piece of new information.


Of course, improved clustering methods may also improve performance.

a clusterlearning algorithm that permits clusters to split and/or merge, as in Petrov et al.


We demonstrate an application of the new inference algorithm to non-projective dependency parsing.

we show how to implement two popular supervised learning approaches for this task: globally-normalized log-linear models and max-margin models.


Using the chain rule, the derivative of the log partition-function in Proposition 1 is To perform the derivative, we use the identity and the fact that ∂L h ,m (θ)/∂θ h,m is nonzero for only a few h , m .

when h = 0, the marginals are given by and for h > 0, the marginals are given by where δ h,m is the Kronecker delta.


One is how to learn a statistical model to estimate the conditional probability , and the other is how to generate confusion set C of a given query q We take a feature-based approach to model the posterior probability .

we use the maximum entropy model (Berger et al., 1996) for this task: where exp( ) is the normalization factor; , is a feature function defined over query q and correction candidate c , while is the corresponding feature weight.


In the first, we explore the use of features extracted from syntactic and semantic processing at a level that is robust to changes in genre and language.

we seek to go beyond the kind of bag of local words features employed in earlier systems (Bagga & Baldwin, 1998;Gooi & Allan, 2004;Pedersen et al., 2005) that did not attempt to exploit deep semantic features that are difficult to extract, and to go beyond the kind of biographical information (Mann & Yarowsky, 2003) that is unlikely to occur with great frequency (such as place of birth, or family relationships) in many of the documents returned by typical search engines.


We believe that the skewed long-tailed distribution that characterizes this data requires the use of clustering algorithms tailored to this distribution.

the large number of singleton clusters is an issue that confounds the standard clustering methods we have been employing.


In that case, the backoff estimate is: The backoff alphas (α) are a normalization factor that accounts for the discounted mass.

where −2 −1 > 0 simply says that the trigram was seen in training data.


We will decompose t into a pair of a quotient (t q ) and a remainder (t r ).


In this case, we have the following code, which is simply t in unary.

we write out 1  t zeros followed by a 1.


For example, to fix the storage per n-gram at around 13 bits: If we solve for λ, we obtain 0000 , 20 / 1   .

set P to a prime near N 000 , 20 and then do as much Stolcke pruning as necessary to meet the memory constraint.


We tested values of α in {0, +∞} ∪ {10 q | q ∈ {0, 1, ..., 16}}.

recall that α = 0 ignores the morphology model probabilities altogether (using an unweighted lattice), 6 in a correspondence of a morphemes in one string with b in the other, the penalty is a + b − 2, since the morpheme on each side is not in violation.


In comparison to previous bootstrapping algorithms developed for this problem, our approach aims to improve the quality of the seed clusters by employing seed words that are both distributionally and morphologically reliable.

we present a novel method for combining morphological and distributional information for seed selection.


Most notably, our approach aims to improve the quality of the seed clusters by employing seed words that are both distributionally and morphologically reliable.


Furthermore, given our emphasis on resource-scarce languages, our approach does not rely on any language resources.

the morphological information that it exploits is provided by an unsupervised morphological analyzer.


As mentioned in the introduction, our proposed solution to both tasks is to use an unsupervised morphological analyzer that can be built just from an unannotated corpus.

we have implemented an unsupervised morphological analyzer that outperforms Goldsmith's (2001) Linguistica and Creutz and Lagus's (2005) Morfessor for our English and Bengali datasets and compares favorably to the bestperforming morphological parsers in MorphoChallenge 2005 3 (see Dasgupta and Ng (2007)).


We denote the resulting set of clusters by C. Populating the clusters with words.

next, for each word w in our vocabulary, we check whether w can be assigned to any of the clusters in C. we assign w to a cluster c if w can combine with each of its primary keys and at least half of its secondary keys.


As mentioned above, we use distributional features to represent an instance created from a word w. The distributional features are created based on Schütze's (1995) method.

the left context and the right context of w are each encoded using the most frequent 500 words from the vocabulary.


To address this problem, we augment the feature set with suffixal features.

we create one binary feature for each of the 30 most frequent suffixes that we employed in the previous section.


Since we have a small seed set (5K words for English and 8K words for Bengali) and a large number of unlabeled words, we believe that it is most natural to apply a weakly supervised learning algorithm to bootstrap the clusters.

we employ a version of selftraining together with SVM as the underlying learning algorithm.


In fact, comparing rows 1 and 2, we outperform the baseline in each of the three steps of our algorithm.

our system yields F1-scores of 73% and 77% for the entire English and Bengali test sets, thus outperforming the baseline by 14% and 18% for English and Bengali, respectively.


At the subcluster level, the algorithm imposes a hard clustering on the words.

no word appears in more than one sub-cluster.


In contrast to existing algorithms developed for this problem, our algorithm is designed to (1) operate under a resource-scarce setting in which no languagespecific tools or resources are available and (2) more tightly integrate morphological information with the distributional POS induction framework.

our algorithm (1) improves the quality of the seed clusters by employing seed words that are distributionally and morphologically reliable and (2) uses support vector learning to combine morphological and distributional information.


Note further that the score associated with a particular operation may depend on any features computable from the respective positions in the two sequences.

in the experiments reported in this paper, we assume that each local function a is defined in terms and that these features have the functional form Σ * × N → R. each feature takes a sequence and an index and returns a real value.


The task poses a latent affinity between strings, but we can measure it only indirectly.

we can order some of the target sequences according to their affinity with a source sequence s. In this case, we train as follows: Order a sample of the target sequences according to this partial order.


Using a large list of English words, we define an edit affinity that is sensitive only to consonants.

the affinity between two words is the maximum number of consonant selfsubstitutions, with any substitutions involving the first five consonants counting for five normal substitutions.


In the ontology population settings traditional WSD approaches cannot be directly applied since entities are not reported into dictionaries, making the lexical entailment alternative more viable.

we model the problem of ontology population as the problem of recognizing for each mention of an entity of a particular coarsegrained type (e.g., location) the fine-grained concept (e.g., lake or mountain) that can be substituted in texts preserving the meaning.


Given a set of candidate concepts belonging to generic ontological types (e.g., people or locations), and a set of pre-recognized mentions of entities of these types in the corpus (e.g., Newton, Ontario), we assign the entity to the class whose lexicalization is more frequently entailed in the corpus.

as training set to learn the fine-grained category models, we use all the occurrences of their corresponding expressions in the same corpus (e.g., we collected all occurrences in context of the word scientist to describe the concept scientist).


In the multi-class setting, supervised learning approaches can be used.

we can apply a one-versus-all learning methodology, in which each class e i is trained from both positive (i.e., all the occurrences of e i in the corpus) and negative examples (i.e., all the occurrences of the words in the set {e j |j = i}).


When re-translating the source sentence, for each partially expanded decoding path, we compute the ratio of word pairs that satisfy such ordering constraints 2 .

given a partially expanded path is covered by a full decoding path Q (from other system outputs), we denote the relationship as (s i < s j ) ∈ Q.


The VI is the sum of these conditional entropies.

given a corpus labeled with hidden states and POS tags, if p(y),p(t) andp(y, t) are the empirical probabilities of a hidden state y, a POS tag t, and the cooccurance of y and t respectively, then the mutual information I, entropies H and variation of information VI are defined as follows: As Meilǎ (2003) shows, VI is a metric on the space of probability distributions whose value reflects the divergence between the two distributions, and only takes the value zero when the two distributions are identical.


We used the Forward-Backward algorithm to perform Expectation-Maximization, which is a procedure that iteratively re-estimates the model parameters (θ, φ), converging on a local maximum of the likelihood.

if the parameter estimate at time is (θ ( ) , φ ( ) ), then the re-estimated parameters at time + 1 are: where n x,y is the number of times observation x occurs with state y, n y ,y is the number of times state y follows y and n y is the number of occurences of state y; all expectations are taken with respect to the model (θ ( ) , φ ( ) ).


MacKay (1997) and Beal (2003) describe Variational Bayesian (VB) inference for HMMs in detail, and Kurihara and Sato (2006) describe VB for PCFGs (which only involves a minor modification to the M-step of the Inside-Outside algorithm).

the E-step for VB inference for HMMs is the same as in EM, while the M-step is as follows: Figure 4: The conditional distribution for state y i used in the Gibbs sampler, which conditions on the states y −i for all observations except x i .


To detect a coordinate structure involving a key bunsetsu, b n , we consider only a partial matrix (denoted A n ), that is, the upper right part of b n ( Figure  1).

to specify correspondences between bunsetsus in potential pre-and post-conjuncts, a path is defined as follows: where a path represents a series of elements from a non-zero element in the lowest row in A n to an element in the leftmost column in A n .


The F-measure of our method is slightly higher than that of the baseline method (KNP).


This low accuracy is attributed to the out-of-domain training corpus.

the parser is trained on a newspaper corpus, whereas the test corpus is obtained from the web, because of the non-availability of a tagged web corpus that is large enough to train a supervised parser.


This algorithm is proved to converge (i.e., there are no more updates) in the separable case (Collins, 2002a).

1 if there exist weight vector U (with ||U || = 1), δ (> 0), and R (> 0) that satisfy: the number of updates is at most R 2 /δ 2 .


To deal with this problem, we first relaxed our objective.

the modified objective was to find a good model from those with the form: we first generate n-best candidates {y n } under the local model, Φ l (x, y) • α.


However, by examining the Algorithm 4.2: Perceptron with local and non-local features (parameters: if y 1 ̸ = yi * then (y 1 represents the best in {y n }) proofs in Collins (2002a), we can see that the essential condition for convergence is that the weights are always updated using some y (̸ = y * ) that satisfies: • α ≤ 0 (≤ C in the case of a perceptron with a margin).

2y does not necessarily need to be the exact best candidate or the exact second-best candidate.


With the larger goal of automatically constructing a Pan-Chinese lexical resource, this work aims at taking an existing semantic classificatory structure as leverage and incorporating new words into it.

it is important to see if the classification could accommodate new words from heterogeneous data sources, and whether simple similarity measures and clustering methods could cope with such variation.


Hence, the current study aims at taking an existing Chinese thesaurus, namely the Tongyici Cilin 同義詞詞林, as leverage and extending it with lexical items specific to individual Chinese speech communities.

the feasibility depends on the following issues: (1) Can lexical items from various Chinese speech communities, that is, from such heterogeneous sources, be classified as effectively with methods shown to work for clustering closely related words from presumably the same, or homogenous, source?


For the present study, we made use of the subcorpora collected over the 9-year period 1995-2004 from Beijing (BJ), Hong Kong (HK), Singapore (SG), and Taipei (TW).

we made use of the financial news sections in these subcorpora, from which we extracted feature vectors for comparing similarity between a given target word and a thesaurus class, which is further explained in Section 4.3.


Kwong and Tsou (2006) observed that among the unique lexical items found from the individual subcorpora, only about 30-40% are covered by Cilin, but not necessarily in the expected senses.

cilin could in fact be enriched with over 60% of the unique items from various regions.


To evaluate the effectiveness of the automatic classification, we adopted a simple baseline measure by ranking the 94 subclasses in descending order of the number of words they cover.

assuming the bigger the subclass size, the more likely it covers a new term, thus we compared the top-ranking subclasses with the classifications obtained from the automatic method using the cosine measure.


We use a two-stage approach to enhance the reliability of summarization.

we add a process of low-quality review detection before the summarization process, so that the summarization result is obtained based on the high-quality reviews only.


The basic idea is that: several candidate translation models are prepared in training stage.


To test our online model optimization method, we divide the baseline corpus according to the origins of sub corpus.

the FBIS, HK_ Hansards and HK_News models are used as three submodels and the baseline model is used as general model.


Story is broadly defined as segment of text containing topically related sentences.

the task may require segmenting a stream of broadcast news, addressed by the Topic Detection and Tracking (TDT) evaluation project (Wayne, 2000;Allan, 2002).


GLSA begins with a matrix of pair-wise term similarities S, computes its eigenvectors U and uses the first k of them to represent terms and documents, for details see (Matveeva et al., 2005).

the justification for this approach is the theorem by Eckart and Young (Golub and Reinsch, 1971) stating that inner product similarities between the term vectors based on the eigenvectors of S represent the best elementwise approximation to the entries in S. the inner product similarity in the GLSA space preserves the semantic similarities in S. Since our representation will try to preserve semantic similarities in S it is important to have a matrix of similarities which is linguistically motivated.


We use our selectional preference models to provide the probability that a candidate is representative of the typical objects of the verb.

if the object might typically occur in such a relationship then this should lessen the chance that this verb-object combination is non-compositional.


There is, however, a large and growing body of work concerned with the summarization of short documents, with evaluations typically focusing on news articles.

a significant number of summarization systems have been proposed during the recent Document Understanding Conference exercises (DUC) -annual evaluations that usually draw the participation of 20-30 teams every year.


We constructed a data set starting from the observation that several English and literature courses make use of books that are sometimes also available in the form of abstracts -meant to ease the access of students to the content of the books.

we have identified two main publishers that make summaries available online for books studied in the U.S. high-school and college systems: Grade Saver (http://www.gradesaver.com) and Cliff's Notes (http://www.cliffsnotes.com/).


We decided to make several changes to our initial system, in order to account for the specifics of the data set we work with.

our data set consists of very large documents, and correspondingly the summarization of such documents requires techniques that account for their length.


Once the text is divided into segments, we generate a separate summary for each segment, and consequently create a final summary by collecting sentences from the individual segment summaries in a round-robin fashion.

starting with the ranked list of sentences generated by the summarization algorithm for each segment, we pick one sentence at a time from each segment summary until we reach the desired book-summary length.


Instead, since we noticed that the summaries generated with our system and with TextRank covered different sentences, we implemented a method that combines the top ranked sentences from the two methods.

the combination method picks one sentence at a time from the summary generated by our system for each segment, followed by one sentence selected from the summary generated by the TextRank method, and so on.


6 Second, it showed that systems developed for the summarization of short documents do not fare well when applied to very long documents such as books, and instead a better performance can be achieved with a system that accounts for the length of the documents.

the book summarization system we developed was found to lead to more than 30% relative error rate reduction with respect to an existing state-of-the-art summarization tool.


For COARSE, the PTB tags were even further reduced to 15 category tags, 5 which is still six more than the core consensus tag set used in Yarkowsky and Ngai (2001).

cOARSE was measured by comparing the first letter of each tag.


A drawback of vector space models is the difficulty of interpreting what some degree of "generic semantic similarity" between two target words means in linguistic terms.

this similarity is not sensitive to selectional preferences over specific semantic relations, and thus cannot model the plausibility data we are interested in.


In order to satisfy our homogeneity criteria, a clustering must assign only those datapoints that are members of a single class to a single cluster.

the class distribution within each cluster should be skewed to a single class, that is, zero entropy.


However, our completeness criterion is not measured at all.

they do not address the question of whether all members of a given class are included in a single cluster.


(The precision and recall for each class is 3 5 .)

for each class, the best or "matched" cluster contains 3 of 5 elements of the class (Recall) and 3 of 5 elements of the cluster are members of the class (Precision).


A second class of clustering evaluation techniques is based on a combinatorial approach which examines the number of pairs of data points that are clustered similarly in the target and hypothesized clustering.

each pair of points can either be 1) clustered together in both clusterings (N 11 ), 2) clustered separately in both clusterings (N 00 ), 3) clustered together in the hypothesized but not the target clustering (N 01 ) or 4) clustered together in the target but not in the hypothesized clustering (N 10 ).


Dom's Q 0 measure (Dom, 2001) uses conditional entropy, H(C|K) to calculate the goodness of a clustering solution.

given the hypothesized partition, what is the number of bits necessary to represent the true clustering?


While this is an application of the minimum description length (MDL) principle (Rissanen, 1978;Rissanen, 1989), it does not provide an intuitive manner for assessing our two competing criteria of homogeneity and completeness.

at what point does an increase in conditional entropy (homogeneity) justify a reduction in the number of clusters (completeness).


This is due to an increased homogeneity ( H (C|K) H(C) ) and a relatively stable completeness ( H (K|C) H(K) ).

inclusion of more clusters leads to clusters with a more skewed within-cluster distribution and a equivalent distribution of cluster memberships within classes.


A learning algorithm for this model is a supervised learning algorithm.

an explicit topic model learns model parameter using a training data set of tuples such as (documents, topics).


In this paper, we focus on the explicit topic model.

we propose a novel model that is based on PMM but fundamentally improved.


One point on the simplex indicates one mixture ratio π of the three topics.

the point indicates multiple topics with the mixture ratio.


PMM generates documents assuming that each mixture ratio is equal.

pMM generates only documents with multiple topics that indicates the center point of the 2Dsimplex in Figure 1.


On the contrary, PDMM generates documents assuming that mixture ratio π follows Dirichlet distribution.

pDMM can generate documents with multiple topics whose weights can be generated by Dirichlet distribution.


Despite these encouraging results, we believe that additional research on improved estimation of probabilities in N-gram-or phrase-based statistical machine translation systems is needed.

the problem of generalization to new translations seems to be promising to us.


Each classifier predicts a small set of possible values and hence can be highly accurate.

the basic classifiers do not suffer from problems of data sparseness.


There is a potential risk of redundancy if we represent related statistics using the log-frequency BF scheme presented in Talbot and Osborne (2007).

we do not need to store information explicitly that is necessarily implied by the presence of another item in the training set, if that item can be identified efficiently at query time when needed.


Assuming that statistics are entered consistently from the same corpus, we can take advantage of the monotonicity of the n-gram event space to place upper bounds on the frequencies of events to be retrieved from the filter prior to querying it, thereby reducing the a priori probability of a negative and consequently the error rate.

since the log-frequency BF scheme will never underestimate an item's frequency, we can apply the following inequality recursively and bound the frequency of an n-gram by that of its least frequent subsequence, We use this to reduce the error rate of an interpolated BF language model described below.


Distinct suffix counts of 2 and above are stored by subtracting this proxy count and converting to the log quantisation scheme described above, i.e., In testing for a suffix count, we first query the item c(w 1 , ..., w n ) = 1 as a proxy for s(w 1 , ..., w n ) = 1 and, if found, query the filter for incrementally larger suffix counts, taking the reconstructed suffix count of an n-gram with a non-zero n-gram count to be the expected value, i.e., Having created a BF containing these events, the algorithm we use to compute the interpolated WB estimate makes use of the inequalities described above to reduce the a priori probability of querying for a negative.

we bound the count of each numerator in the maximum likelihood term by the count of the corresponding denominator and the count of distinct suffixes of an n-gram by its respective token frequency.


The technique based on Lin's similarity measure does outperform the baseline, but still suffers from bad precision, along with Padó and Lapata's association measure.

the similarity and association measures seem to be too noisy to be used directly for ranking antecedents.


Whatever the disease or impairment considered, oral communication is impossible for these persons who have in addition serious difficulties to control physically their environment.

they are not able to use standard input devices of a computer.


During translation decoding, the domain specific language model was used as an additional feature in the log-linear combination according to the PHARAOH decoder's option.

the general and domain specific language models are combined by log-linear rather than linear interpolation.


The improvements over the baseline of our method in both BLEU and NIST (Doddington 2002) score were greater than those for both CLM and SMix.

our method showed improvent in both the BLEU and NIST scores, this is in contrast to the CLM and SMix methods which both degraded the translation performance in terms of the NIST score.


We will compare a variety of training criteria for statisti-cal machine translation.

we are considering criteria for the log-linear parameters or model scaling factors.


Hence, for simplicity, we only consider anaphoric zero pronouns (AZP) in this work.

we only attempt to resolve a coreferential zero pronoun to noun phrases preceding it.


In this paper, we are only concerned with grammars that define proper probability distributions over trees, such that the probability of all derivations sum up to 1 and no probability mass gets lost in derivations that never reach a terminal yield.

we require (if t(d) is the tree derived by derivation d): For simplicity, but without loss of generality, we assume there are no recursions on the start symbol.


We compute a word-pair's entropy by considering it as a distribution over features, in a manner that is analogous to the feature entropy defined in (Turney, 2006).

the normalized entropy of a word-pair wp is: is the conditional probability of f occurring in the context of wp.


The relational similarity between two wordpairs is the cosine measure of their corresponding row vectors in the reduced feature space.

letÂ i denote the ith row vector of A, then the relational similarity between wordpairs wp i 1 and wp i 2 is: The primary difference between our algorithm and LRA is that LRA also includes each word's synonyms in the computation.


Then, by transitivity, wp 1 and wp 3 are also likely to be relationally similar.

the relational similarity between two word-pairs can be reinforced by other word-pairs through transitivity.


The significant overlap between SGT and LRA-S' outputs allowed us to evaluate SGT using the samples collected from the previous surveys instead of conducting a new round of human grading.

we identified previously graded samples that  Table 4: Examples of Good and Poor Lexical Analogies Generated had also been generated by SGT, and used these samples as the evaluation data points for SGT.


One of our immediate tasks is to complement dependency patterns with additional features.

we expect semantic features such as word definitions from machine-readable dictionaries to improve our system's ability to differentiate between different senses of polysemic words, as well as different granularities of semantic relations.


We have proposed a new method to determine semantic distance in a possibly resource-poor language by combining its text with a knowledge source in a different, preferably resource-rich, language.

we combined German text with an English thesaurus to create cross-lingual distributional profiles of concepts-the strengths of association between English thesaurus senses (concepts) of German words and co-occurring German words-using a German-English bilingual lexicon and a bootstrapping algorithm designed to overcome ambiguities of word-senses and translations.


Formally, we define the probability n (t) i of finding the particle at node n i ∈ V at time t as the sum of all ways in which the particle could have reached n i from any other node at the previous time-step: where P (n i | n j ) is the conditional probability of moving to n i given that the particle is at n j .

we construct the transition distribution such that P (n i | n j ) > 0 whenever WordNet specifies a local link relationship of the form j → i.


The skew divergence 2 accounts for zeros in q by mixing in a small amount of p. Lee found that as α → 1, the performance of skew divergence on natural language tasks improves.

it outperforms most other models and even beats pure KL divergence modified to avoid zeros with sophisticated smoothing models.


The features of relation type, attachment point as well as the distance characterize the topological relations among the relevant etrees.

the attachment point and distance features can explicitly capture important information hidden in the standard path feature.


Compared to (Liu and Sarkar, 2006), we have used a more sophisticated learning algorithm and a richer set of syntactic LTAG-based features in this task.

in this paper we built a strong baseline system using a standard set of features and did a thorough comparison between this strong baseline and our proposed system with LTAG-based features.


Thus, our method predicts the correct actions by measuring the distance between the codeword string[k] and the predicted binary (later extended to real-valued) sequences string output.

our method selects the action that is the closest to the outputs of the learned model.


Table 2 shows that PARENT-ANCESTOR METHOD is more accurate than the other two   Kudo and Matsumoto (2002) 89.19% 46.64% Proposed method 89.54% 47.38% Sekine (2000) 87.20% 40.76% Kudo and Matsumoto (2000) 89.09% 46.17% Kudo and Matsumoto (2002) 89.29% 47.53% Sassano (2004) 89.56% 48.35% w/o Rich Sassano (2004)  methods.

the accuracy of dependency analysis improves by utilizing the redundant information.