Research "into the automatic acquisition of subcategorization frames (SCFS) from corpora is starting to produce large-scale computational lexicons which include valuable frequency information.

the accuracy of the resulting lexicons shows room for improvement.


This could make the results appear better for in-corpora experiments.

in the crosscorpora experiments training and testing example come from different documents.


When trained as a single classifier (e.g., (Roth and Zelenko, 1998) uses each -tagged example as a positive example for and a negative example for all other tags.

the SM classifier is trained on a -tagged example of word , by using it as a positive example for and a negative example only for the effective confusion set.


Furthermore, during classification the president consults the same members that were used to prepare its training set.

in crossvalidation stacking the president is tested using members that have received more training than those that prepared its training set.


The same memory-based learner was used as the president.

we experimented with several configurations, varying the neighborhood size (k) from 1 to 10, and providing the president with the m best wordattributes, as in Section 1, with m ranging from 50 to 700 by 50.


Our goals are more general than those of information extraction, and so this work should be helpful for that task.

our approach will not solve issues surrounding previously unseen proper nouns, which are often important for information extraction tasks.


In the training set, these nouns appear only in NCs that have been labeled as belonging to relation 22.

if we look at relations 14 and 15, we find a wider range of words, and in some cases  Table 1.


This unfortunately eliminates acronyms like "U. S." and phrasal verbs like "throw up."

discarding some words may be worthwhile if the final list of n-grams is richer in terms of MRD headwords.


These formulations suggest that several of the probabilistic algorithms we have seen include non-compositionality measures already.

since the probabilistic algorithms rely only on distributional information obtained by considering juxtaposed words, they tend to incorporate a significant amount of non-semantic information such as syntax.


The problem of abbreviation processing has attracted relatively little attention in NLP field.

technical documents use a lot of abbreviations to represent domainspecific knowledge.


This may underestimate the difculty of the Brown corpus by including sentences from the same documents in training and test sets.

because of the variation within the Brown corpus, we felt that a single contiguous test section might not be representative.


However, its priors cause it to incorrectly predict that aa belongs to class 1.

maximizing CL will push the prior for sense 1 arbitrarily close to zero.


If we want to be guaranteed a non-deficient joint interpretation, we can require equality.

if we relax the equality then we have a larger feasible space which may give better values of our objective.


If w occurs with s in an example where other good indicator words are present, then those other words' large weights will explain the occurrence of s, and without w having to have a large weight, its expected count with s in that instance will approach 1.

if no trigger words occur in that instance, there will be no other explanation for s other than the presence of w and the other non-indicative words.


Our technique could easily be combined with these techniques, presumably leading to even better results.

since we build our decision lists from last to first, rather than first to last, the local probability is not available as the list is being built.


If one wants a probabilistic decision list learner, this is clearly the algorithm to use.

if probabilities are not needed, then TBL can produce lower error rates, with still fewer rules.


However, if probabilities are not needed, then TBL can produce lower error rates, with still fewer rules.

if one wants either the lowest entropies or highest accuracies, then it appears that linear models, such as maxent or the perceptron algorithm with margin work even better, at the expense of producing much larger models.


The recent work of Pedersen (2001a) and  evaluated a variety of learning algorithms on the SENSEVAL-1 data set.

all of these research efforts concentrate only on evaluating different learning algorithms, without systematically considering their interaction with knowledge sources.


In SENSEVAL-2, the various Duluth systems (Pedersen, 2001b) attempted to investigate whether features or learning algorithms are more important.

relative contribution of knowledge sources was not reported and only two main types of algorithms (Naive Bayes and decision tree) were tested.


However, they reported recall of only 56.8%.

our implementation of SVM using the two knowledge sources of surrounding words and local collocations achieves recall of 61.8%.


As a hill-climbing procedure, the algorithm terminates when removal of any of the rules in the ruleset fails to improve performance.

to most existing algorithms for coreference resolution, RULE-SELECT establishes a tighter connection between the classification-and clustering-level decisions for coreference resolution and ensures that system performance is optimized with respect to the coreference scoring function.


Charniak (1997) calls this a "Treebank grammar" and gambles that assigning 0 probability to rules unseen in training data will not hurt parsing accuracy too much.

there are four reasons not to use a Treebank grammar.


Indeed a good deal of syntax induction work has been carried out in just this framework (Stolcke and Omohundro, 1994;Chen, 1996;De Marcken, 1996;Grünwald, 1996;Osborne and Briscoe, 1997).

all such work to date has adopted rather simple prior distributions.


The very rarity of these rules makes it impossible to create a table like Table 1.

rare rules can be measured in the aggregate, and the result suggests that the same kinds of transformations are indeed useful-perhaps even more useful-in predicting them.


It is worthwhile to compare the statistical approach here with some other approaches: • Transformation models are similar to graphical models: they allow similar patterns of deductive and abductive inference from observations.

the vertices of a transformation graph do not represent different random variables, but rather mutually exclusive values of the same random variable, whose probabilities sum to 1.


Using movie reviews as data, we find that standard machine learning techniques definitively outperform human-produced baselines.

the three machine learning methods we employed (Naive Bayes, maximum entropy classification, and support vector machines) do not perform as well on sentiment classification as on traditional topic-based categorization.


The bulk of such work has focused on topical categorization, attempting to sort documents according to their subject matter (e.g., sports vs. politics).

recent years have seen rapid growth in on-line discussion groups and review sites (e.g., the New York Times' Books web page) where a crucial characteristic of the posted articles is their sentiment, or overall opinion towards the subject matter -for example, whether a product review is positive or negative.


They also handily beat our two human-selected-unigram baselines of 58% and 64%, and, furthermore, perform well in comparison to the 69% baseline achieved via limited access to the test-data statistics, although the improvement in the case of SVMs is not so large.

in topic-based classification, all three classifiers have been reported to use bagof-unigram features to achieve accuracies of 90% and above for particular categories (Joachims, 1998;Nigam et al., 1999) 9 -and such results are for settings with more than two classes.


This would not rule out the possibility that bigram presence is as equally useful a feature as unigram presence; in fact, Pedersen (2001) found that bigrams alone can be effective features for word sense disambiguation.

comparing line (4) to line (2) shows that relying just on bigrams causes accuracy to decline by as much as 5.8 percentage points.


11 Alternatively, we could have tried integrating frequency information into MaxEnt.

feature/class functions are traditionally defined as binary (Berger et al., 1996); hence, explicitly incorporating frequencies would require different functions for each count (or count bin), making training impractical.


12 This serves as a crude form of word sense disambiguation (Wilks and Stevenson, 1998): for example, it would distinguish the different usages of "love" in "I love this movie" (indicating sentiment orientation) versus "This is a love story" (neutral with respect to sentiment).

the effect of this information seems to be a wash: as depicted in line (5) of Figure 3, the accuracy improves slightly for Naive Bayes but declines for SVMs, and the performance of MaxEnt is unchanged.


In terms of relative performance, Naive Bayes tends to do the worst and SVMs tend to do the best, although the differences aren't very large.

we were not able to achieve accuracies on the sentiment classification problem comparable to those reported for standard topic-based categorization, despite the several different types of features we tried.


We also assume the existence of a special Null word in the source language that generates words in the target language.

we define a different model that better constrains and conditions generation from Null.


Thus this probability distribution provides prior knowledge of the possible translations of a word based only on its part of speech.

p( A "¨£ z f D g ) should not be too sharp or 4 Since we are only concerned with alignment here and not generation of candidate translations the factor p( e,eT) can be ignored and we omit it from the equations for the rest of the paper.


Following (Briscoe and Carroll, 1993), conflict resolution is based on contextual information extracted from the so called Instantaneous Description or Configuration: a stack, representing the control memory of the LR parser, and a lookahead sequence, here limited to one symbol.

1 while Briscoe and Carroll invested on massive parallel computation of the possible parsing paths, with pruning and posterior ranking, we ex-periment with a simple greedy depth-first technique with limited amount of backtracking, that resembles to a certain extent the commitment/recovery models from the psycholinguistic research on human language processing, supported by the occurrence of "garden paths".


The reason for such increase in structure is not quite a particular decision of ours, but a consequence of using a sound grammar under the TAG grammatical formalism.

9 having concluded our manifesto, we understand that algorithms that try to keep precision as high as the recall necessarily have losses in recall compared to if they ignored the precision, and therefore in order to have fair comparison with them and to improve the credibility of our results, we flattened the parse trees in a post-processing step, using a simple rule-based technique on top of some frequency measures for individual grammar trees gathered by (Xia, 2001) and the result is presented in the bottom lines of the table.


In practice, the instances of this class of error are all cases where the computer can't detect the error for certain.

for all Type B errors, once detected, the correction that needs to be made is clear, at least to a human observer with access to the annotation guidelines.


To compensate, statistical models were used to separate the meaningful semantic associations from the spurious ones.

our work aims to identify "strong" syntactic heuristics that can isolate instances of general structures that reliably identify the desired semantic relations.


At prima facie, the Viterbi alignment for the first sentence pair appears incorrect because we, as humans, have a natural tendency to build alignments between the smallest phrases possible.

note that the choice made by our model is quite reasonable.


The model described in this paper cannot learn that the English word "not" corresponds to the French words "ne" and "pas".

our model learns to deal with negation by memorizing longer phrase translation equivalents, such as ("ne est pas", "is not"); ("est inadmissible", "is not good enough"); and ("ne est pas ici", "is not here").


And Marcu (2001) extracts phrase translations from automatically aligned corpora and uses them in conjunction with a word-for-word statistical translation system.

none of these approaches learn simultaneously the translation of phrases/templates and the translation of words.


We have chosen to present MBR decoding using the IBM-3 statistical MT models implemented via WFSTs.

mBR decoding is not restricted to this framework.


We have presented these alignment loss functions to explore how linguistic knowledge might be incorporated into machine translation systems without building detailed statistical models of these linguistic features.

we stress that the MBR decoding procedures described here do not preclude the construction of complex MT models that incorporate linguistic features.


In principle it can assist in the production of a target text with minimal disruption to a translator's normal routine.

recent evaluations of a prototype prediction system showed that it significantly decreased the productivity of most translators who used it.


Moreover, sentence (3) omits the goal argument entirely.

as Figure 2 shows, the combination of these verbalizations, as computed by our multiple-sequence alignment method, exhibits high structural similarity to the semantic input: the indicated "sausage" structures correspond closely to the three arguments of show-from.


Librarians and search professionals have traditionally favored Boolean keyword search systems, which, when successful, return a small set of relevant hits.

the success of these systems critically depends on the choice of the right keywords and the appropriate Boolean operators.


(2001) also iteratively reformulate queries based partly on the search results.

their mechanism for query reformulation is heuristic-based.


However, those words are not necessarily useful for retrieval.

low-frequency words appearing in specific documents are often effective query terms.


For example, in the case where "kankitsu (citrus)" is not listed in the dictionary, this word should be transcribed as /ka N ki tsu/.

it is possible that this word is mistakenly transcribed, such as /ka N ke tsu/ and /ka N ke tsu ke ko/.


Then, in the second stage, we replace detected OOV words with identified index terms so as to complete the transcription, and re-perform text retrieval to obtain final outputs.

we do not re-perform speech recognition in the second stage.


In spoken document retrieval, an open-vocabulary method, which combines recognition methods for words and syllables in target speech documents, was also proposed (Wechsler et al., 1998).

this method requires an additional computation for recognizing syllables, and thus is expensive.


6 Query Completion As explained in Section 3, the basis of the query completion module is to correspond OOV words detected by speech recognition (Section 4) to index terms used for text retrieval (Section 5).

to identify corresponding index terms efficiently, we limit the number of documents in the first stage retrieval.


In principle, terms that are indexed in topranked documents (those retrieved in the first stage) and have the same sound with detected OOV words can be corresponding terms.

a single sound often corresponds to multiple words.


score) when we use hierarchical structure.

the computation of the former is far more efficient than the latter.


Recent work by Banko and Brill (2001) suggests that this would not necessarily be true if very large training corpora were available.

their results are limited by the simplicity of their evaluation task and individual classifiers.


The simplest evaluation measure is direct comparison of the extracted thesaurus with a manuallycreated gold standard (Grefenstette, 1994).

on smaller corpora direct matching is often too   Bernard, 1990) and Roget's (Roget, 1911) thesauri and the head ordered Moby (Ward, 1996) thesaurus.


Grefenstette (1998) observes that this approach suffers from an acute data sparseness problem if the corpus counts are obtained from a conventional corpus such as the British National Corpus (BNC) (Burnard, 1995).

as Grefenstette (1998) demonstrates, this problem can be overcome by obtaining counts through web searches, instead of relying on the BNC.


Because these two LMs tightly integrate the word information jointly with the tag distribution, the trigram information is already represented.

the cSuperARV LM and Chelba's and Charniak's parserbased LMs have much lower correlations, indicating they have much lower overlap with the trigram.


This assumption can be formulated as a co-occurrence model for headword prediction: that is, the probability of a headword is determined by the occurrence of other headwords within a window.

in our experiments, we instead used an interpolated probability First, co-occurrence models do not predict words from left to right, and are thus very difficult to interpolate with trigram models for decoding.


In the TREC QA track, there is no distinction made in scoring between returning a wrong answer to a question for which an answer exists and returning no answer.

to deploy a real system, we need the capability of making a trade-off between precision and recall, allowing the system not to answer a subset of questions, in hopes of attaining high accuracy for the questions which it does answer.


From the Tables 3 and 4, it is clear that the 7-gram and 9-gram models are quite similar to the 5-gram model both in the performance and in the distribution of correct/over-generated named entities.

variable length models have distribution of correct/over-generated named entities a lit-  tle different from that of the 5-gram model.


what is a crossing when moving from English to French is not guaranteed to be a crossing when moving from French to English).

we only pursue translation direction since that is the one for which we have parsed data.


The major reason may be that the resolution algorithm relies on surface features and does not have access to world or domain knowledge, which we did not want to depend upon since we were mainly interested in cheap features.

the string ident and substring match features did not perform very well either.


Other researchers like Vieira and Poesio (2000) used information about the syntactic structure and compared only the syntactic heads of the phrases.


In our two-phase approach it is straightforward for the second-phase classifier to take the length of a candidate phone number into account.

standard named entity taggers that use trigram features do not exploit this information, and doing so would entail significant changes to the underlying models and parameter estimation procedures.


Mandarin Chinese is, of course, not resource-deficient for language modeling -100s of millions of words are available on-line.

we choose it for our experiments partly because it is sufficiently different from English to pose a real challenge, and because the availability of large text corpora in fact permits us to simulate controlled resource deficiency.


Looking at the broader context they appear in can provide additional insight: if the types of information expressed in the contexts are similar, then the specific information expressed in the sentences is more likely to be the same.

if the types of information in the two contexts are unrelated, chances are that the sentences should not be aligned.


We compiled two collections from the Encyclopedia Britannica and Britannica Elementary.

to the long (up to 15-page) detailed articles of the Encyclopedia Britannica, Britannica Elementary contains one-to two-page entries targeted towards children.


The two models may contain overlapping information: in many cases, the lexical cue corresponds to the immediate head-word the EE depends on.

other surrounding words (which frequently correspond to the headword of grandparent of the empty node) often carry important information, especially for distinguishing NP-NP and PRO-NP nodes.


Therefore our new tree kernel is more lexicalized.

it immediately follows that the lexicalized tree kernel is well-defined.


Then we combine the results of individual SVMs with simple combination.

the overall performance does not improve.


Naive Bayes classifiers are fairly resistant to class skewness, which can only exert its influence on classifier prediction via the class priors.

decision lists suffer from skewed class distributions.


Frequency counts of feature-value pairs in these classifiers are updated independently, and thus a single instance can possibly contribute to the discovery of more than one useful feature-value pair.

some classifiers such as decision trees are not able to take advantage of this redundancy because of their intrinsic nature of recursive data partitioning.


(2003a) also investigate instance selection methods for co-training, but their goal is primarily to use selection methods as a means to explore the trade-off between maximizing coverage and maximizing accuracy.

6 our focus 6 McCallum and Nigam (1998)   here is on examining whether a more conservative ranking method can alleviate the problem of performance deterioration.


If the underlying learners have indeed induced two different hypotheses from the data, then each classifier can potentially acquire informative instances from the other and yield performance improvements very rapidly.

our ranking method is more conservative in that it places more emphasis on maintaining labeled data accuracy than the B&M method.


This inequality constraint falls into a type of fat constraints, (Khudanpur, 1995).

as noted in (Chen and Rosenfeld, 2000), this type of constraint has not yet been applied nor evaluated for NLPs.


For this reason, HMMs have been standardly used with current word-current label, and previous label(s)-current label features.

if we incorporate information about the neighboring words and/or information about more detailed characteristics of the current word directly to our model, rather than propagating it through the previous labels, we may hope to learn a better classifier.


The best performing models of label sequence learning are MEMMs or PMMs (also known as Maximum Entropy models) whose features are carefully designed for the specific tasks (Ratnaparkhi, 1999;Toutanova and Manning, 2000).

maximum entropy models suffer from the so called label bias problem, the problem of making local decisions (Lafferty et al., 2001).


To  overcome the numerical problems of working with a product of a large number of small probabilities, usually the logarithm of the likelihood of the data is optimized.

most of the time, these systems, sequence labelling systems in particular, are tested with respect to their error rate on test data, i.e.


Sequential Exp-loss function: This loss function, was first introduced in (Collins, 2000) for NLP tasks with a structured output domain.

there, the sum is not over the whole possible label sequence set, but over the best label sequences generated by an external mechanism.


Thus, one only needs to modify those sum [i, y] that satisfy f * (x i , y)=1, and to make changes to their corresponding normalizing factors z [i].

to what is shown in Berger et al 1996's paper, here is how the different values in this variant of the IFS algorithm are computed.


Furthermore, as the length of context increases, the ratio for the Kneser-Ney smoothed model becomes greater -a clear sign of over-parameterization.

the ratio for the neural network model changes very little even when the length of the context increases from 4 (2H) to 8 (3H-1OP).


Figure 4: Learning curves As we expected, the learning curve of the training data in EM iteration 1 is not as smooth as that in EM iteration 0, and even more so for the heldout data.


In 3, "other risk factors for Mr. Cray's company" refers to a set of risk factors excluding the designer's age.

in list-contexts such as (4), the antecedent is available both anaphorically and structurally, as the left conjunct of the anaphor.


The algorithm's performance with this feature set is encouraging.

the semantic knowledge the algorithm relies on is not sufficient for many cases of other-anaphors (Section 4.2).


In this way, our zero resolver creates a 'general purpose' candidate list.

some of the candidates are inappropriate for certain zeros.


Siblings When CP is wa or mo, it is not clear whether 7 is a subject.

a verb rarely has the same entity in two or more cases.


Many of these algorithms are, in principle, language-independent.

when applying these algorithms to languages such as Chinese and Japanese, we must deal with certain language-specific issues: for example, should we build a character-based model or a word-based model?


On one hand, the word-based model is attractive since it allows the system to inspect a larger window of text, which may lead to more informative decisions.

a word segmenter is not error-prone and these errors may propagate and result in errors in NE recognition.


Our HMM classifier for English uses a set of word-features to indicate whether a word contains all capitalized letters, only digits, or capitalized letters and period, as described in (Bikel et al., 1999).


In the previously described methods, also known as voting, each classifier gave its entire vote to one classification -its own output.

equation (2) allows for classifiers to give partial credit to alternative classifications, through the probability @ # © u e!


For example, suppose that we have a set of documents as in Table 1.

some possible virtual examples generated from Document 1 by GenerateByDeletion algorithm are´ ¾ ¿ GenerateByAddition algorithm is: 1.


We have reduced the problem to a polynomial size QP, which, in principle, can be solved using standard QP toolkits.

although the number of variables and constraints in the factored dual is polynomial in the size of the data, the number of coefficients in the quadratic term in the objective is very large: quadratic in the number of sentences and dependent on the sixth power of sentence length.


7 Nonetheless, generative approaches are vastly cheaper to train, since they must only collect counts from the training set.

the max-margin approach does have the potential to incorporate many new kinds of features over the input, and the current feature set allows limited lexicalization in cubic time, unlike other lexicalized models (including the Collins model which it outperforms in the present limited experiments).


Unlike the other two strategies, predictions must sometimes be made when there is no immediately adjacent complete node in the tree.

these comparisons are not conclusive, because the choice of features for the state representation may also have an important role in the differences.


They report 95% coverage and 75% average recall and precision on sentences of length ≤ 40 with 490 popped edges; this is ten times the minimum number of steps.

to get complete coverage, they required 1760 popped edges, which is a factor of 37 greater than the minimum.


For example, using a 1.5GB newspaper corpus, here are the 20 most associated paths to "X solves Y" generated by DIRT: Y is solved by X, X resolves Y, X finds a solution to Y, X tries to solve Y, X deals with Y, Y is resolved by X, X addresses Y, X seeks a solution to Y, X does something about Y, X solution to Y, Y is resolved in X, Y is solved through X, X rectifies Y, X copes with Y, X overcomes Y, X eases Y, X tackles Y, X alleviates Y, X corrects Y, X is a solution to Y, X makes Y worse, X irons out Y This list of associated paths looks tantalizingly close to the kind of axioms that would prove useful in an inference system.

dIRT only outputs pairs of paths that have some semantic relation.


To conclude, several approaches exhaustively process different types of corpora, obtaining varying scales of output.

the Web is a huge promising resource, but current Web-based methods suffer serious scalability constraints.


The last approach is the most general with respect to the template form.

its processing time increases exponentially with the size of the templates.


On one hand, an anchor set should correspond to a sufficiently specific setting, so that entailment would hold between its different occurrences.

it should be sufficiently frequent to appear with different entailing templates.


The challenge of these tasks varies by the degree of parallel-ness of the input multilingual documents.

the non-parallel corpora used so far in the previous work tend to be quite comparable.


Since previous works were carried out on different corpora, in different language pairs, we cannot directly compare our method against them.

we implement a baseline method that follows the same "find-topic-extract-sentence" principle as in earlier work.


In the HMM model we incorporated auxiliary LMs by interpolation, which is not possible here since there is no LM per se, but rather N-gram features.

we can use the same trick as we used for prosodic features.


Note that the distance in words from the previous paragraph boundary (Dist w ) is a good indicator for a paragraph break in the English news domain.

this feature is less useful for the other two languages.


They removed an additional 2% of the training data due to issues involving the named entity tagger splitting corpus tokens into multiple words.

where these issues occurred in tagging the section 23 test sentences, they were manually corrected.


This random selection method should act to further reduce the correlation between trees and Breiman notes that it gets around the problem caused by categorical inputs with large numbers of values.

he leaves the number of values chosen unspecified.


The Random Forest approach outperforms the Bayesian method and the Decision Tree method.

it does not perform as well as the SVM classifier.


We mention the performance of their method where appropriate below.

our results are compared to human annotation of chunked data, while theirs (and other supervised results) are compared to manually annotated full sentences.


In terms of overall results, the MBL model outperforms the Maxent model by 3 to 4 points F-score.

all our results lie broadly in the range of existing systems with a similar architecture (i.e.


Some of the plausible variables which might explain the variance are the number of semantic roles per frame, the amount of training data, and the number of verbs per frame.

we suggest that a fourth variable might have a more decisive influence.


Kaufman and Rousseeuw (1990)), which estimates the "cost" of a cluster as the sum of squared distances between each vector and the cluster centroid Under this view, a good cluster is one with a low cost, and the goal of the clustering algorithm is to minimise the average distance to the centroid.

for our purposes it is more convenient for a good cluster to have a high rating.


As for the two statistical frameworks, uniformity is better correlated with the Maxent model than with the MBL model, even though MBL performs better on the evaluation.

this does not mean that the correlation will become weaker for semantic role labelling systems performing at higher levels of accuracy.


Since uniformity is defined in terms of a quality function, clustering would be the natural method to employ for this task.

this method is only viable for frames with a large amount of annotation.


We find that the parser performs well on the object extraction cases found in the Penn Treebank, given the difficulty of the task.

the parser performs poorly on questions from TREC, due to the small number of questions in the Penn Treebank.


This small study only provides anecdotal evidence for the reasons the parser is unable to recover some long-range object dependencies.

the analysis suggests that the parser fails largely for the same reasons it fails on other WSJ sentences: wrong attachment decisions are being made; the lexical coverage of the supertagger is lacking for some verbs; the model is sometimes biased towards incorrect lexical categories; and the supertagger is occasionally led astray by incorrect POS tags.


The model is very much like a Hidden Markov Model in which the summary is the observed sequence.

using a standard HMM would not allow us to account for phrases in the summary.


That is, when given an abstract containing "the man" and a document also containing "the man," a human may prefer to align "the" to "the" and "man" to "man."

a phrase-based model will almost always prefer to align the entire phrase "the man" to "the man."


StoryStation was designed by researchers in conjunction with two teachers and a group of students.

both students and teachers indicated StoryStation would be significantly improved if it were enhanced with an agent that could give feedback about the plot of a story.


Good: A good story shows that the pupil was listening to the story, and can recall the main Class Probability Number of Class 1 (Excellent) 0.175 18 2 (Good) 0.320 33 3 (Fair) 0.184 19 4 (Poor) 0.320 33 Table 1: Distribution of Story Ratings events and links in the plot.

the pupil shows no deeper understanding of the plot, which can often be detected by the pupil leaving out an important link or emphasizing the wrong details.


A general plot analysis agent would be more useful than our current system, which is successful by virtue of the story rewriting task being less complex than full story understanding.

our system fulfills an immediate need in the StoryStation application, in contrast to more traditional story-understanding and story-generation systems, which are usually used as testing grounds for theoretical ideas in artificial intelligence.


Indeed the total perplexity on the test corpus is 21.3 and the one obtained with the specific LMs is 17.8, so a gain of 16% compared to the 26.8% obtained previously.

this result is not surprising as the previous method was designed for specifically decreasing the perplexity measure.


As we have seen in the previous section, specific dialog situations (like those obtained with the hierarchical clustering) proved to be more efficient than the semantic channel (represented by the calltype labels) for clustering utterances in relation with the language used, at least from the perplexity point of view.

for clustering dialogs rather than utterances, the semantic channel is the main channel that we are going to use because in this case we want to characterize the whole interaction between a system and a user rather than just the language used.


Their methodology provides striking results within a limited domain characterized by a high frequency of stereotypical sentence types.

as we show below, the approach may be of limited generality, even within the training domain.


6 We gathered all phrase differs by less than 5%), we have presented the average of the directional AERs.

5 following SMT practice of augmenting data with a bilingual lexicon, we did append an identity lexicon to the training data.


The model is thus deficient in that it assigns a large portion of its probability mass to impossible cases: those instances which have words in the context which do not match those in the sentence.

because the sentences are always observed, we only consider instances in the set of consistent cases, so the deficiency should be irrelevant for the purpose of reasoning about sense and SCF.


the sense model again believes that the sense 2:42:04 is most likely.

the SCF model correctly gives high weight to the NP frame, which when combined with the joint distribution, gives much more probability to the sense 2:30:01.


The frequencies for attributes and values were again collected as in the first experiment.

these data were used in a different way.


1 The maximum-entropy (ME) principle, which prescribes choosing the model that maximizes the entropy out of all models that satisfy given feature constraints, can be seen as a built-in regularization mechanism that avoids overfitting the training data.

it is only a weak regularizer that cannot avoid overfitting in situations where the number of training examples is significantly smaller than the number of features.


(2003) assess the computational complexity for standard gradient-based optimization with the full feature set by ≈ cmp 2 τ , for a multiple c of p line minimizations for p derivatives over m data points, each of which has cost τ .

for grafting, the cost is assessed by adding up the costs for feature testing and optimization for s grafting steps as ≈ (msp+ 1 3 cms 3 )τ .


In any case, since this is engineering, the rationalization for a feature is far less important than the model's overall performance increase.

science would demand that, at some point, we analyze the multitude of features in a state-of-the-art lexicalized statistical parsing model.


From a history-based grammar perspective, there are 727,930 types of history contexts from which futures are generated.


This is of particular concern when a head word-which the top-down model generates at its highest point in the tree-influences an attachment decision.

inspecting the lowentropy word-generation histories of P M w revealed that almost all such cases are when the model is generating a preterminal, and are thus of little to no consequence vis-a-vis syntactic disambiguation.


In machine translation, intuitively, the informative content words should be emphasized more for better adequacy of the translation quality.

the standard statistical translation approach does not take account how informative and thereby, how important a word is, in its translation model.


Any statistical translation can be used in (1) to calculate the phrase translation probability.

in our experiment we typically see now significant difference in translation results when using lexicons trained from different alignment models.


Allowing mto-n matching of up to two nodes on either side of the parallel treebank allows for limited nonisomorphism between the trees.

even given this flexibility, requiring alignments to match two input trees rather than one often makes tree-totree alignment more constrained than tree-to-string alignment.


In some cases, differences in the number of level may be handled by the tree-to-tree model, for example by grouping the subject NP and its base NP child together as a single elementary tree.

this introduces unnecessary variability into the alignment process.


Previous work in CRFs assumed that observation sequence (word) boundaries were fixed.

word boundaries are not clear in Japanese, and hence a straightforward application of CRFs is not possible.


Statistical parsers tend to scale exponentially in sentence length, unless a narrow beam is employed, which leads to globally poorer parses.

the bracketer described in this paper scales linearly in [[Confidence]   the length of the sentence to find the globally optimal solution.


We use a Viterbi-like dynamic programming decoding algorithm, where transition probabilities are governed by the discriminative tagging model.

the tags generated by our decoder are not the same as those predicted by the maximum entropy model.


To solve these problems in the tagging model would be nearly impossible, without giving up on efficiency.

our decoder is able to produce n-best lists using exact A * search that very frequently contain globally superior taggings, even though the simple tagging model cannot recognize them as such.


The first thing worth noticing in this table is that in general, when one system achieves higher precision, the other system achieves higher recall, which is not surprising.

in the last row, corresponding to proper nouns, the RR2 system outperforms the COL03 (this is the "Full" implementation) in both precision and recall, suggesting that our system is better able to capture the phrasing of proper nouns.


That is, this is the performance attainable given a chunker that identifies base NPs perfectly (at 100% precision).

since this hypothetical system only chunks base NPs, it misses all non-base NPs and thus achieves a recall of only 73.0, yielding an overall F-score below our system's performance.


In the original formulation of BWI, boundaries are identified without reference to the location of the opposing boundary.

we might expect that the end of a name, say, would be easier to identify if we know where it begins.


As a specific NLP task, we will consider partof-speech (POS) tagging.

the problem addressed comes up in any NLP task which is tackled by the statistical approach and which makes use of a Bayes decision rule.


Chinese part-of-speech (POS) tagging assigns one POS tag to each word in a Chinese sentence.

since words are not demarcated in a Chinese sentence, Chinese POS tagging requires word segmentation as a prerequisite.


When a paired t-test was carried out at the level of significance 0.01, the all-at-once approach was found to be significantly better than the one-at-a-time approach for POS tagging accuracy, although the difference was insignificant for word segmentation.

experim ent Num ber POS Accuracy(%) Figure 6: CTB 10-fold CV POS tagging accuracy using an all-at-once approach the time required for training and testing is increased significantly for the all-atonce approach.


It can be shown that, if we smooth the A model with a Gaussian prior on the feature weights that is centered at 0 -following the approach in (Chen and Rosenfeld, 2000) for smoothing maximum entropy models -then the MinDiv update equations for estimating A on the adaptation data are identical to the MAP adaptation procedure we proposed 5 .

we wish to point out that the equivalence holds only if the feature set for the new model A is F background ∪ F adapt .


Thus, a spell checker built according to this formulation could suggest the correction detroittigers because this alternative occurs frequently enough in the employed query log.

detroittigers itself could be corrected to detroit tigers if presented as a stand-alone query to this spell checker, based on similar query-log frequency facts, which naturally leads to the idea of an iterative correction approach.


Most of features including " " are assigned a negative weight (negative opinion).

only one feature " (hard to cut off)" has a positive weight.


There has been some previous use of machine learning to classify email messages (Cohen 1996;Sahami et al., 1998;Rennie, 2000;Segal & Kephart, 2000).

to our knowledge, none of these systems has investigated learning methods for assigning email speech acts.


Like a proposal, the message involves both a commitment and a request.

while a proposal is associated with a new task, an amendment is a suggested modification of an already-proposed task.


In Experiment 1, we replicated the entropy rate effect reported by Charniak (2002, 2003) and showed that it generalizes to a larger range of sentence positions and also holds for individual sentences, not just averaged over all sentences with the same position.

we also found that a simple baseline model based on sentence length achieves a correlation with sentence position.


Significant improvements in both perplexity (PPL) and word error rate (WER) over backoff smoothing were reported after interpolating the neural network models with the baseline backoff models.

the neural network models rely on interpolation with ¢ -gram models, and use ¢ -gram models exclusively for low frequency words.


Most of the studies focused on improving ¢ -gram language models by adopting various smoothing techniques in growing and using DTs (Bahl et al., 1989;Potamianos and Jelinek, 1998).


Standard -test 3 shows that the improvements are significant at ¡ 0.001 and ¡ 0.05 level respectively.

we notice that the improvement in WER using the trigram with 40M words is not as much as the trigram with 20M words.


While segmentation algorithms, such as TEXTTILING (Hearst, 1994) and its recent successors using the inter-paragraph similarity matrix (Choi, 2000), all themselves use nonstructural cosine similarity as a measure of semantic proximity between paragraphs.

the distance function so far has been largely defined and used ad hoc, usually by a tf.idf weighting scheme (Salton and Yang, 1973) and a simple cosine similarity, equivalently, an Euclidean dot product.


As a method of tackling clusters of texts, the text classification task has recently made great advances with a Naïve Bayes or SVM classifiers (for example, (Joachims, 1998)).

they all aim at classifying texts into a few predefined clusters, and cannot deal with a document that fits neither of the clusters.


We can see from both results that metric distance produces a better retrieval over the tf.idf and dot product.

refinements in precision are certain (average p = 0.0243) but subtle.


They report that the effectiveness of A increases as the number of the training pairs S increases; this requires O(n 2 ) sample points from n training data, and must be optimized by a computationally expensive Newton-Raphson iteration.

our method uses only linear algebra, and can induce an ideal metric using all the training data at the same time.


Regular roots such as p.s.q yield forms such as hpsqh.

the irregular roots n.p.l, i.c.g, q.w.m and g.n.n in this pattern yield the seemingly similar forms hplh, hcgh, hqmh and hgnh, respectively.


There are 8 different human judges for DUC 2004 Task 2, and 4 for DUC 2004 Task 4.

a subset of exactly 4 different human judges produced model summaries for any given cluster.


To ease the training of the maximum entropy model, bootstrapping is used to help supervised learning.

to avoid error propagations from word segmentation and NER, we directly extract Chinese NEs and make the alignment from plain text without word segmentation.


So far a lot of research has been conducted in the field of machine translation and knowledge acquisition, including both statistical approaches (Cherry and Lin, 2003;Probst and Brown, 2002;Wang et al., 2002;Och and Ney, 2000;Melamed, 2000;Vogel et al., 1996) and symbolic approaches (Huang and Choi, 2000;Ker and Chang, 1997).

these approaches do not work well on the task of NE alignment.


They only carry out the alignment between words and do not consider the case of complex phrases like some multi-word NEs.

iBM Models allow at most one word in the source language to correspond to a word in the target language (Koehn et al., 2003;Marcu, 2001).


YASMET requires supervised learning for the training of the maximum entropy model.

it is not easy to acquire a large annotated training set.


Surely, a good translation has to adequately capture the meaning of the foreign original.

pinning down all the nuances is hard, and often differences in emphasis are introduced based on the interpretation of the translator.


The obvious next step would be to learn from the entire neighborhood -similar to KNN classification.

due to the sparsity of the data and because different groups of neighbors capture different aspects of the test question, we choose to cluster the neighborhood instead.


These particular content features are intuitive and highly indicative of a correct answer.

in sparse clusters, the content features have less information content and are more vague.


In the context of Web surfing, it is unusual for a page to include multiple or partial links to another page, and hence the original PageRank definition for graph-based ranking is assuming unweighted graphs.

in our model the graphs are build from natural language texts, and may include multiple or partial links between the units (vertices) that are extracted from text.


The summary evaluation performed in SUMMAC (Mani et al., 1999) followed that strategy.

extrinsic evaluations are time-consuming to set up and can thus not be used for the day-to-day evaluation needed during system development.


One of us then used the Kuwait consensus agreement to annotate the 16 machine summaries for that text which were created by different participants in DUC-2002, an annotation which could be done rather quickly.

a small number of missing factoids were detected, for instance the (incorrect) factoid that Saudi Arabia was invaded, that the invasion happened on a Monday night, and that Kuwait City is Kuwait's only sizable town.


Of course, the suggestion might be made that the system ranking will most likely also be stabilised by scoring summaries for more texts, even with such a low (or even lower) N per text.

in that case, the measure only yields information at the macro level: it merely gives an ordering between systems.


The definition of is as follows, where  In the preliminary experiment, a weight-pushing operation (Mohri and Riley, 2001) was also effective for deleting negative v transition of our fullexpansion models.

pushing causes an imbalance of weights among paths if the WFST is not deterministic.


In this paper, we applied the state merging method to a fully-expanded WFST and showed the effectiveness of this approach.

the state merging method itself is general and independent of the fully-expanded WFST.


In other words, the kth sentence chosen is the one with the largest index value in the kth right singular vector in matrix The summarization method proposed by Gong and Liu has some disadvantages as well, the main of which is that it is necessary to use the same number of dimensions as is the number of sentences we want to choose for a summary.

the higher the number of dimensions of reduced space is, the less significant topic we take into a summary.


Figure 2 illustrates the tectogramatical tree structure of the following sentence:  In the PDT the intonation center is not annotated.

the annotators were instructed to use their judgement where the IC would be if they uttered the sentence.


They are shown in the figure 3, which shows that, actually, after using only 1% of the training data (4,947 instances), the classifiers already perform very well, and adding more training data improves the results only slightly.

for RIPPER, adding more data causes a decrease in performance, and as we mentioned earlier, even an impossibility of building a classifier.


It is worth noting that all configurations of this algorithm are computationally intensive, mainly because of Step 2.

since our aim is to provide transcripts for browsing audio recordings, we do not have to correct errors in real time.


In the other methods, it is used inside the co-reference module of the IE pipeline, to find the (single) locally-best state.

other textual features of the state candidate should contribute to establishing the relations to a location mention, besides the raw distance.


The simplest approach is to treat each alignmentsystem output as a separate feature upon which we build a classifier.

when only a few alignment systems are combined, this feature space is not sufficient to distinguish between instances.


In particular, it is a slightly better pair according to the Dice value than the correct the-les.

the latter alignment has the advantage that major-grands follows it.


6 Of course, symmetrizing Model 4 by intersecting alignments from both directions does yield an improved AER of 6.9, so, while our model does do surprisingly well with cheaply obtained count-based features, Model 4 does still outperform it so far.

our model can 4 It is important to note that while our matching algorithm has no first-order effects, the features can encode such effects in this way, or in better ways -e.g.


In this method, we merely run our matching algorithm and update weights based on the difference between the predicted and target matchings.

the performance of the average perceptron learner on the same feature set is much lower, only 8.1, not even breaking the AER of its best single feature (the intersected Model 4 predictions).


For example, we estimate the conditional probability of linking not to ne...pas by considering the number of sentence pairs in which not occurs in the English sentence and both ne and pas occur in the French sentence, compared to the number of times not is linked to both ne and pas in pairs of corresponding sentences.

when we make this estimate in the CLP-based model, we do not count a link between not and ne...pas if the same instance of not, ne, or pas is linked to any other words.


As we will discuss in Section 4, the majority of our coreference-specific features are over pairs of chunks: the proposed new mention and an antecedent.

since in general a proposed mention can have well more than one antecedent, we are left with a decision about how to combine this information.


Then sentence set recall is R = M/A and precision is P = M/S.

set-based recall and precision do not average well, especially when the assessor set sizes A vary widely across topics.


There are similar patterns to those seen in the relevant sentence data, with the 2003 assessors clearly being more liberal in judging.

the pattern is reversed for topic types, with more sentences being considered relevant and novel for the opinion topics than for the event topics.


The majority of the documents in the data set are relevant, and so many of the topic terms are present throughout the documents.

the assessor was often looking for a finer-grained level of information than what exists at the document level.


We observe that the scores for a small fraction of new stories that were initially missed (between scores 0.8 and 1) are decreased by the model-based NED system while a larger fraction (between scores  0.1 and 0.4) is also decreased by a small amount.

the major impact of using SVM modelbased NED systems appears to be in detecting old stories.


We convert a tree to feature vector Then we use a procedure that chooses the DP procedure or the inner product procedure depending on maliciousness: This procedure returns the same value as the original calculation.

naively, if |V (T i )| (the number of feature subtrees such that # F i (T i ) = 0) is small enough, we can expect a speed-up because the cost of calculating the inner product is O(|V since |V (T i )| might increase as the training set becomes larger, we need a way to scale the speed-up to large data.


However, the speed-up might be smaller since without node marks the number of subtrees increases while the DP procedure becomes simpler.

the FREQTM conversion for marked labeled ordered trees might be made faster by exploiting the mark information for pruning.


The results showed that the LCS-based measure is comparable to Ngram-based automatic evaluation methods.

these methods tend to be strongly influenced by word order.


ROUGE-L is better than both ROUGE-3 and ROUGE-4 but worse than ROUGE-1 or ROUGE-2.

& § © x ª 's correlation coefficients ( ) do not change very much with respect to l .


We limited our first study reported here to linear classifiers, in which conversion can be performed by simple ordering according to the score of each document.

approaching the problem as "learning how to order things" allowed us to design our sampling and training mechanisms in a novel and, we believe, more powerful way.


Similarly to the global weighting, we assigned each occurrence of a term to its local weighting bin l, but this time by simply capping tf at the total number of local weighting bins |L|: ) Let's note that this particular representation does not really need rounding since tf is already a positive integer.

in a more general case, tf can be normalized by document length (as is done in BM25 and language models) and thus local weighting would become a continuous function.


The WFST composition gives the word-to-word alignments between the sentences.

to obtain the phrase alignments, we need to construct additional FSTs not described here.


Our goal is to estimate the reordering model parameters P (b|x, u) for each phrase-pair (x, u) in this inventory.

when translating a given test set, only a subset of the phrase-pairs is needed.


The WtoP alignment model includes the first two of these.

distortion, which allows hypothesized words to be distributed throughout the target sentence, is difficult to incorporate into a model that supports efficient DP-based search.


Notably, simply increasing the amount of bitext used in training need not improve AER.

larger aligned bitexts can give improved phrase pair coverage of the test set.


A block δ [] invokes both the inner and outer generations simultaneously in Bracket Model A (BM-A).

the generative process is usually more effective in the inner part as δ [] is generally small and accurate.


On the one hand, the ability to provide fully correct phrasal alignments is impaired by the occurrence of high-frequency function words and/or words that are not exact translations of the words in the other language.

we have observed that most alignment systems are capable of providing partially correct phrasal alignments.


So, they potentially lead to better solutions.

the error rate of a finite set of training samples is usually a step function of model parameters, and cannot be easily minimized.


Second, the number of candidate features is not too large, and these features are not highly correlated.

neither of the assumptions holds in our case.


When the two domains are very similar to the background domain (such as Yomiuri), discriminative methods outperform MAP by a large margin.

the margin is smaller when the two domains are substantially different (such as Encarta and Shincho).


In the absence of robust mechanisms for assessing the reliability of the decoded inputs, the system will take the misunderstanding as fact and will act based on invalid information.

in a nonunderstanding the system fails to obtain an interpretation of the input altogether.


The task therefore has to be fixed, known in advance: for instance the slots that the system queries the user about (in a slot-filling system) are fixed.

in the RavenClaw architecture, the dialog task tree (e.g.


In the past, the idea of using perceptual categories has been dismissed as impractical due to the high cost of hand annotation.

with advances in weakly supervised learning, it is possible to train prosodic event classifiers with only a small amount of hand-labeled data by leveraging information in syntactic parses of unlabeled data.


Our strategy is similar to that proposed in (Nöth et al., 2000), which uses categorical labels defined in terms of syntactic structure and pause duration.

their system's category definitions are without reference to human perception, while we leverage learned relations between perceptual events and syntax with other acoustic cues, without predetermining the relation or requiring a direct coupling to syntax.


It is an open question as to whether the conclusions will hold for errorful ASR transcripts and automatically detected SU boundaries.

there is reason to believe that relative gains from using prosody may be larger than those observed here for reference transcripts (though overall performance will degrade), based on prior work combining prosody and lexical cues to detect other language structures (Shriberg and Stolcke, 2004).


In principle it could be any probability distribution.

largely for the sake of technical convenience, we assume it is one component of a multinomial distribution known as the Dirichlet distribution.


A two-step approach also avoids the creation of illegal chunk sequences, such as "B-SAT I-NUC".

a potential drawback is that the number of training examples for the labeller is reduced as the instances to be classified are chunks rather than tokens.


The two BoosTexter models also perform significantly worse than Spade on segmentation.

the higher WDiff for Spade on the segmentation task suggests that the boundaries predicted by our models contain more "near misses" than those predicted by Spade.


As with the one-step method, the stacked model performs (insignificantly) better than its unstacked counterpart on the segmentation task.

on the labelling task, the stacked variant performs significantly worse.


Discriminative training methods strive to optimize the parameters of a model by minimizing SR, as in Equation 4.

(4) cannot be optimized directly by regular gradient-based procedures as it is a piecewise constant function of λ and its gradient is undefined.


First, all discriminative methods significantly outperform the linear interpolation (statistically significant according to the t-test at p < 0.01).

the differences among three discriminative methods are very subtle and most of them are not statistically significant.


The algorithm cannot be guaranteed to terminate (since it is possible to write arbitrary Turing machines in Dyna).

if it does terminate, it should return values from a valid model of the program, i.e., values that simultaneously satisfy all the equations expressed by the program.


This strategy for computing the gradient ∂goal/∂a via the chain rule is an example of automatic differentiation in the reverse mode (Griewank and Corliss, 1991), known in the neural network community as back-propagation.

what if goal might be computed only approximately, by early stopping before convergence ( §4.5)?


MDPs provide us with a principled way to deal with these elements and their relationships.

dealing with the most general case results in models that are very cumbersome and which hide the conceptual simplicity of our approach.


Without noise, the performance (dotted triangles) is only slightly worse than that of the original policy.

when noise objects are added (solid triangles) the training is no longer slowed down.


Question series were also the fundamental structure used in the QACIAD challenge (Question Answering Challenge for Information Access Dialogue) of NTCIR4.

there are some important differences between the QACIAD and TREC series.


Linguistic analy-sis is useful because full parsing captures long distance dependencies between the answers and the query terms, and provides more information for inference.

merely linguistic analysis may not be enough.


The TREC question answering evaluation is based on human judgments (Voorhees, 2004).

such a manual procedure is costly and time consuming.


The advantage is that it does not require human annotation.

it only works for certain types of questions that have fixed anchors, such as "where was X born".


(2004) uses a similar approach for unsupervised pattern learning and generalization to soft pattern matching.

the method is actually used for sentence selection rather than answer snippet selection.


Perfect knowledge of informer spans can enhance accuracy from 79.4% to 88% using linear SVMs on standard benchmarks.

standard heuristics based on shallow pattern-matching give only a 3% improvement, showing that the notion of an informer is non-trivial.


We also note that the post-2003 TREC task has encountered evaluation problems, because it is difficult to agree on which nuggets should be included in the multi-snippet definitions (Hildebrandt et al., 2004).

our experimental results of Section 4 indicate strong inter-assessor agreement for single-snippet answers, suggesting that it is easier to agree upon what constitutes an acceptable single-snippet definition.


they found the additional attribute to lead to no significant improvements, and, hence, we do not use it.


Considering additional features tailored to the NFL domain could further enhance performance.

feature selection is not one of the main objectives of this work.


(2004), who use machine learning techniques to identify propositional opinions and their holders (sources).

their work is more limited in scope than ours in several ways.


Due to its complex nature, it is not uncommon that the mention detection task itself is also divided into a number of smaller sub-tasks.

in this paper, we adopt an integrated classification approach to this problem that yields a monolithic structure.


One can argue that the large number of classes and data sparsity is an important issue here that it might have significant effect on performance.

several attempts to divide the task into simpler subtasks have failed to yield a system with a better performance than that of the integrated system.


Strapping may be harder in cases like gender induction: it is hard to stumble into the kind of detailed seed used by Cucerzan and Yarowsky (2003).

we suspect that fertile seeds exist that are much smaller than their lists of 50-60 words.


13 Our basic idea was to conflate x and y into a pseudoword x-y.

to get a pseudoword with only two senses, we tried to focus on the particular senses of x and y that were selected by t. We constructed about 500 pseudoword tokens by using only x and y tokens that appeared in sentences that contained t, or in sentences resembling those under a TF-IDF measure.


There are several metrics that can be used for this purpose, see for instance (Budanitsky and Hirst, 2001) for an overview.

most of them rely on measures of semantic distance computed on semantic networks, and thus they are limited by the availability of explicitly encoded semantic relations (e.g.


The algorithm was later improved through a method for simulated annealing (Cowie et al., 1992), which solved the combinatorial explosion of word senses, while still finding an optimal solution.

recent comparative evaluations of different variants of the Lesk algorithm have shown that the performance of the original algorithm is significantly exceeded by an algorithm variation that relies on the overlap between word senses and current context (Vasilescu et al., 2004).


some of these words were fairly obscure, did not occur frequently enough in one of the domain corpora or were simply too polysemous.


A naive choice of source transliteration unit is a single character.

single characters lack contextual information, and their combinations may generate too many unlikely candidates.


The CRF is discriminative and avoids label/observation bias by using a model that is constrained only in that the conditional distribution factorizes over an undirected Markov chain.

most popular training procedures for a CRF are time-consuming and complex processes.


A naive implementation of this algorithm requires O(n 2 ) invocations of local classifiers, where n is the number of the words in the sentence, because we need to update the probabilities over the words at each iteration.

a k-th order Markov assumption obviously allows us to skip most of the probability updates, resulting in O(kn) invocations of local classifiers.


The first search strategy resulted in a relatively low inclusion rate; the second achieved a much higher inclusion rate.

because such English pages were limited, and on average only 45 unique snippets could be found for each f, which resulted in a maximum inclusion rate of 85.8%.


Therefore, even if some of these words are missing, numerical Information Extraction methods can use the remaining salient words and discard the noise generated by ASR errors.

this phenomenon is not true for tasks related to the extraction of fine grained entities, like Named Entities.


These improvements are not significant enough to justify the use of this kind of metadata information for improving the general performance of both ASR and NER processes.

if we focus now on the entities occurring in the newsletters corresponding to the exact days of the unmatched corpus, the improvement is much more significant, as presented in the next section.


We could restrict the features to local scope on the candidate parses, allowing dynamic-programming to be used to train the model with a packed representation.

even with these restrictions, finding arg max t h p(t, h | s, Θ) is NP-hard, and the Viterbi approximation arg max t,h p(t, h | s, Θ) -or other approximations -would have to be used (see Matsuzaki et al.


The maximal accuracy was 58.59% which was obtained after 7 iterations.

to the experiments on data2, the accuracy decreased by more than 1.5% when the training was continued.


Andreas Eisele (unpublished work) implemented a statistical disambiguator for German based on weighted finite-state transducers as described in the introduction.

his system fails to represent and disambiguate the ambiguities observed in compounds with three or more elements and similar constructions with structural ambiguities.


These systems have shown that accurate projective dependency parsers can be automatically learned from parsed data.

non-projective analyses have recently attracted some interest, not only for languages with freer word order but also for English.


The present work is related to that of Hirakawa (2001) who, like us, reduces the problem of dependency parsing to spanning tree search.

his parsing method uses a branch and bound algorithm that is exponential in the worst case, even though it appears to perform reasonably in limited experiments.


We illustrate here the application of the Chu-Liu-Edmonds algorithm to dependency parsing on the simple example x = John saw Mary, with directed graph representation G x , The first step of the algorithm is to find, for each word, the highest scoring incoming edge If the result were a tree, it would have to be the maximum spanning tree.

in this case we have a cycle, so we will contract it into a single node and recalculate edge weights according to Figure 3.


In that algorithm, the single highest scoring tree (or structure) is used to update the weight vector.

mIRA aggressively updates w to maximize the margin between the correct tree and the highest scoring tree, which has been shown to lead to increased accuracy.


The results also show that in terms of Accuracy, factored MIRA performs better than single-best MIRA.

for the factored model, we do have O(n 2 ) margin constraints, which results in a significant increase in training time over single-best MIRA.


Unlike the Reuters titles or the proverbs, the BNC sentences have typically no added creativity.

we decided to add this set of negative examples to our experimental setting, in order to observe the level of difficulty of a humorrecognition task when performed with respect to simple text.


The results of the recent Senseval-3 competition (Mihalcea et al., 2004) have shown that supervised WSD methods can yield up to 72.9% accuracy 1 on words for which manually sense-tagged data are available.

supervised methods suffer from the so-called knowledge acquisition bottleneck: they need large quantities of high quality annotated data to produce reliable results.


However, the first translation means "an exit" or "to export".

the second translation ( ) is monosemous and should be used.


In our experiments, we used all sense examples that we built for a sense (with an upper bound of 6000).

the distribution of senses in English text often does not match the distribution of their corresponding Chinese translations in Chinese text.


Second, since the engine is local, network latency is minimal.

to support IE, we must also execute the second stage of the algorithm (see the beginning of this section).


Thus, we see that a small number of repetitions can yield high confidence in an extraction.

when the sample size increases so that n = 20, 000, and the other parameters are unchanged, then P (x ∈ C) drops to 19.6%.


For the relations with a small number of correct instances, Country and CapitalOf, KNOWITNOW is able to identify 70-80% as many instances as KNOW-ITALL at precision 0.9.

corp and ceoOf have a huge number of correct instances and a long tail of low frequency extractions that KNOWITNOW has difficulty distinguishing from noise.


Traditionally, these sub-word units are determined by the phones or phonemes of a language (depending on desired detail of representation).

phonetic (or phonemic) representation has its pitfalls (cf.


Briscoe experimented with several triggers for starting a new word -at every phone, at the beginnings of syllables, at the beginnings of syllables with unreduced vowels, and at the beginnings of word boundaries.

the latter three require oracle information as to where word or syllable boundaries can occur.


It is possible that by focusing more on the training of annotator pairs, particularly on joint training, agreement might improve.

that would also result in a bias, which is probably not preferable to actual perception.


Moreover, sequencing shows potential for contributing in one case.

observations also point to three issues: first, the current data set appears to be too small.


Traditional question answering systems adopt the framework of parsing questions, searching for relevant documents, and then pinpointing and generating answers.

this framework includes potential dangers.


Inspired by this, we consider taking N-grams in sentences as our features.

n-gram features not closely related to the topic will bring more noise into the system.


Trigram features actually bring enough evidence in classification.

when we investigated 4-grams language features in the collected data, most of them are very sparse in the feature space of all the cases.


our approach in the use of metadata, Web-derived answer patterns and context bring incremental gains to RC performance.

the actual gain levels are much reduced.


For those cases, our association data provides a useful basis for detecting missing links in GermaNet, which can be used to enhance the taxonomy.

a large proportion of the "no relation" associations represent instances of verbverb relations not targeted by GermaNet.


There is hardly any overall difference in accuracy between the shallow and the deep classifier.

it seems that the shallow classifier in its current form has very little potential outside of the CD subset whereas the deep classifier shows a more promising performance for several subsets.


Using only the case tagger it is not possible to classify all the nouns in this set.

it is possible to capture some nouns in this set by applying a heuristic: © If a noun follows the preposition zum, zur, am, im, ins, or ans −→ Class I.


This assumption holds well on hand-corrected parse trees and simplifies significantly the SRL process because only one syntactic constituent has to be correctly classified in order to recognize one semantic argument.

this approach is limited when using automatically-generated syntactic trees.


Several studies have established that there is considerable variance in semantic role assignment performance across different semantic roles within systems (Carreras and Màrquez, 2004;Carreras and Màrquez, 2005;Pado and Boleda Torrent, 2004).

these studies used either the PropBank semantic role paradigm (Carreras and Màrquez) or a limited of experimental conditions (Pado and Boleda).


Since we used the gold standard features provided by FrameNet and did not introduce implementation-or feature-specific knowledge, this points to a general limitation of syntax-based models.

semantic features behave completely differently; their contribution is not limited by a role's confusability.


The results from these papers indicate that on corpus sizes up to 60,000 parallel sentences, the restructuring operations yielded a large improvement in translation quality, but the morphological decomposition provided only a slight additional benefit.

since German is not as morphologically complex as Czech, we might expect a larger benefit from morphological analysis in Czech.


Our final set of experiments used the same input format as the Modified Lemma experiments.

in this set of experiments, we changed the model used to calculate the word-to-word alignment probabilities.


The morpheme-based model in Equation 2 is similar to the modified lemma model in that it removes much of the differentiation between Czech wordforms, but leaves the differences that are most likely to appear as inflection on English words.

it also performs an additional smoothing function.


Fitting a model based onF α=0.75 , which gives increased weight to recall compared withF α=0.5 , led to higher recall as expected.

we also expected that the F α=0.75 score of theF α=0.75 trained classifier would be higher than the F α=0.75 score of theF α=0.5 trained classifier.


The classifiers discussed in this paper are logistic regression models.


Evita extracts the values of these two attributes using basic pattern-matching techniques over the approapriate verbal, nominal, or adjectival chunk.

the identification of event class cannot rely on linguistic cues such as the morphology of the expression.


This computation shows an average output for all participants of about 6,500 sentences and a median of 6,981 -out of a total of 8,343 sentences.

this total includes some amount of header material, not only the headline, but the document ID and other identifiers, the date and some shorthand messages from the wire services to its clients.


There may be cases where e 1 and e 2 belong to predicate-argument structures that have no argument in common.

because the dependency graph is always connected, we are guaranteed to find a shortest path between the two entities.


ROUGE-W is also based on LCS, but assigns higher weights to sequences that have fewer gaps.

these metrics still do not distinguish among translations with the same LCS but different number of shorter sized subsequences, also indicative of overlap.


The fact that current automatic MT evaluation metrics including BLANC do not correlate well with human judgments at the sentence level, does not mean we should ignore this need and focus only on system level evaluation.

further research is required to improve these metrics.


While in traditional word-based statistical models (Brown et al., 1993) the atomic unit that translation operates on is the word, phrase-based methods acknowledge the significant role played in language by multiword expressions, thus incorporating in a statistical framework the insight behind Example-Based Machine Translation (Somers, 1999).

phrase-based models proposed so far only deal with multi-word units that are sequences of contiguous words on both the source and the target side.


The word posterior probability is then calculated by summing over all target sentences containing word e in a position which is Levenshtein-aligned to i: , where The confidence of word e then depends on the source sentence f J 1 as well as the target sentence e I 1 , because the whole target sentence is relevant for the Levenshtein alignment.

to the approaches presented in section 4, the phrase-based confidence measures do not not use the context information at the sentence level, but only at the phrase level.


Such documents provide many training instances, where a word in one language is translated into another.

the data is only partially labeled in that we are not given a word-to-word alignment between the two languages, and thus we do not know what every word in the source language S translates to in the target language T .


, b k be the set of words that align to a in the aligned sentence t. In general, we can consider U a = {U a,s→t } s∈Da to be the candidate set of translations for a in T , where D a is the set of source language sentences containing a.

this definition is quite noisy: a word b i might have been aligned with a arbitrarily; or, b i might be a word that itself corresponds to a multiword translation in S. Thus, we also align the sentences in the T → S direction, and require that each b i in the phrase aligns either with a or with nothing.


In the community of parsing, labeled recall and labeled precision on phrase structures are often used for evaluation.

in our experiments we cannot evaluate our parser with respect to the phrase structures in PTB.


The fscore on the oracle in top 10 in the development data is 88.5%, while the f-score of the top candidate is 87.3%, as shown in Table 1.

we are not satisfied with the score on oracle, which is not good enough for post-processing, i.e.


The parser proposed in this paper is an incremental parser, so the accuracy on dependency is lower than that for chart parsers, for example like those reported in (Collins, 1999;Charniak, 2000).

5 it should be noted that the dependencies computed by our parser are deeper than those calculated by parsers working directly on PTB.


While Church's technique was developed with speech recognition in mind, we will show that it is useful for investigating psycholinguistic phenomenon.

the connection between cognitive phenomenon and engineering approaches go in both directions: it is possible that syntactic parsers could be improved using a model of syntactic priming, just as speech recognition has been improved using models of lexical priming.


This strongly indicates that the parallelism effect is an instance of a general processing mechanism, such as syntactic priming (Bock, 1986), rather than specific to coordination, as suggested by (Frazier and Clifton, 2001).

we also found that the parallelism effect is strongest in coordinate structures, which could explain why comprehension experiments so far failed to demonstrate the effect for other structural configurations (Frazier et al., 2000).


7 For most standard association measures utilized for terminology extraction, the frequency of occurrence of the term candidates either plays a major role (e.g., C-value), or has at least a significant impact on the assignment of the degree of termhood (e.g., t-test).

frequency of occurrence in a training corpus may be misleading regarding the decision whether or not a multi-word expression is a term.


We observe that backward constituent alignment-based models (1-4) perform similarly to word-based projection models (the F-score ranges between 0.40 and 0.45).

they obtain considerably higher precision (albeit lower recall) than the word-based models.


It is possible to correct more errors per token by iterating the correction process.

iterative correction cannot guarantee that the result is optimal under the model.


Our lexicon-free chunking algorithm placed an erroneous boundary at 11.3% of word split points for Arabic test data (Section 4.3).

correction performance was identical to that of error- free chunking.


Goshtasby and Ehrich (1988) present a lexicon-free method based on probabilistic relaxation labeling.

they use the probabilities assigned to individual characters by the OCR system, which is not always available.


Let X be a random variable with distribution P true (x), such that no direct observations of it exist.

we may have some indirect observations of X and have built several models of , each parameterized by some parameter vector θ i .


Intelligent language technologies capable of full semantic interpretation of domain-general text remain an elusive goal.

statistical advances have made it possible to address core pieces of the problem.


At least for the simplest model, these estimations do not vary with a larger corpus or one lacking in noise.

the increase in performance seen here for the more specific model, albeit small, may indicate that richer probability models may require more or cleaner training data.


A characteristic of genitives is that they are very productive, as the construction can be given various semantic interpretations.

in some situations, the number of interpretations can be reduced by employing world knowledge.


This includes the seminal paper of (Gildea and Jurafsky, 2002), Senseval 3 and coNLL competitions on automatic labeling of semantic roles detection of noun compound semantics (Lapata, 2000), (Rosario and Hearst, 2001) and many others.

not much work has been done to automatically interpret the genitive constructions.


The new boundary is more specific then the previous boundary and it is closer to the ideal boundary.

we do not know how well it behaves on unseen examples and we are looking for a boundary that classifies with a high accuracy the unseen examples.


But it can not group unseen features (features that do not occur in labeled data) into meaningful clusters since there are no class labels associated with these unseen features.

while given labeled data, unsupervised feature clustering method can not utilize class label information to guide feature clustering procedure.


Their system achieved 56.6% fine-grained score on ELS task of SENSEVAL-3.

with their work, our datadriven method for feature clustering based WSD does not require external knowledge resource.


2 Standard measures for evaluating information retrieval results are precision and recall.

for QA several other specialized measures have been proposed, e.g.


We do not know how the ranking is done internally or how the output is influenced by parameter changes.

we can inspect and evaluate the output of the system.


For example, all RootPOS keywords are marked as required and therefore, the restrictions of RootPOS keywords are useless because they do not alter the query.

in other cases overlapping keyword type definitions do influence the query.


Noteworthy are the rather low word-error rates (20%) in the TREC evaluations, and that recognition errors did not lead to catastrophic failures due to redundancy of news segments and queries.

in our scenario, requirements are rather different.


The simplest phonetic recognizer is a regular recognizer with the vocabulary replaced by the list of phonemes of the language, and the language model replaced by a phoneme M -gram.

such phonetic language model is much weaker than a word language model.


In both case, the set of words to index is known.

indexing phoneme lattices is very different, because theoretically any phoneme string could be an indexing item.


Note that several incorrectly spelled words, including "equpment" itself, are given as candidate corrections.

the language model derived from the query logs assigns a low probability to the incorrect candidates.


In the case of a correctly spelled query, the most likely candidate correction is the word itself.

occasionally there is a correctly spelled but infrequent word within a small edit distance of another more common word.


This method is simple and relatively effective if a search engine returns a hit-list which contains a certain number of relative documents in the upper part.

unless this assumption holds, it usually gives a worse ranking than the initial search.


The performance of query expansion or relevance feedback is usually evaluated on a residual collection where seen documents are removed.

we compare our method with pseudo feedback based ones, thus we do not use residual collection in the following experiments.


As for the number of training examples, performance of 20 and 50 does not differ so much in all the number of expansion terms.

performance of 100 is clearly worse than of 20 and 50.


In TREC-8, above two methods uses TREC1-5 disks for query expansion and a phase extraction technique.

we do not adopt these methods in our experiments 4 .


The comparisons can be characterized by a moving window, where successive overlapping comparisons are advanced by one unit of text.

hearst (1994hearst ( , 1997 and Foltz et al.


For example, when measuring comprehension,  use the unit of the sentence, as opposed to the more standard unit of the proposition, because LSA is most correlated with comprehension at that level.

when using LSA to segment text, Foltz et al.


Clearly this problem would be trivial for a cue phrase based approach, which could learn the finite set of problem introductions.

the current lexical approach does not have this luxury: words in the problem statements recur throughout the following dialogue.


Given the smaller number of test cases, 22, this F-measure of .22 is not significantly different from .17.

the Foltz method is significantly higher than both of these, p < .10.


Depending on the application, segments may be tokens, phrases, or sentences.

in this paper we primarily focus on segmenting sentences into tokens.


At first sight, it might appear that we have made the segmentation problem intractably harder by turning it into a classification problem with a number of labels exponential on the length of the instance.

we can bound the number of labels under consideration and take advantage of the structure of labels to find the k most likely labels efficiently.


Sequential tagging with the BIO tag set has proven quite accurate for shallow parsing and named entity extraction tasks (Kudo and Matsumoto, 2001;Sha and Pereira, 2003;Tjong Kim Sang and De Meulder, 2003).

this approach can only identify non-overlapping, contiguous segments.


The learning algorithm in Section 3.1 seeks a separator through the origin, though, our experimental results suggest that this tends to favor precision at the expense of recall.

at test time we can use a separation threshold different from zero.


Natural spoken language is often regarded as the obvious choice for a human-computer interface.

despite significant research efforts in automatic speech recognition (ASR) (Huang et al., 2001), existing ASR systems are still not sufficiently robust to a wide variety of speaking conditions, noise, accented speakers, etc.


Its computation is straightforward if the question classifies the document set in a completely disjointed manner.

the retrieved documents may belong to two or more categories for some questions, or may not belong to any category.


The system can generate question to which the user's query corresponds using this metadata.

some documents are related with multiple versions, or may not belong to any category.


All of the results so far have used the learned policy for the system interacting with the corresponding policy that was learned for the user.

there is no guarantee that a real user will behave like the learned policy.


The two main handtagged corpora are PropBank (Palmer et al., 2003) and FrameNet (Baker et al., 1998), the former of which currently has broader coverage.

even PropBank, which is based on the 1M word WSJ section of the Penn Treebank, is insufficient in quantity and genre to exhibit many things.


This work bears some similarity to the substantial literature on automatic subcategorization frame acquisition (see, e.g., Manning (1993), Briscoe and Carroll (1997), and Korhonen (2002)).

that research is focused on acquiring verbs' syntactic behavior, and we are focused on the acquisition of verbs' linking behavior.


Additionally, as noted earlier, Excite questions are often ungrammatical and make parsing less likely to succeed.

the baseline system, by definition, does not output semantic representations, so that its outcome is of little use for further reasoning, as required by question answering or general information extraction systems.


A trivial case is when both sentences are identical, word for word.

paraphrases often employ different words or syntactic structures to express the same concept.


Consider a filter that simply predicts that every sentence is incorrectly parsed: it would have an overall accuracy of 55% on our WSJ corpus, not too much worse than WOODWARD's classification accuracy of 66% on this data.

such a filter would be useless because it filters out every correctly parsed sentence.


It is generally accepted, however, that WordNet senses are far too fine-grained (Agirre and Lopez de Lacalle Lekuona (2003) and citations therein).

published thesauri, such as Roget's and Macquarie, group near-synonymous and semantically related words into a relatively small number of categories-typically between 800 and 1100-that roughly correspond to very coarse concepts or senses (Yarowsky, 1992).


A distributional measure of concept-distance can be used to populate a small 812 ¢ 812 concept-concept distance matrix where a cell m i j , pertaining to concepts c i and c j , contains the distance between the two concepts.

a word-word distance matrix for a conservative vocabulary of 100,000 word types will have a size 100,000 ¢ 100,000, and a WordNet-based concept-concept distance matrix will have a size 75,000 ¢ 75,000 just for nouns.


As correction ratio is determined by the product of a number of ratios, each evaluating the various stages of malapropism correction (identifying suspects, raising alarms, and applying the correction), we believe it is a better indicator of overall performance than correction performance, which is a not-so-elegant product of an F-score and accuracy.

no matter which of the two is chosen as the bottom-line performance statistic, the results show that the newly proposed distributional concept-distance measures are clearly superior to word-distance measures.


We expect that the WordNet-based measures will perform poorly when other parts of speech are involved, as those hierarchies of WordNet are not as extensively developed.

our DPC-based measures do not rely on any hierarchies (even if they exist in a thesaurus) but on sets of words that unambiguously represent each sense.


Both DPW-and WordNet-based measures have large space and time requirements for precomputing and storing all possible distance values for a language.

by using the categories of a thesaurus as very coarse concepts, precomputing and storing all possible distance values for our DPC-based measures requires a matrix of size only about 800 ¢800.


When using a phrase-based translation model, one can easily extract the phrase pair (THE MUTUAL; the mutual) and use it during the phrase-based model estimation phrase and in decoding.

within the xRS trans-ducer framework that we use, it is impossible to extract an equivalent syntactified phrase translation rule that subsumes the same phrase pair because valid xRS translation rules cannot be multiheaded.


Because the decoder only proposes phrase translations that are in the phrasetable (ie, that have non-zero count), it never requires estimates for pairss,t having c(s,t) = 0.

1 probability mass is reserved for the set of unseen translations, implying that probability mass is subtracted from the seen translations.


Smoothing relative frequencies with an additional Zens-Ney phrasetable gives about the same gain as Kneser-Ney smoothing on its own.

combining Kneser-Ney with Zens-Ney gives a clear gain over any other method (statistically significant for all language pairs except en→es and en→de) demonstrating that these approaches are complementary.


We hypothesize that this correlation captures the student involvement in the tutoring process: more involved students will try harder thus expressing more certainty or uncertainty.

less involved students will have fewer certain/uncertain/mixed turns and, in consequence, more neutral turns.


With a smaller training set the proportion of new entities is far too small to be of use.

as said, the overall final accuracy of 85.5% (see Table 7) does not significantly improve over the baseline.


We did not investigate the influence of treebank refinement in this study.

we would like to note that by a combination of suffix analysis and smoothing, Dubey (2005) was able to obtain an F-score of 85.2 for Negra.


Furthermore, end-to-end systems like speech recognizers (Roark et al., 2004) and automatic translators (Och, 2003) use increasingly sophisticated discriminative models, which generalize well to new data that is drawn from the same distribution as the training data.

in many situations we may have a source domain with plentiful labeled training data, but we need to process material from a target domain with a different distribution from the source domain and no labeled data.


It provides a discriminative online learning algorithm which when combined with a rich feature set reaches state-of-the-art performance across multiple languages.

within this framework one can only define features over single attachment decisions.


(2005b) use the Chu-Liu-Edmonds (CLE) algorithm to solve the maximum spanning tree problem.

global constraints cannot be incorporated into the CLE algorithm (McDonald et al., 2005b).


They are based on the concept of eliminating subtours (cycles), cuts (disconnections) or requiring intervertex flows (paths).

in practice these formulations cause long solve times -as the first two meth- ods yield an exponential number of constraints.


For the plain MST problem it is sufficient to set k = 1 and only take the best scoring label for each token pair.

if we want a constraint which forbids duplicate subjects we need to provide additional labels to fall back on.


The best performing system ; note: this system is different to our baseline) achieves 79.2% labelled accuracy while our baseline system achieves 78.6% and our constrained version 79.8%.

a significant difference is only observed between our baseline and our constraintbased system.


We see that solve time can be reduced by 80% while only losing a marginal amount of accuracy when we set q to 10.

we are unable to reach the 20 seconds solve time of the CLE algorithm.


Such aspectual distinctions have been alive and well in the linguistic literature since at least the late 60s (Vendler, 1967).

the use of the term event in natural language processing work has often diverged quite considerably from this linguistic notion.


Because our approximation of Evita uses a feature-based statistical machine learning algorithm instead of the rule-based Evita algorithm, it cannot predict how well Evita would perform if it had not used the same data for training and testing.

it can give us an approximation of how well a model can perform using information similar to that of Evita.


The implemented parsers of models 1 and 2 were around four times faster than the previous model without a loss of accuracy.

what surprised us is not the speed of the models, but the fact that they were as accurate as the previous model, though they do not use any phrasestructure-based probabilities.


Segment Ordering We introduce a new method for learning temporal ordering.

to existing methods that focus on pairwise ordering, we explore strategies for global temporal inference.


An abdominal examination revealed a soft systolic bruit... and a neurologic examination was normal... order between the two events is consistent with our interpretation of the text, therefore we cannot determine the precedence relation between the segments S5 and S7.

to many existing temporal representations (Allen, 1984;Pustejovsky et al., 2003), TDAG is a coarse annotation scheme: it does not capture interval overlap and distinguishes only a subset of commonly used ordering relations.


For instance, the presence of the temporal anchor last year indicates the lack of temporal continuity between the current and the previous sentence.

many of these predictors are heavily context-dependent and, thus, cannot be considered independently.