We then evaluate on the data set the completeness of the generic segmenter.

we will show that we can effectively adapt the generic segmenter to the four different bakeoff data sets, each of which simulates an application subset.


However, according to the above definition, the relative frequency of CASs can be much higher because most single characters in Chinese can be words by themselves, and as a result, almost all two-character words can be CASs.


As described in Section 4, with a context model, NWI can be performed simultaneously with other word segmentation tasks (e.g., word breaking, NER, and morphological analysis) in a unified manner.

it is difficult to develop a training corpus where new words are annotated because "we usually do not know what we don't know."


As mentioned earlier, to achieve a fair comparison, we compare the previously mentioned four systems only in terms of NER precision and recall and the number of OAS errors.

we find that due to the different annotation specifications used by these systems, it is still very difficult to compare their results automatically.


(In fact, the coefficient for sim R , which depends on only one of the three quantities, log p(lso(c 1 , c 2 )), improves only in the third digit.)

with the present paucity of evidence, this connection remains hypothetical.


(2004) found that the Jiang-Conrath and Lesk measures gave the best accuracy in their task of finding predominant word senses, with the results of the two being "comparable" but Jiang-Conrath being far more efficient.

corley and Mihalcea (2005) found little difference between the measures when using them in an algorithm for computing text similarity.


Last, Weeds experimented with distributional measures in real-word spelling correction, much as we have defined it in Hirst and Budanitsky (2005) and in Section 5.1 above, but replacing the semantic relatedness measures with distributional similarity measures.

she varied the experimental procedure in a number of ways, with the consequence that her results are not directly comparable to ours: her test data was the British National Corpus; scope was measured in words, not paragraphs; and relatedness thresholds were replaced by considering the k words most similar to the target word (and k was a parameter).


25 It's very unclear how ad hoc semantic relationships could be quantified in any meaningful way, let alone compared with prior quantifications of the classical and non-classical relationships.

ad hoc relationships accounted for only a small fraction of those reported by Morris's subjects (Morris 2006).


They attempt to describe Hebrew verb inflection as a concatenation of prefix+base+suffix, implementable by the Two-Level model.

they conclude that "The Two-Level rules are not the natural way to describe .


Since FSRAs are inherently nondeterministic (see the discussion of linearization below), their minimization is related to the problem of non-deterministic finite-state automata (NFA) minimization, which is known to be NP-hard.

6 while FSRA arc minimization is NP-hard, FSRA state minimization is different.


In these cases the simple detection of the patterns leads to the discovery of part-whole relations.

there are many ambiguous expressions that are explicit but convey part-whole relations only in some contexts.


Berland and Charniak (1999) also used Hearst's algorithm to find part-whole patterns.

they focused only on the first five patterns that occur frequently in their corpus.


For example, both the relationships apartment#1, woman#1, No and hand#1, woman#1, Yes are mapped into the more general type entity#1, entity#1, Yes/No .

the first example is negative (a POS-SESSION relation), while the second one is a positive example.


Moreover, their system was tested only on a working list of predefined highly probable wholes for their corpus based on the genitive syntactic patterns.

the ISS system can disambiguate any pair of concepts, provided they are in WordNet or can be classified by NERD.


By generalizing the method to all the parts and wholes from our testing corpus, the accuracy of the system will fall.

to be able to test the system on their six whole concepts we would need thousands of positive and negative examples for each such word.


Coleman targets the book at students in a variety of disciplines, including arts, humanities, linguistics, psychology, and speech science, as well as early science and engineering students who want a glimpse into natural language and speech processing.

since it assumes prior knowledge of basic linguistics, the text is likely to be less accessible to traditional science and engineering students.


This chapter contains no exercises for the student.

it does provide a list of materials to assist the student in learning more about C programming, digital signal processing, the Klatt synthesizer, speech recognition, Prolog, computational linguistics, and probabilistic grammars.


If only the schoolmen had measured instead of classifying, how much they might have learnt!

(page 41 Grassmann was recognized for his foundational work on analytical geometry; for example, Hermann Weyl (1949)  Today probably the best approach to analytic geometry is by means of the vector concept, following the procedure of Grassmann's Ausdehnungslehre.


On the one hand, verb classes reduce redundancy in verb descriptions since they encode the common properties of verbs.

verb classes can predict and refine properties of a verb that received insufficient empirical evidence, with reference to verbs in the same class: Under this criterion, a verb classification is especially useful for the pervasive problem of data sparseness in NLP, where little or no knowledge is provided for rare events.


We believe that our approach can be applied to a variety of situations in which vagueness affects referring expressions including, for example, r color terms (Section 9.3); r nouns that allow different degrees of strictness (Section 9.5); r degrees of salience (Section 9.4); and r imprecise pointing (Section 9.5).

we have also met some considerable obstacles on our way: Expressive choice (Sections 4 and 7).


The low frequency or absence of a bigram in the BNC may be due to chance.

the World Wide Web is big enough that a negative result is more reliable.


The rules achieved a coverage of 75% on the test set with an accuracy of 55%.

16 a baseline algorithm of taking the first word in each string as the peripheral concept covers 100% of the strings, but with only 16% accuracy.


The formulas for Sim att and Sim sty involve relatively straightforward matching.

sim den requires the matching of complex interlingual structures.


In our work, the form that the peripheral nuances can take is not restricted, because the list of peripheral nuances is open-ended.

it may be possible to keep the form unrestricted but add restrictions for the most important types of peripheral nuances.


If, in spite of Hone and Graham's remarks, one wants to evaluate user satisfaction with the survey given in Table 1, the question that arises is whether the target to be predicted should really be the sum of all the user-satisfaction scores.

our experiments (Section 2.3) showed a remarkable, but expected, difference in the significance of the predictors when taking different satisfaction-measure sums or even individual scores as the target to be predicted.


To be able to see the difference between these two equations, note that Comp was significant for US (p < 0.02), but removed by backward elimination.

the data from the second WOZ experiment gave the following performance equations: with 26% and 46% of the variance explained, respectively.


All the predictors from the first performance equation (i.e., NND, NRD, Comp) were insignificant (p > 0.1) for DM in the second experiment.

the only predictor from the second performance equation that was significant for DM (p < 0.004) in the first experiment, but removed by backward elimination, was GDR.


On the one hand, this could indicate that the selected individual user-satisfaction measures really measure the performance of the dialogue manager and consequently illustrate the obvious difference between both dialogue-management manners.

one could argue that this simply means that the individual user-satisfaction measures are not appropriate measures of attitude because people are likely to vary in the way they interpret the item wording (Hone and Graham 2000).


The authors remark that Keller storage is "a very simple modification of our earlier code for Cooper storage."

these storage mechanisms are found wanting.


But this implies that each natural language item accepts only arguments of some one fixed type.

this is not true for natural language, where conjunctions, verbs, and pretty much any functional term that accepts arguments at all can accept arguments of different types.


In a first attempt, we tried to obtain a general German HTML corpus using the meaningless query der die das, i.e., the three German definite articles.

queries of this and a similar form did not lead to satisfactory results: As a consequence of Google's ranking mechanism, which prefers "authorities" (Brin and Page 1998), mainly portals of big organizations, companies, and others were retrieved.


In principle, the same tendency was observed in the documents of the parallel German corpora.


In particular, all tuples in the training data S Tr necessarily also occur in the statistics corpus C St , and therefore no vectors in the training data S Tr involve data unseen in the statistics corpus.

the testing data S Te can be expected to include tuples that did not also occur in the statistics corpus, and so the classifier might not generalize to these tuples using the features we calculate.


This choice apparently misclassifies predication adjuncts as arguments.

for some cases, such as obligatory predication adjuncts, an argument status might in fact be more appropriate than an adjunct status.


On the one hand, this result indicates that the notion of argument is not entirely derivative from the attachment site.

it shows that some features developed to distinguish arguments from adjuncts could improve the disambiguation of the attachment site.


1998), measuring textual cohesion (Morris and Hirst 1991), and word sense disambiguation (Lesk 1986).

since measures of relational similarity are not as well developed as measures of attributional similarity, the potential applications of relational similarity are not as well known.


Intuitively, we might expect that lexicon-based algorithms would be better at capturing synonymy than corpusbased algorithms, since lexicons, such as WordNet, explicitly provide synonymy information that is only implicit in a corpus.

experiments do not support this intuition.


Early versions of SME only mapped identical relations, but later versions of SME allowed similar, nonidentical relations to match (Falkenhainer 1990).

the focus of research in analogy making has been on the mapping process as a whole, rather than measuring the similarity between any two particular relations; hence, the similarity measures used in SME at the level of individual connections are somewhat rudimentary.


animal:eat::inflation:reduce interesting relations, such as antonymy, that do not occur in noun-modifier pairs.

noun-modifier pairs are interesting due to their high frequency in English.


It would seem that there is a problem here because the training vectors would be relatively dense, since they would presumably be derived from a large corpus, but the new unlabeled vector for John Smith and Hardcom Corporation would be very sparse, since these entities might be mentioned only once in the given document.

this is not a new problem for the VSM; it is the standard situation when the VSM is used for information retrieval.


Like Hearst (1992) and Berland and Charniak (1999), they use manually generated rules to mine text for their desired relation.

they supplement their manual rules with automatically learned constraints, to increase the precision of the rules.


In Table 7, we see that pumping:volume has slipped through the filtering in step 2, although it is not a good alternate for quart:volume.

table 10 shows that all four analogies that involve pumping:volume are dropped here, in step 12.


The confidence intervals are calculated using the Binomial Exact Test (Agresti 1990  Without SVD (compare column 1 to 2 in Table 17), performance drops, but the drop is not statistically significant with 95% confidence, according to the Fisher Exact Test (Agresti 1990).

we hypothesize that the drop in performance would be significant with a larger set of word pairs.


In step 6, we no longer have two columns for each pattern P, one for "word 1 P word 2 " and another for "word 2 P word 1 ."

to be fair, we kept the total number of columns at 8,000.


In step 5, for each pair A:B, we add B:A, yielding 4,800 pairs.

some pairs are dropped because they correspond to zero vectors and a few words do not appear in Lin's thesaurus.


2005;Karsenty and Botherel 2005;Sturm and Boves 2005), and on the use of error-recovery strategies that are based on analyses of human-human dialogues (Skantze 2005), including the use of facial expressions (Barkhuysen, Krahmer, and Swerts 2005).

as ASR accuracy improves, dialogue systems will be called upon to handle ever more complex tasks and ever less restricted vocabularies.


This text offers an introduction to the field of linguistics for students who have already some background in computer science but lack special training in linguistics.

those using this textbook in the classroom should consider providing supplementary materials that discuss the fundamental algorithms and techniques for speech and language processing as well as certain linguistic areas, such as phonetics and pragmatics.


Dependency-based approaches, which understand the verb as the center of the sentence structure and describe this structure on the basis of binary relations between heads and their modifiers, have been for a long time a matter of Continental syntactic theories rather than of the mainstream syntactic approaches on the other side of the Atlantic.

the notion of head can be found also in Bloomfield (1933) when referring to the names of the main constituents of the sentence, that is, NP (noun phrase, with N as its head) and VP (verb phrase, with V as its head).


The Praguian concepts met a favorable response within continental linguistics (one should mention in this connection especially the works by British linguists M. A. K. Halliday and H. W. Kirkwood, several German linguists such as J. Esser, R. Bartsch, and J. Jacobs, the French linguist J.-M. Zemb, and others).

only the syntactic or word order consequences (or, as the case may be, conditions) of different sentence articulations into topic and focus were mostly taken into account, and its relevance for and effects on the coherence of discourse.


One could argue that it is the presence of structures with quantification rather than the topic-focus articulation of the quoted examples that is responsible for the indicated semantic distinction.

the Praguian writings from the sixties convincingly demonstrate that it is not difficult to find sentences without quantification that exhibit the same phenomenon (for reasons I will mention in a minute, in the examples, the capitals indicate the intonation center): Russian is spoken in SIBERIA versus In Siberia, RUSSIAN is spoken, or John works on his dissertation on WEEKENDS versus On weekends, John works on his DISSERTATION.


Thus, on the contrary, annotation may and should bring an additional value to the corpus.

there are some necessary conditions for an annotation to fulfil this aim: Its scenario should be carefully (i.e., systematically and consistently) designed, and it should be based on a sound linguistic theory.


This means that for a given data set, both measures will lead to rejection of the null hypothesis at the same level of significance.

the two measures have different underlying scales, and, numerically, they are not directly comparable to each other.


As the probabilities p, p 1 , and p 2 are determined by maximum-likelihood estimation (MLE) and the null hypothesis includes fewer parameters than the alternative hypothesis, the ratio log λ is asymptotically χ 2 -distributed and can thus be used as a test statistic.

we have decided to employ different hypotheses for abbreviation detection.


Due to the revision of the hypotheses H 0 and H A , the log-likelihood ratio for abbreviation detection is no longer asymptotically χ 2 -distributed.

this is not a disadvantage since the resulting log-likelihood value expresses only one of three crucial properties of abbreviations.


9 Those languages in which the period is not usually used to mark ordinal numbers and for which the test without special treatment of numbers achieved better results are italicized in the following tables.

even if the special treatment of numbers was not turned off for such languages, the resulting increase in the error rate was not very high, maximally 0.03%; see also Section 6.4.5.


Riley induces a decision tree for sentence boundary detection using the following features (Riley 1989, pages 351 and 352): The resulting decision tree is able to classify the periods in the Brown corpus with a very low error rate of only 0.2%, which is 0.82% better than that achieved by Punkt.

the impressive performance of Riley's approach also requires impressive amounts of training data: He calculated the probabilities that a certain word occurs before or after a sentence boundary from 25 million words of AP newswire text.


Then, for close language pairs, tuples are expected to successfully handle those short reordering patterns that are included in the tuple structure, as in the case of "traducciones perfectas : perfect translations" presented in Figure 1.

in the case of distant pairs of languages, for which a large number of long tuples are expected to occur, the approach will more easily fail to provide a good translation model due to tuple sparseness.


On the other hand, in the Spanish-to-English direction, it seems that a little improvement with respect to System D is achieved by using 4-grams.

it is not clear which system performs the best since System E obtains the best BLEU score while System F obtains the best mWER score.


Given the many remaining challenges, which are outlined at the end of the book, the CIRCSIMtutor system can serve as a test bed and allow any researcher just starting out in the area of tutorial dialogue to test new techniques and approaches while avoiding the high costs of building an effective tutorial dialogue system.

the authors never explicitly express the latter as one of their goals.


Although this part of the book extends the introductions made in part 2, it does not provide quite enough detail.

there is enough for a reader to understand what was tried and why, and there are plenty of pointers to relevant project papers that will be useful for anyone who wants to try using the CIRCSIM-tutor system as a test bed.


This carries the additional danger that annotations intended to serve for several, often not yet defined, applications may in fact not be useful for any.

the task is organized, annotators typically have no stake in the end result.


The concept of word posterior probabilities based on the fixed target position allows for easy calculation over word graphs and N-best lists.


The absolute values are not needed for the translation process, because the search is performed using the maximum approximation (see Equations (1) and (2)).

to this, the actual values of the weights make a difference for confidence estimation, because the summation over the sentence probabilities is performed.


The statistical models presented in Section 2.2 can be used to estimate the confidence of target words as first described in Ueffing and Ney (2005b).

to the approaches presented in Section 4, the direct phrase-based confidence measures do not use the context information at the sentence level, but only at the phrase level.


This section also contains details on which confidence measures were combined.

the focus of this work is on word posterior probabilities as stand-alone confidence measures.


Similarly to the EPPS data, the domain is basically unrestricted, because a wide range of different topics is covered.

the vocabulary size and the training corpus are much larger than in the EPPS collection, as the corpus statistics presented in Table 3 show.


r The combination of several different word posterior probabilities into one confidence measure yields better confidence estimation performance than the best single feature.

the word posterior probabilities proposed here proved to be strong stand-alone features (see also experiments reported in Blatz et al.


The sample dialogues provided were certainly impressive.

no transcripts of real-world dialogues were provided and therefore it cannot be determined whether the methods and theories developed in UC were robust enough for practical use.


This fact allows the separation of the process of representing the concepts expressed in a document from the use of the relations between concepts for deduction or reasoning processes.

formalisms, theories, and algorithms either designed for domain document representation or reasoning may be made independent from the chosen domain ontology and can also be applied to different domains, thus enhancing system portability between domains.


On the one hand, current systems are overly influenced by the specific characteristics and requirements of the domains, from the different types of questions to be answered to the heterogeneity of the knowledge available for the domain.

the known methodological proposals (Minock 2005) are so general that they could be used to design any kind of information system.


Ideally, we would like to match structured representations derived from the question with those derived from MEDLINE citations (taking into consideration other EBMrelevant factors).

we do not have access to the computational resources necessary to apply knowledge extractors to the 15 million plus citations in the MEDLINE database and directly index their results.


Below, we enumerate the relevant indicator terms by clinical task.

there is a set of negative indicators common to all tasks; these were extracted from the set of genomics articles provided for the secondary task in the TREC 2004 genomics track evaluation (Hersh, Bhupatiraju, and Corley 2004); examples include genetics and cell physiology.


Both the EBM and combination rerankers significantly outperform the termbased reranker (at the 1% level, on all metrics, on both development and test set), with the exception of MRR on the development set.

for all metrics, on both the development set and test set, there is no significant difference between the EBM and combination reranker (which combines both term-based and EBM-based evidence).


Their study also illustrates the importance of semantic classes and relations.

extraction of outcome statements from secondary sources (meta-analyses, in this case) differs from extraction of outcomes from MEDLINE citations because secondary sources represent knowledge that has already been distilled by humans (which may limit its scope).


A number of studies (e.g., Hildebrandt, Katz, and Lin 2004) have pointed out shortcomings of the original nugget scoring model, although a number of these issues have been recently addressed Demner-Fushman 2005a, 2006b).

adaptation of the nugget evaluation methodology to a domain as specific as clinical medicine is an endeavor that has yet to be undertaken.


The answers to such queries can be produced by simple interrogation of the database, because they do not require inferences over the repository of patient records.

the query interface is also coupled with a data-mining module to provide answers to more complex queries, such as The query interface can also be used for accessing information about individual patients.


to these findings, our consultation with cancer clinicians revealed that questions posed in a clinical research setting tend to have a more complex nature and to be directed at groups of patients, searching for relationships rather than simple values: What is the average time of relapse in Acute Myeloid Leukaemia for patients with a complete response after two cycles of treatment?


The treatment profile (Taxol) and the outcome measure (survival rate) have a content that can be easily specified-a single choice from a menu would suffice.

the set of relevant patients requires a very elaborate description because there are so many qualifications.


Furthermore, significant differences were found between subjects' performance on the first query they composed compared to the second, third, and the fourth (each at p < .01 on the Tukey HSD test).

application of the same test showed no significant difference in subjects' performance on the second versus third, second versus fourth, or the third versus fourth composed query.


This means that once a consistent set is found, no descendants of that set can be maximal, and that subtree can be pruned from the search space.

consistent subsets that are maximal in their branch of the tree may turn out not to be globally maximal.


In their case, paths are mapped onto pairs representing a dependency relation r and the end word w (see the discussion in Section 2): Any plausible and computationally feasible function can be used as basis mapping.

in this article we restrict ourselves to models which use a word-based basis mapping.


The global co-occurrence frequency of a basis element b and a target t is computed by the function f : The global co-occurrence frequency f (b, t) could be used directly as the matrix value M[b] [t].

as Lowe (2001) notes, raw counts are likely to give misleading results.


A smaller parameter space could have resulted from collapsing the context selection and path value functions into one parameter, for example, by defining context selection directly as a function from (anchored) paths to their path values, and thus assigning a value of zero to all paths π ∈ cont(t).

we refrained from doing this for two reasons, a methodological one and a technical one.


In this experiment we follow McDonald and Brew's (2004) methodology in simulating semantic priming.

because our primary focus is on the representation of the semantic space, we do not adopt their incremental model of semantic processing.


In theory, an automatically acquired sense ranker should have a good accuracy on all ambiguous words in order to do well on WSD.

in practice the sense ranker's performance depends crucially on its ability to correctly predict the first sense for highly frequent and highly ambiguous words.


This result indicates that the model performs well when trained on a small corpus and that its good performance cannot be attributed solely to corpus size.

it also suggests that a large increase in corpus size is necessary to obtain substantial improvements with the present sense ranking strategy, which uses distributional similarity as a corrective for taxonomy-based similarity: Accuracy increases by approximately 4% when our corpus size increases by a factor of 20.


If two items are added to a cell that are equivalent except for their weights or back-pointers, then they are merged (in the MT decoding literature, this is also known as hypothesis recombination), with the merged item taking its weight and back-pointers from the better of the two equivalent items.

(if we are interested in finding the k-best derivations, the merged item gets the multiset of all the tuples of back-pointers from the equivalent items.


No one algorithm can cater to all types of settings.

our data do suggest quite strongly that, at least in the situation in which our subjects found themselves, a law of diminishing returns is in operation.


Her long involvement with IR arose (as someone who subsisted for an inordinately long time on soft money) by the need to find a new line of research in the aftermath of the ALPAC Report and the subsequent difficulties in getting machine translation work funded.

she was always well qualified to work in IR, a topic addressed in her very early publications (see, for example, Masterman, Needham, and Spärck Jones 1958), although in a rather different context to her later work.


gests that the proposed MLE is a big improvement over the MF baseline for twoway associations, but the improvement becomes less and less noticeable with higher order associations.

this observation is not surprising, because the number of degrees of freedom, 2 m − (m + 1), increases exponentially with m. In order words, the margin constraints are most effective for small m, but the effectiveness decreases rapidly with m. smoothing becomes more and more important as m increases, as shown in Figure 24(b), partly because of the data sparsity in high order associations.


In particular, it uses only intersections, which is always smaller than a s = |K 1 ∩ K 2 | available in the samples.

our algorithm takes advantage of all useful samples up to D s = min(max(K 1 ), max(K 2 )), particularly all a s intersections.


the resemblance R(a) is a convex function of a, the delta method also underestimates the variance.

figure 29 shows that the errors are not very large, and become negligible with reasonably large sample sizes (e.g., 50).


When the (S[ng]\NP 1 )/NP 2 passing is combined with the NP the buck, the lexical head of the NP 2 is instantiated with buck.

similarly, when the adverb just (s\NP 1 )/(s\NP) 2 is applied to passing the buck, a dependency between just and passing is created: because (s\NP 1 )/(s\NP) 2 is a modifier category, the head of the resulting s[ng]\NP is passing, not just (and no dependency is established between just and its NP 1 ).


In written English, certain types of NP-extraposition require a comma before or after the extraposed noun phrase: Because any predicative noun phrase could be used in this manner, this construction is also potentially problematic for the coverage of our grammar and lexicon.

the fact that a comma is required allows us to use a small number of binary type-changing rules (which do not project any dependencies), such as: The translation algorithm presumes that the trees in the Penn Treebank map directly to the desired CCG derivations.


Although the Treebank does not explicitly indicate coordination, it can generally be inferred from the presence of a conjunction.

in list-like nominal coordinations, the conjuncts are only separated by commas or semicolons, and may be difficult to distinguish from appositives.


Typically, frequency information for rare words in the training data is used to estimate parameters for unknown words (and when these rare or unknown words are encountered during parsing, additional information may be obtained from a POS-tagger (Collins 1997)).

in a lexicalized formalism such as CCG, there is the additional problem of missing lexical entries for known words.


Niyogi uses the language of statistical physics to give an evolutionary account of language change and language origins, which for some readers might be a little daunting.

an effort has been made to reach out to different disciplines and together with a peppering of practical examples, The Computational Nature of Language Learning and Evolution will not only be of interest to researchers modeling the evolution of language, but also deserves attention from adventurous historical linguists.


It is a requirement that she be 17 years old.

on the specific reading must gets an epistemic interpretation.


The Bikel parser's performance (without changing attachments) is slightly lower than C&B's, O&M's, and TM&N's.

for the trilexical case, the difference is not statistically significant for NA-discard.


None of these papers attempt to improve a parser in a realistic parsing situation.

a few studies were published recently that do evaluate PP attachment without oracles (Olteanu 2004;Atterer and Schütze 2006;Foth and Menzel 2006).


This has computational advantages (no potentially infinite Z(Θ) terms to compute).

the set of conditional distributions of labels given terminals that can be expressed by MEMMs is strictly smaller than those expressible by HMMs (and by extension, Mealy MRFs).


(1999) because their grammars are hand-written and constraining enough to allow the analyses for each sentence to be enumerated.

for grammars with wider coverage it is often not possible to enumerate the analyses for each sentence in the training data.


In fact, the results given in Hockenmaier (2003b) are lower than previous results.

, Hockenmaier (2003b) reports that the increased complexity of the model reduces the effectiveness of the dynamic programming used in the parser, and hence a more aggressive beam search is required to produce reasonable parse times.


One solution is to only keep a small number of charts in memory at any one time, and to keep reading in the charts on each iteration.

given that the L-BFGS algorithm takes hundreds of iterations to converge, this approach would be infeasibly slow.


In this article we wanted to use a gold standard which is easily accessible to other researchers.

there are some differences between the dependency scheme used by our parser and CCGbank.


The training of the dependency model already uses most of the RAM available on the cluster.

it is possible to use smaller β values for training the dependency model if we also apply the two types of normal-form restriction used by the normalform model.


The results are encouraging because a much smaller amount of corpus data is needed compared to our approach.

their method has only been applied to an artificially constructed test set, rather than a publicly available corpus, and has yet to be applied in a domain-specific setting, which is the chief motivation of our work.


This measure is based on Lin's information-theoretic similarity theorem (Lin 1997 The similarity between A and B is measured by the ratio between the amount of information needed to state the commonality of A and B and the information needed to fully describe what A and B are.

in our application, if T(w) is the set of features f such that i(w, f ) is positive, then the similarity between two words, w and n, is due to this choice of dss and the openness of the domain, we restrict ourselves to only considering words with a total feature frequency of at least 10.


No doubt analysts following this protocol will also achieve excellent interannotator agreement.

we obviously can't use the set of analyses they produce as empirical evidence for the theory.


In fact their definition of directed relations is similar to the definition of "nuclearity" in Mann and Thompson's (1988) theory of coherence relations, and this makes explicit reference to the analyst's intuitions of segment importance.

these decisions are all local to single relations between pairs of text segments.


in "Question answering supported by multiple levels of information extraction" focus on applying information-extraction techniques at retrieval time.

echihabi et al., in their chapter "How to select an answer string," use statistical and information-theoretic techniques, which interpret an answer as a "translation" of the question.


The notion of transfer is supposed to explain children's nonlinear learning curve, as well as their ability to generalize their lexical knowledge to novel items.

the details of this central mechanism are left unspecified: it is not clear how and under which constraints the transfer between two items can take place.


The LSA model is lexicalized: coherence amounts to quantifying the degree of semantic similarity between sentences.

our model does not incorporate any notion of similarity: coherence is encoded in terms of transition sequences that are document-specific rather than sentence-specific.


Both LSA and our entity-grid model are local-they model sentence-to-sentence transitions without being aware of global document structure.

the content models developed by Barzilay and Lee (2004) learn to represent more global text properties by capturing topics and the order in which these topics appear in texts from the same domain.


First, note that the entity-grid model significantly outperforms LSA on both domains (p < .01 using a Sign test, see Table 5).

to our model, LSA is neither entity-based nor unlexicalized: It measures the degree of semantic overlap across successive sentences, without handling discourse entities in a special way (all content words in a sentence contribute towards its meaning).


In the chart in Figure 12, these signs are packed into an equivalence class.

figure 14 shows that the values of CONT, that is, predicate-argument structures, have different values, and the signs as they are cannot be equivalent.


The method just described is the essence of our solution for the tractable estimation of maximum entropy models on exponentially many HPSG parse trees.

the problem of computational cost remains.


For example, PP-attachment ambiguity cannot be resolved with only syntactic preferences.

the results show that a model with only semantic features performs significantly worse than one with syntactic features.


Zero-pronoun resolution is also a difficult problem.

we found that most were indirectly caused by errors of argument/modifier distinction in to-infinitive clauses.


A possible problem with this method is in the approximation of exponentially many parse trees by a polynomial-size sample.

their method has an advantage in that any features on parse results can be incorporated into a model, whereas our method forces feature functions to be defined locally on conjunctive nodes.


, 2002 also proposed a maximum entropy model for probabilistic modeling of LFG parsing.

similarly to the previous studies on HPSG parsing, these groups had no solution to the problem of exponential explosion of unpacked parse results.


The proposed algorithm is an essential solution to the problem of estimating probabilistic models on exponentially many complete structures.

the applicability of this algorithm relies on the constraint that features are defined locally in conjunctive nodes.


When we define more global features, such as co-occurrences of structures at distant places in a sentence, conjunctive nodes must be expanded so that they include all structures that are necessary to define these features.

this obviously increases the number of conjunctive nodes, and consequently, the cost of parameter estimation increases.


Sampling techniques (Rosenfeld 1997;Chen and Rosenfeld 1999b;Osborne 2000;Malouf and van Noord 2004) allow us to define any features on complete structures without any constraints.

they force us to employ approximation methods for tractable computation.


Readers of Arabic will notice that the translation of like is indeed a comparative, not a verb meaning enjoy, as in the legend.

they will also notice that flies translates as the noun, rather than the verb, just as the story foretold.


The gain in performance from the combination step is consistently between two and three F 1 points.

a combination approach increases system complexity and penalizes efficiency.


This opens the possibility of exploiting dependencies among the different verbs in the sentence.

the complexity may grow significantly, and results so far are inconclusive (Carreras, Màrquez, and Chrupała 2004;Surdeanu et al.


The predicate widen shares the phrase the trade gap with expect as an ARG1 argument.

as expect is a raising verb, widen's subject is not in its typical position either, and we should expect to find it in the same position as expected's subject.


For example, [VP [V NP]] is an SST which has two non-terminal symbols, V and NP, as leaves.

]] is not an SST as it violates the production VP→V NP.


On the one hand, this makes the learning and classification phases more complex because they involve more instances.

our results are not biased by the quality of the heuristics, leading to more meaningful findings.


We note that marking the argument node simplifies the generalization process as it improves both tasks by about 3.5 and 2.5 absolute percentage points, respectively.

row Poly shows that the polynomial kernel using state-of-the-art features (Moschitti et al.


The previous section has shown that if a state-of-the-art model 12 is adopted, then the tree kernel contribution is marginal.

if a non state-of-the-art model is adopted tree kernels can play a significant role.


It should be noted that a more accurate baseline can be provided by using the Viterbi-style search (see Section 4.4.1).

the experiments in Section 6.5 show that the heuristics produce the same accuracy (at least when the complete task is carried out).


(c) The use of fast tree kernels (Moschitti 2006a) along with the proposed tree representations makes the learning and classification much faster, so that the overall running time is comparable with polynomial kernels.

when used with SVMs their running time on very large data sets (e. g., millions of instances) becomes prohibitive.


For example, a noun phrase headed by ("today") is very likely to be a temporal element; so is a prepositional phrase with the head word ("at").

for prepositional phrases, the preposition is not always the most informative element.


Throughout this article, when full parsing information is available, we assume that the system is presented with the full phrase-structure parse tree as defined in the Penn Treebank (Marcus, Marcinkiewicz, and Santorini 1993) but without trace and functional tags.

when only shallow parsing information is available, the full parse tree is reduced to only the chunks and the clause constituents.


The predictions are combined to form argument candidates.

we can employ a simple heuristic to filter out some candidates that are obviously not arguments.


Previous approaches usually rely on dynamic programming to resolve nonoverlapping/embedding constraints (i.e., Constraint 4) when the constraint structure is sequential.

they are not able to handle more expressive constraints such as those that take long-distance dependencies and counting dependencies into account (Roth and Yih 2005).


The F 1 difference is 1.5 points when using the gold-standard data.

when automatic parsers are used, the shallow parsing-based system is, in fact, slightly better; although the difference is not statistically significant.


Specifically, these argument candidates often overlap and differ only in one or two words.

the pruning heuristic based on full parsing never outputs overlapping candidates and consequently provides input that is easier for the next stage to handle.


For example, an assignment that has two A1 arguments clearly violates the non-duplication constraint.

if an assignment has no predicted arguments at all, it still satisfies all the constraints.


Each possible labeling of the argument is associated with a variable which is then used to set up the inference procedure.

the final prediction will be likely dominated by the system that produces more candidates, which is system B in this example.


However, their chunk-based approach was very weak-only chunks were considered as possible candidates; hence, it is not very surprising that the boundaries of the arguments could not be reliably found.

our shallow parse-based system does not have these restrictions on the argument boundaries and therefore performs much better at this stage, providing a more fair comparison.


The performance of argument Identification is essentially the same as when training and testing on WSJ.

argument Classification is 6 percentage points worse (80.1% vs. 86.1%) when training and testing on Brown than when training and testing on WSJ.


It first describes the IR models used, which is interesting for someone unfamiliar with the field.

the discussion of the integration of IE does not seem convincing and there is once again a lack of real examples which would strengthen the arguments.


In theory, there is one correct label for any given act.

in practice human coders disagree, choosing different labels for the same act (sometimes even with divergences that make one question whether there is one correct answer).


For data with a very strong correlation between the input features A and the output labels B, the turning point below which performance is spuriously high occurs at around κ = 0.55 (Figure 3d), a value the community holds to be pretty low but which is not unknown in published work.

when the underlying relationship to be learned is moderate or strong (Figures 3b and 3c), the spuriously high results already occur for κ values commonly held to be tolerable.


Some factors such as morphology (gender, number, animacy, and case) or syntax (e.g., the role of binding and commanding relations [Chomsky 1981]) are "eliminating," forbidding certain NPs from being antecedents.

many others are "preferential," giving more preference to certain candidates over others; examples include: r Sentence-based factors: Pronouns in one clause prefer to refer to the NP that is the subject of the previous clause (Crawley, Stevenson, and Kleinman 1990).


Thus, it enables a relatively large number of candidates to be processed.

as our twin-candidate model imposes no constraints that enforce transitivity of the preference relation, the preference classifier would likely output C 1 C 2 , C 2 C 3 , and C 3 C 1 .


r For most of our experiments we use as input the gold-standard tags from the treebank.

in our last experiments we evaluate the impact of automatic statistical morphological disambiguation on the performance of our best performing parser.


Such a post-processor could be worth developing if the WW U accuracy obtained with this model proves to be higher than all of the other models, that is, if this is the best way of finding the correct dependencies between words without considering which IGs are connected.

as we will see in Section 4.4, this model does not give the best WW U .


19 This approach has some advantages over the probabilistic parser, in that r it can process both left-to-right and right-to-left dependencies due to its parsing algorithm, r it assigns dependency labels simultaneously with dependencies and can use these as features in the history-based model, and r it does not necessarily require expert knowledge about the choice of linguistically relevant features to use in the representations because SVM training involves implicit feature selection.

we still exclude sentences with non-projective dependencies during training.


We take the minor POS category plus the case and possessive agreement markers for nominals and participle adjectives to make up the POS feature of each IG.

21 we do not employ dynamic selection of these features and just use the same strategy for both dependents and the heads.


We can see the benefit of using inflectional features separately and split into atomic components, by comparing the first line of the table with the best results for the IG-based model in Table 4.

we can also note the improvement that lexicalized models bring: 24 to the probabilistic parser, lexicalization using root information rather than surface form gives better performance, even though the difference is not statistically significant.


1 Moreover, a surprising variety of problems are attackable with FSTs, from part-of-speech tagging to letter-to-sound conversion to name transliteration.

language problems like machine translation break this mold, because they involve massive re-ordering of symbols, and because the transformation processes seem sensitive to hierarchical tree structure.


consists only of q S(x0, x1), although we want the English-to-Arabic transformation to apply only when it faces the entire structure q S(PRO, VP(V, NP)).

we can simulate lookahead using states, as in these productions:  By omitting rules like qpro NP → ..., we ensure that the entire production sequence will dead-end unless the first child of the input tree is in fact PRO.


This modification adds to our new transducer model all the contextual information specified in Yamada and Knight (2001).

upon closer inspection one can see that the exact transducer is in fact overspecified in the reordering, or r rules.


Regular roots such as p.s.q yield forms such as hpsqh.

the irregular roots n.p.l, i.c.g, q.w.m, and g.n.n in this pattern yield the seemingly similar forms hplh, hcgh, hqmh, and hgnh, respectively.


These are the kinds of errors which are most difficult to fix.

in many cases the system's errors are relatively easy to overcome.


Once you have gone down this route, it's very hard to consider releasing the resulting software.

if you plan from the start to distribute your software, you will inevitably be guided by considerations that are important to your potential audience.


It is true that releasing software that is both usable and reliable requires a strong hand to guide system development, and that's a skill that many researchers don't think they have.


This is not necessarily a bad thing, and might address concerns such as those raised by Chuch (2005) about very conservative reviewing in our field and the resulting tendency to prefer incremental improvements.

the other path is to accept (and in fact insist) that highly detailed empirical studies must be reproducible to be credible, and that it is unreasonable to expect that reproducibility be possible based on the description provided in a publication.


This data is usually used to motivate and inspire a new hand-built dialogue system or to modify an existing one.

given the existence of such data, it should be possible to exploit machine learning methods to automatically build and optimize a new dialogue system.


Simulated users are generally preferred due to the much smaller development effort involved, and the fact that trialand-error training with humans is tedious for the users.

the issues of how to construct and then evaluate simulated users are open problems.


We show that all four algorithms give competitive accuracy, although the non-projective list-based algorithm generally outperforms the projective algorithms for languages with a non-negligible proportion of non-projective constructions.

the projective algorithms often produce comparable results when combined with the technique known as pseudo-projective parsing.


The fact that both the head and the dependent are kept in either λ 2 or β makes it possible to construct non-projective dependency graphs, because the NO-ARC n transition allows a node to be passed from λ 1 to λ 2 even if it does not (yet) have a head.

an arc can only be added between two nodes i and j if the dependent end of the arc is not the artificial root 0 and does not already have a head, which would violate ROOT and SINGLE-HEAD, respectively, and if there is no path connecting the dependent to the head, which would cause a violation of ACYCLICITY.


We conjecture that an additional necessary condition is an annotation style that favors more deeply embedded structures, giving rise to chains of left-headed structures where each node is dependent on the preceding one, which increases the number of points at which an incorrect decision can be made by an arcstandard parser.

we have not yet fully verified the extent to which this condition holds for all the data sets where the arc-eager parsers outperform their arc-standard counterparts.


Before we consider the evaluation of efficiency in both learning and parsing, it is worth pointing out that the results will be heavily dependent on the choice of support vector machines for classification, and cannot be directly generalized to the use of deterministic incremental parsing algorithms together with other kinds of classifiers.

because support vector machines constitute the state of the art in classifierbased parsing, it is still worth examining how learning and parsing times vary with the parsing algorithm while parameters of learning and classification are kept constant.


Donner and Eliasziw (1987) propose a more general form of significance test for arbitrary levels of agreement.

krippendorff (2004a, Section 11.4.2) states that the distribution of α is unknown, so confidence intervals must be obtained by bootstrapping; a software package for doing this is described in Hayes and krippendorff (2007).


These results show that agreement coefficients should not be used as indicators of the suitability of annotated data for machine learning.

the purpose of reliability studies is not to find out whether annotations can be generalized, but whether they capture some kind of observable reality.


The probability Pr(t) represents the well-formedness of t and it is generally called the language model probability (n-gram models are usually adopted [Jelinek 1998]).

pr(s|t) represents the relationship between the two sentences (the source and its translation).


Moreover, in practice the summation operator is replaced with the maximization operator, which in turn reduces the contribution of each individual source word in generating a target word.

modeling word sequences rather than single words in both the alignment and lexicon models cause significant improvement in translation quality (Och and Ney 2004).


Other techniques are heuristics based on the previous computation of word alignments in the training corpus (Zens, Och, and Ney 2002;Koehn, Och, and Marcu 2003).

as for AT, the weights λ i in Equation 13are usually optimized using held-out data.


Specifically in the field of language translation, it is often argued that natural languages are so complex that these simple models are never able to cope with the required source-target mappings.

one should take into account that the complexity of the mapping between the source and target domains of a transducer is not always directly related to the complexity of the domains themselves.


For a fixed source sentence, if no pruning is applied in the production of the word graph, it represents all possible sequences of target words for which the posterior probability is greater than zero, according to the models used.

because of the pruning generally needed to render the problem computationally feasible, the resulting word graph only represents a subset of the possible translations.


In general, the different techniques perform similarly for the various translation directions.

the English-Spanish language pair is the one for which the best translations can be produced.


In cases where these texts were quite unrelated to the training data, the system did not significantly help the human translators to increase their productivity.

when the test texts were reasonably well related to the training data, high productivity gains were registeredclose to what could be expected according to the KSR/MAR empirical results.


This measure does not take into account that the OSOs differ in length.

this information is necessary to estimate reliably the performance of an information ordering approach, as we discuss in Karamanis and Mellish (2005a) in more detail.


16 Barzilay and Lapata (2005) compare the OSO with just 20 alternative orderings, often sampled out of several millions.

barzilay and Lee (2004) enumerate exhaustively each possible ordering, which might become impractical as the search space grows factorially.


Karamanis and Mellish (2005b) conducted an experiment in the MPIRO domain using Lapata's methodology which supplements the work reported in this article.

such an approach is less practical for much larger collections of texts such as NEWS and ACCS.


Following McKeown (1985), Kibble and Power argue in favor of an integrated approach for concept-to-text generation in which the same centering features are used at different stages in the generation pipeline.

our study suggests that features such as CHEAPNESS and the centering transitions are not particularly relevant to information ordering.


If this were not the case, then, because ERCS(x) must contain at least one coordinate with an L (else there would be no way to reject {x}), the presence of x in S would place an L in a W-unique coordinate in S − {x}.

this would make it impossible to associate a ranking with at least one partitioning of S into accepted and rejected subsets contra the assumption that S is shatterable.


Nonetheless, at these recall values, the two measures have relatively low precision (compared to the other measures), suggesting that both measures also put many idiomatic pairs near the bottom of the list.

the precision-recall curve of Fixedness syn shows that its performance is consistently much better than that of PMI: Even at the recall level of 90%, its precision is close to 70% (cf.


We return to these results later (Section 7.2.3) to offer some reasons as to why this might be the case.

the results using CFORM confirm our hypothesis that canonical forms-which reflect the overall behavior of a verb+noun type-are strongly informative about the class of a token.


This method is similar to ours, in that it also uses type-based knowledge to determine the class of each token in context.

their method is supervised, whereas our methods are unsupervised.


In this approach, linguistic-quantitative analyses come as a later step to facilitate the interpretation of discourse types.

the bottom-up approach begins with a linguisticquantitative analysis based on the automatic segmentation of texts into discourse units on the basis of vocabulary distributional patterns.


In this article, relation disambiguation experiments are only presented for Factotum, given that the others do not readily provide sufficient training data.

the other inventories are discussed because each provides relation types incorporated into the inventory used below for the definition analysis (see Section 3.5).


The full Cyc KB is proprietary, which has hindered its adoption in natural language processing.

to encourage broader usage, portions of the KB have been made freely available to the public.


This has the advantage of allowing more training data to be used in the derivation of the clues indicative of each semantic role.

if there were sufficient annotations for particular preposition, then it would be advantageous to have a dedicated classifier.


Currently, FrameNet roles are always mapped to the same common inventory role (e.g., place to location).

this should account for the frame of the annotation and perhaps other context information.


His probabilistic model computes the probability of a preposition p given a noun-noun pair n 1 − n 2 and finds the most likely preposition paraphrase p * = argmax p P(p|n 1 , n 2 ).

as Lauer noticed, this model requires a very large training corpus to estimate these probabilities.


This approach works most of the time for relations such as LOCATION and TEMPORAL because in both English and Romance languages they rely mostly on prepositions indicating location and time and less on underspecified prepositions such as of or de.

a closer look at these relations shows that some of the noun-noun pairs that encode them are not symmetric and this is true for both English and Romance.


Moreover, each cross-linguistic study requires translated data, which is not easy to obtain in electronic form, especially for most of the world's languages.

more and more parallel corpora in various languages are expected to be forthcoming.


(103) some(x, mouse(x), def(y, table(y), appear(e 1 , x) ∧ from(e 2 , e 1 , e 3 ) ∧ under(e 3 , u, y))) On this analysis, the first preposition from is transitive, and it takes the PP under the table, or rather the event introduced by this preposition, as an argument.

to ERG, this analysis allows the last argument of a preposition like from to be an event, e 3 , as well as an individual.


For example, Gapp (1995) defines the area of proximity as the region within ten times the size of the landmark object in each direction.

neither of these approaches consider the effect that the locations of other objects in the scene have on the proximity.


Finally, Regier and Carlson (2001) developed a vector sum algorithm to compute the applicability of a projective relation between a landmark and a target.

as with previous topological models, none of these models consider the influence of other objects in the context of the landmark target relationship.


These three chapters will most likely be of least interest to the computational linguistics community.

two topics that might be of interest are the use of wildcard queries by users and spelling correction of queries, discussed in Chapter 3.


The intermediate referents e d ρ,t in this framework correspond to the traditional notion of compositional semantics (Frege 1892), in which meanings of composed constituents (at higher levels in the HHMM hierarchy) are derived from meanings of component constituents (at lower levels in the hierarchy).

in addition to the referents A graphical representation of the dependencies in the referential semantic language model described in this article (compare with Figure 2).


These hierarchic regular expressions are defined to resemble expansion rules in a context free grammar (CFG).

unlike CFGs, HHMMs have memory limits on nesting, in the form of a maximum depth D beyond which no expansion may take place.


These domains also provide an ideal proving ground for a referential semantic language model, because directives in these domains mostly refer to a world model that is shared by the user and the interfaced application, and because the idiosyncratic language used in such domains makes it more resistant to domain-independent corpus training than other domains.

domains such as database query (e.g., of airline reservations), dictation, or information extraction are less likely to benefit from a referential semantic language model, because the world model in such domains is not shared by either the speaker (in database query) or by the interfaced application (in dictation or information extraction), 7 or because these domains are relatively fixed, so the expense of maintaining linguistic training corpora in these domains can often be justified.


Ordinarily this is thought of as being mediated by syntax, which is covered in this article only through a relatively simple framework of bounded recursive HHMM state transitions.

the bounded HHMM representation used in this paper has been applied (without semantics) to rich syntactic parsing as well, using a transformed grammar to minimize stack usage to cases of center-expansion (Schuler et al.


2004) and semantic role labeling (Punyakanok, Roth, and Yih 2005), we first pursue a purely data-driven approach where the predicate of a multimodal command and its arguments are determined by classifiers trained on an annotated corpus of multimodal data.

given the limited amount of data available, this approach does not provide an improvement over the grammar-based approach.


Given the higher chance of error in speech recognition compared to gesture recognition, we focus on processing the speech recognition output to achieve robust multimodal understanding.

these techniques are also equally applicable to gesture recognition output.


Johnston (1998aJohnston ( , 1998b utilized techniques from natural language processing (unification-based grammars and chart parsers) to extend the unification-based approach and enable handling of inputs with more than one gesture, visual parsing, and more flexible and declarative encoding of temporal and spatial constraints.

to the unification-based approaches, which separate speech parsing and multimodal integration into separate processing stages, Bangalore (2000, 2005) proposed a one-stage approach to multimodal understanding in which a single grammar specified the integration and understanding of multimodal language.


The brittleness of using a grammar as a language model is typically alleviated by building SLMs that capture the distribution of the user's interactions in an application domain.

such SLMs are trained on large amounts of spoken interactions collected in that domain-a tedious task in itself, in speech-only systems, but an often insurmountable task in multimodal systems.


The techniques are presented in the context of SLMs, since spoken language interaction tends to be a dominant mode in our application and has higher perplexity than the gesture interactions.

most of these techniques can also be applied to improve the robustness of the gesture recognition component in applications with higher gesture language perplexity.


In a grammar-based speech-only system, if the language model of ASR is derived directly from the grammar, then every ASR output can be parsed and assigned a meaning by the grammar.

using an SLM results in ASR outputs that may not be parsable by the grammar and hence cannot be assigned a meaning by the grammar.


Although a grammar could be written so as to be easily portable across applications, it suffers from being too prescriptive and has no metric for the relative likelihood of users' utterances.

in the data-driven approach a weighted grammar is automatically induced from a corpus and the weights can be interpreted as a measure of the relative likelihoods of users' utterances.


As mentioned earlier, a hand-crafted grammar typically suffers from the problem of being too restrictive and inadequate to cover the variations and extra-grammaticality of users' input.

an n-gram language model derives its robustness by permitting all strings over an alphabet, albeit with different likelihoods.


It is immediately apparent that the hand-crafted grammar as a language model performs poorly and a language model trained on the collected domain-specific corpus performs significantly better than models trained on derived data.

it is encouraging to note that a model trained on a derived corpus (obtained from combining the migrated out-of-domain corpus and a corpus created by sampling the in-domain grammar) is within 10% word accuracy as compared to the model trained on the collected corpus.


There is some more recent work on using structured classification approaches to transduce sentences to logical forms (Papineni, Roukos, and Ward 1997;Thompson and Mooney 2003;Zettlemoyer and Collins 2005).

it is not clear how to extend these approaches to apply to lattice input-an important requirement for multimodal processing.


We adopted the edit-based technique used on speech utterances to improve robustness of multimodal understanding.

unlike a speech utterance, a gesture string has a structured representation.


However, scalability of such systems is a bottleneck due to the heavy cost of authoring and maintenance of rule sets and inevitable brittleness due to lack of coverage.

data-driven approaches are robust and provide a simple process of developing applications given availability of data from the application domain.


This can be complex and expensive, involving a detailed labeling guide and instructions for annotators.

in this approach, if data is used, all that is needed is transcription of the audio, a far more straightforward annotation task.


Many approaches to automatic sentiment analysis begin with a large lexicon of words marked with their prior polarity (also called semantic orientation).

the contextual polarity of the phrase in which a particular instance of a word appears may be quite different from the word's prior polarity.


Acquiring the polarity of words and phrases is undeniably important, and there are still open research challenges, such as addressing the sentiments of different senses of words (Esuli and Sebastiani 2006b;Wiebe and Mihalcea 2006), and so on.

what the polarity of a given word or phrase is when it is used in a particular context is another problem entirely.


The adjective well is considered positive, and indeed it is positive in this context.

the same is not true for the words reason and reasonable.


1966;Hatzivassiloglou and McKeown 1997), we largely retained their original polarity.

we did change the polarity of a word if we strongly disagreed with its original class.


In terms of accuracy, this classifier does not perform much worse than the BoosTexter and TiMBL classifiers that use all the neutral-polar features: The SVM word+priorpol baseline classifier has an accuracy of 75.6%, and both the BoosTexter and TiMBL classifiers have an accuracy of 76.5%.

the BoosTexter and TiMBL classifiers using all the features perform notably better in terms of polar recall and F-measure.


Also, because the set of polar instances being classified is the same for all the algorithms, condition 1 allows us to compare the performance of the polarity features across the different algorithms.


For the positive class, precisions and recalls for the word+priorpol baseline range from 63.7 to 76.7.

it is with the positive class that polarity features seem to help the most.


It follows that a larger lexicon will have a greater coverage of sentiment expressions.

expanding the lexicon with automatically acquired prior-polarity tags may result in an even greater proportion of neutral instances to contend with.


As can be seen, some of the top features are either too specific (landlocked, airspace), and are thus less reliable, or too general (destination, ambition), thus not indicative and may co-occur with many different types of words.

intuitively more characteristic features of country, like population and governor, occur further down the Table 2 The top 20 most similar words for country (and their ranks) in the similarity list of LIN, followed by the next four words in the similarity list that were judged as entailing at least in one direction.


The resulting increase in precision was much smaller, of about 2%, showing that most of the potential benefit is exploited in the first bootstrapping iteration (which is not uncommon for natural language data).

computing the bootstrapping weight twice increases computation time significantly, which led us to suggest a single bootstrapping iteration as a reasonable cost-effectiveness tradeoff for our data.


Table 9 shows that most of the top 10 common features for country-state are now ranked highly for both words.

there are only two common features (among the top 100 features) for the incorrect pair country-party, both with quite low ranks (compare with Table 7), while the rest of the common features for this pair did not pass the top 100 cutoff.


In recent work on distributional similarity (Curran 2004;Weeds and Weir 2005) a variety of alternative weighting functions were compared.

the quality of these weighting functions was evaluated only through their impact on the performance of a particular word similarity measure, as we did in Section 5.


For example, Allen's (1995) Natural Language Understanding presented mostly a symbolic approach to NLP, whereas Manning and Schütze's (1999) Foundations of Statistical Natural Language Processing presented an exclusively statistical approach.

there is also a negative side to the wide coverage-it is probably impossible to present material in an order that would satisfy audiences from different backgrounds, in particular, linguists vs. computer scientists and engineers.


It's probably best not to bother with linguistic software in the early stages of linguistic description.

things change once the descriptive notation has stabilized, and a "linguistic exploration" workflow is established.


are established, the recognizer when it observes A can conduct a search leading to the maximizing word string W. In the case of an artificial grammar, such as the New Raleigh grammar, the model P(W) is in fact equal to the actual probability P(W).

furthermore, because the set S of word strings W over which we maximize can be listed, the difficulty of the task can be measured approximately (because the acoustic similarity of words is not taken into account) by entropy: because the participants in the ARPA project introduced the false measure "branching factor" (which was the arithmetic mean of the out-of-state branching of their finite-state grammar), we replaced H(W) as a measure of difficulty by perplexity, defined by PP 2 H (W) It turned out that the New Raleigh grammar had approximate perplexity 7 whereas the Resource Management grammar had 2.


My former colleagues will not tell me: Theirs is a very hush-hush operation!

we did start working on machine translation (MT) in 1987.


The issue is more important when dealing with small-to-moderate data sets.

even for a 130K test set (Sections 22-24 of the Wall Street Journal corpus, standardly used as a test set in POS-tagging benchmarks), it is useful to know the estimated noise rate, as it is not clear that all reported improvements in performance would come out significant.


the samples match not because the classifier gets the label right, but because it overuses the same label as the human coder" (Reidsma and Carletta 2008, page 232).

if disagreements are random classification noise (the label of any instance can be flipped with a certain probability), a performance estimate based on observed data would often be lower than performance on the real data, because the noise that corrupted it was ignored by the classifier (see Figure 2(d) therein).


Learning in the presence of noise is an active research area in machine learning.

annotation noise is different from existing well-understood noise models.


It seems our method is over 10% below state of the art in precision on the MSR data.

we find that multiword expressions are consistently segmented into smaller words.


For instance, despite the fact that the sentences He is affected by AIDS and HIV is a virus express closely-related concepts, their similarity is zero in the VSM because they have no words in common (they are represented by orthogonal vectors).

due to the ambiguity of the word virus, the similarity between the sentences The laptop has been infected by a virus and HIV is a virus is greater than zero, even though they convey very different messages.


Weather forecast generation is probably the closest that NLG comes to a "standard" application domain, and hence seems a good choice for validation studies from this perspective.

though, one could also argue that weather-forecast generators are atypical in that the language they generate tends to be very simple, even by the standards of NLG systems: very limited syntax (which differs from conventional English), very small vocabulary, no real text structure above the sentence level, and so on.


This procedure identified the same four significant differences in Accuracy as in Table 4-namely, SUMTIME is significantly better than all systems except pCRU-greedy and Template.

it identified only three significant differences in Claritynamely, SUMTIME is significantly better than Template and pCRU-random, and pCRUgreedy is better than pCRU-random (this is a subset of the significant differences in Clarity shown in Table 4).


In principle, it would have been preferable to ask the forecasters to write texts based on content tuples (the actual input to the systems), but this is not a natural task for forecasters (they write texts from data, not from content tuples).

asking them to rewrite corpus texts meant they were unable to change the content, and focused on lexical choice and syntactic structure, as intended.


If only texts which communicate the same content at the content tuple level are included in the analysis, then  and SE are strongly correlated with clarity judgments (r > 0.95), and BLEU-4 also correlates significantly (Table 8).

no metric correlates significantly with human accuracy judgments under any analysis.


Both experiments required similar time commitments from similar subjects.

subjects (and the domain experts who facilitated subject recruitment) were much more enthusiastic about testing the effectiveness of a system which they might themselves use; they were less enthused about testing hypotheses about correlations between NLG evaluation metrics.


But if we are applying multiple hypothesis corrections, then there is a major drawback to including a large number of metrics in the study, which is that this will make it more difficult to find statistically significant correlations.

perhaps it is wrong to use such strict statistics in computational linguistics, and indeed we are aware of many reports in computational linguistics which present one-tailed p-values, do not apply multiple hypothesis corrections, present post hoc analyses as significant, and/or use parametric tests to analyse data which does not have the characteristics assumed by the parametric test.


Finally, evaluation with metrics is entirely reproducible.

despite the cost-effectiveness and other appealing aspects of automatic metrics in shared tasks, we do not believe that shared tasks in NLG should use automatic metrics as the sole evaluation criterion.


Because our results suggest that current automatic metrics are not highly accurate predictors of the quality of texts produced by NLG systems, we recommend developers be cautious in using metrics for diagnostic evaluation, and do not use metrics for automatic parameter tuning.

automatic metrics do have a potential advantage in small diagnostic evaluations, which is that they are not influenced by the individual preferences of a small number of human subjects.


is perhaps the best metric to use for this purpose (out of the ones we investigated).

existing metrics should not be used to evaluate the content of texts.


One way around this difficulty is to stipulate that all rules must be binary from the outset, as in Inversion Transduction Grammar (ITG) (Wu 1997) and the binary SCFG employed by the Hiero system (Chiang 2005) to model the hierarchical phrases.

the rule extraction method of Galley et al.


These two binarizations are no different in the translation-model-only decoding described previously, just as in monolingual parsing.

in the source-channel approach to machine translation, we need to combine probabilities from the translation model (TM) (an SCFG) with the language model (an n-gram), which has been shown to be very important for translation quality (Chiang 2005).


This is primarily of theoretical interest, as we found that they constitute a small fraction of all rules, and removing these did not affect our Chinese-to-English translation results.

non-binarizable rules are shown to be important in explaining existing hand-aligned data, especially for other language pairs such as German-English (see Section 5.2, as well as Wellington, Waxmonsky, and Melamed [2006]).


Any combination of two subsets of the rule's nonterminals involves the indices for the spans of each subset.

some of the indices are tied together: If we are joining two spans into one span in the new item, one of the original spans' end-points must be equal to another span's beginning point.


Sometimes requests match each other quite well, suggesting an approach where a new request is matched with an old one, and the corresponding response is reused.

analysis of our corpus shows that this does not occur very often, because unlike response e-mails, request e-mails exhibit a high language variability: There are many customers who write these e-mails, and they differ in their background, level of expertise, and pattern of language usage.


Hence, what is important when matching a request to a response is the number of (significant) terms in common, rather than their frequency.

when matching a request to a request, or a request to a request-response pair, term frequency would be more indicative of the goodness of the match, as the document also has a request component.


As argued in the Introduction, when it comes to obtaining information quickly on-line, this option may be preferable to having to wait for a human-generated response.

the document-level approach is an all-or-nothing approach: If there is insufficient evidence for a complete response, then no automated response is generated.


This is because SC 1 includes sentences from responses whose requests mention different faulty products, for example, monitor, printer, or notebook.

if there are more cases of faulty monitors in the corpus than other faulty products, then requests about repairing monitors will have a higher prediction probability than requests about repairing other products.


However, if there are more cases of faulty monitors in the corpus than other faulty products, then requests about repairing monitors will have a higher prediction probability than requests about repairing other products.

to prediction probability, SVM reliability reflects its overall performance (on the training data), and is independent of particular requests.


As seen in cluster SC 2 in Figure 3, it is possible for an SC to be strongly predicted without it being sufficiently cohesive for a confident selection of a representative sentence.

sometimes the ambiguity can be resolved through cues in the request.


At first glance, using the best method may appear too lenient, as in practice, we cannot always automatically select this method in advance.

these averages also suffer from the fact that in many cases only the Doc-Ret method is applicable, but its performance is poor.


This is because we hoped that our trial subjects would prefer Sent-Hybrid to Sent-Pred, as the former is designed to better tailor a response to a request.

we cannot determine from this result whether indeed there is no difference between the sentencebased methods, or whether such a difference simply could not be observed from our test sample of at most 80 cases, which constitutes 1.8% of the corpus used in our automatic evaluation (as indicated previously, it would be quite difficult to conduct user studies with a much larger data set).


Following Lekakos and Giaglis (2007), one approach for achieving this objective consists of applying supervised learning, where a winning method is selected for each case in the training set, all the training cases are labeled accordingly, and then the system is trained to predict a winner for unseen cases.

in our situation, there is not always one single winner (two methods can perform similarly well for a given request), and there are different ways to pick winners (for example, based on F-score or precision).


For example, for w = 0.5, the precision and recall values of Cluster 16 (Figure 8(b)) translate to F-scores of 0.895 and 0.865 for Doc-Pred and Sent-Pred, respectively, leading to a choice of Doc-Pred.

for w = 0.75, the respective F-scores are 0.897 and 0.914, leading to a choice of Sent-Pred.


We evaluate the meta-learning system by looking at the quality of the response produced by the method selected by this system, where, as done in Section 4, quality is measured using F-score and precision.

here we employ 5-fold crossvalidation (instead of 10-fold) to ensure that we get a good spread of selected methods in each testing split.


Note that the baselines do not have an estimated precision because they do not use meta-learning.

for completeness, we implement the practical system for them as well, with a threshold of 0.8 on actual precision.


Because the results of all the methods are comparable, no learning is required: At each stage of the "cascade of methods," the method that performs best is selected.

to these two systems, our system employs methods that are not comparable, because they use different metrics.


Such technologies require significant human input, and are difficult to create and maintain (Delic and Lahaix 1998).

the techniques examined in this article are corpus-based and data-driven.


The operator is assisted in this task by the retrieval results: The system highlights the request-relevant sentences in the ranked responses.

there is no attempt to automatically generate a single response.


A fine example of this cross-over is Chapter 9, "Kernel-Based Machine Translation," in which a novel approach to estimating translation models is presented.

this promise is not entirely fulfilled, as some contributions either fail to make use of machine learning or are somewhat obscure, unlikely to impact on the mainstream SMT community.


This is an interesting problem with relevance for commercial MT systems which must avoid nonsensical literal translations of named entities.

the treatment takes the form of a system description and fails to make use of any machine learning, thus feeling somewhat out of place in this book.


This model maintains a bounded store of complete but unattached constituents as a buffer, and operates on them using a variety of specialized memory manipulation operations, deferring certain attachment decisions until the contents of this buffer indicate it is safe to do so.

(the model described in this article maintains a store of incomplete constituents using ordinary stack-like push and pop operations, defined to allow constituents to be composed before being completely recognized.)


This is essentially an ambiguity between arc-eager (in-element) and arc-standard (cross-element) composition strategies, as described by Abney and Johnson (1991).

an ordinary (purely arc-standard) parser with an unbounded stack would only hypothesize analysis (b), avoiding this ambiguity.


Semantic distance can also be measured more explicitly, by using the relations in an ontology as the direct encoding of semantic association.

such approaches have generally been limited to calculating the distance between two individual concepts, rather than capturing the distance between two sets of concepts corresponding to two texts.


For example, researchers have developed measures of semantic distance between texts that apply distributional distances to concept vectors of frequencies rather than to word vectors (McCarthy 2000;Mohammad and Hirst 2006).

these approaches only make pairwise comparisions between the elements of the concept vectors, and do not take into account the important ontological relations among the concepts.


In order to capture such relations, other methods have instead integrated distributional information into an ontological method.

such approaches have heretofore been limited to measuring distance between two individual concepts.


If i ∈ D, b(i) is set to the negative of the normalized demand frequency, −f D (i), since demand is indicated by a value less than zero.

i may be part of both the supply and demand profiles, and then b(i) must be set to the net supply/demand at node i.


Such methods estimate the appropriate probability distribution over a set of concepts to represent a given bag of nouns as a whole (Li and Abe 1998;Clark and Weir 2002).

such techniques still start with a mapping of each word to all of its immediate concepts.


Given the Table 3 Average accuracies by the network flow method (NF), Manhattan distance (Man), skew divergence (skew div), and Jensen-Shannon divergence (JS) on different profiles: original ("raw"), Li  Our experiments use semantic profiles created directly from the word frequencies, as described earlier.

research has explored the possibility of generalizing this kind of "raw" data to a semantic profile that more appropriately reflects the coherent concepts expressed in the original set of weighted concept nodes.


The experiment revealed that the stories in the corpus contained multiple mentions of characters (on average, 64 mentions per story, excluding pronouns).

the 14 stories contained only 22 location markers, mostly street names.


For example, the nature of events such as dropping, stumbling, or recognizing is that they occur instantaneously and, therefore, are achievements.

events such as playing golf or writing an essay last for some time, so they are processes.


Such situations are referred to as accomplishments.

playing golf or talking on the phone does not imply that the process must end with a specific conclusion and the situation is atelic.


As we suspected, the relationship is not simple or monotonic.

one can identify distinct peaks at 3, 25, and 100-150 semantic classes.


We addressed the normalization problem using a constrained linear solver and the crossentropy problem using numerical optimization.

our experiments showed the difference in WSD performance to be less than 1% in each case.


Using a dictionary in this way provides an objective method for selecting experimental expressions and indicating their gold standard source words.

it results in a data set of blends that are sufficiently established in the language to appear in a dictionary.


We began by using CELEX, because it contains rich phonological information that some of our features draw on.

in our analysis of the results, we noted that for many expressions the correct candidate pair is not in the candidate set.


is much higher than for MAC-CONF (118M vs. 34M).

the average for non-source words in the candidate sets is similar across these data sets (11M vs. 9M).


At first glance, it may appear that this makes the task of source word identification easier for blends, since there is more source word material available to work with.

acronyms have two properties that help in their identification.


Also, decisions on when references are given within the body of a chapter or postponed to end-of-chapter notes are inconsistent, some technical terms (e.g., 'spurious ambiguity') are not explained at the first point at which they are mentioned, and others (e.g., 'gap degree') are not explained at all.

given that these problems are mostly textual and in general do not impede the reader's understanding, this book serves as a very useful and up to date survey of the burgeoning research area of dependency parsing.


The requested origin and destination are used as a filter when selecting the set of available options by querying the database.

the requested arrival or departure time-if specified-is used in the corresponding attribute's evaluation function to give a higher score to flights that are closer to the specified time.


The next-highest-ranked flight is a morning flight, which does not have any attributes that are compellingly better than those of the top choice, and is therefore skipped.

the third option presents an interesting trade-off: even though business class is not available, it is a direct flight, so it is also included.


As an alternative, we could have associated a theme or rheme predication with the edge tones, which would be more in line with Steedman's (2000a) approach.

doing so would make it necessary to include one such predication per phrase in the logical forms, thereby anticipating the desired number of output theme and rheme phrases in the realizer's input.


In one sense, defining a target-cost component to direct the search towards finding suitable intonation is not difficult, as a simple penalty for not matching the target intonation suffices.

standard unit selection techniques only ever take into account local effects, and no provision is made to ensure a suitable global intonation contour.


Here, the APML version was annotated with the target accents, namely L+H* for Lufthansa and none for flight, whereas the ALL version was annotated with a L+H* !H* pattern, which was also considered acceptable.

with the APML version the pitch drops from an f0 value of 293 Hz to 177 Hz, whereas with the ALL version the pitch only drops from 260 Hz to 209 Hz.


Pon- Barry, Weng, and Varges (2006) found that fewer dialogue turns were necessary when the system proactively suggested refinements and relaxations.

as argued in Demberg and Moore (2006), there are several limitations to the SR approach.


's more comprehensive study, their system predicts accent placement (but not type), break indices, and edge tones based on features extracted from the SURGE realizer (Elhadad 1993), deep semantic and discourse features, including semantic type, semantic abnormality and given/new status, and surface features, such as part of speech and word informativeness.

neither of these CTS approaches makes use of the theme/rheme distinction, or the notion of kontrast that stems from Rooth's (1992) work on alternative sets, both of which are crucial to Steedman's (2000a) theory of how information structure constrains prosodic choices.


In an approach that is more similar in spirit to ours, Bulyko and Ostendorf (2002) likewise aim to reproduce distinctive intonational patterns in a limited domain.

unlike our approach, theirs makes use of simple templates for generating paraphrases, as their focus is on how deferring the final choice of wording and prosodic realization to their synthesizer enables them to achieve more natural sounding synthetic speech.


Subsequently,  found correlations between results in MDS spaces and standard mean opinion score (MOS) tests, but as the MOS tests did not correspond to single dimensions in the MDS space, they suggested that it may be possible to design more informative evaluations by asking subjects to specifically rate each factor of interest (e.g., prosody), where each factor relates to one dimension in the MDS space.

as no specific method is suggested to guarantee reliable prosodic judgments from naive listeners, we have left this question for future research, opting instead to augment the listener ratings gathered in our perception experiment with an expert prosody evaluation and an f0 analysis of the theme phrases.


Similarly to what we present in Section 3.1, those authors propose constructing a comparator by using an SVM to compare two sentences or texts with multiple features.

neither further applied this approach to obtain readability assessment based on sorting.


Here, what kinds of corpora the regression method requires in the modern machine learning context has not been clarified because of the lack of previous work in machine learning regression.

our empirical results, shown in Section 8, suggest that the two sets of difficult and easy training data will not be sufficient, and machine learning regression requires texts labeled with scores for different levels.


This must have been due to the different natures of the training and test data.

the texts of the Asahi newspaper articles and TD2-{M,F}-J are controlled under a similar standard (in terms of vocabulary, syntax, and so forth), which would account for the difference from the case in English.


We would like to initialize EM with all the rules that might conceivably be used to explain the training data.

this set is too large to practically enumerate.


No major breakthrough came for speech technology-I am still typing this.

language technology did change almost beyond recognition.


By now, it is generally accepted that the problems that Winograd, Hovy, and others tried to tackle are very complex, and that the current emphasis on more well-delimited problems is probably a good thing.

it is not difficult to come up with computational applications for which a better understanding would be required of language as a process and the effects language may have on a user (interactive virtual agents which try to persuade a user to do something, for example).


If we take empty cepts into account, the product for {(1, 2)} can be rewritten as Similarly, the product for {(1, 2), (2, 2)} now becomes Note that after adding the link (2, 2), the new product still has more factors than the old product.

the new product is not necessarily always smaller than the old one.


We observe that C→E, union, and grow-diag-final weight recall higher because F-measure decreases when α increases.

e→C, intersection, refined method, and Cross-eM weight precision higher.


The original form should be (1 − p 0 ) φ 0 .

this assumption results in a problem for our search algorithm that begins with an empty alignment (see Algorithm 1), for which φ 0 is J and the feature value h m 3 (f, e, a) is negative infinity.


It could be argued that it is premature to survey an area of research that has shown promise but has not yet been tested for a long enough period (and in enough systems).

we believe this argument actually strengthens the motivation for a survey that can encourage the community to use paraphrases by providing an applicationindependent, cohesive, and condensed discussion of data-driven paraphrase generation techniques.


Individual lexical items having the same meaning are usually referred to as lexical paraphrases or, more commonly, synonyms, for example, hot, warm and eat, consume .

lexical paraphrasing cannot be restricted strictly to the concept of synonymy.


For some of the applications we discuss subsequently, the use of paraphrases in the manner described may not yet be the norm.

wherever applicable, we cite recent research that promises gains in performance by using paraphrases for these applications.


We present similar observations in Section 3.5 and highlight that although more recent translation techniquesspecifically ones that use phrases as units of translation-are better suited to the task of generating paraphrases than the competitive linking approach, they continue to suffer from the same problem of low precision.

such techniques can take good advantage of large bilingual corpora and capture a much larger variety of paraphrastic phenomena.


In addition, the paraphrases produced are of better quality than other approaches employing parallel corpora for paraphrase induction discussed so far.

the approach does have a couple of drawbacks: r No paraphrases for unseen data.


In fact, the corpus used (Huang, Graff, and Doddington 2002) also contains, besides the 11 human translations, 6 translations of the same sentence by machine translation systems available on the Internet at the time.

no experiments are performed with the automatic translations.


The idea of enlisting named entities as proxies for detecting semantic equivalence is interesting and has certainly been explored before (see the discussion regarding Paşca and Dienes [2005] in Section 3.2).


As far as the quality of acquired paraphrases is concerned, this approach easily outperforms almost all other sentential paraphrasing approaches surveyed in this article.

a paraphrase is produced only if the incoming sentence matches some existing template, which leads to a strong bias favoring quality over coverage.


Both approaches use word lattices to represent and induce paraphrases since a lattice can efficiently and compactly encode n-gram similarities (sets of shared overlapping word sequences) between a large number of sentences.

the two approaches are also different in that Pang, Knight, and Marcu use the parse trees of all sentences in a cluster to compute the alignment (and build the lattice), whereas Barzilay and Lee use only surface level information.


Note that both of these techniques rely on a secondary language to provide the cues for generating paraphrases in the primary language.

wu and Zhou rely on a pre-compiled bilingual dictionary to discover these cues whereas Bannard and Callison-Burch have an entirely datadriven discovery process.