While these evaluations are our focus, because many more opinion piece than subjective-element data exist, we do evaluate the clues learned from the opinion piece data on the subjective-element data as well.

we cross-validate the results both ways between the two types of annotations.


In fact, as we shall see below, uniqueness in the corpus is an informative feature for subjectivity classification.

we do not want to discard low-frequency clues, because they are a valuable source of information, and we do not want to evaluate individual low-frequency lexical items, because the results would be unreliable.


We did not want to discard all low-frequency words from consideration but cannot effectively judge the suitability of individual words.

to decide whether to retain a word as a PSE, we consider the precision not of the individual word, but of the word together with a cluster of words similar to it.


This hypothesis is realized as follows: Each match of each member of each type of PSE is considered to be a PSE instance.

among them, there are 11 members in PSEinsts for the five phrases sunny, radiant, exhilarating, and splendidly, and is it that, one for each of the matches mentioned above.


Competing versions of the central definitions and claims of the theory have also been proposed: For example, different definitions of backward-looking center (CB) can be found in Weinstein (1983, 1995) and Gordon, Grosz, and Gillion (1993).

a researcher wishing to test the predictions of centering, or to use it for practical applications, is confronted with a large number of possible instantiations of the theory.


But without further specification of these concepts it is impossible to evaluate the claims above, just as it is not possible to evaluate the predictions of, say, "government and binding theory" without providing an explicit definition of "command" or "argument".

a considerable amount of research has been concerned with establishing the best specification for what are, essentially, parameters of the theory.


One exception is that we weren't able to reach acceptable agreement on a feature of NPs often claimed to affect ranking, thematic roles (Sidner 1979;Cote 1988;Stevenson, Crawley, and Kleinman 1994); the agreement value in this case was κ = .35.

we were not able to evaluate ranking functions based on thematic roles.


With indirect realization, 3.4% of utterances violate Rule 1 (GJW 95), as opposed to 2.7% with generalized Suri and direct realization.

rule 1 (GJW 95) (+451, −16) and rule 1 (GJW 83) (+318, −149) are still verified, whereas rule 1 (Gordon et al.)


With simple grammatical function, both the coffers and brass are CBs of (s8), which is therefore treated by our script as not having a well-defined CB.

the pronominalization of a non-CB, premiere partie marquetry, is not counted as a violation.


With the IF and IS instantiations and grammatical-function ranking, we find many more RSH than SSH and many more RET than "pure" CON (i.e., without counting Establishments); indeed, in the IS instantiation with GFTHERELIN ranking, RET are the second most common transition.

rule 2 (BFP) is verified with is instantiations only at the .05 level, and with IF instantiations only if second-person pronouns are counted as realizations of CFs.


Note that when p is a point distribution, H(p) = 0 and hence H(p||q) = D(p||q).

in particular: when, as here, φ x is a point distribution, we can restate the maximum-likelihood criterion as instructing us to choose the model that minimizes the total divergence between the empirical labeling distributions φ x and the model's prediction distributions π x .


Since it is unlabeled at time t, we know that φ (t) x is the uniform distribution u: must also be the uniform distribution; otherwise example x would have been labeled in a previous iteration.

the value of H(x) = H(φ x ||π x ) at the beginning of iteration t is H 0 : After the training step, the value is H 1 : If π x remains unchanged in the training step, then the new distribution π , like the old one, is the uniform distribution, and the example remains unlabeled.


It is not hard to see that this is the update rule that DL-EM-X computes, using the intermediate values: If we consider the case where α is the set of labeled examples and expand φ x in (19), we obtain This is the update rule that DL-EM-Λ computes.

we see that DL-EM-X reduces D X , and DL-EM-Λ reduces D Λ .


First, we assume that syntactic realization serves only to distinguish the Cp from all other referents, which are ranked on the same level: Thus effectively subject > others.

secondly, we assume that the system already knows, from the argument structure of the proposition, which entities can occur in subject position: in realizing a proposition ban(fda, elixir), both arguments are potential Cps because active and passive realizations are both allowed; for contain(elixir, gestodene), only elixir is a potential Cp because we disallow Gestodene is contained by Elixir.


All bilingual phrases that are consistent with the alignment are extracted together with the align- ment within this bilingual phrase.

we obtain a count N(z) of how often an alignment template occurred in the aligned training corpus.


Hence, we also include a dependence on the word positions in the lexicon model p(e | f , i, j): Here, [(i , j) ∈ A] is 1 if (i , j) ∈ A and 0 otherwise.

the word e i depends not only on the aligned French word f j , but also on the number of preceding French words aligned with e i and on the number of the preceding English words aligned with f j .


Hence, there is a data estimation problem especially for words that rarely occur.

we linearly interpolate the models p(e | f ) and p(e | f , i, j).


Note that this tuning depends on the starting point of the model parameters and is not guaranteed to converge to the global optimum on the training data.

this approach is limited to a very small number of model parameters.


This makes the comparison of the probabilities problematic.

we integrate an admissible estimation of the remaining probabilities to arrive at a complete translation (Section 5.6) Many of the other search approaches suggested in the literature do not meet the described aims: • Neither optimal A* search (Och, Ueffing, and Ney 2001) nor optimal integer programming (Germann et al.


Not all feature functions have this nice property: For the conventional lexicon feature function (LEX), we obtain an additional term in our decision rule which depends on the full sentence.

this feature function will not be integrated in the dynamic programming search but instead will be used to rerank the set of candidate translations produced by the search.


we integrate an admissible estimation of the remaining probabilities to arrive at a complete translation.


If we have an estimation r X (j) of the optimal score for translating position j, then the value of the heuristic function R X (n) for a node n can be inferred by summing over the contribution for every position j that is not in the coverage vector c(n) (here X denotes different possibilities to choose the heuristic function): The situation in the case of the alignment template approach is more complicated, as not every word is translated alone, but typically the words are translated in context.

the basic quantity for the heuristic function in the case of the alignment template approach is a function r(Z) that assigns to every alignment template instantiation Z a maximal probability.


So far, in machine translation research there is no generally accepted criterion for the evaluation of experimental results.


Yet long n-grams are seen rarely and are therefore rarely used on unseen data.

we expect that extending the history length will at some point not improve further translation quality.


See Table 4  error seen when removing such published features as the verb-intervening component of the distance metric, which results in an F-measure error increase of 9.86%, or the subcat feature, which results in a 7.62% increase in F-measure error.

33 while the collection of unpublished details presented in Sections 4-7 is disparate, in toto those details are every bit as important to overall parsing performance as certain of the published features.


Compared to introductory (e.g., Hopcroft and Ullman 1979) or advanced (e.g., Martín-Vide, Mitrana, and Pȃun 2004) books on formal language theory, Kracht's book emphasizes those aspects of formal language theory that are relevant to the study of natural languages, whereas the former do not.

kracht's book gives a uniform introduction, which currently does not exist at this level, to an important area of mathematical linguistics.


Figure 7 illustrates the idea, very simplistically.

for the user's input, Who supplies green parts?, suppose we have an initial interpretation, say as a dependency tree.


Each of these structures conveys information, through its form and associated content detail, about what is more or less important in the source text from the relevant point of view, and can therefore be used for summarizing.

the linguistic structure leads to a contrastive summary emphasizing key book-type features (LS), the communicative structure to a justification for action summary (CS), and the world structure to a simple descriptive summary (WS), without the presentational or functional character of the other two.


This has not been just as an exporter of the oftenmisused notions of recall and precision, but through its emphasis on careful methodology and on evaluation geared to the task's function in its context.

retrieval is not about indexing per se, but about delivering relevant documents to users.


Figure 12 sketches, in a very abbreviated form drawn from a fuller illustration, the kind of decompositional analysis required to design and conduct an evaluation of a task system operating in some context, from a particular point of view.

if we imagine having a natural language interface to a house design system, we suppose that we want to discover whether this is effective as a training device for architecture students, and choose comparison with a menu system as the way of doing this.


(1998) show that under the same assumption, minimization of ExpLoss using the feature selection method ensures that the distribution of margins on training data develops in such a way that good generalization performance on test examples is guaranteed.

far in this article we have presented boosting as a feature selection approach.


Note that x i,1 may not be identical to the gold-standard parse; in some cases the parser may fail to propose the correct parse anywhere in its list of candidates.

9 our training data consist of a set of parses, fx i, j : i ¼ 1, : : : , n, j ¼ 1, : : : , n i g, together with scores Scoreðx i, j Þ and log-probabilities Lðx i, j Þ.


Each a i can take any value in the reals.

the ranking function for a parse tree x implied by a parameter vectorā a is defined as Given a new test sentence s, with parses x j for j = 1, : : : , N, the output of the model is the highest-scoring tree under the ranking function arg max Fðx,ā aÞ Fðx,ā aÞ can be interpreted as a measure of how plausible a parse x is, with higher scores meaning that x is more plausible.


The ranking error rate is the number of times a lower-scoring parse is (incorrectly) ranked above the best parse: where again, gpÄ is one if p is true, zero otherwise.

in the ranking problem we define the margin for each example x i,j such that i = 1, : : : , n, j = 2, : : : , n i , as M ij ðā aÞ is the difference in ranking score between the correct parse of a sentence and a competing parse The ranking error is zero if all margins are positive.


It follows that in many cases, the sets A þ k Ã and A À k Ã will be much smaller than the overall size of the training set.

when updating the model fromā a to Updðā a, k Ã , d Ã Þ, the values W þ k and W À k remain unchanged for many features and do not need to be recalculated.


We first define the quantity T as follows:  This is a measure of the number of updates to the W þ k and W À k variables required in making a pass over the entire training set.

it is a measure of the amount of computation that the naive algorithm for ExpLoss, presented in Figure 3, requires for each round of feature selection.


The function in equation (28) can be optimized using variants of gradient descent, which in practice require tens or at most hundreds of passes over the training data (see, e.g., Sha and Pereira 2003).

log-linear models with a Gaussian prior are likely to be comparable in terms of efficiency to the feature selection approach described in this article (in the experimental section, we showed that for the parsereranking task, the efficient boosting algorithm requires computation that is equivalent to around 40 passes over the training data).


Both Della Pietra, Della Pietra, and Lafferty (1997) and McCallum (2003) describe methods that induce conjunctions of ''base'' features, in a way similar to decision tree learners.

a relatively small number of base features can lead to a very large number of possible conjoined features.


The fact that the water is going into the bottle is inherent in the description ''destination''; the preposition merely adds the specific information that the water will end up inside the bottle.

arguments should properly be associated with the NP heads of prepositional phrases.


(wsj_1957) Rush is being affected by a framing decision, currently being reconsidered, wherein rush was taken to mean cause to move quickly.

the entity in motion is tagged Arg1, as in Congress in Congress would have rushed to pass a private relief bill.


The preliminary version of the data used in these experiments was not tagged for WordNet word sense or PropBank frameset.

the system neither predicts the frameset nor uses it as a feature.


In the chunked representation, base-level constituent boundaries and labels are present, but there are no dependencies between constituents, as shown by the following sample sentence: Our chunks were derived from the treebank trees using the conversion described by Tjong Kim Sang and Buchholz (2000).

the experiments were carried out using Table 13 Accuracy of semantic-role prediction for unknown boundaries (the system must identify the correct constituents as arguments and give them the correct roles).


In particular, we assume that two words filling the same semantic requirement share the same contextual word sense.

learning semantic requirements also leads us to induce word senses.


Filtering: Finally, for each aggregation of two similar positions, we select the intersection of their features; that is, the features of a basic cluster are those words that appear in both positions.

of this process, we obtain a set of basic clusters, each augmented by reliable features.


In order to resolve this problem, we use a weight very similar to that proposed in Grefenstette (1994).

we employ, on the one hand, the general structure of the Lin coefficient, and on the other, the weight proposed by Grefenstette.


However, its framework may be responsible for a certain ''lack of consensus,'' in the sense that the chapters are quite independent, and there isn't a clear formulation of the fundamental advances in the field.

argument schemas, which are such an advance and pervade the entire book, are briefly reintroduced in every chapter, instead of being presented in one authoritative chapter.


Furthermore, for the right-hand side a of ρ, inner(a) = 1.

to obtain the required PFA (Σ, Q, q 0 , q f , t, p M ), we now define the probability that such a relative frequency estimator p M minimizes the KL distance between p G and p M on the domain L(M) is proven in the appendix.


The simplified forms of the functions inner and outer from section 3 are commonly called forward and backward, respectively, and they are defined by systems of linear equations.

we can compute exact solutions, as opposed to approximate solutions by iteration.


However, the formal properties of nonlocal MCTAG are not well understood, and it is assumed that the formalism is not polynomially parsable.

this approach is no longer pursued, but it has influenced the different subsequent proposals.


However, the formalism allows one to generate tree descriptions with underspecified dominance relations, and the process of resolving the remaining dominance links is nonlocal.

one may have the same problem as in the case of DSG and V-TAG.


That is, the root node of the derived tree is considered to belong only to the verspricht tree.

an adjunction at that node is an adjunction at the verspricht tree.


However, this standard TAG view is not completely justified: In the derived tree, the root node and the lower S node might as well be considered to belong to reparieren, since they are results of identifying the root node of reparieren with the root and the foot node of verspricht.

5 we propose that the two nodes in question belong to both verspricht and reparieren.


, γ n and β. Consequently, if an elementary tree γ is added to an elementary tree γ, and if there is then a sequence of adjunctions at root or foot nodes starting from γ , then each of these adjunctions can be considered an adjunction at γ, since it takes place at a node shared by γ, γ , and all the subsequently adjoined trees.

one way to define SN-MCTAG refers to the standard TAG derivation tree in the following way.


Secondly, V-TAG is nonlocal in the sense of nonlocal MC-TAG, whereas SN-MCTAG is local, even though the locality is not based on the parent relation in the derivation tree, as is the case in standard local MCTAG, but on the SNdominance relation in the derivation tree.

of the locality, we do not need dominance links (i.e., dominance constraints that have to be satisfied by the derived tree) in SN-MCTAG, in contrast to other TAG variants for scrambling.


The locality condition put on the derivation sufficiently constrains the possibilities for attaching the trees from elementary tree sets: Different trees from a tree set attach to different nodes of the same elementary tree.

the dominance relations among these different nodes determine the dominance relations among the different trees from the tree set.


So the only condition for adding such a tree set is that the verb it depends on has already been added, since the tree of this verb provides the substitution node for the initial tree.

since the lexical material is always left of the foot node, one obtains that Note that in Figure 7, for a scrambled n i , the substitution node is filled with an empty node, while the n is adjoined higher at a node that is not yet available in the elementary structure of v i .


However, it would be desirable to stay inside the class of mildly context-sensitive languages.

in the following, we define a restricted version, RSN-MCTAG, that limits the number of relevant secondary SN-daughters of an elementary tree.


The number of primary SN-daughters of a specific elementary tree is limited, since the primary SN-daughters correspond to substitutions/adjunctions at pairwise different nodes and the number of nodes in an elementary tree is limited.

the number of relevant secondary SN-daughters for a node is limited as well.


A scrambled element always adjoins to a VP node, and the scrambled element is to the left of the foot node.

it precedes everything that is below or on the right of the VP node to which it adjoins.


In example (5), the reparieren-VP is scrambled out of the versuchen-VP, with dem Kunden intervening between the two.

the tree sets are used for reparieren and dem Kunden.


In example (6), we have the same scrambling as in example 4, and additionally, das Fahrrad is scrambled out of the reparieren-VP and the versuchen-VP (depth two).

in this case one needs tree sets for Fahrrad, er, versuchen, and reparieren.


Furthermore, as shown in Boullier (1998b), simple RCGs in particular are even weakly equivalent to linear context-free rewriting systems (Weir 1988).

one obtains that the languages generated by simple RSN-MCTAGs are mildly context-sensitive.


First, more than one tree can be added to a node.


As already mentioned, in an RSN-MCTAG, the number of substitutions and (primary or secondary) adjunctions that can occur at each node is limited (see Lemma 3).

the number of predicates needed in the corresponding RCG is limited as well.


The construction algorithm and a sketch of the proof are presented in the appendix.

of this theorem, the following corollary holds: For a given n, the string languages generated by RSN-MCTAGs of arity n are mildly context-sensitive, and they are in particular polynomially parsable.


The expected net increase in the number of word senses in L D during one step of the process is −t + (1 − t) = 1 − 2t, since the probability that a word sense is lost by obsolescence is t and the probability that a word sense is gained is 1 − t. If r denotes the expected net increase in the number of words in L D during one step of the process, then we must have 1−2t r = m, since m is a constant.

note that the number of words in L D would be constant if and only if t = 0.5.


We should mention in passing that the very exhaustiveness of the OED makes it completely unsuitable (in the present context) as an accurate representation of the English language, since 90% of the senses listed are unknown to the majority of educated native English speakers (Winchester 2003).

our model cannot be expected to provide a faithful prediction of the evolution of the OED, since we assume that the set of word senses in a dictionary is an approximation of those available to people who create new senses for existing words.


The groups marked by the annotators consisted of 536 different discourse segment group types (for example, groups that included the first two discourse segments of each text were marked 31 times by both annotators; groups that included the first three discourse segments of each text were marked 6 times by both annotators).

the confusion matrix had 536 rows and columns.


(1999): Based on the observation that he cannot identify characteristic cue phrases for elaboration relations (e.g., because would be a characteristic cue phrase for cause-effect), Knott argues that elaboration relations are more permissive than other types of coherence relations (e.g., cause-effect, similarity, contrast).

knott argues, elaboration relations would be better described in terms of focus structures (cf.


According to our discourse segmentation guidelines (cf.

section 2.1), 1 and 2 should be one single discourse segment: either same relation between 1 and 2 (cf.


Furthermore, even if one could find a way of augmenting tree structures to account for crossed dependencies and nodes with multiple parents, there would have to be a mechanism for unifying the tree structure with the augmentation features.

in terms of derivational complexity, trees would just shift the burden from having to derive a less constrained data structure to having to derive a unification of trees and features or coindexation.


This correlation is also confirmed by the results of a linear regression (R 2 = 0.9574, p < 10 −4 ; after removing the elaboration data point: R 2 = 0.8146, p < .0005; note that these correlations do not include the data point elaboration + similarity).

although removing certain types of coherence relations (the same ones as for crossed dependencies, i.e., elaboration and similarity; cf.


We hypothesize that the difference in grammatical features such as auxiliaries, number, and tense has a secondary effect when the meaning of sentences is being compared.

we represent in the dependency tree only nonauxiliary words with their associated grammatical features.


This overpruning may happen if either the input to the fusion algorithm is noisy or the alignment has failed to recognize similar subtrees.

we perform a more conservative pruning, deleting only the self-contained components which can be removed without leaving ungrammatical sentences.


Moreover, in some cases substitution is mandatory.

of subtree insertions and deletions, the words used in the basis tree may not be a good choice after the transformations, and the best verbalization might be achieved by using a paraphrase of them from another theme sentence.


Improper placement of punctuation is a limitation of our implementation of the sentence fusion algorithm that we are well aware of.

13 in our grammaticality evaluation (following the DUC procedure), the judge was asked to ignore punctuation.


Using an existing externally produced resource is quicker, but the gold standard may contain many more frames than those which occur in the data from which the test lexicon is induced or, indeed, may omit relevant correct frames contained in the data.

systems generally score better against custom-made, manually established gold standards.


Direct comparison of subcategorization frame acquisition systems is difficult because of variations in the number of frames extracted, the number of test verbs, the gold standards used, the size of the test data, and the level of detail in the subcategorization frames (e.g., whether they are parameterized for specific prepositions).

in order to establish a baseline against which to compare our results, following Schulte in Walde (2002b), we assigned the two most frequent frame types (transitive and intransitive) by default to each verb and compared this "artificial" lexicon to the gold standard.


We conjecture that the main reason for this is that the amended approach to the annotation of obliques is slightly less precise and conservative than the largely -CLR-tag-driven approach taken for Penn-II.

we record an increase in recall and a drop in precision.


Our results show that (a) the knowledge encoded in WordNet is often insufficient, especially for anaphorantecedent relations that exploit subjective or context-dependent knowledge; (b) for otheranaphora, the Web-based method outperforms the WordNet-based method; (c) for definite NP coreference, the Web-based method yields results comparable to those obtained using WordNet over the whole data set and outperforms the WordNet-based method on subsets of the data set; (d) in both case studies, the BNC-based method is worse than the other methods because of data sparseness.

in our studies, the Web-based method alleviated the lexical knowledge gap often encountered in anaphora resolution and handled examples with context-dependent relations between anaphor and antecedent.


For the examples above, we need the knowledge that magazines are periodicals, that hoods are parts of jackets, that costs can be or can be viewed as repercussions of an event, and that institutional homes are facilities.

many resolution systems that handle these phenomena (Vieira and Poesio 2000;Harabagiu, Bunescu, and Maiorano 2001;Ng and Cardie 2002b;Modjeska 2002;Gardent, Manuelian, and Kow 2003, among others) rely on hand-crafted resources of lexico-semantic knowledge, such as the WordNet lexical hierarchy (Fellbaum 1998).


We follow the rationale that the most frequent of these instantiated patterns determines the most likely antecedent.

should the head noun of an antecedent candidate and the anaphor co-occur in a pattern although they do not stand in the lexical relationship considered (because of pattern ambiguity, noise in the corpus, or spurious occurrences), this need not prove a problem as long as the correct antecedent candidate co-occurs more frequently with the anaphor.


As the patterns can be elaborate, most manually controlled and linguistically processed corpora are too small to determine the pattern frequencies reliably.

the size of the corpora used in some previous approaches leads to data sparseness (Berland and Charniak 1999), and the extraction procedure can therefore require extensive smoothing.


Therefore, the size of the corpora used in some previous approaches leads to data sparseness (Berland and Charniak 1999), and the extraction procedure can therefore require extensive smoothing.

as a further extension, we suggest using the largest corpus available, the Web, in the above procedure.


21 Because each noun is a synonym of itself, anaphors in StrSet v1 /StrSet v2 that do have a string-matching antecedent candidate will already be covered by the WordNet look-up prior to back-off in almost all cases: Back-off string matching will take effect only if the anaphor/antecedent head noun is not in WordNet at all.

the described back-off will most of the time just amount to a recency back-off.


In our scheme we extract the bankruptcy code as anaphoric but our method of extracting candidate antecedents does not include bankruptcy-law.

there are four anaphors in our data set with no correct/lenient antecedent extracted.


Typical examples involve named-entity recognition and inverted queries.

in Example (15), the anaphor the union is coreferent with the first occurrence of the union, a case easily resolved by string matching.


Indeed, one of the problems in comparing algorithm results for coreference is that such a large number of anaphors are covered by simple string matching, leaving only a small data set on which the lexical methods can differ.

strset v2n contains 331 of 477 cases (268 of which are assigned correctly by baselinestr v2n ), so that improvements by the other methods are confined to the set of the remaining 146 anaphors.


(16) Conseco Inc., in a move to generate about $200 million in tax deductions, said it induced five of its top executives to exercise stock options to purchase about 3.6 million common shares of the financial-services concern.


Of course, in Spoken Dialogue Technology, Michael McTear will not give an answer to the question of whether talking to domestic appliances makes sense, but he indicates that even a normal household, for instance, may offer a wide field of application for spokenlanguage dialogue systems in the near future.

his book primarily focuses on theory and practice of these systems.


Any agent that passes the Turing Test has conversational verbal behavior indistinguishable from that of humans.

any agent that passes the Turing Test is intelligent.


However, high performance in one application area is not necessarily correlated with high performance in another application area (Weeds and Weir 2003a).

it is not clear that the same characteristics that make a distributional similarity measure useful in one application will make it useful in another.


However, since the use of the CR framework requires parameter optimization, here, we randomly select 60% of the nouns to form a development set and use the remaining 40% as a test set.

any parameters are optimized over the development set nouns and performance measured at these settings over the test set.


First, the differences in mean similarities over the development set and the test set are minimal.

performance of the models with respect to different parameter settings appears stable across different words.


The major difference appears to be the use of MI, and hence we predicted that there would be a close relationship between sim lin and the harmonic mean in the additive MI-based CRM.

this relationship is shown below: Now if tP I(w 1 , c) = tP I(w 2 , c), it follows: when the additive MI-based model is used, γ = 1 and the condition tP I(w 1 , c) = tP I(w 2 , c) holds, the CR framework reduces to sim lin .


Each of n's neighbors, m, is given a vote that is equal to the difference in frequencies of the co-occurrences (m, v 1 ) and (m, v 2 ) and that it casts to the verb with which it occurs most frequently.

we distinguish between cases where a neighbor occurs with each verb approximately the same number of times and where a neighbor occurs with one verb significantly more often than the other.


High-precision neighbors were more useful in the language modeling task of pseudo-disambiguation and high-recall neighbors were more highly correlated with WordNet-derived neighbor sets.

similarity appears to be inherently asymmetric.


Traditional sentence alignment algorithms (Gale and Church 1991;Wu 1994;Fung and Church 1994;Melamed 1999;Moore 2002) are designed to align sentences in parallel corpora and operate on the assumption that there are no reorderings and only limited insertions and deletions between the two renderings of a parallel document.

they perform poorly on comparable, non-parallel texts.


According to the ME principle, the optimal parametric form of the model of our data, taking into account the constraints imposed by the feature functions, is a log linear combination of these functions.

for our classification problem, we have: where c i is the class (c 0 ="parallel", c 1 ="not parallel"), Z(sp) is a normalization factor, and f ij are the feature functions (indexed both by class and by feature).


According to this model, given foreign sentence ( f j 1<=j<=m ), English sentence (e i1<=i<=l ), and translation probabilities t( f j |e i ), the best alignment f → e is obtained by linking each foreign word f j to its most likely English translation argmax e i t( f j |e i ).

each foreign word is aligned to exactly one English word (or to a special NULL token).


Due to its simplicity, this model has several shortcomings, some more structural than others (see Moore [2004] for a discussion).

we use a version that is augmented with two simple heuristics that attempt to alleviate some of these shortcomings.


Note that in each of these experiments the only resource used by our system is the initial, out-of-domain parallel corpus.

the experiments fit in the framework of interest described in Section 1, which assumes the availability of (limited amounts of) out-of-domain training data and (large amounts of) in-domain comparable data.


Values above 0.5 indicate that the classifier considers the particular sentence pair to be parallel; the higher the value, the higher the classifier's confidence.

in order to obtain higher precision, we can choose to define as parallel only those pairs for which the classifier probability is above a certain threshold.


Our main goal is to extract, from an in-domain comparable corpus, parallel training data that improves the performance of an out-of-domain-trained SMT system.

we evaluate our extracted corpora by showing that adding them to the out-of-domain training data of a baseline MT system improves its performance.


For each initial corpus, we use the same amount of human-translated data as there is extracted data (see Table 2).

for each language pair and each initial parallel corpus, we compare 3 MT systems: Baseline, PlusExtracted, and UpperBound.


Alignments computed using this model and a noisy, automatically learned, dictionary will contain many incorrect links.

if two sentences share several content words, these incorrect links together with the correct links between the to each other, and according to our dictionary, the Arabic word for "Mexico" can be translated as any of the country names listed in the example.


Moreover, comparable article pairs, even those similar in content, may exhibit great differences at the sentence level (reorderings, additions, etc).

they pose hard problems for the dynamic programming alignment approach.


Moreover, out-of-domain corpora introduce additional difficulties related to limited dictionary coverage.

the ability to reliably judge sentence pairs in isolation is crucial.


In all these numbers, care should be taken to note that the humans were instructed to produce phrase alignments only when word alignments were impossible.

it is entirely likely that summary word i is aligned to document word j and summary word i + 1 is aligned to document word j + 1, in which case we count this as two singleton alignments, rather than an alignment of length two.


There is also a null state for each document position r ∅,i .

the output alphabet consists of each word found in S, plus the end-of-sentence word ω.


When transitioning from the start state p to state r i,i , the value we use is a jump length of i.

if we begin at the first word in the document, we incur a transition probability of j1.


Unfortunately, in such model selection problems, likelihood increases monotonically with model complexity.

eM will find for us the most complex model; in our case, this will correspond to a model in which the entire summary is produced at once, and no generalization will be possible.


For example, automatic speech recognition (ASR) systems prefer longer "words" to achieve higher accuracy, whereas in-formation retrieval (IR) systems prefer shorter "words" to obtain higher recall rates (Wu 2003).

we do not assume that an application-independent universal word segmentation standard exists.


(1999) do not use any linguistic knowledge.

the identified OOV words are likely to be linguistically implausible, and consequently, additional manual checking is needed for some subsequent tasks such as parsing.


nk-blind gives a more representative performance measure by taking into account multiple judges.


As mentioned in Section 1, MSRSeg is designed as an adaptive segmenter that consists of two components: (1) a generic segmenter that can adapt to different domain vocabularies, and (2) a set of output adaptors, learned from application data, for adapting to different application-specific standards.

we evaluate MSRSeg using five corpora, each corresponding to a different standard, and consistent train-test splits, as shown in Table 4.


We notice in Table 4 that the adaptation data sets (i.e., training corpora for the four Bakeoff standards) are much smaller than the MSR training set.

the experimental setting is a good simulation of the adaptation paradigm described above.


Second, the output adaptors of MSRSeg can automatically pick different subsets (i.e., segmentation units) from the superset according to different applications.

there are two criteria for evaluating an adaptive segmenter: how complete the superset is and how effective the adaptation is.


Though the revision clarifies the definition in principle, it requires a judgment of the syntactic and semantic sense of the segmentation-a task where an agreement cannot be reached easily among different human annotators.

we only use the CAS measure in a pilot study.


is not differentiable with respect to λ λ λ, and there are many local minima on the error surface.

we use an alternative loss function, minimum squared error (MSE) in equation 5, where Score(.)


Fortunately, there are only a few hundred Chinese characters that are particularly common in transliterations.

an FN candidate would be generated given s if it contains only characters stored in a transliterated name character list (which contains 618 Chinese characters).


Given an input sentence s, all word candidates are ranked by their normalized class model score λf (.).

the number of candidates (i.e., the size of the lattice) is controlled by two parameters: r Number threshold: The maximum number of candidates cannot be larger than a given threshold; r Score threshold: The difference between the class model score of the top-ranked candidates and the bottom-ranked candidates cannot be larger than a given threshold.


Worse still, some LNs tagged in one system are tagged as ONs in another system.


Unfortunately, because of the sheer number of methods measuring similarity, as well as those measuring distance as the "opposite" of similarity, this would have made for an awkward presentation.

we have to ask the reader to rely on context when interpreting what exactly the expressions semantic distance, semantically distant, and semantically close mean in each particular case.


In mathematical terms, for any concept c in the taxonomy, let p(c) be the probability of encountering an instance of concept c. Following the standard definition from information theory, the information content of c, IC(c), is then − log p(c).

we can define the semantic similarity of a pair of concepts c 1 and c 2 , as Notice that p is monotonic as one moves up the taxonomy: if c 1 IS-A c 2 then p(c 1 ) ≤ p(c 2 ).


For example, whenever we encounter a nickel, we have encountered a coin (Figure 1), so p(nickel) ≤ p(coin).

the higher the position of the most-specific subsumer for given two concepts in the taxonomy (i.e., the more abstract it is), the lower their similarity.


Formally, where W(c) is the set of words (nouns) in the corpus whose senses are subsumed by concept c, and N is the total number of word (noun) tokens in the corpus that are also present in WordNet.

resnik's approach attempts to deal with the problem of varying link distances (see Section 2.5) by generally downplaying the role of network edges in the determination of the degree of semantic proximity: Edges are used solely for locating superordinates of a pair of concepts; in particular, the number of links does not figure in any of the formulas pertaining to the method; and numerical evidence comes from corpus statistics, which are associated with nodes.


We focus on measures that use WordNet (Fellbaum 1998) as their knowledge source (to keep that as a constant) and that permit straightforward implementation as functions in a programming language.

we select the following five measures: Hirst and St-Onge's (Section 2.4), Jiang and Conrath's (Section 2.6.2), Leacock and Chodorow's (Section 2.5.3), Lin's (Section 2.6.3), and Resnik's (Section 2.6.1).


Precision (P S ), recall (R S ), and F-measure (F S ) for malapropism suspicion by measure and scope.

the Jiang-Conrath measure does best for the suspicion phase (and is optimal with scope = 5).


Doing just this in FSRA is insufficient because using the same registers might cause undesired effects: The result might be affected by the content left in the registers after dealing with a substring from L(A 1 ).

this basic construction is used with care.


Notice that it cannot be based on the first alternative (adding registers) due to the fact that the number of iterations in Kleene star is not limited, and therefore the number of registers needed cannot be bounded.

the second alternative is used: Register operations are added to delete the content of registers.


The number of possible sequences of register operations along an -loop is unbounded, but it is easy to prove that there are only finitely many equivalence classes of such sequences: Two sequences are in the same equivalence class if and only if they have the same effect on the state of the machine; since each machine has a finite number of configurations (see theorem 1), there are only finitely many such equivalence classes.

the basic idea behind the construction is as follows: If there exists an -path from q 1 to q 2 with the register operations a over its arcs, and an arc (q 2 , σ, b , q 3 ) Figure 9 removal paradigm.


The formal construction is similar in spirit to the -removal paradigm in weighted automata (Mohri 2000), where weights along an -path need to be gathered.

we suppress the formal construction and the proof of its correctness.


For example, the series of register operations (W, 1, a), (R, 1, a), (W, 1, b) is equal in its effect to the series (W, 1, b) .

we show an algorithm for optimizing a given FSRA by minimizing the series of register operations over its arcs and removing redundant arcs and states.


Generally, a FSRA is linearized if it is optimized, -free, and given a current state and a new input symbol, and at most one transition can be traversed.

if the transition relation includes two arcs of the form (q, σ, a , q 1 ), (q, σ, b , q 2 ), then a and b must be a contradicting series of register operations.


Evidently, given an FSRA A, it can be verified in polynomial time that A is linearized.


By the definition of linearized FSRA, A does not contain -arcs.

∈ L(A ) iff the initial state of A is also a final one.


Notice that the different symbols in L n have no significance except the pattern they create.

fSRAs are extended in order to be able to identify a pattern without actually distinguishing between different symbols in it.


For instance, genitives can encode relations such as PART-WHOLE  MAKE-PRODUCE (Mary's novel -if Mary wrote it).

any attempt to interpret genitive constructions has to deal with the semantic analysis of the two noun constituents.


Then, we retrieved all the sentences that contained the two words in at least one of the target patterns.

we obtained sentences containing the pair of words linked by patterns such as door of car, car's door, car has door, car with four doors, car door, etc.


In the genitive cluster, for example, there were 18,936 such pattern instances, of which 325 encoded part-whole relations, while 18,611 did not.

for the genitive cluster we used a training corpus of 27,961 positive examples (325 pairs of concepts in a part-whole relation extracted from corpus "A" and 27,636 extracted from WordNet as selected pairs) and 18,611 negative examples (the non-part-whole relations extracted from corpus "A").


Since the two nodes have opposite default values, the default value (D A j ) needs to be used when the rules for the child node (A j ) do not hold.

a new rule, specific to the example A j , needs to be created, for handling all the instances of A j : if A j then R A j ∪ D A j .


Its only ambiguous node (node 1.4.2.1.2.2 : writing#2-writing#2) has the default value Yes.

a specific rule (Rule 2.1.1 from Table 12) needs to be created for the example (Part=writing#2 and Whole=writing#2), Step 4.


Unfortunately, in our LA Times testing corpus we couldn't find more than ten parts for each of their proposed whole objects.

we are unable to replicate their work using our text collection.


Our classification rule learning approach is based on the WordNet semantic classes of the two concepts that represent the part and the whole, respectively.

if the WSD system fails to annotate the concepts with the correct senses, the ISS system can generate wrong semantic classes, which leads to wrong conclusions.


For example, the concepts import and export are not listed in WordNet as denoting the act of importing/exporting commodities from a foreign country.

relations such as import of sweater and export of milk are mis-classified.


If a certain class of negative or positive examples is not seen in the training data (and therefore it is not captured by the classification rules), the system cannot classify its instances.

the larger and more diverse the training data, the better the classification rules.


However, York#1 is defined in WordNet as the House of York, the English royal house that reigned from 1461 to 1485.

the ISS system will consider York#1 a group instead of an entity, yielding an erroneous result.


In addition, we aimed at a (rather linguistically than technically based) choice of selectional preferences that represents a useful compromise for the conceptual needs of the verb classes.

this choice of features best utilizes the meaning-behavior relationship and will be applied in a largescale clustering experiment (cf.


The verbs in each verb class show similarities with respect to their argument structure, in that they all can be used both as transitives and intransitives.

argument structure alone does not distinguish the classes, and subcategorization information is refined by thematic relations.


To make this possible, we have to use inequalities, that is, Values of the form '> α' or '< α', instead of Values of the form '= α'.

we compile the KB into a more elaborate form by replacing equalities by inequalities of the form size(x) > α or size(x) < α.


The problem at hand suggests a simpler approach that will always prefer logically stronger inequalities over logically weaker ones, even when they do not remove more distractors.

5 (size(x) > m is preferred over size(x) > n iff m > n; conversely, size(x) < m is preferred over size(x) < n iff m < n.) This is reflected by the order in which the properties are listed above: Once a sizerelated property is selected, later size-related properties do not remove any distractors and will therefore not be included in the description.


Assuming the analysis of Section 2.1, the n large mouse/mice is semantically equivalent to the n largest mouse/mice.

there is no need to distinguish between the two at the level of CD.


(This comes down to counting the number of elements in the extension of the inequality.)

if the number of inequalities in the description is n i then the complexity is O(n d n i ).


Therefore, if the number of inequalities in the description is n i then the complexity is O(n d n i ).

the complexity of GRE in the gradable case is determined by three steps: The first is quadratic and can be performed off-line, the second has a worst-case running time of O(n v n a ), and the third one has a worst-case running time of O(n d n i ).


Thus, the complexity of GRE in the gradable case is determined by three steps: The first is quadratic and can be performed off-line, the second has a worst-case running time of O(n v n a ), and the third one has a worst-case running time of O(n d n i ).

gradable GRE takes only polynomial time, and if we focus on the part that cannot be done off-line, it takes only linear time.


Once again, the most time-consuming part of the calculation can be performed off-line, since it is the same for all referring expressions.

the worst-case time complexity of interpretation is as follows: The part that can be computed off-line takes O(n d log n d ) calculations.


The last two are grouped together in a class ATTITUDE-STYLE DISTINCTIONS because they are expressed by similar syntactic constructions in the text of CTRW.

the algorithm to be described in Section 2.2 will treat them together.


From 60 randomly selected sentences, 52 were correctly dealt with (41 needed no split, 11 were correctly split).


Unlike most other NLG systems, HALogen already incorporates some collocational knowledge implicitly encoded in its language model (bigrams or trigrams), but this is mainly knowledge of collocations between content words and function words.

in its integration into Xenon, the collocational knowledge acquired in Section 3 will be useful, as it includes collocations between near-synonyms and other nearby content words.


In such cases, Xenon cannot be expected to make the correct choice, or, more precisely, the other choices are equally correct, at least as far as Xenon's LKB is concerned.

the Test data set used in Experiment 1 (English only) and Experiment 2 (English and French).


Sometimes more than one translation of a French near-synonym could be correct, but in this conservative evaluation, the solution is the near-synonym that was used in the equivalent English sentence.

test cases that would be considered correct by a human judge are harshly penalized.


Nevertheless, the accuracy of the prediction US for US is indicated by the q ratio: The definitions 2and 6and the equality (4) lead to the equality which shows that as soon as US > US 0 2 the prediction for the normalized values is (usually by a large margin) not as good as the prediction for the unnormalized values.

after predicting the normalized user-satisfaction values, these values should be transformed back to the original scale to guarantee more accurate predictions.


Moreover, some individual usersatisfaction measures could not be well modeled.

we think that it would be more appropriate to take the sum of the user-satisfaction scores that are likely to measure the selected aspect (e.g., dialogue-manager performance) of the system's performance.


Only if the users are disencumbered from speech-recognition errors are they able to reliably assess the mutual influence of the observable less significant contributors to their satisfaction with the dialogue manager's performance.

in our WOZ experiments (Section 2), which were carried out in order to evaluate the performance of the dialogue manager, speech understanding (i.e., speech recognition and natural-language understanding) was performed by a human.


This case is diametrically opposite in that here WordNet encodes much information about both senses, but such "noisy" knowledge does not help discriminate.

a number of semantic interconnection patterns are presented to the validator, indicating the relevance of both senses for tagging, but no evidence in favor of the choice of sense #1 (which is most appropriate in the sentence).


Furthermore, a small number of texts found in the Web is obtained via optical character recognition (OCR), which may again lead to garbled words.

of these and other error sources, the Web contains a considerable number of "bad" pages with language material that is inappropriate for corpus construction.


We applied each rule to D(English), introducing one error at the first possible position, for each entry of the appropriate form.

we obtained a list with 1,223,128 garbled strings.


Since arguments are obligatory complements of a verb, whereas adjuncts are not, we expect knowledge of a given verb to be more informative with respect to the probability of existence of an argument than of an adjunct.

we expect that the predictive power of a verb with regard to its complements will be greater for arguments than for adjuncts.


To be able to use the relative clause construction, there must be two properties that are being predicated of the same entity.

the probability that a PP is able to be paraphrased, and therefore that it is an adjunct, can be approximated by the probability of its occurrence following a construction headed by a copular verb, be, become, appear, seem, remain (Quirk et al.


Since WordNet has a much finer-grained top-level classification for nouns than for verbs, we found that grouping head nouns into classes yielded useful generalizations, but it did not do so for verbs.

we calculate head dependency in three different variants: One measure is based on PP-internal noun tokens, another variant is based on noun classes for the PPinternal noun position, and another variant is based on classes for both the PP-internal and the head noun position, as indicated in equation 8.


Performance measures on this set will support comparison across methodologies.

we conclude that the mapping we have assumed is coherent with the judgments of a native speaker, although the agreement is not perfect.


For automatically labeled data, diagnostics based on linguistic theory improve the performance even further.

the hypotheses we were testing with these experiments are confirmed.


(1989) suggest that the assessment of similarity in semantic networks can in fact be thought of as involving just taxonomic (IS-A) links, to the exclusion of other link types; that view will also be taken here, although admittedly it excludes some potentially useful information.

semantic similarity is a specific type of attributional similarity.


In the text "food at the plant," the relation strongly suggests that the plant is an industrial plant, since living organisms are not usually considered as locations.

an algorithm for classifying semantic relations (as in Section 7) should be helpful for word sense disambiguation.


Turney and Littman (2005) used the AltaVista search engine to obtain the frequency information required to build vectors for the VSM.

their corpus was the set of all Web pages indexed by AltaVista.


If a phrase is n words long, there are n − 2 intervening words between the members of the given word pair (e.g., between quart and volume).

a phrase with n words generates 2 (n−2) patterns.


P = "in" P = "* of" P = "of *" P = "* *" freq("quart P volume") 4 1 5 19 freq("volume P quart") 10 0 2 16 more weight to columns (patterns) with frequencies that vary substantially from one row (word pair) to the next, and less weight to columns that are uniform.

we weight the cell x i,j by w j = 1 − H j / log(m), which varies from 0 when p i,j is uniform to 1 when entropy is minimal.


The matrix is designed so that the row vector for A:B is different from the row vector for B:A only by a permutation of the elements.

the same permutation distinguishes the row vectors for C:D and D:C. the cosine of the angle between A:B and C:D must be identical to the cosine of the angle between B:A and D:C (see equation 7).


There is no row for B:A unless B:A also appears in the input set.

the number of rows in the matrix dropped from 17,232 to 8,616.


In step 4, we selected the top 8,000 patterns (instead of the top 4,000), distinguishing the pattern "word 1 P word 2 " from the pattern "word 2 P word 1 " (instead of considering them equivalent).

a pattern P with a high frequency is likely to appear in two columns, in both possible orders, but a lower frequency pattern might appear in only one column, in only one possible order.


For example, in user initiative mode, the system allows the user to specify any number of attributes in a single utterance.

the system will let the user ignore specific questions.


Above we showed that corrections were significantly longer, louder, higher in pitch excursion, and followed longer pauses than other turns.

we expected that these features would be useful in identifying corrections automatically.


Note also that prediction from raw prosodic features alone (19.68%) is not improved by the inclusion of their normalized versions (PROS, DIA = all, error = 20.33%).

aSRderived features and prosodic features seem to provide equally successful classifications of user corrections.


The major concern is to explicate, as clearly as possible, the themes that have shaped research on temporal language, and following the publication timeline is not the best way to do this.

although the collection opens in 1967 (with Vendler), the second paper fast-forwards the reader to 1991 (James Pustejovsky on event structure), the third takes us to 1986 (Emmon Bach on the algebra of events), and the fourth catapults us all the way back to 1947 (Reichenbach), after which we wend our way forward (with the occasional temporal bounce) toward more recent work.


The units of the system were understood as constituting hierarchies in which some of them function as certain parts of the others.

for example, phonemes were defined and delimited one against the other on a functional basis (two different phonemes can distinguish two morphemes), and the established repertory of distinctive features gave a firm foundation to the description of the system of phonemes as a structured whole.


annotation does not "spoil" a corpus, because the annotation is done "in addition" to the raw corpus.

on the contrary, annotation may and should bring an additional value to the corpus.


For each text, an order was randomly sampled from each bin.

our set of materials consisted of 8 × 8 = 64 texts.


It simply assumes that all token-final periods in a test corpus represent sentence boundaries.


Evaluations in all three experimental settings are performed with respect to the same standard system configuration, which is defined in terms of the following parameters: In the three experimental settings considered, which are presented in the following subsections, a total of seven different system configurations are evaluated in both translation directions, English to Spanish and Spanish to English.

a total of 14 different translation experiments are performed.


Another important result is that the quality of Spanish-to-English translations is significantly and consistently better than those obtained in English-to-Spanish translations.

significant efforts should be dedicated towards properly exploiting morphological analysis and synthesis methods for improving English-to-Spanish translation quality.


Apart from the more practical issues, the book also explores the suitability of the chosen learning paradigm, memory-based learning (Stanfill and Waltz 1986), for NLP problems.

the book is a valuable source of information for a wide range of readers from the linguist interested in applying machine-learning techniques or the machine-learning specialist with no prior experience in NLP to the expert in machine learning wanting to learn more about the appropriateness of the MBLP bias for NLP problems.


The posterior probability of a target word e can depend on its occurrence in position i of the target sentence, for example, or on the number of times the word is contained in the sentence.

several different definitions of posterior probabilities will be introduced and investigated in the following discussion.


For this purpose, it is compared to a threshold t. All words that have confidence above this threshold are tagged as correct, and all others are tagged as incorrect, translations.

the binary classifier is defined as The threshold t is optimized on a distinct development set beforehand.


If two hypotheses represent the same information with respect to translation and language models, they will be assigned the same probabilities by these models in the future.

the outcome of the search is not altered, but the processing time can be significantly decreased if only the more promising of the two hypotheses is considered for further expansion.


For a given source sentence f J 1 and a target word e, we want to determine a sort of marginal probability Q(e, f J 1 ).

we extract all source phrases f j+s j that occur in the given source sentence f J 1 .


We modified this confidence measure because we found that the average lexicon probability used there is dominated by the maximum.

we determine the maximal translation probability of the target word e over the source sentence words: where f 0 is the "empty" source word (Brown et al.


The nature of a particular restricted domain affects the kinds of questions asked and answers that can be expected.

different restricted domains benefit from different QA techniques.


The choice of domain affects the kind of users to target.

for each domain it is important to determine the kinds of questions asked in the specific domain (question style and terminology used are two important factors to consider), the sort of information that is most commonly requested, and the level of detail expected in the answers.


The actual domain provides a specific context to the question-answering process.

the set of senses available to words is typically a subset of all the available senses.


In some domains, however, these terms may be used widely.

open-domain ontologies will need to be complemented with terminology lists or local ontologies.


Co-occurring problems must be taken into consideration in the differential diagnosis and cause determination search tasks because knowledge of the problems is typically incomplete in these scenarios.

physicians would normally be interested in any problems mentioned in the abstracts in addition to the primary problem specified in the query frame.


8 Clinical question answering is an emerging area of research that has only recently begun to receive serious attention.

there exist relatively few points of comparison to our own work, as the research space is sparsely populated.


The Copy operation here applies to the actual instance selected: It does not create a new instance of the same type.

after the Paste operation, the doctor instance fills two slots, both the actor and the actee of the event.


However, the configuration of the search space is such that most nodes are either leaves or very close to leaves, and only a few nodes have a large number of descendants.

root pruning mostly removes very small subtrees and has only a small effect on the overall cost of the algorithm.


Our strategy in the present study was to define a small number of generic parameterizations, evaluate the resulting models on a development set, and select a broadly optimal model for testing on unseen data.

our models were not specifically tuned for the tasks at hand and we have only explored a relatively small subset of the parameter space.


The model is formally a synchronous context-free grammar but is learned from a parallel text without any syntactic annotations.

it can be seen as combining fundamental ideas from both syntax-based translation and phrase-based translation.


Following Och and Ney (2002), we depart from the traditional noisy-channel approach and use a more general log-linear model over derivations D: where the φ i are features defined on derivations and the λ i are feature weights.

one of the features is an m-gram language model P LM (e); the remainder of the features we will define as products of functions on the rules used in a derivation: we can rewrite P(D) as The factors other than the language model factor can be put into a particularly convenient form.


This function induces a weight function over derivations: If we define then the probability model becomes It is easy to write dynamic-programming algorithms to find the highest-weight translation or k-best translations with a weighted synchronous CFG.

it is problematic that w(D) does not include the language model, which is extremely important for translation quality.


Then, for each cell, we throw out any item that has a score worse than: r β plus the best score in the same cell, or r the score of the bth best item in the same cell.

in the +LM parser, the score of an item [X, i, j; e] in the chart does not reflect the LM probability of generating the first (m − 1) words of e. two items [X, i, j; e] and [X, i, j; e ] are not directly comparable.


In actuality, the grid may be a cube (one dimension for rules and two dimensions for two nonterminals) with up to b 3 elements, whereas the target chart cell can hold at most b items (where b is the limit on the size of the cell imposed during pruning).

the vast majority of computed items are pruned.


These phrases are combined using the glue rules only, which makes the grammar equivalent to a conventional phrase-based model with monotone translation.

this system represents the nearest phrase-based equivalent to our model, to provide a controlled test of the effect of hierarchical phrases.


When relations (i.e., two-place properties) are taken into account at all (e.g., Dale and Haddock 1991;Krahmer and Theune 2002), the motivating examples are kept so small that it is reasonable to assume that speaker and hearer know all the relevant facts in advance.

search is not much of an issue (i.e., resolution is easy): The hearer can identify the referent by simply intersecting the denotations of the properties in the description, for example, intersecting the set of girls with the set of individuals who wear a yellow dress (both in the domain).


A j }), the description (x, P), even if minimally distinguishing, cannot lead to DE or LO.

(x, P) can be uttered in any position d within the subtree denoted by SCOPE(x, {A 1 , .


We would like to show that the decisions taken by FI and SL are sensible, that is, that they produce descriptions that are neither too short nor too long.

: S1: We want to consider situations in which FI and SL have produced an incomplete description, and investigate how much gain could have been made by using a complete description in those cases.


Understanding language is one of the great challenges of science, and languagerelated technology is one of the great opportunities of Information Technology.

many different kinds of researchers work on language issues.


However, if we substitute the relevant values into the formula for AER (Equation (3)), we see that 1 − AER for both of the hypothesized alignments is 0.5.

aER does not share the property of F-Measure (with α = 0.5) that unbalanced precision and recall are penalized.


After applying the random permutation, document IDs will be uniformly random.

we can construct the random sample by picking any D s documents.


We find it convenient, as an informal practical metaphor, to think of the large box as physical memory.

the sketches reside in physical memory, and the rest are paged out to disk.


We calculate, from sketches K 1 and K 2 , the sample inner product a s,(1,2) = |{6}| = 1, and the corresponding corpus sample size, denoted by D s,(1,2) = min(max(K 1 ), max(K 2 )) = min(11, 12) = 11.


Simpler data models often result in simpler estimation methods but with larger errors.

it is obvious that our proposed method has smaller estimator errors.


Broder's method was designed to estimate resemblance.

this section will compare the proposed method with Broder's sketches in terms of resemblance, R. We can compute R from our estimated associationâ MLE : R MLE is slightly biased.


Without function composition, the category of regularly would have to be ((S\NP)/NP)\((S\NP)/NP), but (crossed) composition allows the ordinary category (S\NP)\(S\NP) to also work in this case.

if the parent (and head) category C is of the form X/$, the algorithm strips off all outermost forward arguments /$ (and syntactic features) from C to obtain C .


Particle-verb constructions are difficult to identify in the Treebank, because particles can be found as PRT, ADVP-CLR, and ADVP.

verbs in the CCGbank grammar do not subcategorize for particles, which are instead treated as adverbial modifiers.


This eschews the computational problem that a grammar for compound nouns induces all possible binary bracketings, but is linguistically incorrect.

a similar problem arises in compound nouns that involve internal coordination: (NP (NN cotton) (CC and) (NN acetate) (NNS fibers)) We include the following (linguistically incorrect) rule in our grammar, which yields a default dependency structure corresponding to N/N coordination: This rule allows us to translate the above tree as follows: our current grammar does not distinguish between appositives and NP coordination, even though appositives should be analyzed as predicative modifiers.


Because this distinction requires manual inspection on a caseby-case basis, we were unable to modify the Treebank analysis.

all CCGbank relative pronouns have categories of the form (NP i \NP i )/(S/NP i ), rather than (N i \N i )/(S/NP i ).


The inclusion of sufficient instances of the lesser frequent sluice types would have involved selecting a much larger sample.

it was decided to abstract over the true frequencies to create a balanced sample whose size was manageable enough to make the manual annotation feasible.


In the three baselines considered, Short Answer acts as the default category.

even though the recall is high (given that Short Answer is the class with the highest probability), precision tends to be quite low.


Just as in computation in general, process is not the mere implementation of some non-process account of something, but supplies the motivational account.

in computational linguistics, the mechanisms for selecting word senses provide interpretations for what it is for a word to have a sense in relation to a linguistic structure, and for what a linguistic structure is when built by disambiguation.


They still are, but that's no skin off our noses: We have more interesting things to do.

while we may believe that computational linguistics continues to develop views of language that are worthwhile (whether because valid or just stimulating), as in thinking about process, and that linguistics ought to take notice, computational linguistics is solidly growing in its own right.


The oracle approach to PP attachment differs from many other tasks in NLP like part-of-speech tagging or parsing, in which usually no data based on manual annotation are part of the input.

published evaluation results for tagging and parsing accurately reflect the performance we would expect in an application whereas PP reattachment results arguably do not.


There are many more things that can go wrong in a long natural sentence than in a 5-word artificial one.

the parser recognizes the PP ambiguity less often in this experiment without an oracle than in Experiment 1, where an oracle was available.


The relative scores of trees with the same yield are unaffected, because they are all scaled equally.

g Θ defines the same conditional distribution over trees given sentences as g Θ , which implies that g Θ and g Θ have the same highest scoring parses.


Given a fixed set of productions, G is the set of WCFGs with exactly those productions (i.e., they vary only in the production weights), G Z<∞ is the subset of G that defines (joint) probability distributions over trees (i.e., that have a finite partition function Z) and P Z<∞ is the set of probability distributions defined by grammars in G Z<∞ .

chi (1999) and Abney, McAllester, and Pereira (1999) proved that P Z<∞ is the same as P PcFG , the set of probability distributions defined by the PcFG G PcFG with the same productions as G. even though the set of WcFGs properly includes the set of PcFGs, WcFGs define exactly the same probability distributions over trees as PcFGs.


One possible way to increase the accuracy of LTAG supertagging is to use a Maximum Entropy, rather than HMM, tagger (as discussed previously), but this is likely to result in an improvement of only a few percentage points.

whether the difference in supertagging accuracy is due to the nature of the formalisms, the supertagging methods used, or properties of the extracted grammars, is an open question.


In order to distinguish different argument slots, the arguments are numbered from left to right.

the subject relation for a transitive verb is represented as (S\NP 1 )/NP 2 , 1 .


The expression can be expanded further: The reason for this manipulation is that the expected recall score for π is now written in terms of a sum over the individual dependencies in π, rather than a sum over each dependency structure for S. The inner sum is over all derivations which contain a particular individual dependency τ.

the final score for a dependency structure π is a sum of the scores for each dependency τ in π; and the score for a dependency τ is the sum of the probabilities of those derivations producing τ.


We can measure the remaining differences as follows: Comparing the CCGbank dependencies in Section 00 against those generated by running our parser over the derivations in 00 gives labeled precision and recall values of 99.80% and 99.18%, respectively.

there are a small number of dependencies in CCGbank which the current version of the parser can never get right.


For the CCG parser, the accuracy did not degrade when using the new adaptive parsing strategy.

the accuracy and efficiency of the parser were not tuned separately: The configuration used to obtain the speed results was also used to obtain the accuracy results in Sections 10.2 and 11.


9 Treating the DepBank GRs as a gold standard, and comparing these with the CCGbank GRs, gave precision and recall scores of only 76.23% and 79.56%, respectively.

given the current mapping, the perfect CCGbank parser would achieve an F-score of only 77.86% when evaluated against DepBank.


However, in order that the parser evaluation be as fair and informative as possible, it is important that the parser output conform as closely to the gold standard as possible.

it is appropriate to use any general transformation rules, as long as they are simple and not specific to the test set, to achieve this.


For example, if an ncmod GR is incorrectly labeled with xmod, but is otherwise correct, it will be correct for all levels which subsume both ncmod and xmod, for example mod.

the scores at the most general level in the GR hierarchy (dependent) correspond to unlabeled accuracy scores.


(For example, the table indicates that 12.9% of nouns in the Senseval-2 data, and 16.7% in Senseval-3, have five or fewer occurrences in SemCor.)

although SemCor may cover many frequently occurring word types in a given document, there are likely to be a substantial proportion for which there is very little or no information available.


For many words, and particularly for nouns, there is very little or no data in SemCor; Table 2 shows that 81.9% of nouns (excluding multiwords) listed in WordNet do not occur at all in SemCor.

even for English, which has substantial manually sense-tagged resources, coverage is severely limited for many words.


Nugues is discussing the fact that, if you sample a corpus for n-grams, some will not occur in your sample at all, but it would be a mistake to consider the unseen ones to be infinitely rare (frequency 0).

the counts need to be adjusted: Good-Turing estimation.


Although the point of departure for the book was the ELSNET summer school held in 2002, the decision to make a book was made considerably later.

the work on the chapters was only initiated in 2004.


For instance, in some Centering papers, entities are ranked with respect to their grammatical function (Brennan, Friedman, and Pollard 1987;Walker, Iida, and Cote 1994;Grosz, Joshi, and Weinstein 1995), and in others with respect to their position in Prince's (1981) givenness hierarchy (Strube and Hahn 1999) or their thematic role (Sidner 1979).

two "instantiations" of the same theory make different predictions for the same input.


From a design viewpoint, we emphasize automatic computation for both the underlying discourse representation and the inference procedure.

our work is complementary to computational models developed on manually annotated data (Miltsakaki and Kukich 2000;Hasler 2004;Poesio et al.


Because we are interested in an automatically constructed model, we have to take into account computational and learning issues when considering alternative representations.

our exploration of the parameter space is guided by three considerations: the linguistic importance of a parameter, the accuracy of its automatic computation, and the size of the resulting feature space.


Because existing methods for coreference resolution take into consideration the order of noun phrases in a text, the accuracy of these tools on permuted sentence sequences is close to random.

we opt to resolve coreference within the original text.


Because human summaries often have identical (high) scores, we eliminated pairs of such summaries from the training set.

the resulting training corpus consisted of 144 summaries.


For example, difficult texts tend to have twice as many entities as easy ones.

easy and difficult texts are represented by entity transition sequences with different probabilities (e.g., the sequences [S S] and [S O] are more probable in difficult texts).


A dynamic programming algorithm is proposed for maximum entropy estimation without unpacking feature forests.

probabilistic modeling of any data structures is possible when they are represented by feature forests.


Generally, a feature forest represents an exponential number of trees with a polynomial number of nodes.

complete structures, such as tag sequences and parse trees with ambiguities, can be represented in a tractable form.


As noted in this section, the number of nodes in a feature forest is usually polynomial even when that of the unpacked trees is exponential.

we can efficiently compute model expectations with polynomial computational complexity.


The difference between the two approaches is that feature forests use a simpler generic data structure to represent packed forest structures.

without assuming what feature forests represent, our algorithm can be applied to various tasks, including theirs.


computer science is continually subjected to governmental reviews seeking assurance that we know what we are doing, and are doing enough for the economy.


Third, the problem of ambiguity for grammars of this size was much worse than anyone had expected.

opinion changed, and MT became seen as one of the hardest cognitive problems, requiring deep understanding of semantics and knowledge of the world, as well as full syntactic processing.


Moreover, some simple rules can be used to join constituents and fix a considerable portion of the mismatches (Toutanova, Haghighi, and Manning 2005).

it has become a common practice to use full parse trees as the main source for solving SRL.


Note that these measures do not exclude the core arguments but instead consider the core plus a coarse version of the modifier arguments.

for COARSEARGM ALL we count {0} as a true positive span, {1, 2} , {3, 4}, and {7, 8, 9} as false positive, and {1, 2, 3, 4} and {7, 8, 9} as false negative.


The post-processing rule is the following: For every constituent that bears a core argument label ARGX, if there is a preceding constituent with the same label, re-label the current constituent C-ARGX.

according to our algorithm, all constituents having the same core argument label are part of the same argument, and all constituents having the same modifier labels are separate arguments by themselves.


Training a model which has such a huge number of classes is infeasible if the model does not factorize due to strong independence assumptions.

in order to be able to incorporate long-range dependencies in our models, we chose to adopt a re-ranking approach (Collins 2000), which selects from likely assignments generated by a model which makes stronger independence assumptions.


If an argument does not correspond to any constituent in a parse tree, or a constituent exists but is not attached or labeled correctly, our model will have a very hard time guessing the correct labeling.

if the syntactic parser makes errors, these errors influence directly the semantic role labeling system.


The rule for deciding when to add continuation labels was the same as for our systems for the February 2004 data described in Section 4.3: A constituent label becomes continuing if and only if it is a core argument label and there is another constituent with the same core argument label to the left.

for the CoNLL 2005 shared task we employ the same semantic role labeling system, just using a different post-processing rule to map to CoNLL-style labelings of sets of words.


Although the literature on SRL is extensive, there is almost no study of the use of tree kernels for its solution.

the reported research is mainly based on diverse natural language learning problems tackled by means of tree kernels.


Regardless of the adopted feature space, our multiclassification approach suffers from the problem of selecting both boundaries and argument roles independently of the whole structures.

it is possible that (a) two labeled nodes refer to the same arguments (node overlaps) and (b) invalid role sequences are generated (e. g., Arg0, Arg0, Arg0, .


Indeed, we selected the best alternative produced by the Viterbi algorithm according to the goldstandard score, and we obtained an F1 of 84.76 for n = 5.

the critical aspect resides in the selection of the best annotations, which should be carried out by an automatic re-ranker.


For example, ("this") (CL) ("book") plays the same semantic role even though it is the subject in " /this /CL /book /publish /AS" and the object in " /this /CL /publishing /house /publish /ASP /this /CL /book."

each verb will belong to a class with a symbol representing each of the three dimensions.


When a verb is less polysemous, the arguments of the verb tend to be realized in a more uniform manner in syntax.

the argument labels are easier to predict from their structure.


The underlying system architecture can largely affect the outcome of our study.

to make the conclusions of our experimental study as applicable as possible to general SRL systems, the architecture of our SRL system follows the most widely used two-step design.


Specifically, our goal is to investigate at what stage full parsing information is most helpful relative to a shallow parsingbased system.

our experiments were designed so that the compared systems are as similar as possible, and the addition of the full parse tree-based features is the only difference.


When only shallow parsing is available, the system does not have a pruning stage, and also does not have constituents to begin with.

conceptually, the system has to consider all possible subsequences (i.e., consecutive words) in a sentence as potential argument candidates.


The recall and precision of the  full parsing-based system are around 2 to 3 percentage points higher than the shallow parsing-based system on the gold-standard data.

the F 1 score is 2.5 percentage points higher.


As before, in order to compare the systems that use full parsing and shallow parsing information, we need to see the impact on the overall performance.

we built two semantic role systems based on full parsing and shallow parsing information.


Precision is improved by 1 to 2 percentage points but recall is decreased a little.

the gain in F 1 is about 0.5 to 1 point.


Their shallow parse-based system achieved 60.4% precision and 51.4% recall and their full system achieved 80.6% precision and 67.1% recall on the same data set (but 84% precision and 75% recall with the full data set).

due to the use of different architectures and data set sizes, the questions of "how much one can gain from full parsing over shallow parsing when using the full PropBank data set" and "what are the sources of the performance gain" were left open.


Following Cohen (1988), for a five-way distinction Aron and Aron (page 527) would consider 0.06 to represent a small real relationship-that is, one with not much effect-and 0.3, a large one.

we describe 0.06 as "weak," 0.45 as "very strong," and intermediate points as "moderate" and "strong.


The single-candidate model, which only considers the candidates of an anaphor in isolation, is incapable of effectively capturing the preference relationship between candidates for its training.

the learned classifier cannot produce reliable results for preference determination during resolution.


The singlecandidate model assumes that the probability that C k is the antecedent is only dependent on the anaphor ana and C k , and independent of all the other candidates.

that is: the probability of a candidate C k being the antecedent can be approximated using the classification result on the instance describing the anaphor and C k alone.


Simply looking at a candidate alone cannot explain this, and may possibly result in inconsistent training instances (i.e., the same feature vector but different class labels).

the confidence values returned by the learned classifier cannot reliably reflect the preference relationship between candidates.


Formally, the model considers the probability that a candidate is the antecedent as the probability that the candidate is preferred over all the other competing candidates.

that is: Assuming that the preference between C k and C i is independent of the preference between C k and the candidates other than C i , we have: : this suggests that the probability that a candidate C k is the antecedent can be estimated using the classification results on the set of instances describing C k and each of the other competing candidates.


The Tournament Elimination scheme has a computational complexity of O(N), where N is the number of the candidates.

it enables a relatively large number of candidates to be processed.


One possible reason could be that the classifier in our systems can make a correct preference judgement (with accuracy above 92% as in our test) in the cases where one candidate is the antecedent and the other is not.

the simple linear search can find the final antecedent as well as the Round Robin method.


This is due to the fact that the number of features in the twincandidate model is nearly double that in the single-candidate model.

the twin-candidate model requires more training data than the single-candidate model to avoid the data sparseness problem.


The baseline system with the single-candidate model tends to yield higher recall while the system with the twin-candidate model tends to produce higher precision.

a fairer comparison of the two systems is to examine the precision rates that these systems achieve under the same recall rates.


In this article, we focus on dependency-based parsing of Turkish, a language that is characterized by rich agglutinative morphology, free constituent order, and predominantly head-final syntactic constructions.

turkish can be viewed as the representative of a class of languages that are very different from English and most other languages that have been studied in the parsing literature.


r The first set includes issues relating to the treatment of morphology in syntactic parsing, which becomes crucial when dealing with languages where the most important clues to syntactic functions are often found in the morphology rather than in word order patterns.

for Turkish, it has previously been shown that parsing accuracy can be improved by taking morphologically defined units rather than word forms as the basic units of syntactic structure .


It should also be noted that the IG-based model improves not only the AS U accuracy but also the word-to-word accuracy compared, to the word-based models.

the IG-based model not only helps to recover the relations between correct IGs but also to find the correct head word.


When we investigate these dependencies, we see that 70% of them are connected to a head adjacent to the dependent and the parser finds 90% of these dependencies correctly.

we can say that the parser has no problem in finding adjacent head-initial dependencies.


For example, suppose we want to transform NP(DET, N) (but not, say, NP(ADJ, N)) into the tree N using rules in T. Although this is a simple xT rule, the closest we can get with T would be q NP(x0, x1) → q.N x1, but we cannot check both subtrees without emitting two independent subtrees in the output (which rules out producing just N).

xT is a bit more powerful than T. Derivation trees for a transducer M = (Σ, ∆, Q, Q i , R) are T R (trees labeled by rules) isomorphic to complete leftmost M-derivations.


We denote tied rules by marking each rule in the same tied class with the symbol @ and a common integer.

the JJ(JJ NN) and NN(JJ NN) reordering rules described previously are modified as follows: All reordering rules with the same input and output variable sequence are in the same tied class, and thus receive the same probability, independent of their parent symbols.


On top of that, the script dictates that many particles, including four of the most frequent prepositions, the definite article, the coordinating conjunction, and some subordinating conjunctions, all attach to the words which immediately follow them.

a form such as mhgr can be read as a lexeme ('immigrant'), as m-hgr ('from Hagar'), or even as m-h-gr ('from the foreigner').


The actual values that the function returns were chosen empirically by counting the number of occurrences of each class in the training data.

"good" candidates make up 74.26% of the data, hence the value the function returns for "good" roots is set to 0.7426.


This also requires learning as much as possible from the initial set of data.

in this article we investigate using a fixed corpus of dialogues to automatically optimize dialogue systems.


The difficulty is that, at any point in the dialogue, the system cannot be sure what will happen in the remainder of the dialogue, and thus cannot be sure what effect its actions will have on the total reward at the end of the dialogue.

the system must choose an action based on the average reward it has observed previously after it has performed that action in states similar to the current one.


In particular, if u is very low, then maximizing Q hybrid (s, a) is equivalent to maximizing S data (s, a).

a very low u is equivalent to the policy that always chooses the most probable action, which we will call the "SL policy."


We also deduct 1 point for each action performed by the system, to penalize longer dialogues.

the maximum possible score is 197 (i.e., 200 minus 3 system actions: ask for all the user information in one action, then confirm all the four slots in one action and offer a flight).


To allow direct comparisons between the different values of U and UO, these scores are plotted against the proportion of decisions that are different from that which the pure SL policy would choose.

the SL policy (average reward 139.8) is plotted at 0 (which is Average dialogue score plotted against the proportion of decisions that diverge from the SL policy, for different values of the unobservable state reward U and reward offset UO.


This can be explained by the fact that the SL policy is the result of merging all policies of the COMMUNICATOR systems.

it can be thought of as a form of multi-version system, where decisions are made based on what the majority of systems would do.


They were also designed or trained to complete the task, rather than to fill the slots.

the evaluation does not reflect all the capabilities or behaviors of the systems.


Both the stack and the buffer will be represented as lists, although the stack will have its head (or top) to the right for reasons of perspicuity.

σ|i represents a stack with top i and tail σ, and j|β represents a buffer with head j and tail β.


The arc-eager parser differs from the arc-standard one by attaching right dependents (using RIGHT-ARC e l transitions) as soon as possible, that is, before the right dependent has found all its right dependents.

the RIGHT-ARC e l transitions cannot replace the head-dependent structure with the head, as in the arc-standard system, but must store both the head and the dependent on the stack for further processing.


The list λ 1 has its head to the right and stores nodes in descending order, and the list λ 2 has its head to the left and stores nodes in ascending order.

λ 1 |i represents a list with head i and tail λ 1 , whereas j|λ 2 represents a list with head j and tail λ 2 .


Finally, if p has a head k in G x , then k must be in λ c q and any j such that j ∈ λ c q and k < j must either be a dependent of p in G x or must have a head to the left in both G x and G x (anything else would again be inconsistent with the assumption that G x is projective).

we can construct a transition sequence C 0,m such that G c m = G x , by starting in c 0 = c s (x) and applying exactly the same q transitions as in C 0,q , followed by as many LEFT-ARC p l transitions as there are left dependents of p in G x , interleaving NO-ARC p transitions whenever the node at the head of λ 1 already has a head, followed by a RIGHT-ARC p l transition if p has a head in G x .


The result for Chinese is expected, given that it is the only data set that does not contain any non-projective dependencies, but the difference in accuracy is very slight (0.19 percentage points).

it seems that the non-projective parser can also be used without loss in accuracy for languages with very few non-projective structures.


The difference between the coefficients is only in the interpretation of "chance agreement": π-style coefficients calculate the chance of agreement among arbitrary coders, whereas κ-style coefficients calculate the chance of agreement among the coders who produced the reliability data.

the choice of coefficient should not depend on the magnitude of the divergence between the coders, but rather on the desired interpretation of chance agreement.


All of these tests were based on the same two assumptions: that every unit (utterance) is assigned to exactly one category (dialogue act), and that these categories are distinct.

again, unweighted measures, and in particular K, tend to be used for measuring inter-coder agreement.


We found that even though our coders by and large agreed on the interpretation of anaphoric expressions, virtually no coder ever identified all the mentions of a discourse entity.

even though the values of α and K obtained by using the ID of the antecedent as label were pretty similar, the values obtained when using anaphoric chains as labels were drastically different.


Second, it is difficult to replicate results presented in previous work because there is no standard corpus, and no standard evaluation methodology.

comparisons across systems are few and far between.


However, because of the pruning generally needed to render the problem computationally feasible, the resulting word graph only represents a subset of the possible translations.

it may happen that the user sets prefixes which cannot be found in the word graph.


However, the problem is that in many cases the sentence hypotheses in the n-best list differ in only one or two words.

we introduce the additional requirement that the first four words of the nbest hypotheses must be different.


It may be the case that a user prefix ends in an incomplete word during the interactive translation process.

it is necessary to start the translation completion with a word whose prefix matches this unfinished word.


In translation from a source language to a target language, there are some words which are translated identically (because they have the same spelling in both languages).

we identify them in the corpus and replace them with some generic tags to help the translation system.


Post-processing takes place after the translation in order to hide the internal representation of the text from the user.

the user will only work with an output which is very similar to human-generated texts.


benefits of the n-best increased accuracy.

this feature should be offered to the users as an option.


Our main result is that the simplest metric (which relies exclusively on NOCB transitions) sets a baseline that cannot be outperformed by other metrics that make use of additional centering-related features.

we provide substantial insight into the role of centering as an information ordering constraint and offer researchers working on text generation a simple, yet robust, baseline to use against their own information ordering approaches during system development.


In fact, because M.BFP fails to outperform the baseline, the latter may be considered as the most promising solution for information ordering in these cases too by applying Occam's razor.

m.NOCB is shown to be the best performing metric across all four corpora.


The meaning of an ERC is that at least one constraint whose corresponding coordinate contains a W outranks all of the constraints whose coordinates contain L's.

w, e, L, L means that C 1 outranks both C 3 and C 4 , while L, L, w, w means that either C 3 or C 4 outranks both C 1 and C 2 .


However, because there is no ranking of which the members of an inconsistent ERC set are all true, Ω is not shatterable because there is no ranking of which the ERCs in Γ are true while the ERCs in Ω − Γ are false.

consistency under partial negation is both necessary and sufficient for shatterability.


However, this would make it impossible to associate a ranking with at least one partitioning of S into accepted and rejected subsets contra the assumption that S is shatterable.

if C j is a coordinate in S − {x} that is not W-unique, then, for every partial exclusion that rejects F, under each partitioning of S − {x} into T and F, there is at least one other coordinate C i that fuses to W. C j could be removed from CON while preserving the shatterability of S − {x}.