In (Qiu and Frei, 1993), it is observed that one of the reasons is that one tried to determine expansion terms according to each original query term separately, which may introduce much noise.

they proposed to determine the expansion terms by summing up the relations of a candidate expansion term to each of the query terms.


this model is indeed a biterm model (Srikanth and Srihari, 2002), i.e., we allow two terms be separated within some distance.


In a query, if two words are strongly associated, this also means that their association is more meaningful to the query, thus should be weighted higher.

a natural way to assign a probability to a biterm in the query is to use mutual information, which denotes the strength of association between two words.


If it is negative, we consider that the biterm is not meaningful, and is ignored.

we arrive at the following probability function: means all the meaningful biterms in the query.


It turns out that the best weighting is the one with MI.

in the next section, we will only report the results with the second option.


However, neither this difference nor the denominator in (4) affects the classification.

replacing the true probabilities with their estimates, we can define the classifier whereθ denotes the parameters of the probabilistic model learned from the training data.


Assume that two occurrences of entity pairs with similar contexts, are tend to hold the same relation type.

unsupervised relation extraction problem can be formulated as partitioning collections of entity pairs into clusters according to the similarity of contexts, with each cluster containing only entity pairs labeled by the same relation type.


Each row of Y corresponds to a context vector of entity pair, and the above process can be considered as transforming the original context vectors in a d-dimensional space to new context vectors in the K-dimensional space.

the rows of Y will cluster upon mutually orthogonal points on the K dimensional sphere,rather than on the coordinate axes.


As the rows of the K eigenvectors clustered along mutually orthogonal vectors, their projections in a lower dimensional space will cluster along radial directions.

the general picture will be of q clusters elongated in the radial direction, with possibly some clusters very near the origin (when the subspace is orthogonal to some of the discarded eigenvectors).


In particular, it is important to incorporate lower-order information based on subsets of the conditioning information.

we assume a structural form of the conditional probability tables which implements a more sophisticated type of smoothing -interpolated Witten-Bell (Witten and Bell, 1991).


Starting from the current BN, the set of next candidate structures is defined as follows: For every unplaced variable R in REMAINING, and for every subset Sub of size at most two from the already placed variables in CHOSEN, consider adding R with parents Sub ∪Y to the current BN.

the number of candidate structures for extending a current BN is on the order of m 3 , where m is the number of variables.


Note that our feature selection algorithm adds an input variable or a variable conjunction with all of its possible values in a single step of the search.

we are adding hundreds or thousands of binary features at each step, as opposed to only one as in (Della Pietra et al., 1997).


3 The highest reported result for independent classification of core arguments is 96.0% for a log-linear model using more than 20 additional basic features (Toutanova et al., 2005).

our resulting models with 93.5% and 93.9% accuracy compare favorably to the SVM model with polynomial kernel and show the importance of structure learning.


For discriminative log-linear model structure learning: (i) the gaussian prior was fixed to be very weak, serving only to keep the weights away from infinity (σ = 100) and (ii) the conjunction selection was restricted to correspond to a Bayes Net structure with no features for subsets of feature conjunctions.

the only difference between the class of our modified discriminative log-linear models and the class of models considered in (Pernkopf and Bilmes, 2005;Grossman and Domingos, 2004) is that we do not restrict the parameters to be interpretable as probabilities.


We represent C ij by a maximum-entropy classifier over the binary variable S ij , which is true if and only if field set F i belongs to the same record as field set F j .

we model the conditional distribution where f k is a binary feature function that computes attributes over the field sets, and Λ = {λ k } is the set of real-valued weights that are the parameters of the maximum-entropy model.


For example, in Korean, the Hangul-Jamo decomposition leads to fairly redundant strings of Korean characters thereby making transliterated characters to be relatively far from each other.

korean requires a larger relative distance bound.


The structure recognition problem dealt with here is closely related to the field of linear text segmentation with the goal to partition text into coherent blocks, but on a single level.

our task generalizes linear text segmentation to multiple levels.


Assuming an undirected graphical model G over an observed entity x and a set of discrete, interdependent random variables 4 y, a conditional random field describes the conditional distribution: The normalization term Z(x) sums over all possible joint outcomes of y, i.e.,  In practice, for efficiency reasons, independence assumptions have to be made about variables y ∈ y, so G is restricted to small cliques (say, (|c| ≤ 3).

the sufficient statistics only depend on a limited number of variables y c ⊆ y; they can, however, access the whole observed entity x.


It should be noted that the labels are unequally distributed, especially in chain 0 (there are very few BEGIN labels).

the baseline is substantially high for this chain, and other measures may be better suited for evaluating segmentation quality (cf.


We have shown that segmentation results improve when fewer errors are present in the automatically generated annotation.

manual correction of the reference annotation will yield further improvements.


If the relation denoted by R is functional, then typically the object y is a function of the subject x.

our discussion focuses on this possibility, though the analysis is easily extended to the symmetric case.


being the negative lookahead operator) returns matches of regex R b but only if they do not match R a .

by replacing R 5 1 in our original regex with R 5 1b =(?!


We present a generative model of how a given annotator, knowing the true θ, stochastically chooses rationales.

observing the rationales helps us infer the true θ.


The crucial point is that the second factor depends on θ (since r i is supposed to reflect the relation between x i and y i that is modeled by θ).

the learner has an incentive to modify θ in a way that increases the second factor, even if this somewhat decreases the first factor on training data.


However, crucially, our true goal is not to predict the rationales but to recover the classifier parameters θ.

if great tends to be highlighted, then the model should not be permitted to explain this directly by increasing some feature φ great , but only indirectly by increasing θ great .


We discuss an interesting characteristic of ρ(w): if D is large enough, then, with high probability, ρ(w) values are greater for topical words than for non-topical words.

ρ(w) can be used as a mean to measure the topicality of w. Obviously, the quality of this measure depends on the quality of estimating q(w), i.e.


The proposition states that in corpora of practical size 4 the set of topical words can be almost perfectly detected, simply by taking words with the highest ρ ratios.

the core D k will consist of k documents, each of which contains more topical words than any document from D \ D k .


This is likely to be the case when richer representations as suggested above are used, necessitating a return to Gibbs sampling.

in future work we plan to experiment with classifiers whose decision function is cheaper to compute, such as neural networks and decision trees.


Because exact probabilistic inference is #Pcomplete, HOLMES uses approximate methods, but even these techniques have problems if the Markov network gets too large.

hOLMES creates the network incrementally.


However, the source-side of a rule usually corresponds to multiple target-sides in multiple rules.

during decoding, the decoder should select a correct target-side for a source-side.


During decoding, when a rule is selected and applied to a source text, both lexical translations (for terminals) and reorderings (for nonterminals) are determined.

rule selection affects both lexical translation and phrase reordering.


For each ambiguous source-side of translation rules, a maximum entropy based rule selection (MERS) model is built.

the MERS models can help the decoder to perform a context-dependent rule selection.


We consider the rule selection task as a multi-class classification task: for a source syntactic tree T , each corresponding target string e is a label.

during decoding, when a TAT T , e is selected, T is classified into label e , actually.


The same source sub-tree may have different parent nodes in different training examples.

this feature may provide information for distinguishing source sub-trees.


Since the MERS models are only built for ambiguous source trees, the first feature P rs ( e| T , T (X k )) for non-ambiguous source tree will be set to 1.0.

the decoder will prefer to use non-ambiguous TATs.


However, nonambiguous TATs usually occur only once in the training corpus, which are not reliable.

we use this feature to reward ambiguous TATs.


This is because one of the strengths of the maximum entropy model is that it can incorporate various features to perform classification.

using all features defined in Section 3.2, we obtain statistically significant improvements (the last row of Table 4).


Then, this raw similarity measure is transformed to a surface similarity score between 0 and 1 through an exponential mapping,   i .

and ρ is a smoothing factor that characterizes the mapping, as ρ approaches infinity, p e e  backs off to the exact match model.


Both LMP-and LCS-based methods achieve similar performance but the computation of LMP is faster.

we only report results of the LMP-based smoothed similarity model.


However, when searching for the optimal alignment, the TER-based method uses a strict surface hard match for counting edits.

it is not able to handle synonym matching well.


Beside the word posteriors, we use language model scores and a word count as features for confusion network decoding.

for an M-way system combination that uses N LMs, a total of M+N+1 decoding parameters, including M-1 system weights, one rank smoothing factor, N language model weights, and one weight for the word count feature, are optimized using Powell's method (Brent, 1973) to maximize BLEU score on a development set 4 .


No MT outputs on our dev set are available from them at present.

we directly adopt system combination parameters trained for the previous 8way system combination, except the system weights, which are re-set by the following heuristics: First, the total system weight mass 1.0 is evenly divided among the three groups of single systems: {Sys-1~8}, {Sys-9~12}, and {Sys-13~15}.


However, the subsequent classification cannot provide constraint and correction to the results of the preceding step, which will lead to the accumulation and propagation of the classification errors.

the performance of sentiment analysis of sentences is often not satisfactory.


That is because the redundancy between sentimental strength labels makes the classifier confused to determine the polarity of the sentence.

we should deal with the sentiment analysis in a hierarchical frame which can consider the redundancy between the different classes and make full use of the subjective and polarity information implicitly contained in sentimental strength classes.


For the transition feature, the same strategy is used.

the complexity of its is the number of the training samples, N is the average sentence length, j F is the average number of activated features in the j th layer, l is the number of the original labels and m is the number of the layers.


In the different layers of sentiment analysis task, the granularities of classification are different.

when an observation cannot be correctly labeled on a smaller-granularity label set, our method will use the larger-granularity labels in the upper layer to control the final classification labels.


The reason lies in that: the reviewers often follow some writing patterns, for example some reviewers prefer to concede an opposite factor before expressing his/her real sentiment.

we divide a review into five parts, and assign each sentence with the serial number of the part which the sentence belongs to.


Data Statistics for Movies Reviews Corpus There is a problem in the dataset that more than 70% of the sentences are labeled as "Neu" and labels are seriously unbalanced.


Further training of a machine learning algorithm, as in experiments two and three, is required in order to build a subjectivity analysis tool.

this fourth experiment is an evaluation of the resources generated in the target language, which represents an upper bound on the performance of any machine learning algorithm that would be trained on these resources.


As English is a language with fewer inflections when compared to Romanian, which accommodates for gender and case as a suffix to the base form of a word, the automatic translation into English is closer to a human translation (experiment three).

labeling this data using the OpinionFinder tool and projecting the labels onto a fully inflected humangenerated Romanian text provides more accurate classification results, as compared to a setup where the training is carried out on machine-translated Romanian text (experiment two).


Intuitively, it is better to provide a ranking of emotions according to their popularity rather than associating a single reader emotion with a document.

current writer-emotion analysis techniques for classifying a document into a single emotion category are not suitable for analyzing reader emotions.


The coefficient for the data in Figure 7 is 0.726 with p-value < 0.01.

pairwise emotion classification accuracy reflects the emotional ambiguity experienced by readers.


In the figure, the error generated by EDR is less than those by NN, and all the differences are statistically significant with pvalue < 0.01.

eDR's use of regression leads to better estimation of f i than the NN.


We will show how BP's "message-passing" discipline offers a principled way for higher-order features to incrementally adjust the numerical edge weights that are fed to a fast first-order parser.

the first-order parser is influenced by higher-order interactions among edges-but not asymptotically slowed down by considering the interactions itself.


BP's behavior in our setup can be understood intuitively as follows.

inasmuch as the first-order parser finds that edge e is probable, the higher-order features will kick in and discourage other edges e to the extent that they prefer not to coexist with e. 2 the next call to the first-order parser assigns lower probabilities to parses that contain these e .


The binary factor PAIR ij,k fires with some value iff L ij and L k are both true.

it penalizes or rewards a pair of links for being simultaneously present.


We perform these combinatorial sums by calling a first-order parsing algorithm, with edge weightsq ij .

as outlined in §2, a first-order parser is called each time we propagate through the global TREE or PTREE constraint, using edge weights that include the first-order LINK factors but also multiply in any current messages from higher-order factors.


5 The CoNLL-X shared task actually involves thirteen languages; our experiments do not include Czech (the largest dataset), due to time constraints.

the average results plotted in the last rows of Tables 3, 4, and 5 are not directly comparable with previously published averages over thirteen languages.


Apparently, the number of constituents produced in parsing is not affected by for-statement implementations.

we assume that the running time of CKY is primarily determined by the number of constituents generated in parsing.


The difference merely lies on what intermediate ICs are used.

given a grammar and an input sentence, no matter what binarization is used, the CKY parsing will generate the same set of CCs.


We call such an IC failed, otherwise it is successful.

if a binarization can help generate fewer failed ICs then parsing would be more efficient.


If ctr is zero, those ICs are surely failed.

the smaller num(w) is and the larger ctr(w) is, the better w would be.


The key fact in lexicalized parsers is that we cannot explicitly write down all rules and compute their probabilities precisely, due to the great number of rules and the severe data sparsity problem.

in Collins (1997) grammar rules are already factorized into a set of probabilities.


This method exhibits problems, because input sentences very often overlap and complement each other at the same time.

there is a trade-off between non-redundancy and completeness of the output.


Barzilay & McKeown (2005) present a sentence fusion method for multi-document news summarization which crucially relies on the assumption that information appearing in many sources is important.

their method produces an intersection of input sentences by, first, finding the centroid of the input, second, augmenting it with information from other sentences and, finally, pruning a predefined set of constituents (e.g.


Obviously, even a superbly written scientific paper will not be perceived as very readable by a lay person and a great novel might not be appreciated by a third grader.

the vast majority of prior work on readability deals with labeling texts with the appropriate school grade level.


1 One of the selected articles was missing from the Penn Treebank.

results that do not require syntactic information (Tables 1, 2, 4, and 6) are over all thirty articles, while Tables 3, 5, and 7 report results for the twenty-nine articles with Treebank parse trees.


Because the CCG labels for a given sentence are so specific, many times there are no matches.

we also investigated a looser constraint.


The most important result from this evaluation is that both our training regimes realise substantial gains from the introduction of the language model feature.

we can be confident that our model is capable of modelling the distribution over translations even when the space over all derivations is intractable to dynamically program exactly.


Due to their co-occurring with the lamp object, words dresser and lamp are both likely to be associated with the lamp object in the translation models.

word dresser is likely to be incorrectly acquired for the lamp object.


For example, the knowledge that the word lamp is more semantically related to the object lamp can help the system avoid associat-ing the word dresser to the lamp object.

we are interested in investigating the use of semantic knowledge in word acquisition.


Although the system is aware that green is some word describing the green chair, it does not know that word green refers to the chair's color while the word chair refers to the chair's semantic type.

after learning the word-entity associations p(w|e) by the translation models, we need to further ground words to domain concepts of entity properties.


As described in Section 6, a document entity is only clustered one time.

36% of these clustering instances are 'skipped' because they contain only already clustered document entities.


As seen in the example above, a surface Arabic word may be translated as a phrase in English.

the Arabic data in its raw surface form (from a statistical viewpoint) is much more sparse which decreases the efficiency of training significantly.


Hence, it is not possible to give a final word on the superiority of SVM or CRF in the NER task, and it is necessary to conduct a per class study, as the one we present in this paper, in order to determine the right ML approach and features to use for each class.

our best global NER system combined   the results obtained from both ML approaches.


Thus, while many structural innovations in the modeling approach have been made, those innovations have generally been tested on systems with features whose strength has not been established, and compared to weak pairwise baselines.

it is possible that some modeling innovations may have less impact or applicability when applied to a stronger baseline system.


However, for some pairs of mentions from an equivalence class, there is little or no direct evidence in the text that the mentions are coreferential.

training pc on all pairs of mentions within an equivalence class may not lead to a good predictor.


Anaphoricity Ng and Cardie (2002a) and Denis and Baldridge (2007) show that when used effectively, explicitly predicting anaphoricity can be helpful.

we learn a separate classifier to detect whether a mention is anaphoric (that is, whether it is not the first mention in its equivalence class), and use that classifier's output as a feature for the coreference model.


When many of the competing sentences are included in the summary, the issue of information overlap between parts of the output comes up, and a mechanism for addressing redundancy is needed.

our summarization systems employ a final level of analysis: before being added to the final output, the sentences deemed to be important are compared to each other and only those that are not too similar to other candidates are included in the final answer or summary.


The samples z i are guaranteed (in the limit) to consistently approximate the conditional distribution P (z|x, θ) (or P (z|x) later).

the average counts of phrase pairs in the samples converge to expected counts under the model.


Each phrase pair in each outcome contributes to these posteriors the probability of adding a new pair, deciding whether it is null, and generating the phrase pair along with its contribution to the distortion probability.

this is all captured in a succinct potential function ψ( e, f ) = outcome o 0 is chosen with probability Operators in a Gibbs sampler require certain conditions to guarantee the correctness of the sampler.


However, after choosing o 2 , when we reapply the operator to positions e 1 and 4 These are two sufficient conditions to guarantee that the Metropolis-Hastings acceptance ratio of the sampling step is 1.  e 2 , we freeze all alignments except e 1 , null and e 2 , null , which prevents us from returning to o 0 .

we fail to satisfy the second condition.


However the two point-wise Gibbs samplers investigated here, which resample the label of each word conditioned on the labels of its neighbours (amongst other things) only require O(m) steps per sample (where m is the number of HMM states), while EM, VB and the sentence-blocked Gibbs samplers require O(m 2 ) steps per sample.

for HMMs with many states it is possible to perform one or two orders of magnitude more iterations of the point-wise Gibbs samplers in the same run-time as the other samplers, so it is plausible that they would yield better results.


Starting from single-wordbased translation approaches, significant improvements have been made through advances in modeling, availability of larger corpora and more powerful computers.

substantial progress made in the past enables today's MT systems to achieve acceptable results in terms of translation quality for specific language pairs such as Arabic-English.


Several constraints and setups will be described later on in more detail, but as an introduction one can think of the following interpretation which is depicted in Figure 1: Using a phrase-based MT approach, a source word f is triggered by its translation e which is part of the phrase being considered, whereas another target word e outside this phrase serves as an additional trigger in order to allow for more fine-grained distinction of a specific word sense.

this cross-lingual trigger model can be seen as a combination of a lexicon model (i.e.


The advantage of this reranking approach is that we can directly test the obtained models since we already have fully generated translations.

we can apply the triplet lexicon model based on p(f |e, e ) and its inverse counterpart p(e|f, f ) directly.


For the various setups, there are no big differences in the TER score which might be an effect of optimization on BLEU.

for further experiments using the constraints from Section 3.2, we use both singletons and empty words as the default.


The motivation of this approach is to add an additional second trigger to a translation lexicon component which can come from a more global context (on a sentence level) and allow for a more fine-grained lexical choice given a specific context.

the method is related to word sense disambiguation approaches.


In this experiment the system used propositions for which the hit number exceeded 1,000 hits using Google.


In the evaluation, the participants expressed the opinion that an utterance like (xx ha yy) is unnatural and using a modality like (maa)("well"), (moo)("anyway") is very natural.

we can say that the modality expressions make the utterances of the system seem more natural.


When a disagreement has been made between two volunteers, one of the authors has made the final judgment.

177 relevant Q&A pairs have been found in total for the 32 short queries.


With traditional parallel programming models (e.g., MPI), the developer shoulders the burden of explicitly managing concurrency.

a significant amount of the programmer's attention is devoted to system-level details, leaving less time for focusing on the actual problem.


Features for the classifier are designed using the rules of substring substitution.

we can obtain, efficiently, candidates of destination strings and negative instances for training.


What is worse, the distance/similarity metrics do not specifically derive destination strings to which the classifier is likely to assign positive labels.

we can no longer use the efficient algorithm as a candidate generator (in Section 2.4) with these features.


For each string pair (s, t) in LR-SPL 9 , LRNOM, and LRAGR tables, we generated two instances s, t and t, t .

a system is expected to leave the string t unchanged.


a known vocabulary), the average number of positive substitution rules relevant to source strings was as small as 23.9 (in XTAG morphology data).

the candidate generator performed 23.9 substitution operations for a given string.


As described in Section 1, this task is related closely to various applications.

we specifically examine several prior studies that are relevant to this paper in terms of technical aspects.


In the sentence What year did the Vietnam War end?, the newspaper-trained POS tagger incorrectly assigns the POS tag NN (common noun) to the verb end, since verb-final sentences are atypical for the PTB.

the supertagger is virtually certain (greater than 99% probability) that the correct CCG lexical category for end is N (noun).


At iteration 0, the user does not supply any correct text prefix to the system, for this reason p is shown as empty.

the IMT system has to provide an initial complete translation s h , as it were a conventional SMT system.


In our algorithm, dependency is defined on complicated hidden structures instead of on a graph.

long distance dependency in a graph becomes local in hidden structures, which is desirable from linguistic considerations.


In HTM, the content of a document is decided by the topics of the document as well as the topics of the documents which the document cites.

contents of documents can be 'propagated' along the hyperlinks.


As a result contents of documents can be 'propagated' along the hyperlinks.

for example, suppose web page A cites page B and page B cites page C, then the content of page A is influenced by that of page B, and the content of page B is further influenced by the content of page C. HTM is able to more accurately represent the contents of hypertexts, and thus is more useful for text processing such as topic discovery and document classification.


There may not always be enough players available online to match a human player with another human player.

one important part of designing an online game is building a bot which can function in the place of a player.


However, it is also not structured; we simply get pairs of related words.

the essential question for this game is what kind of data we get.


First, statistics about selectional preferences of verbs are not too difficult to extract from the web (although in some cases they might be somewhat noisy).

the motivation for extracting this data using a game is not as apparent.


For example, words that are part of an adjunct phrase in the key sentence might not be good candidates to elaborate upon.

one might want to weight these seed words lower, to reduce their influence on triggering schematically associated words.


Now because each word excluding the root must be pushed to the stack once and popped off once during the parsing process, the number of actions Inputs: training examples (x i , y i ) Initialization: set w = 0 Algorithm: // R training iterations; N examples for t = 1..R, i = 1..N :   needed to parse a sentence is always 2n − 1, where n is the length of the sentence.

the decoder has linear time complexity, given a fixed beam size.


H(v) can be interpreted as the entropy of v's neighborhood.


As noted in Section 3, Adsorption ignores node type and hence the final ranked extraction may also contain classes along with instances.

in addition to finding new instances for classes, it also finds additional class labels similar to the seed class labels with which Adsorption was run, at no extra cost.


The phrases in each cluster are expected to capture most syntactic variation in the expression of a given relationship.

the largest clusters are well suited as positive examples for training a relationship extractor (Rosenfeld and Feldman, 2006).


As in this work, they use the vector model and cosine similarity to define a measure of similarity between relations, but build relation vectors out of all instances of each frequently co-occurring entity pair.

each mention of the same co-occurring pair is assumed to express the same relationship.


Minimum Bayes-Risk (MBR) decoding (Bickel and Doksum, 1977) aims to address this mismatch by selecting the hypothesis that minimizes the expected error in classification.

it directly incorporates the loss function into the decision criterion.


In contrast with the model of (DeNero et al., 2006), who define the segmentations over the source sentence f alone, our model employs bilingual containers thereby segmenting both source and target sides simultaneously.

unlike (DeNero et al., 2006), our model does not need to generate the word-alignments explicitly, as these are embedded in the segmentations.


For instance, given the type PERSON, there is a higher probability of generating "he" than "it".

we maintain a list of compatible semantic types for each pronoun, and give a pronoun a count of (1 + λ P ) if it is compatible with the drawn semantic type; otherwise, we give it a count of λ P .


We do not predetermine anaphoricity of a mention, but rather fuse it into the integrated resolution process.

our model is inherently joint among mentions and subtasks.


Pronouns tend to resolve with mentions that are semantically compatible with them.

we introduce predicates that represent entity type, number, and gender: Type(x, e!


The core of our model (head mixture) tends to cluster non-pronouns with the same head.

we initialized by setting all weights to zero, and running the same learning algorithm on the base MLN, while assuming that in the ground truth, nonpronouns are clustered by their heads.


Not all referential expressions in a given document are anaphors: some expressions introduce a discourse entity, rather than accessing an existing one.

coreference resolvers must have a way of identifying such "discourse-new" expressions.


The independence assumption from the first approximation can be relaxed without increasing the asymptotic time complexity by considering as independent only chunks of contiguous POS tags that are at least a certain number of tokens apart.

the probability of the tag sequence will be approximated with the product of the probabilities of the tag chunks, where the exact probability of each chunk is computed in constant time with the constrained forward-backward procedure.


On the other hand, the nodes sailors and thought are adjacent in the dependency tree of the sentence.

their relationship can be easily captured as a dependency tree feature using the same window size.


We enforce an independence assumption between POS tags.

ifz 1 = t i 1 t i 2 ...t i k specifies the tags at k positions in the sentence, then p(z 1 |x) is rewritten as: 3.


We are interested in the probability that h(x) is erroneous, given that h(x) satisfies the constraints: Here, the second step is Bayes' rule plus definitions, the third step is by the weak initial hypothesis assumption, and the last step is by algebra.

in order to get a probability of error at most η, we need The proof of Corollary 1 is straightforward.


The task was a simplified classification task from Timebank in that only one verb, the main verb, of each sentence was used.

the task can be viewed as ordering the main events in pairwise sentences rather than the entire document.


Our Integer Linear Programming framework uses the following objective function: with added constraints: where x ij represents the ith pair of events classified as the jth relation of m relations.

each pair of events generates m variables.


Unlike the previous evaluation, we also use this entire dataset for testing, not just for training.

all event-event relations in Bethard as well as Timebank are used to expand the dataset with transitive closure and are used in training and testing.


Statistical MT systems, the currently dominant research paradigm, typically do not address this issue directly.

when evaluated for tense, current MT systems often perform miserably.


"He also revealed that Russia intended to provide weapons to Iran within the next ten years."

in our Chinese "tense" annotation task, we annotate both absolute and relative tenses.


That is, future situations may or may not happen.

future situations are often linked to possibilities, not just to situations that will definitely happen.


A downside of this approach is, however, that N -best lists can only capture a very small fraction of the search space.

the line optimization algorithm needs to repeatedly translate the development corpus and enlarge the candidate repositories with newly found hypotheses in order to avoid overfitting on C s and preventing the optimization procedure from stopping in a poor local optimum.


This operation neither changes the number of line segments nor their relative order in the envelope, and therefore it preserves the structure of the convex hull.

we can propagate the resulting envelope over an outgoing arc ε to a successor node v ½ headÔεÕ.


This is, however, easy to accomplish since the combined envelope is simply the convex hull of the union over the line sets which constitute the individual envelopes.

by merging the arrays that store the line segments for the incoming arcs and applying Algorithm 1 to the resulting array we obtain the combined upper envelope for all partial translation candidates that are associated with paths starting at the source node s and ending in v ½ .


Proof: Suppose that such a continuation exists, then this continuation can be represented as a line g, and sincef has been discarded from the envelope, the path associated with g must also be a valid continuation for the line segments f 1 , ..., f N that constitute the envelope.

it follows that maxÔf 1 g, ..., f N gÕ maxÔf 1 , ..., f N Õ g f g for some γ È R. This, however, is in contradiction with the premise thatf To keep track of the phrase expansions when propagating an envelope over an outgoing arc ε È tailÔvÕ, the phrase label ϕ v,headÔεÕ has to be appended from the right to all partial translation hypotheses in the envelope.


It is also very important to know how similar the morphology is between the two languages, as two languages which are morphologically complex in very similar ways, could be relatively easy to translate.

we also include a measure of the family relatedness of languages in our analysis.


Interpreting natural language is such a complex task that writing a perfect set of rules would be extremely challenging.

a more ideal solution would be a learning-based method that can exploit ideas from compositional semantics while providing the flexibility to the rigid application of the heuristic rules.


Conversely, while existing MT aligners can make use of dictionaries, they are not designed to harness other sources of information on degrees of semantic relatedness.

the tools and techniques of MT alignment may not transfer readily to NLI alignment.


While none of the aligners rivals the performance of the LCC system, all achieve respectable results, and the Stanford and MANLI aligners outperform the average RTE2 entry.

even if alignment quality does not determine inferential validity, many NLI systems could be improved by harnessing a well-designed NLI aligner.


For nodes along a chosen decipherment path, this sum will be 1, and for others, it will be 0.

3 we create one "subject to" constraint for each node (" " stands for space).


In contrast, the classbased language model groups the words that have similar functions of syntax or semantics into meaningful classes.

it handles the questions of data sparsity and generalization of unseen event.


Moreover, a SuperARV language model was presented (Wang and Harper, 2002), in which lexical features and syntactic constraints were tightly integrated into a linguistic structure of SuperARV serving as a class in the model.

these knowledge was integrated in the representation level, and then the joint probabilities of words and corresponding SuperARVs were estimated.


This is because not only those unknown words cannot be recognized correctly, but the words surrounding them will be affected.

many efforts have been made to deal with the issue of OOV words (Martins et al., 2006;Galescu, 2003;Bazzi and Glass, 2001), and various model units smaller than words have been examined to recognize OOVs from speech, such as phonemes (Bazzi and Glass, 2000a), variable-length phoneme sequence (Bazzi and Glass, 2001), syllable (Bazzi and Glass, 2000b) and sub-word (Galescu, 2003).


Hence, taking names recognition into account, a WFST, including all possible segmentations as well as recognized candidates of names, can be obtained as below, denoted as W F ST words : (1) POS information is integrated as follows.

(2) the desired analyzer, a combined WFST that represents multi-level linguistic knowledge sources, has been obtained.


The presented analyzer models linguistic knowledge at different levels, which will be useful to find an optimal words sequence among a large number of speech recognition hypotheses.

in this research, the analyzer is incorporated after the first pass recognition, and the n-best hypotheses are reranked according to the total path scores adjusted with the analyzer scores as follows.


While we may have existing transcripts from general lectures or written text on the precise topic, training data that matches both the style and topic of the target lecture is often scarce.

past research has investigated various adaptation and interpolation techniques to make use of partially matched corpora (Bellegarda, 2004).


Note that with β(hw) = 1, the model degenerates to the original modified Kneser-Ney formulation.

furthermore, β only specifies the relative weighting among n-grams with a common history h. scaling β(hw) by an arbitrary function g(h) has no effect on the model.


However, as the size of each partition increases, this ratio increases to 1, since most n-grams have a non-zero probability of appearing in each partition.

an alternative is to compute the normalized entropy of the n-gram distribution across the S partitions, or h is the fraction of an n-gram appearing in partition s. For example, the normalized entropy of the unigram C is h(C) = −1 log 3 [ 2 6 log 2 6 + 4 6 log 4 6 + 0] = .58.


However, the 1.0% WER drop for the interpolated model significantly exceeds the sum of the individual reductions.

as we will examine in more detail in Section 5.1, n-gram weighting allows probabilities to be shifted from less relevant n-grams in one component to more specific n-grams in another.


Fortunately, the features that yield the lowest perplexity and WER on the development set also yield one of the lowest perplexities and WERs, respectively, on the test set.

during feature selection for speech recognition applications, we should consider the development set WER.


For t doc , values close to 1 correspond to n-grams ending frequently on topic words with uneven distribution across documents.

as expected, the signs of the optimized parameter values are flipped.


N -gram weighting and other approaches to crossdomain language modeling require a matched development set for model parameter tuning.

for future work, we plan to investigate the use of the initial recognition hypotheses as the development set, as well as manually transcribing a subset of the test set utterances.


In addition, there is always a reference sentence whose length is shorter than that of a I 1 , such that the brevity penalty is also 1.0.


Step 6 can be performed in polynomial time (Hopcroft and Karp, 1973) O((C + n) 5/2 ); all other steps in linear time O(C + n).

cN-1BLEU can be decided in polynomial time O((c + n) 5/2 ).


Stepping along a non-empty edge will often decrease the local precision, though.

a simple algorithm will prefer paths with shorter hypotheses, which leads to a suboptimal total BLEU score, because of the brevity penalty.


2 The Pr(Q) is constant across all documents.

finding a document to maximize Pr(D is rel|Q) is equivalent to finding a document that maximizes Pr(Q|D is rel).


Of course, each time the number of grammar categories is doubled, the number of binary productions is increased by a factor of eight.

while our final grammars used few categories, the number of total active (non-zero) productions was still substantial (see Section 7).


Rather, they present only a coarse trace of the true underlying processes.

learning a grammar for parsing requires the estimation of a more highly articulated model than the naive CFG embodied by such treebanks.


In terms of its semantics, a multi-scale grammar is simply a compact encoding of a fully refined latent variable grammar, in which identically weighted refinements of productions have been collapsed to the coarsest possible scale.

rather than attempting to control the degree to which categories are split, multi-scale grammars simply encode productions at varying scales.


In this paper, we propose a technique that automatically takes into account certain characteristics of the domains of interest, and accurately predicts parser performance on data from these new domains.

we have a cheap (no annotation involved) and effective recipe for measuring the performance of a statistical parser on any given domain.


On the modeling side, however, a vector combination function that ignores the relation will assign the same representation to (2a) and (2b).

existing models are systematically unable to capture this class of phenomena.


This task can also be addressed with kernel methods, which project items into an implicit feature space for efficient similarity computation.

vector space methods and kernel methods have both been used for NLP tasks based on similarity, notably Information Retrieval and Textual Entailment.


Rather than yielding a single, joint vector for the whole expression, our procedure for computing meaning in context results in one context-adapted meaning representation per word, similar to the output of a WSD system.

our model can be combined with any formalism representing the structure of an expression.


Difficult sentences (those with rare words and greater length) are overrepresented in the test set as the discrimination of short sentences containing only frequent words can easily be done by simplistic models.

a test set of "easy" sentences would not distinguish good models from bad models.


Previous work on non-categorical representations of words has viewed these representations as an intermediate step for arriving at categorical parts of speech (Redington et al., 1998;Schütze, 1995;Clark, 2003).

all of these papers evaluate their results by comparing induced categories to gold-standard parts of speech.


Regarding features: both words and structure of the text (e.g., word order) can be used to infer subjectivity.

the features we employ, such as words and word n-grams, are expected to be useful as a (coarse) proxy to grammatical and phrase features.


Furthermore, the feature set using the words in the question with best answer together (q_bestans) exhibit higher performance than question with all answers (q_allans).

we will only consider questions and best answers in the following experiments with GE and CoCQA.


A dialog in this domain usually has a single goal, to create an air-travel itinerary which may include hotel and car reservations.

the entire dialog corresponds to one task.


Predicted segments provide more context to the clustering algorithm that induces the HMM states.

a more robust state representation is obtained.


We argue that while the terms of interest lie in that subset of interest in the graph (the terms whose rank increased substantially), not all of those words are equally interesting or important and rather than simply looking at absolute changes in rank we focus on the relative-rank differential RRD metric.

figure 3 shows the sorted values of the relative rank differential list (with 3 .


One of the difficult aspects of trying to predict CS points is how to evaluate the performance of the learner since switching is intrinsically motivated and there are no forced switches (Sankoff, 1998b).

standard classification measures for this task such as precision, recall, F-measure, or accuracy, are not the best approach for measuring the effectiveness of a CS predictor.


It is unrealistic to just consider F-measure, or accuracy, as truthfully reflecting how well the learners generalize to the task.

we evaluated the classifier's performance using two different criteria, which are discussed in the next sections.


From the resulting 30 sets, we removed 2 sets because one or more of the generator schemes produced a monolingual sentence.


Evaluation of our approach cannot be done based only on the gold-standard set since there is no sin-gle right answer in this task.

we complemented the evaluation by involving judgements from bilingual speakers.


However, non-antonymous semantically related words such as hypernyms, holonyms, meronyms, and nearsynonyms also tend to occur together more often than chance.

separating antonyms from them has proven to be difficult.


Of course, a number of nonantonymous words also tend to co-occur more often than chance-commonly known as collocations.

strong co-occurrence is not a sufficient condition for detecting antonyms, but these results show that it can be a useful cue.


(the distributional hypothesis of closeness).

just like the co-occurrence hypothesis, occurrence in similar contexts is not sufficient, but rather yet another useful cue towards detecting antonyms.


In order to best evaluate a computational measure of antonymy, we need a task that not only requires knowing whether two words are antonymous but also whether one word pair is more antonymous than another pair.

we evaluated our system on a set of closest-opposite questions.


The corpus and the experiment are the largest of their kind, as far as we know.

we found that a standard supervised WSD method works well for the idiom identification and achieved an accuracy of 89.25% and 88.86% with/without idiomspecific features and that the most effective idiom-specific feature is the one involving the adjacency of idiom constituents.


In the Japanese language, there are basically three character systems: Hiragana, Katakana, and Chinese characters.

you can write an idiom in different characters.


Acquired instances tend to become noisy as the iteration proceeds, so it is important to terminate before semantic drift occurs.

we have another configuration: (f) stopping criterion.


1 This similarity of Equations (1), (2) and HITS is not discussed in (Pantel and Pennacchiotti, 2006).

of the above discussion, semantic drift in simplified Espresso seems to be inevitable as the iteration proceeds, since the principal eigenvector of A need not resemble seed vector i 0 .


Given these definitions, we see that the (M T M ) n term of smaller n corresponds to lower-order cooccurrence, which is accurate but sparse, and the (M T M ) n term of larger n corresponds to higherorder co-occurrence, which is dense but possibly giving too much weight on unrelated instances extracted by generic patterns.

it is expected that setting diffusion factor β to a small value prevents semantic drift and also takes higher order pattern vectors into account.


using the capitalization input by web searchers is misleading and actually hurts performance.


On the other hand, if the word " " does appear in the training data but it has only a translation "gruel" as that is the meaning in the formal text, the translation system may wrongly translate " " into "gruel" for the informal text where the word " " is more likely to mean "like".

as a text-normalization step, it is desirable to transform the informal text into its standard formal equivalent before feeding it into a general-purpose text-processing system.


While multilingual learning can address ambiguities in each language, it must be flexible enough to accommodate cross-lingual variations such as tag inventory and syntactic structure.

of such variations, two languages often select and order their tags differently even when expressing the same meaning.


Especially the overlapping ambiguity, as shown in experiments of (Wang, 2006), will cause more performance decrement of CIR.

a CIR system with a word segmenter better solving the overlapping ambiguity, may achieve better performance.


As we mentioned above, segmentation decision is based on IAS value.

the number of IAS degree's class label is also correspondent to the number of segmentation class label.


We cannot easily classify the gap between " " and " " as segmented or unsegmented.

beside these two class labels, we define another class label, semisegmented, which means that the gap between two characters could be segmented or unsegmented, either will be right.


pointed out that the average length of words in usage is 1.59 (Nie et al., 2000).

we stop the segmentation iterative when the length of sub sentence is 2 or less than 2.


Our segmentation approach can achieve good performance even with small training corpus, which indicates that Ranking SVM has generalization ability.

we can use a relative small corpus to train Ranking SVM, saving labeling effort.


However, we argue that querying long sequences should not be explicitly discouraged, if in fact they contain more information.

we also propose the total token entropy (TTE) measure: For most sequence labeling tasks, however, it is more appropriate to consider the entropy of the label sequence y as a whole, rather than some aggregate of individual token entropies.


Note that, at query time, ∇ (L; θ) should be nearly zero since converged at the previous round of training.

we can approximate ∇ (L + x,ŷ ; θ) ≈ ∇ ( x,ŷ ; θ) for computational efficiency, because the training instances are assumed to be independent.


However, it is common in similar situations to approximate the Fisher information matrix with its diagonal (Nyffenegger et al., 2006).

we estimate I x (θ) using: and I U (θ) using: For CRFs, the partial derivative at the root of each element in the diagonal vector is given by: which is similar to the equation used to compute the training gradient, but without a regularization term.


The expected gradient length and Fisher information methods are the most computationally expensive, because they must first perform inference over the possible labelings and then calculate gradients for each candidate label sequence.

they take eight to ten minutes (upwards of a half hour on the larger corpora) for each query.


Morphophonological changes are often sensitive to phonemes (whereas x and y may consist of graphemes); syllable boundaries; a conjugation class; morpheme boundaries; and the position of the change within the form.

as mentioned in section 2.1, we enrich the alignment string A so that it specifies additional latent variables to which features may wish to refer.


Here x i ∈ X (the input space of the classifier, and corresponds to vectors of features) and y i ∈ Y (the space of classifier outputs, and for our case is the space of non-negative integers).

|Y| = 2 yields binary classification while |Y| > 2 yields multi-class.


We use w ij ∈ W to denote the weight of the edge between vertices i and j. W is referred to as the weight (or affinity) matrix of G. As will be seen shortly, the input features x i effect the final classification results via W, i.e., the graph.

graph construction is crucial to the success of any graph-based SSL algorithm.


Next, we see that the two objective functions in fact have identical solutions when the optimization enforces the constraint that p and q are equal: Indeed, as α gets large, the solutions considered viable are those only where p = q.

we thus have that: the two objectives should yield the same solution as long as α ≥ w ij for all i, j.


For unrestricted text, the complexity and subjectivity of annotation render it essentially infeasible; even pre-specifying the target predicates and objects is very difficult.

to apply semantic parsing beyond limited domains, it is crucial to develop unsupervised methods that do not rely on labeled meanings.


Moreover, while ambiguities are present at the lexical level, they quickly diminish when more words are present.

a lambda form can usually only belong to a small number of clusters, if not a unique one.


However, we cannot extract complex paths from the unlabeled graph g, as it does not contain semantic role information.

we use the same list of complex paths extracted from s. Note that this introduces an unavoidable asymmetry into our similarity computation.


Without any constraint, all word pairs in an input sequence must be considered by the classifier, leading to poor computational efficiency and unnecessary performance loss.

the training sample for SRL task needs to be pruned properly.


Here we report that all SRL sub-tasks are tackled in one integrated model, while the predicate disambiguation sub-task was performed individually in both of our previous systems.

this is our first attempt at a full integrated SRL system.


We observe that any finite-state automaton can also be encoded as a hypergraph (in which every hyperedge is an ordinary edge that connects a single antecedent to a consequent).

the methods of this paper apply directly to the simpler case of hypothesis lattices as well.


However, r, s, and t can be positive or negative, and we cannot directly take the log of a negative number.

we represent real numbers as ordered pairs.


We will consider some other choices in Sections 4.3-4.4 below.

for generality, we conclude this section by stating the precise technical conditions needed to construct E P,R and E P,R,S,T : • P is a semiring • R is a P -module (e.g, a vector space), meaning that it comes equipped with an associative and commutative addition operation with an identity element 0, and also a multiplication operation As a matter of notation, note that above and in Tables 1-2, we overload "+" to denote any of the addition operations within P, R, S, T ; overload "0" to denote their respective additive identities; and overload concatenation to denote any of the multiplication operations within or between The second and third rows define the operations between two elements p1, r1, s1, t1 and p2, r2, s2, t2 , while the last two rows define the identities.


From Expectations to Gradients One perspective is that our semiring fundamentally finds expectations.

we must be finding ∇Z by formulating it as a certain expectation r. Specifically, to be rs T , a matrix.


We can rewrite the sum as The second term is constant with respect to λ d , and so is φ d (e k , f ).

renaming those two quantities offest Λ (e k ) and slope(e k ), we get if we plot the score for a candidate translation vs. λ d , that candidate will be represented by a line.


While the results just presented indicate those strategies provide a good approximation of some 'true' RYPT score, label percolation was a strategy based primarily on intuition, and choosing maxLen = 4 for frontier set construction was based on examining a limited amount of preliminary data.

and in addition to encouraging empricial results, we felt a more rigorous quantitative analysis was in order, especially with future, more ambitious annotation projects on the horizon.


In decoding, we build both the source and the target dependency structures simultaneously in chart parsing over the source string.

we can compute the source dependency LM score in the same way we compute the target side score, using a procedure described in (Shen et al., 2008).


Source dependency trees are composed of fragments embedded in the translation rules.

a source dependency LM score can be viewed as a measure whether the translation rules are put together in a way similar to the training data.


In general GE objective functions are not convex.

we initialize 0th-order CRF parameters using a sliding window logistic regression model trained with GE.


Informally, via this message, F tells U : Here is what I would like your value to be, based on the messages that my other neighboring variables have sent me about their values, and how I would prefer you to relate to them.

each edge of the factor graph maintains two messages µ V →F , µ F →V .


A distribution over strings is naturally represented by a WFSA.

belief propagation translates to our setting as follows: • Each message is a WFSA.


Unfortunately, the benefit to easily and effectively model the target linguistic phenomena is reduced by the the implicit nature of the kernel space, which prevents to directly observe the most relevant features.

even very accurate models generally fail in providing useful feedback for improving our understanding of the problems at study.


those that are (almost) irrelevant for the SVM.

the chance of this resulting in an improvement is rather low.


While SVM-CW is simple, we show that SVM-CW can improve the performance of the system more effectively and more efficiently than other methods proven to be successful in other NLP tasks earlier.

sVM-CW with our feature vector is comprised of a PPI system with five different models, of which each model is superior to the best model in the original PPI extraction task, which used only the single corpus.


We also added the other label O for fields in text that are not represented in the database.

our final DB schema is {author, title, date, venue, volume, number, pages, editor, publisher, series, O}.


Again the observational tests used for alignment are: (1) exact token match tests whether the source-target tokens are string identical, (2) approximate token match produces a binary feature after binning the Jaro-Winkler edit distance (Cohen et al., 2003) between the tokens, (3) substring token match tests whether one token is a substring of the other, (4) prefix token match returns true if the prefixes match for lengths {1, 2, 3, 4}, (5) suffix token match returns true if the prefixes match for lengths {1, 2, 3, 4}, and (6) exact and approximate token matches at offsets {−1, −1} and {+1, +1} around the alignment.

a conditional model lets us use these arbitrary helpful features that cannot be exploited tractably in a generative model.


To determine whether a noun phrase instance NP i should be extracted as a filler for an event role, GLACIER computes the joint probability that NP i : (1) appears in an event sentence, and (2) is a legitimate filler for the event role.

gLACIER is designed for noun phrase extraction and, mathematically, its decisions are based on the following joint probability: where S NP i is the sentence containing noun phrase NP i .


The SVM model, on the other hand, achieves an 89% accuracy when trained with the larger MUC-4 training data, compared to 84% accuracy for the model trained from the 100 manually labeled documents.

the sentential event recognizer models used in our unified IE framework (described in Section 4.4) are trained with this 1300 document training set.


For example, a statement that Al Qaeda is believed to be responsible for an attack would typically appear after the event description.

the sentential event recognizer tends to generate low probabilities for such sentences.


In the last sentence in Figure 2, for example, GLACIER correctly makes the inference that the policemen in the bus (which was traveling on the bridge) are likely the victims of the terrorist event.

we see that our system manages to balance the influence of the two probability components to make extraction decisions that would be impossible to make by relying only on the local phrasal context.


Yet, valuable as they are, free-style reviews contain much noisy data and are tedious to read through in order to reach an overall conclusion.

we conducted this study to automatically process and evaluate product reviews in order to generate both numerical evaluation and textual summaries of users' opinions, with the ultimate goal of adding value to real systems such as a restaurant-guide dialogue system.


Simply reversing the polarity of sentiment on the appearance of negations may result in inaccurate interpretation of sentiment expressions.

a system which attempts to quantify sentiment while ignoring adverbials is missing a significant component of the sentiment score, especially if the adverbial is a negative word.


However, this method is likely to lead to a large number of rare combinations, thus suffering from sparse data problems.

an interesting question to ask is whether it is feasible to assign to each adverb a perturbation score, which adjusts the score of the associated adjective up or down by a fixed scalar value.


We decided to retain as the evaluation set all reviews which obtained at least one topic extraction from both systems.

the two systems processed exactly the same data with exactly the same definitions of "ground truth".


To evaluate the contribution of clustering, we replaced the mapping tables used by both systems with the edited one used by the ground truth computation.

both systems made use of the same mapping table, removing this variable from consideration.


We found that over 80% were well formed, correct, and informative.

a lower precision here does not necessarily mean poor performance, but instead shows that the pros/cons provided by users are often incomplete.


Also, notice that the opinions reinforce a particular stance, i.e., a prorubbery-material stance.

the scheme links the opinions via reinforcing relations.


The hypothesis in using discourse information for polarity classification is that the global discourse view will improve upon a classification with only a local view.

we implement a local classifier to bootstrap the classification process, and then implement classifiers that use discourse information from the scheme annotations, over it.


The value of a relational feature, for a given DA, depends on the polarity of the discourse neighbors of that DA.

the relational features incorporate discourse and neighbor information; that is, they incorporate the information about the frame and target relations in conjunction with the polarity of the discourse neighbors.


For example, one of the features in the first row is Percent of neighbors with polarity type positive, that are related via a reinforcing frame relation.

each feature for the relational classifier identifies neighbors for a given instance via a specific relation (f , t, f or t , obtained from the scheme annotations) and factors in their polarity values (a, obtained from the classifier predictions from the previous round).


This example is a reinforcing discourse where the speaker is arguing for the number keypad, which is an alternative to the scrolling option.

he argues against the scrolling, and argues for entering the number (which is a capability of the number keypad).


Words like wondering, thinking, de-bating are used when the user is posing a question or expressing doubts.

such phrases usually do not contribute an opinion, especially if they are in the vicinity of the if connective.


It is observed that in most cases, the condition clause contains no opinion whereas the consequent clause reflects the sentiment of the entire sentence.

this method uses (in a different way) only the above consequent classifier.


As we can see, comparative sentences are entirely different from conditional sentences.

their methods cannot be directly applied to conditional sentences.


We train a different classifier for each lexicon entry for which we have training data.

our approach is like targeted WSD (in contrast to allwords WSD), with two labels: S and O.


Note that, although generally in the SENSEVAL datasets, training and test data are provided separately, a few target words from SENSEVAL1 do not have both training and testing data.

we opted to combine the training and test data into one dataset, and then perform 10-fold cross validation experiments.


We were interested in how well the system would perform on more and less ambiguous words.

we split the words into three subsets according to their majority-class baselines, and report separate results: S1 (9 words), S2 (18 words), and S3 (12 words) have majority-class baselines in the intervals [50%,70%) , [70%,90%), and [90%,100%), respectively.


An instance that is not in a subjective expression is, by definition, being used with an objective sense.

these instances are false hits of subjectivity clues.


However, the only parts of the MPQA corpus for which SWSD could affect performance is the subset con-taining instances of the words in the SWSD system's coverage.

for the classifiers in this section, the data used is the SenMPQA dataset, which consists of the sentences in the MPQA Corpus that contain at least one instance of the 39 keywords.


Second, if a keyword with an O sense appears in a subjective expression, then the subjectivity is not due to that keyword but rather due to something else.

the presence of another lexicon entry "explains away" the presence of the O sense in the subjective expression, and we do not want SWSD to overrule the contextual classifier.


But, an S sense is more likely to appear in a P(olar) expression.

we consider confidence (rule R3): If the contextual classifier labels an instance as N, but SWSD determines it has an S sense and the contextual classifier's confidence is low, 6 then R3 flips the contextual classifier's label to P. Rule R4 is analogous to R2 in the previous section: If the contextual classifier labels an instance as P, but (1) SWSD determines that it has an O sense, (2) the contextual classifier's confidence is low, and (3) there is no other subjective keyword in the same expression, then R2 flips the contextual classifier's label to N. We compare the performance of the original neutral/polar classifier (O N/P ) and sense-aware variants using R3 and R4.


We wanted to see if the improvements in the Acc NP NR NF NgP NgR NgF PsP PsR PsF O P s/N g/N 77.6 80.9 94.6 87.2 60.4 29.4 39.5 52.2 32.4 40.0 R4 80.6 81.2 98.7 89.1 82.1 29.4 43.2 68.6 32.4 44.0 The sense-aware variant of the overall two-part system is the same as the original except that we apply R4 to the output of the first step (flipping some of the neutral/polar classifier's P labels to N).


Each adjunction operation involves a head treelet that dominates a modifier treelet.

for any treelet t, we can consider its descendants, that is, the entire set of treelets that are directly or indirectly dominated by t. We define a π-constituent for treelet t to be the subset of source-language words dominated by t and its descendants.


But, like the other models, MM-LDA's learned topics do not correspond directly with the label set.

these models fall short as a solution to the credit attribution problem.


We also applied our method to text classification on the del.icio.us dataset, where the documents are naturally multiply labeled (more than 89%) and where the tags are less inherently similar than in the Yahoo subcategories.

we expect Labeled LDA to do better credit assignment on this subset and consequently to show improved performance as a classifier, and indeed this is the case.


However, human labelling is time-consuming.

in this study we focus on unsupervised approach.


As reported in , most manually assigned keyphrases turn out to be noun groups.

we annotate the document with POS tags using Stanford Log-Linear Tagger 3 , and then extract the noun groups whose pattern is zero or more adjectives followed by one or more nouns.


In practice, only a small fraction of keyphrases are single-word.

as a part of postprocessing process, we have to use a frequent word list to filter out the terms that are too common to be keyphrases.


Wikipedia-based relatedness is computed according to global statistical information on Wikipedia.

it is more precise than cooccurrence-based relatedness, which is reflected in the performance of the keyphrase extraction.


This pattern serves a few goals beyond the discovery of connectedness.

putting '*'s inside the pattern rather than using "from L to" allowed us to avoid arbitrary truncation of multiword expressions as in 'from Moscow to St. Petersburg' and reduced the probability of capturing unrelated sentence parts like 'from Moscow to cover deficit'.


allows to obtain only the first 1K snippets, in case we get more than 10K hits, we extend our queries by adding the most common term sequences appearing before or after the pattern.

if for the query "from * to Moscow" we got more than 10K hits and among the snippets we see '... bus from X to Moscow...' we create an extended pattern 'bus from * to Moscow' and use the term extraction in Section 3.2 to get additional terms.


Let C 1 , C 2 be sets of pairs assigned to labels L 1 , L 2 .

we merge two labels if one of the following conditions holds: either one group is nearly included in the other or each group shares nearly half with the other group.


For example, we linked (MEDI-CAL SPECIALTIES -gynaecology.n) to the Gynaecology Wikipage.

we could include the adjective gynaecologic, pointing to the Gynaecology page, into the MEDICAL SPECIALTIES frame for sentences like "Fellowship training in a gynaecologic subspeciality can range from one to four years".


Automatic metrics have been criticized for a variety of reasons (Babych and Hartley, 2004;Callison-Burch et al., 2006;Chiang et al., 2008), and it is clear that they only loosely approximate human judgments.

having people evaluate translation output would be preferable, if it were more practical.


These evaluations are used as a "Go / No Go" determinant of whether teams will continue to receive funding.

each team have a strong incentive to get as good a result as possible under the metric.


This is a strong but standard model; better, more complex models could be used, but the gains are likely to be small.

we opted for simplicity in our model so as to focus more on the interaction between the annotator and different levels of machine involvement.


An "informative" example is likely to take longer to annotate if selected early than it would after the annotator has seen many other examples.

it is important to measure annotation time embedded in the context of a particular annotation experiment with the sample selection/labeling strategies of interest.


The motivation for this is that we hypothesise that the supervised classifier will not have a very good performance initially, as it is trained on a very small data set.

its confidence function may also not be very accurate.


Marcu and Echihabi (2002) demonstrated that word pairs extracted from the respective text spans are a good signal of the discourse relation between arguments.

we also consider word pairs as a feature class.


This is further complicated by the language model integration.

we composed each unary rule instance with its descendant rule(s) to create a non-unary rule.


However, as shown in Figure 2, the decoder still generates 801 edges after the hypothesis re-combination.

aggressive pruning with beam search has to be employed to reduce the search space to make the decoding practical.


Intuitively, the size of bucket can be used to approximately indicate the actual number of competing edges on average, and reducing the size of bucket could help reduce the edges generated in a parsing chart by applying the rules in the bucket.

if we can find a method to greedily reduce the size of each bucket ( , ′), we can reduce the overall expected edge competitions when parsing with ′.


only collapse $ 10,000 into an atomic unit, with more than handled compositionally according to the semantics assigned to it by the grammar.

after transferring the BBN annotations to the CCGbank corpus, Rajkumar et al.


The algebraic relation between these matrices can be used to show that any document vector is a linear combination of the vectors representing the words it contains.

within this paradigm it is nat-ural to treat multi-word structures as a "pseudodocument" and represent them via linear combinations of word vectors.


Documents are represented by distributions over topics and topics are distributions over words.

the mixture of topics in any document determines its vocabulary.


The approach of Gildea and Hofmann (1999) overcomes this difficulty by using representations constructed with pLSA, which have a direct probabilistic interpretation.

the probability of an upcoming word given the history can be derived naturally and directly, avoiding the need for ad-hoc transformations.


Now, when we compose these vectors using the multiplicative model and the components definition in (4), we obtain: And by Bayes' theorem: Assuming w 1 and w 2 are independent and applying Bayes' theorem again, h i becomes: By comparing to (4), we can see that the expression on the right hand side gives us something akin to the vector components we would expect when our target is the co-occurrence of w 1 and w 2 .

for the multiplicative model, the combined vector h i can be thought of as an approximation to a vector representing the distributional properties of the phrase w 1 w 2 .


Suppose we were unsure whether a word token x was an instance of w 1 or of w 2 .

it would be reasonable to express the probabilities of context words around this token in terms of the probabilities for w 1 and w 2 , assuming complete uncertainty between them: we could represent x with a vector, based on these probabilities, having the components: Which is exactly the vector averaging approach to semantic composition.


The primary problem with the cosine measure is that, although its values lie between 0 and 1, they do not sum to 1, as probabilities must.


If one model is much weaker than the other, linear interpolation will typically produce a model of intermediate strength (i.e., worse than the better model), with the weaker model contributing a form of smoothing at best.

based on equation 13, we express our semantic probabilities as the product of the unigram probability, p(w), and a semantic component, ∆, which determines the factor by which this probability should be scaled up or down given the context in which it occurs.


Therefore, based on equation 13, we express our semantic probabilities as the product of the unigram probability, p(w), and a semantic component, ∆, which determines the factor by which this probability should be scaled up or down given the context in which it occurs.

it seems reasonable to integrate the n-gram model by replacing the unigram probabilities with the n-gram versions.


The subsequent columns show the words expressing the concept in respective languages (in the example table above, English, Hindi and Marathi).

to express the concept "04321: a youthful male person", there are two lexical elements in English, which constitute a synset.


In (Cook et al., 2007) a syntactic parser was used to collect sentences containing the MWEs in the active and passive voice using heuristics.

examples such as the following (from the BNC) would not be included in their sample: 1. take a chance: 'While he still had a chance of being near Maisie, he would take it'.


The output, for each MWE, is a binary classifier, trained on those sentences.

we target step (2) of MWE identification, which is the difficult one.


In addition, many studies have shown the importance of order and distance in MWE extraction in English (two recent examples are (Dias, 2003;Deane, 2005)).

we develop a supervised classifier based on surface features.


First, there is substantial variation in the dependency tree structures of the non-compositional uses of the expressions as output by the parser.

the syntactic feature classifier was more difficult to learn than the surface feature one, requiring a larger training set.


Second, as described above, the syntactic features consist of general syntactic relations extracted from the parse tree and not type-specific knowledge.

literal or non-expression uses of the MWE's components, which have a close syntactic relation in a given sentence, appear as non-compositional uses of the expression to the classifier.


Because the Moses decoder does not generate function words that are context dependent, it treats a function word as a part of the translation.

we collect possible function words for each translation from the corpora when the conditional probability is larger than a threshold 2 .


To avoid explosion, these approaches generally limit the window size to a small number.

long-span collocations can not be extracted 1 .


In a bilingual sentence pair, a source word is aligned with its corresponding target word, while in a monolingual sentence, a word is aligned with its collocates.

it is reasonable to regard collocation construction as a task of aligning the collocated words in monolingual sentences.


In a monolingual sentence, a word never collocates with itself.


Among them, the log-likelihood ratio measure achieves the best performance.

in this paper, we only show the performance of the loglikelihood ratio measure.


Even if all collocations in the gold set are included in the 200Kbest list, the precision is only 28%.

it is more useful to compare precision curves for collocations in the N-best lists extracted by different methods.


Some collocations related to specific corpora are not included in the set.

we selected the top 1K collocations extracted by our improved method to manually estimate the precision.


We rely on the fact that the level sets of Pr [A y i ,r ] are convex in µ and Σ.

this leads to convex constraints of the form Outer Bound: we can replace the constraint ] ≥ η to achieve an outer bound.


However, sequential converges faster, needing an average of 4.33 iterations compared to 7.56 for parallel across all datasets.

we select sequential for our subsequent experiments.


Furthermore, many real world datasets, especially in NLP, are noisy, and enforcing too many constraints can lead to overfitting.

we consider the impact of reducing the constraint set in terms of both reducing run-time and improving accuracy.


Additionally, k = 5 converges faster than k = ∞ in an average of 2.22 iterations compared with 4.33 iterations.

reducing the number of constraints improves both speed and accuracy.


Each bar reflects the number of mistakes per bin divided by the size of the complete test set (3,000).

the sum of the heights of the corresponding bars in each bin is proportional to test error.


of the dynamic nature of Web, by the time the ranker is trained and deployed, the training data may be more or less out of date.


Often a small change in the data can result in a very different series of splits.

tree-based ranking models are much less robust to noise, as we will show in our experiments.


The exact inference cannot handle the complete model with three sub-mappings, Ψ n , Ψ h , and Ψ v , since the label group defeats the graph transform in Figure 3.

we design two approximate algorithms by employing undergenerating and overgenerating approaches (Finley and Joachims, 2008).


The URL path n-gram, irs, gets maximum MI score.

the query-url path affix features are MI (irs, irs), MI (forms, irs), and MI (irs forms, irs).


As the amount of information in the WWW grows, more of these people are mentioned in different web pages.

a query for a common name in the Web will usually produce a list of results where different people are mentioned.


It is only recently that the web name ambiguity has been approached as a separate problem and defined as an NLP task -Web People Search -on its own (Artiles et al., 2005;Artiles et al., 2007).

it is useful to point out some crucial differences between WSD, CRC and WePS: • WSD typically concentrates in the disambiguation of common words (nouns, verbs, adjectives) for which a relatively small number of senses exist, compared to the hundreds or thousands of people that can share the same name.


This makes it difficult to perform error analysis in terms of the features used to represent the documents.

we have decided to transform the clustering problem into a classification problem: deciding whether two documents refer to the same person.


q k to be log probabilities under the generative model: We modify this definition slightly, be introducing scaling factors c j,a > 0, and defining In our experiments, c j,a is simply a count of the number of times the feature indexed by (j, a) appears in unlabeled data.

more frequent features have their contribution down-weighted in the model.


Step 3: Re-estimation of w and v. In the final step, w 1 and v 1 are estimated as .

w and v are re-estimated to optimize log-likelihood of the labeled examples, with the generative models q 1 estimated in step 2.


While hand-annotated data are very expensive, large-scale unannotated data can be obtained easily.

the use of largescale unannotated data in training is an attractive idea to improve dependency parsing performance.


The reason may be that we are using gold-POS tags for Chinese data 14 .

we did not use cluster-based features for the integrating systems.


Now, since the ranked lists of features are derived from the partition, the presence of these ambiguous reviews can adversely affect the identification of informative features using MMFR.

we remove the ambiguous reviews before deriving informative features from a partition.


Nonetheless, not all expressions require a complicated inference procedure to determine their polarity.

c w δ i still provides useful information about the likely polarity of each word based on the corpus statistics.


For instance, classifying positive class to negative class is more wrong than classifying positive class to neutral class.

the cost of not fixing very incorrect predictions should be higher than the cost of not fixing less incorrect predictions.


Using the same training set used for the lexicon adaptation would be suboptimal, because the features generated from the adapted lexicon will be unrealistically good in that particular data.

we prepared a separate training data for CRFs using 135 documents from the development set.


A word with multiple meanings is listed in multiple thesaurus paragraphs, and so will be assigned semantic orientation labels for each of these paragraphs.

the method assigns a semantic orientation to a word-sense combination similar to the SentiWordNet approach and differing from the General Inquirer and Turney-Littman lexicons.


Suppose we have a collection of reviews R, where each review is written (mainly) about one of the objects in the listing E. The problem now is to correctly associate each r ∈ R with exactly one of e ∈ E. We model each review as a bag of words.

notation such as "w ∈ r" for a word w and a review r makes sense.


E.g., if the object description contains only one token, the standard cosine-normalization technique (Salton et al., 1975) will yield a normalized score of 1 irrespective of the token.

for a review containing the words "Food" and "Casablanca", the standard normalization will yield the same score for a restaurant named "Food" and a restaurant named "Casablanca", ignoring the fact that "Food" is much more likely to be an object-independent term.


After independent annotation, agreement was measured at a percentage agreement of 91/93% and a kappa of 0.80/0.82 for Minnesota and Kent, respectively.

the distinction can be made with high reliability.


In the following, x is the cue word and y a (possible) response word.

p(x) is the probability of observing x, and p(x) refers to the probability of not observing x. Pointwise Mutual Information (hereonin PMI) was introduced in Section 2.


LL, as it most frequently appears in the literature, is not actually a measure of positive association: it also responds to significant negative association.

ll is arguably not suited to the task in hand.


Two of the human datasets, Kent and Minnesota, though collected independently, are based on the same set of 100 cue words established by Kent (1910).

by performing a rank correlation of these two datasets with one another, (each of which was produced by pooling the responses of some 1000 people) we can get a useful upper-bound for correlations: if a computer-based system were to exceed this upper-bound in correlations with either dataset, then we would need to suspect it of over-fitting.


One may note the experiments are conducted over the general test set.

it is not clear if the precision of the NER system really has insignificant impact on the IR relevance or just the impact is diluted in a larger test set.


Three Chinese phoneme sequences starting with "LAI YIN ...", "LAI NA ...", and "LAI NEI ..." were generated from Reinhardt, where "LAI YIN ..." was the best Chinese phoneme sequence based on the probability distribution in Figure 1(c).

m (E G , JC P ), which jointly used Chinese phonemes with English graphemes, generated the correct Chinese transliteration of Reinhardt based on two probability distribution in Figures 1(b) and 1(c).


Evaluating unsupervised PoS tagging is rather difficult mainly due to the fact that the output of such systems are not actual PoS tags but state identifiers.

it is impossible to evaluate performance against a manually annotated gold standard using accuracy.


There are certainly cases that no functional context can help with tagging, since our disambiguation models are encoded by functional context only.

it is worth a closer look to how often the system resorts to random choice, as well as to the disambiguation accuracy of either disambiguation model for open-and closed-class learned from unannotated data.


We can see that the structured feature spaces and their features are encapsulated by a full parse tree of source sentences.

it is critical to understand which portion of a parse tree (i.e.


This is mainly due to the fact that the phrase table in forced decoding is filtered out based on both source and target languages while in normal decoding it is based on source language only.

more phrases are filtered out in the forced decoding.


Specifically, it is only concerned with influencing the translation rule probabilities via the corpus weights; it does not change the set of rules extracted.

it is unable to add new translation rules as in Snover et al.


Using the chain rule, we get equations (4)-(8), where the summation in Equation 6 is over all source-target phrase pairs in the derivation of hypothesis h sm , ξ is the decoding weight assigned to the log phrase translation score, and the summation in Equation 7 is over all training sentences 2 .

in order to compute the derivative of the objective function we first need to calculate for every phrase pair (s k , t k ) in the translation N-best based on Equations 7 and 8, which requires time proportional to the number of occurrences of these phrases in the parallel training data.


Additionally, each hypothesis specifies information about its derivation, i.e., which source-target phrase pairs it consists of.

given an N-best, we can identify the set of unique phrase pairs and use this information in order to perform a filtered accumulation of the statistics needed for calculating the derivative in Equation 8.


We are relying on information available from e to get the best tokenization for f. However, the parallel sentences will not be available for new data we want to translate.

for the new data, we have to rely only on P (f ) to tokenize any new data, which can be obtained by calculating P (f ) = e P (f | e)P (e) With P (f ) from the bilingual model, we can run the Viterbi algorithm in the same manner as monolingual tokenization model for monolingual data.


This reduction is due to a limitation on adjoining: auxiliary trees can only introduce tree material to the left or the right of the node adjoined to.

an auxiliary tree can be classified by direction as left or right adjoining.


This can be justified because P (e|f ) = P (e,f ) P (f ) , and P (f ) is fixed for a given foreign sentence.

: where d etree,ftree is a derivation tree of rules that generates e tree and f tree .


Taking advantage of the dual annotations provided for each sentence in the SSR corpus, we can report double-reference evaluation.

we judge that if a hypothesized cleaned sentence exactly matches either reference sentence cleaned in the same manner we count the cleaned utterance as correct, and otherwise we assign no credit.


A number of natural language tasks such as machine translation (Lopez, 2008) and word sense disambiguation (Banerjee and Pedersen, 2003;McCarthy, 2006), can be framed as semantic distance problems.

developing automatic measures that are in-line with human notions of semantic distance has received much attention.


The "pure" corpus-based distributional profile (a.k.a.

co-occurrence vector, or word association vector), for some target word u, is biased with soft constraints towards each of the concepts c that list u in the thesaurus, to create a distributional profile that is specific to u in the sense that is most related to the other words listed under c. this method can make more finegrained distinctions than the Mohammad and Hirst method, and yet uses word sense information.


So, not surprisingly, the best WordNet-based measures are those that rely only on the is-a hierarchy.

they are good at measuring semantic similarity (e.g., doctor-physician), but not semantic relatedness (e.g., doctor-scalpel).


Human judgments of semantic distance are less consistent on verb-pairs than on noun-pairs, as reflected in inter-rater agreement measures in Resnik and Diab (2000) and others).

not surprisingly, the scores of almost all measures are lower for the verb data than the RG-65 noun data.


In fact, the only contradicting part lies in the second sentence of T against H, that is, whether there are children among the victims.

this pair could also be classified as a Known (K) pair (=E∪C) against Unknown (U) pairs, instead of being classified as a Non-entailment (N) case (=C∪U) against E case in the traditional two-way annotation.


While in lexical semantics, semantic relatedness is a weaker concept than semantic similarity, there is no counterpart at the sentence or text level.

in this paper, we propose a Recognizing Textual Relatedness (RTR) task as a subtask or the first step of RTE.


Our explanation here is that the BoW method (even with syntactic dependency features) is based on overlapping information shared by T and H, which essentially means the more information they share, the more relevant they are, instead of being more similar or the same.

for the "ECU → E/CU " setting, methods based on overlapping information are not the best choice, while for "ECU → U/EC", they are more appropriate.


Even if such annotation issue can be resolved, aligning the term weights with the true importance of each term may not necessarily lead to our real objective -deriving a better similarity measure for the target application.

our learning framework, TWEAK, assumes that we are given only the labels of the pairs of texts being compared, such as whether the two texts are considered similar by human subjects.


Because the pseudo-documents are constructed using the search result snippets instead of regular web documents, special formatting or link information provided by HTML is not very meaningful.

we focused on using features that are available for plain-text documents, including: • Bias: 1 for all examples.


One issue of precision at k is that it does not provide an overall quality measure of the ranking function.

we also present MAP (mean averaged precision), which is a single number that summarizes the performance of the ranking function by considering both precision and recall, and has been shown reliable in evaluating various information retrieval tasks (Manning et al., 2008).


Unfortunately NGD only uses page-counts of words and ignores the context in which the words appear.

it produces inaccurate similarity scores when one or both words between which similarity is computed are polysemous.


Typically two words can co-occur in numerous patterns.

we end up with a large number of parameters in the model.


The "*" operator matches one word or none in a web page.

our wildcard query retrieves snippets in which a and b appear within a window of seven words.


Low frequency patterns often contain misspellings and are not suitable for training.

we selected patterns that occur at least 10 times in the snippet collection.


In the corpus, coordinated elements are always represented with two dependencies.

the phrase they may half see and half imagine the old splendor has two gold-standard dependencies: dobj(see,splendor) and dobj(imagine,splendor).


1 Given the two sentences w, w , our probability distribution over possible graphs considers local features of the parses, the alignment, and both jointly.

we learn what local syntactic configurations tend to occur in each language and how they correspond across languages.


Our novel contribution is to condition the probabilities of the generative actions on the dependency parse of a source-language translation.

our generative model is a quasi-synchronous grammar, exactly as in (Smith and Eisner, 2006a).


We start from a set of words denoting a concept in a given source language.

we may use words like (apple, banana, ...) as the definition of the concept of fruit or (bear, wolf, fox, ...) as the definition of wild animals.


We use the conjecture that terms of the same concept tend to co-appear more frequently than ones belonging to different concepts 3 .

we select a translation of a term co-appearing most frequently with some translation of a different term of the same concept.


Performing an automatic evaluation of such new terms is a challenging task, since there are no exhaustive term lists available.

in order to check how well newly added terms fit the concept definition, we have to use human judges.


The low ranks for WN reflect the ambiguity of definition of some of its classification subtrees.

for the 'body part' concept defined in Word-Net as "any part of an organism such as an organ or extremity" (which is not supposed to require domain-specific knowledge to identify) low scores were given (correctly) by judges to generic terms such as tissue, system, apparatus and process (process defined in WN as "a natural prolongation or projection from a part of an organism"), positioned in WN as direct hyponyms of body parts.


Surprisingly, terms extracted by the cross-lingual framework achieve significantly higher scores not only in comparison to the singlelanguage algorithm but also in comparison to existing WN terms.

while the "native" WN concept and single-language concept extension re-sults get a score of 7/10, terms obtained by the cross-lingual framework obtain an average score of nearly 9/10.


Even if the meanings of the source and target head words are transferred to the pivot language, this is rarely done with the same set of words or definitions.

due to the different word-usage or paraphrases, even semantically identical or very similar head words can have different definitions in different dictionaries.


Thus, due to the different word-usage or paraphrases, even semantically identical or very similar head words can have different definitions in different dictionaries.

performing only lexical overlap, current methods cannot identify the differences between totally different definitions resulted by unrelated concepts, and differences in only nuances resulted by lexicographers describing the same concept, but with different words.


We identify the words' senses comparing each synonym in the WordNet's synonym description with each word from the dictionary definition.

for each common word we arrive at a certain set of senses from the source-pivot definitions (sns((s→I')) and a certain set of senses from the target-pivot definitions (sns((t→I')).


The adjective right has 13 senses according to WordNet, among them 4 were identified from the Japanese to English definition (sns(right)={#1, #3, #5, #10}, all identified through correct) and 5 from the Hungarian to English definition (sns(right)={#1, #3, #5, #6, #10}, through correct or proper).

4 senses are common, and 1 is different.


We expand the source-to-pivot and target-topivot definitions with information from WordNet (synonymy, antonymy and semantic category, respectively).

the similarity of the two expanded pivot language descriptions gives a better indication on the suitability of the translation candidate.