Fourth, Table 6 shows the association between two query n-grams, "form" and "video", that at first glance may not actually look very informative for URL path selection.

notice that the unigram "form" has a strong preference for pdf documents over more standard web pages with an html extension.


Without any additionally ranking information, general URLs (root) tend to be ranked more highly than more specific URLs (path), as the root pages tend to be more popular.

our new features express a preference between "va" and "virginia", and this correctly flips the ranking order.


In addition, NEs tend to be redundant regarding BoW.

if we are able to combine optimally the contributions of the different features, the BoW approach could be improved.


The results show that according to both the Decision Tree results and the upperbound (MaxPWA), adding new features to tokens improves the classification.

taking nonlinguistic features obtains similar results than taking all features.


The UIUC dataset has laid a platform for the follow-up research including (Hacioglu and Ward, 2003;Zhang and Lee, 2003;Li and Roth, 2006; Krishnan et al., 2005;Moschitti et al., 2007).

to Li and Roth (2006)'s approach which makes use of a very rich feature set, we propose to use a compact yet effective feature set.


For example, the best question classifier QC3 outperforms the worst one (QC1) by 1.5%, 2.0%, and 2.0% MRR scores for NE, NE-4 and REG respectively.

it is surprising that the MRR and top5 contribution of NE and NE-4 decreases if QC1 is replaced by QC2, although the top1 score results in performance gain slightly.


For the second layer, the only assumption we make is that there is at most one link between any two words.

we believe that for any interesting linguistic structure, the second layer will be highly dependent on the structure of the first layer.


(2006) presented a self-training approach for phrase structure parsing and the approach was shown to be effective in practice.

their approach depends on a high-quality reranker, while we simply augment the features of an existing parser.


The majority of existing work on text clustering has focused on topic-based clustering, where high accuracies can be achieved even for datasets with a large number of classes (e.g., 20 Newsgroups).

there has been relatively little work on sentiment-based clustering and the related task of unsupervised polarity classification, where the goal is to cluster (or classify) a set of documents (e.g., reviews) according to the polarity (e.g., "thumbs up" or "thumbs down") expressed by the author in an unsupervised manner.


Turney's (2002) work is perhaps one of the most notable examples of unsupervised polarity classification.

while his system learns the semantic orientation of the phrases in a review in an unsupervised manner, this information is used to predict the polarity of a review heuristically.


As expected, CRFs can perform reasonably well (accuracy = 63.9%) even without consulting the dictionary, by learning directly from the data.

having the polarity lexicon boosts the performance significantly (accuracy = 70.4%), demonstrating that lexical resources are very helpful for fine-grained sentiment analysis.


we propose a simple approach to generate a high-coverage semantic orientation lexicon, which includes both individual words and multi-word expressions, using only a Roget-like thesaurus and a handful of affixes.


For example irreverent is negative in most contexts, but positive in the sentence below: Millions of fans follow Moulder's irreverent quest for truth.

as we will show through experiments, the exceptions are far outnumbered by those that abide by the predictions of marking theory.


Thus, the method assigns a semantic orientation to a word-sense combination similar to the SentiWordNet approach and differing from the General Inquirer and Turney-Littman lexicons.

in most natural language tasks, the intended sense of the target word is not explicitly marked.


Theoretically, a much larger Turney-Littman lexicon can be created even though it may be computationally intensive when working with 100 billion words.

mSOL and TLL are created from different sources of information-mSOL from overtly marked words and a thesaurus, and TLL from co-occurrence information.


Both have the form where for RLM, , In both cases, f (w) is monotonically decreasing in the frequency of w in the corpus.

there are several differences between the two cases.


Another difference is that in RLM, P (w) is estimated on reviews with object mentions removed, since the model indicate that P (w) accounts for object-independent review language.


We use evaluations similar to those used before (Rapp, 2002;Pado and Lapata, 2007;Baroni et al., 2008, among others).

whereas most existing studies use only one dataset, or handselected parts thereof, we aim to evaluate measures across four different human datasets.


Some researchers have discovered that supplementing basic syntactic features with information about adjuncts, co-occurrences, tense, and/or voice of the verb have resulted in better performance.

additional information about semantic SPs of verbs has not yielded considerable improvement on verb classification although SPs can be strong indicators of diathesis alternations (Mc-Carthy, 2001) and although fairly precise semantic descriptions, including information about verb se-1 See section 6 for discussion on previous work.


Spectral clustering has been shown to be effective for high dimensional and non-convex data in NLP (Chen et al., 2006) and it has been applied to German verb clustering by Brew and Schulte im Walde (2002).


Inspired by the success of English grapheme-to-phoneme research in speech synthesis, many researchers have proposed phoneme-based English-to-Chinese transliteration models.

such approaches have severely suffered from the errors in Chinese phoneme-to-grapheme conversion.


Previous approaches using Chinese phonemes have relied only on Chinese phonemes in Chinese phoneme-to-grapheme conversion.

the simple use of Chinese phonemes doesn't always provide a good clue to reduce the ambiguity in Chinese phoneme-to-grapheme conversion.


Given the parameters {π 0 , π, φ, K} of the HMM, the joint distribution over hidden states s and observations y can be written (with s 0 = 0): As Johnson (2007) clearly explained, training the HMM with EM leads to poor results in PoS tagging.

we can easily treat the HMM in a fully Bayesian way (MacKay, 1997) by introducing priors on the parameters of the HMM.


For example, under our analysis, the tag 'VBG' has the features [+V, +N, -tense,en], tag 'VBD' [+V, +tense(past), -en], and 'VB' [+V, -tense(finite), -en].

since we do not consider the tense feature to be a structural feature, we do not distinguish 'VBD' from 'VB'; since N(ominal) is a structural feature, 'VBG' remains distinct from both 'VBD' and 'VB'.


For all words in the Nominal class, except for those with the ending -ly, the only possible tag for each is 'NN', since no finer categories of 'NN' exist in our reduced tagset.

for a word with ending -ly falling into the N class, we simply assume that its tag must be 'RB', although this assumption may have a few exceptions.


As shown in Table 3, reducing the dictionary by filtering rare words (with count<= d) has not been a promising track to follow for accomplishing the task with as little information as possible.

by introducing a lexicon acquisition step, we achieve a tagging accuracy of 90.6% for the 24K test data with no prior open-class lexicon, provided with only a minimal lexicon of closed-class items (about 0.6% of the full lexicon), as high as  the best previous performance of 90.4 given a full lexicon (CRF/CE with d = 1) 12 .


Building a lexicon based on induced clusters requires our morphological knowledge of three special endings in English: -ing, -ed and -s; on the other hand, to reduce the feature space used for category induction, we utilize vectors of functional features only, exploiting our knowledge of the role of determiners and modal verbs.

the above information is restricted to the lexicon acquisition model.


The reordering model and the language model are the same in the two experiments.

in forced decoding, we train two translation models, one using training data only while another using both training, dev and test data.


Ideally, we would like to estimate the parameters of the mapping function so as to directly optimize an automatic MT performance evaluation metric, such as TER or BLEU on the full translation search space.

this is extremely computationally intensive for two reasons: (a) optimizing in the full translation search space requires a new decoding pass for each iteration of optimization; and (b) a direct optimization of TER or BLEU requires the use of a derivative free, slowly converging optimization method such as MERT (Och, 2003), because these objective functions are not differentiable.


We varied tokenization of development set and test set to match the training data for each experiment.

as we have implied in the previous paragraph, in the one experiment where P (f | e) was used to segment training data, directly incorporating information from target corpus, tokenization for test and development set is not exactly consistent with tokenization of training corpus.


For both language pairs, this accurately reflects the empirical distribution of token length, as can be seen in Figure 2.

experiments where P (s) was directly optimized performed better, indicating that this parameter should be optimized within the context of a complete system.


By counting and normalizing appropriately over the entire corpus, we can straightforwardly learn the P sub and P adj distributions.

recall that in our model P ifadj is a rule-specific probability, which makes it more difficult to estimate accurately.


The number of bucket cells also effects the overall error rate significantly since smaller ranges reduce the probability of a collision.

too few cells per bucket will result in many full buckets when the bucket hash function is not highly IID.


On the other hand, corpus-based distributional measures of semantic distance, such as cosine and α-skew divergence (Dagan et al., 1999), rely on raw text alone (Weeds et al., 2004;Mohammad, 2008).

when used to rank word pairs in order of semantic distance or correct real-word spelling errors, they have been shown to perform poorly (Weeds et al., 2004;Mohammad and Hirst, 2006).


Mohammad and Hirst (2006) show that their approach performs better than other strictly corpusbased approaches that they experimented with.

all those experiments were on wordpairs that were listed in the thesaurus.


contradiction, instead of entailment (de Marneffe et al., 2008).

according to our best knowledge, the detailed comparison between these strategies has not been fully explored, let alone the impact of the linguistic motivation behind the strategy selection.


And the results show that due to the nature of these approaches based on overlapping information or similarity between T and H, this way of splitting is more reasonable.

rTE systems using semantic role labelers has not shown very promising results, although SrL has been successfully used in many other NLP tasks, e.g.


For example, the functions that construct the traditional TFIDF cosine similarity can be: where N is the size of the document collection for deriving document frequencies, tf and df are the functions computing the term frequency and document frequency, respectively.

tWEAK also takes a specified vector function f sim but assumes a parametric termweighting function tw w .


Notice that we choose these functional forms for their simplicity and good empirical performance shown in preliminary experiments.

other smooth functions can certainly be used.


As shown in Table 1, all three learned termweighting functions lead to better similarity measures compared to the TFIDF scheme in terms of the AUC and MAP scores, where the preference order learning setting performs the best.

for the precision at 3 metric, only the preference learning setting has a higher score than the TFIDF scheme, but the difference is not statistically significant 3 .


In this scenario, textual ads with bid keywords that match the query can enter the auction and have a chance to be shown on the search result page.

as the advertisers may bid on keywords that are not related to their advertisements, it is important for the system to filter irrelevant ads to ensure that users only receive useful information.


By pruning the search space, we can speed up the pattern generation process.

none of these modifications affect the accuracy of the proposed semantic similarity measure because the modified version of the prefixspan algorithm still generates the exact set of patterns that we would obtain if we used the original prefixspan algorithm (i.e.


The RASP parser is based on a manually constructed POS tag-sequence grammar, with a statistical parse selection component and a robust 1 One obvious omission is any form of dependency parser (McDonald et al., 2005;Nivre and Scholz, 2004).

the dependencies returned by these parsers are local, and it would be non-trivial to infer from a series of links whether a long-range dependency had been correctly represented.


RASP has not been designed to capture many of the dependencies in our corpus; for example, the tagsequence grammar has no explicit representation of verb subcategorisation, and so may not know that there is a missing object in the case of extraction from a relative clause (though it does recover some of these dependencies).

rASP is a popular parser used in a number of applications, and it returns dependencies in a suitable format for evaluation, and so we considered it to be an appropriate and useful member of our parser set.


The work which deals with the PTB representation directly, such as Johnson (2002), is difficult for us to evaluate because it does not produce explicit dependencies.

the DCU post-processor is ideal because it does produce dependencies in a GR format.


The lexicalized model in Charniak's parser was first optimized for English and required sophisticated smoothing to deal with sparseness; however, the lexicalized model developed for Chinese works less well.

the PCFG-LA parser learns the latent annotations from the data, without any specification of what precisely should be modeled and how it should be modeled.


The addition of self-labeled data helps on the test set initially but it provides little gain when the labeled training data becomes relatively large.

the PCFG-LA grammar is able to model the training data with different granularities.


It is clear from the figure that the improvement in parsing accuracy from self-training is the result of better bracketing across all span lengths 16 .

even though the automatically labeled training data provides more improvement than the additional treebank labeled data in terms of parsing accuracy, this data is less effective at improving tagging accuracy than the additional treebank labeled training data.


One way to deal with ambiguity is by applying distributional methods, usually requiring a large single-language corpus or, more frequently, parallel corpora.

such corpora are not readily available for many languages and domains.


We can see that while the scores for original WN terms are not perfect (7/10), single-language and cross-lingual concept extension achieve nearly the same scores.

the latter discovers many more new concept terms without reducing quality.


We can also see a decrease in precision when the algorithm is provided with 50% of the concept terms as input and had to discover the remaining 50%.

careful examination of the results shows that this decrease is due to discovery of additional correct terms not present in WordNet.


Among these, only 1 of them (#1) is a correct translation, the rest have similar or totally different meanings.

with the combined scores the faulty translations were eliminated and a new, correct, but previously average scoring translation (#2) was selected (Table 1).


During pre-evaluation type A and type B translations received a score of above 75%, while type C, type D and type E scored low (see §5.2 for details).

type F translations scored close to 80%, therefore from the six translation methods presented above we chose only three (type A, B and F) to construct the dictionary, while the remaining three methods (type C, D and E) are used only indirectly for type F selection.


It is relatively easy to automatically generate the translations of low-frequency keywords, because they tend to be less ambiguous.

the ambiguity of the high frequency words is much higher than their low-frequency counterparts, and as a result conventional methods fail to translate a considerable number of them.


On the contrary, the ambiguity of the high frequency words is much higher than their low-frequency counterparts, and as a result conventional methods fail to translate a considerable number of them.

this discrepancy is not reflected in the traditional recall evaluation, since each word has an equal weight, regardless of its frequency of use.


Nouns, adjectives and adverbs had a relatively high degree of accuracy.

verbs proved to be the most difficult POS to handle.


Spectral method such as Latent Semantic Analysis has been commonly applied for MLDC task.

current techniques strongly rely on the presence of common words between different languages.


This is because there are already enough links between multilingual documents, so we do not necessarily build more links through similarity propagation anymore.

even when there are already many links, our model with propagation still outperforms the model without propagation.


, Φ L and αm is intractable, due to the summation over an exponential number of topic assignments for these held-out documents.

recently developed methods provide efficient, accurate estimates of this probability.


The above results suggest that the Web-based suggestions system performs at least as well as the Aspell system.

it must be highlighted that results on the test set with artificial errors does not guarantee similar performance on real user data.


Explicit word acquisition data is based on interviewing adults regarding their age acquisition process during childhood and so may be unreliable and difficult to obtain for a large representative group of people.

it is possible to reliably collect large quantities of readability data defined as pairs of documents and ages of intended audience.


This raises some practical difficulties with respect to the computational maximization of the likelihood and subsequent estimation of (2).

for long documents containing a large number of words, q d (s, r) is approximately smooth which motivates a maximum likelihood procedure using gradient descent on a smoothed version of q d (s).


As a result, using a very simple metric (a line-initial '>' character) to identify reply lines achieves more than 95% accuracy.

this same simple metric applied to the Enron email data we annotated detects less than 10% of actual reply or forward lines.


Table 2 shows that average performance without n-grams (across two-, three-and nine-zone tasks) for line-based classification drops by 4.67%.

fragment-based classification accuracy drops by less than half this amount -an average of 2.26%.


Relations extracted from Wikipedia are relatively clean.

reliable distributional similarity can be calculated using a large number of documents on the Web.


This Wikipedia-based approach can extract a large volume of hyponymy relations with high accuracy.

it is also true that this approach does not account for many words that usually appear in Web documents; this could be because of the unbalanced topics in Wikipedia or merely because of the incomplete coverage of articles on Wikipedia.


1997) and Bunrui-Goi-Hyo (1996) contain approximately 300,000 words and 96,000 words, respectively.

the extracted hyponymy relations contain approximately 1.2 million hyponyms and are undoubtedly much larger than the existing taxonomies.


Note that the extracted relations have a hierarchical structure because one hypernym of a certain word may also be the hyponym of another hypernym.

we observed that the depth of the hierarchy, on an average, is extremely shallow.


The number of common words that are also included in the Wikipedia relation database are as follows: Hypernyms 28,015 (common hypernyms) Hyponyms 175,022 (common hyponyms) These common hypernyms become candidates for hypernyms for a target word.

the common hyponyms are used as clues for identifying appropriate hypernyms.


WordNet contains 20% of the Animal concepts and 51% of the People concepts learned by our algorithm, which confirms that many of these concepts were considered to be valuable taxonomic terms by the WordNet developers.

our human annotators judged 57% of the Animal and 84% of the People concepts to be correct, which suggests that our algorithm generates a substantial number of additional concepts that could be used to enrich taxonomic structure in WordNet.


This poses a bottleneck, requiring expertise in both machine learning and the application domain.

domain experts often express their knowledge through text; one direct expression is through text designed to aid human learning.


We envision an application scenario in which a designer manually specifies a few glosses for each predicate.

for the purposes of evaluation, it would be unprincipled for the experimenters to handcraft the ideal set of glosses.


Also, we retain twinless system mentions that are nonsingletons, as the resolver should be penalized for identifying spurious coreference relations.

we do not remove twinless mentions in the key partition, as we want to ensure that the resolver makes the correct (non-)coreference decisions for them.


Knowledge of noun phrase anaphoricity might be profitably exploited in coreference resolution to bypass the resolution of non-anaphoric noun phrases.

it is surprising to notice that recent attempts to incorporate automatically acquired anaphoricity information into coreference resolution have been somewhat disappointing.


Experiments show that the proposed method improves the performance by 2.9 and 1.6 to 67.3 and 67.2 in F1-measure on the MUC-6 and MUC-7 corpora, respectively, due to much more gain in precision compared with the loss in recall.

surprisingly, their experiments also show that eliminating non-anaphors using an anaphoricity determination module in advance harms the performance.


Experiments on the NWIRE, NPAPER and BNEWS domains of the ACE 2003 corpus shows that this joint anaphoricity-coreference ILP formulation improves the F1-measure by 0.7-1.0 over the coreferenceonly ILP formulation.

their experiments assume true ACE mentions(i.e.


This algorithm has been shown to converge to a unique solution (Zhu and Ghahramani 2002), which can be obtained without iteration in theory, and the initialization of Y U 0 (the unlabeled data) is not important since Y U 0 does not affect its estimation.

proper initialization of Y U 0 actually helps the algorithm converge more rapidly in practice.


A successful PCDC must accurately extract the relevant context for coreference.


The existence of such rules makes it possible for the disambiguation decisions to be made considering the local context.

the distribution of the PNMs in a corpus is rather random and the relevant coreference context is a dynamic variable depending on the diversity of the corpus, that is, on how many different persons with the same name share a similar context.


A typical case for this situation is when there is a person that is very often mentioned, and few other persons having few mentions; when the number of clusters is passed in the input, the clusters representing the persons who are rarely mentioned are wrongly enriched.

this situation can be avoided if there is a measure of how probable it is to have a certain number of different persons with the same name, each being mentioned very often in a newspaper.


To find out such relationships is computationally very hard.

the analysis carried out further shows that we can avoid making such computations in most of the cases.


The number of different persons is a parameter that cannot be known beforehand.

not all the names behave alike with respect to coreference.


Considering any two PNMs of the same name the similarity of two of the professional contexts guarantees the correct coreference.

two professional contexts are present in only 4% of the cases.


The chance that many different persons carry this name is high.

as both "Barack" and "Obama" are rare American first and last names respectively, almost surely many mentions of this name refer only to one person.


If we knew the distribution function of Y, let's call it F, we would simply determine ξ i from equation 1, where P i =Σp k , k ≤ i : and we would know that in each partition p i the name perplexity is between ξ i-1 and ξ i, with ξ 0 = 0.

we do not know F. Fortunately, we can estimate ξ i .


Combining all these pairwise preferences to find the best global reordering is NP-hard.

we present a non-trivial O(n 3) algorithm, based on chart parsing, that at least finds the best reordering within a certain exponentially large neighborhood.


a novelty is that the grammar weights must themselves be computed by O(n 3 ) dynamic programming.


During training, we iterate the local search as described earlier.

for decoding, we only do a single step of local search, thus restricting reorderings to the ITG neighborhood of the original German.


While discriminative methods show superior alignment accuracy in benchmarks, generative methods are still widely used to produce word alignments for large sentence-aligned corpora.

neither generative nor discriminative alignment methods are reliable enough to yield high quality alignments for SMT, especially for distantly-related language pairs such as Chinese-English and Arabic-English.


According to Och's algorithm, the target phrase "China" breaks the alignment consistency and therefore is not valid candidate.

this is not true for using the weighted matrix shown in Figure 2(c).


This way, each sentence pair may generate any number of potentially overlapping biphrases.

when defining the phrase-based sentence level translation model, phrase overlaps are explicitly disallowed: The source sentence is segmented into disjoint phrases, which are translated independently using conditional phrase-level translation models that have been estimated from extracted biphrase counts.


This means it is straightforward to use probabilistic criteria in learning the model parameters.

systems modelling P (y|x) directly are often plagued by the reference reachability problem.


The prediction strategy outlined in the previous section is simple and conceptually clean.

biphrase overlaps alone may not be enough to enforce fluent output, especially given that bilingual data is typically more scarce than monolingual data.


A partial explanation for the good relative performance could be that the challenge participants had only a week to train their models on the full version of GigaFrEn data, so they may not have had time to take full advantage of it.

many of the top ranked systems relied on external resources that were not available for us.


the maximum number of nodes and the maximum height of fragment, to limit the number of possible fragments.

these heuristics are very subjective and hard to optimize.


It means the pruned forest is able to at least keep all the top n best trees.

because of the sharing nature of the packed forest, it may still contain a large number of additional trees.


These resources are valuable for human-consumption and can also be exploited in order to learn computational resources (Medelyan et al., 2008;Weld et al., 2008;Zesch et al., 2008b;Zesch et al., 2008a).

it is possible to acquire useful resources and knowledge from aggregating behavioral patterns of large groups of people, even in the absence of a conscious effort.


(2005) presented a semantic inference framework which "augments" the text representation with only the right-hand-side of an applied rule, and in this respect is similar to ours.

in their work, both rule application and the semantics of the resulting "augmented" structure were not fully specified.


Both approaches train using a structured perceptron, as we do here.

these models represent a dramatic departure from the existing literature, while ours has clear analogs to the well-known noisy-channel paradigm, which allows for useful comparisons and insights into the advantages of discriminative training.


Given the character domain's lack of sparsity, and the large amount of available training data, we had expected the hybrid generative system to behave only as a strong baseline; instead, it matched the performance of the indicator system.

this is not unprecedented: discriminatively weighted generative models have been shown to outperform purely discriminative competitors in various NLP classification tasks (Raina et al., 2004;Toutanova, 2006), and remain the standard approach in statistical translation modeling (Och, 2003).


Significant efforts have been made in attempt to learn a generic ranking model which can appropriately rank documents for all queries .

web users' query intentions are extremely heterogeneous, which makes it difficult for a generic ranking model to achieve best ranking results for all queries.


There are a few related works to apply multiple ranking models for different query categories.

none of them takes click-through information into consideration.


For example, in Table 5 (a), the NDCG 5 values (0.7822 and 0.7834) are very close to each other.

in Figure  3, we find that with the same amount of pairs, when we use 30,000 or fewer pairs, using dedi-  cated click pairs alone is always better than using generic click pairs alone.


Although the correct translation can also be composed by two phrases 海水 ⇒ and 淡化 ⇒ , its overall translation score cannot beat the incorrect one because the combined phrase translation probability of these two phrases are much smaller than ( |海水 淡化) .

if we intentionally remove the ( | ) feature from the model, the preferred translation can be generated as shown in the result of − because in this way the bad estimation of ( | ) for this phrase is avoided.


Our method is similar to the work proposed by Hildebrand and Vogel (2008).

except the language model and translation length, we only use intra-hypothesis n-gram agreement features as Hildebrand and Vogel did and use additional intra-hypothesis n-gram disagreement features as Li et al.


As each edge in the confusion network only has a single word, it is possible to produce inappropriate translations such as "He is like of apples".

we allow many-to-many mappings in the hypothesis alignment shown in Figure 2(b).


One backbone arc in a lattice can only span one backbone word.

all hypothesis words in an alignment unit must be packed into one hypothesis arc.


Part-of-speech tags can be easily obtained for unannotated data using off-the-shelf POS taggers or PCFG parsers.

the amount of information these tags typically provide is very limited, Figure 1: A parse tree example e.g., while it is helpful to know whether fly is a verb or a noun, knowing that you is a personal pronoun does not carry the information whether it is a subject or an object (given the Penn Tree Bank tagset), which would certainly help to predict the following word.


<unk>, effectively changing the vocabulary thus making perplexity incomparable to models without these factors, without improving WER noticeably.

we do plan to use more overt factors in Machine Translation experiments where a language model faces a wider range of OOV phenomena, such as abbreviations, foreign words, numbers, dates, time, etc.


It would seem that rare lexical items are indeed crucial for SVM classification performance.

in Goldberg and Elhadad (2007), we suggested that the SVM learner is using the rare lexical features for singling out hard cases rather than for learning meaningful generalizations.


This was followed by (Bikel, 2004) who showed that bilexical-information is used in only 1.49% of the decisions in Collins' Model-2 parser, and that removing this information results in "an exceedingly small drop in performance".

uni-lexical information was still considered crucial.


Here, the learning objective is to minimize: Interestingly, for the linear kernel, SVM-anchoring reduces to L2-SVM with C=1.

for the case of non-linear kernels, anchored and L2-SVM produce different results, as the anchoring is applied prior to the kernel expansion.


As we show in Sect.5.4, fine-tuning the C parameter reaches better accuracy than L1-SVM with C=1.

as this fine-tuning is computationally expensive, we first report the comparison L1-SVM/C=1 vs. anchored-SVM, which consistently reached the best results, and was the quickest to train.


Either of the results are state-of-the-art for this task.

even modest pruning (k = 2) hurts the soft-margin model significantly.


Unlike the NPchunking case, here feature pruning has a relatively large impact on the results even for the anchored models.

the anchored models are still far more robust than the soft-margin ones.


When moving outside of the canonic training  corpus, the fully lexicalized model have no advantage over the heavily pruned one.

the pruned models seem to have a small advantage in most cases (though it is hard to tell if the differences are significant).


The two predicate argument relations thus took the same word as their common arguments, and therefore the two errors co-occurred.

one-way inductive relations also exist among errors.


From the sentence in the figure, we can obtain two errors for "Prepositional attachment" around prepositions "to" and "for."

each "Predicate type selection" pattern collects errors around a word whose predicate type is erroneous.


Certainly these templates or word lattices are more useful in such NLP applications as Q&A than simple entailment relations between verbs.

our contention is that entailment certainly holds for some verb pairs (like snore → sleep) by themselves, and that such pairs constitute the core of a future entailment rule database.


Then, if we can directly estimate is large enough.

we cannot estimate P (v r |v l ) directly since it is unlikely that we will observe the verbs v r and v l at the same time.


Thirdly, (Shen et al., 2008) deploys the dependency language model to augment the lexical language model probability be-tween two head words but never seek a full dependency graph.

our approach integrates an incremental parsing capability, that produces the partial dependency structures incrementally while decoding, and thus provides for better guidance for the search of the decoder for more grammatical output.


In these cases, the systems may choose equivalent paraphrases.

the translations using syntactic structures are rather similar.


For example, if in such method, Hypothesis 3 is first aligned to the backbone, followed by Hypothesis 1, we are likely to arrive at the CN in Figure 2b) in which the two instances of Jeep are aligned.

if Hypothesis 1 is aligned to the backbone first, we would still get the CN in Figure 2a).


sometimes the state may contain a few input words that have not been visited.


If resilient changes function and becomes a noun modifier, its modifiers must change category too: There is often a way to analyse around the need for type-changing operations in CCG.

these solutions tend to cause new difficulties, and the resulting category ambiguity is quite problematic (Hockenmaier and Steedman, 2002).


So far, we have focused on replacing the phrasestructure rules added to CCGbank, which are not part of the CCG linguistic theory.

the theory does include some type-changing rules, referred to as type-raising.


Since each of the combinators is hard-coded for speed, this was time-consuming and error prone.

we created a detailed set of regression tests for the new versions which greatly reduced our development time.


The derivs model uses features calculated over the derivations, while the hybrid model uses features calculated on the dependency structures.

unlike the deps model Clark and Curran (2007) describe, the hybrid model uses two sets of derivation-based constraints.


The lexicalised type-changing scheme we have proposed offers many opportunities for favourable analyses, because it allows form and function to be represented simultaneously.

we have limited our changes to replacing the existing CCGbank non-combinatory rules.


In this case we say feature c(s t−1 , s t ) =+, which encourages "reduce".

in Figure 3(b), the source span is still [saw .. Bill], but this time maps onto a much longer span on the Chinese side.


WordNet is just one of the several knowledge sources which have been utilized.

the WordNet based features is not informative compared to other features such as the semantic neighbor feature.


(2006) used speech acts to capture the intentional focus of emails and discussion boards.

they assume that enough labeled data are available for developing speech act recognition models.


(2006)'s work, Ravi and Kim (2007) applied speech act classification to detect unanswered questions.

none of these studies have focused on the semisupervised speech act recognition problem and examined their methods across different genres.


(2005) presented semi-supervised learning to employ auxiliary unlabeled data in call classification, and is closely related to our work.

our approach uses the most discriminative subtree features, which is particularly attractive for reducing the model's size.


Previous work in speech act recognition used a large set of lexical features, e.g., bag-of-words, bigrams and trigrams (Stolcke et al., 2000;Cohen et al., 2004;Ang et al., 2005;Ravi and Kim, 2007).

these methods create a large number of lexical features that might not be necessary for speech act identification.


We can find the correct parts of speech of the composite characters of a word when it is an example word in the dictionary.


The performance of opinion extraction boosts to an f-score 0.80 and the performance of polarity detection an f-score 0.54.

the utilization of structure trios needs the parsing tree of sentences as the prior knowledge.


Virtual evidence (VE), first introduced by Pearl (1988), offers a principled and convenient way of incorporating external knowledge into Bayesian networks.

to standard evidence (also known as observed variables), VE expresses a prior belief over values of random variables.


We let P l denote a prototype list associated with the label l. If x t belongs to P l , we should prefer y t = l as opposed to other values.

to this end, for cases where x t ∈ P l , we set s 1 as if x t is not a prototype, we will always have s 1 (y t , v t , t) = 0 for all hypotheses of y t .


If it is determined that x t is not the start of a sentence, we set s 2 as It is easy to see that this would penalize state transitions within a sentence.

if x t is a sentence start, we set s 2 (y t−1 , y t , v t , t) = 0 for all possible (y t−1 , y t ) pairs.


Lexicalized PCFGs use the structural features on the lexical head of phrasal node in a tree, and get significant improvements for parsing (Collins, 1997;Charniak, 1997;Collins, 1999;Charniak, 2000).

they suffer from the problem of fundamental sparseness of the lexical dependency information.


By splitting and merging alternately, this method can refine the grammars step by step to mitigate the overfitting risk to some extent.

this data-driven method can not solve this problem completely, and we need to find other external information to improve it.


To our knowledge, traditional indexing consistency metrics have not yet been applied to collaboratively tagged data.

experiments on determining tagging quality do follow the same idea.


(2006) automatically suggest tags previously assigned to similar documents.

in Maui (as in Kea) this feature is just one component of the overall model.


Supervised methods for automatic verb classification have been extensively investigated Merlo and Stevenson, 2001;Joanis et al., 2008).

their focus has been limited to a small subset of verb classes, and a limited number of monosemous verbs.


Previous works thus proposed features that express the corpus statistics of syntactic frames.

class boundaries are subtle in some cases; several classes share syntactic frames with each other to a large extent.


One possible direction for this research topic would be to use our model for the semi-automatic construction of verb lexicons, with the help of human curation.

there is also a demand for exploring other types of features that can discriminate among confusing classes.


Enabled by the development of Web 2.0 technology and created by communities of volunteers, CKBs have emerged as a new source of lexical semantic knowledge in recent years.

to LKBs, they are created by persons with diverse personal backgrounds and fields of expertise.


the majority of SR values lie between 0 and 0.1.


The large difference of average document length and query term instances suggests a larger difference of the average relevance scores than 20%.

in the BM25 model the relevance score is decreased with increasing document length and additional occurrences of a query term have little impact after three or four occurrences.


Also SR-WordNet outperforms Wikipedia for hypernymy and hyponymy.

to Wiktionary, no direct information about related words is used to construct the textual representation of concepts.


In contrast to Wiktionary, no direct information about related words is used to construct the textual representation of concepts.

the very short and specific representations are built from glosses and examples which often contain hypernym-hyponym pairs.


On the negative side, our approach is limited in that it requires that X 2 be related to X 1 , while the pivoting language Z does not need to be related to X 1 nor to Y .

we only need one additional parallel corpus (for X 2 -Y ), while pivoting needs two: one for X 1 -Z and one for Z-Y .


Finally, we also report in the last row (in italic) the superior RE result by (Zhou et al., 2007).

to achieve this outcome the authors used the composite kernel CK 1 with several heuristics to define an effective portion of constituent trees.


Using a manually annotated corpus for relation extraction has one particular advantage compared to extraction from plain text: the type of an argument-predicate relation is already annotated; there is no need to determine it by automatic means which are usually error-prone.

the relatively small size of SALSA does not allow to make relevant predictions about the degree of semantic relatedness in the extracted argument-predicate pairs, see section 4.


Most of the literature discusses predicates inferable from nouns.

other parts of speech can support similar inferences.


The calculated F max values are reported in table 2 which shows a correlation between F max values calculated for the "semantic" and "syntactic" procedures.

there is no correlation with human agreement.


An obvious limitation of the presented approach is that it is bounded to manual annotations which are hard to obtain.

since semantic annotations are useful for many different goals in linguistics and NLP, the number of reliable annotated corpora constantly grows.


is an example of a complex keyword, since the words forming the phrase can only express speculation together.

to the minimalist strategy followed when annotating the keywords, the annotation of scopes of the keywords was performed by assigning the scope to the largest syntactic unit possible by including all the elements between the keyword and the target word to the scope (in order to avoid scopes without a keyword) and by including the modifiers of the target word to the scope (Vincze et al., 2008).


For example, the keyword "suggest" has been used in a speculative context in all its occurrences in the abstracts and in the full papers.

"appear" is a real speculation keyword in 86% of its occurrences in the abstracts and in 83% of its occurrences in the full papers, whereas "can" is a real speculation keyword in 12% of its occurrences in the abstracts and in 16% of its occurrences in the full papers.


Our continuous objective can be optimized using simple gradient ascent.

computing critical quantities in the gradient necessitates a novel dynamic program, which we also present here.


For the remainder of this paper, we focus on consensus BLEU.

the techniques herein, including the optimization approach of Section 3, are applicable to many differentiable functions of expected n-gram counts.


This plot exhibits a known issue with MERT training: because new k-best lists are generated at each iteration, the objective function can change drastically between iterations.

coBLEU converges  Table 1: Performance measured by BLEU using a consensus decoding method over translation forests shows an improvement over MERT when using coBLEU training.


The query, ebay official, is assumed to be commercial intent, because a large fraction of the clicks are on ads.

typos tend to have relatively more clicks on "did-you-mean" spelling suggestions.


DIPRE (Dual Iterative Pattern Relation Expansion) (Brin, 1998) is a system based on bootstrapping that exploits the duality between patterns and relations to augment the target relation starting from a small sample.

it only extracts simple relations such as (author, title) pairs from the WWW.


We also found some basic emotions tend to combine together, such as {expect, joy, love}, {anxiety, sorrow}, {angry, hate}.

some emotions have small or scarce possibility appear together, such as joy and hate, surprise and angry.


In English, overt pronouns such as "she" and definite noun phrases such as "the company" are anaphors that refer to preceding entities (antecedents).

in Japanese, anaphors are often omitted, which are called zero pronouns, and zero anaphora resolution is one of the most important techniques for semantic analysis in Japanese.


When we used the decay rates smaller than 0.5, the recall score worsened clearly.

although we expected to obtain higher precision with small decay rate, the highest precision was achieved by the decay rate 0.5.


It is therefore crucial that all annotations at a high level of interpretation are backed up by human annotation with more than one annotator.

annotations of citation function classification typically use only the untested annotation of a single human annotator as gold standard, who is typically the designer of a scheme (Spiegel-Rüsing, 1977;Weinstock, 1971;Nanba and Okumura, 1999;Garzone and Mercer, 2000).


In these cases the guidelines seem to fully suffice for their description, but then again good performance of AIM, FUT and USE is not that surprising, as they are signalled clearly by linguistic and non-linguistic cues.

there are three categories with particularly low distinguishability in both disciplines: ANTISUPP, OWN FAIL and PREV OWN.


The definition of the categories SUPPORT and NOV ADV also seem to be substantially more confusing for CL than for chemistry.

cODI is a category which shows average distinctiveness for cL, but much worse distinctiveness for chemistry.


Overall, only 25 (0.02%) official symbols were ambiguous within the organisms.

when official symbols from 21 organisms were combined, the ambiguity increased substantially to 21, 279 (14.2%) symbols.


It is expected that the first rule would produce good precision.

it can only disambiguate the fraction of entities that happen to have a species word to their immediate left.


High performance can often be achieved if the system is trained and tested on data from the same domain.

the performance of NLP systems often degrades badly when the test data is drawn from a source that is different from the labeled data used to train the system.


It is often used when labeled training data is scarce but unlabeled data is abundant.

for domain adaptation problems, we may have a lot of training data but the target application domain has a different data distribution.


It has also been applied to the problem of domain adaptation for word sense disambiguation in (Chan and Ng, 2007).

active learning requires human intervention.


Regarding the selection-criterion, standard and balanced bootstrapping both select instances which are confidently labeled by SC t to be used for training SC t+1 , in the hope of avoiding using wrongly labeled data in bootstrapping.

instances that are already confidently labeled by SC t may not contain sufficient information which is not in D S , and using them to train SC t+1 may result in SC t+1 performing similarly to SC t .


Balanced bootstrapping has been shown to be more effective for domain adaptation than standard bootstrapping (Jiang and Zhai, 2007) for named entity classification on a subset of the dataset used here.

we found that both methods perform poorly on domain adaptation for NER.


In named entity classification, the names have already been segmented out and only need to be classified with the appropriate class.

for NER, the names also  Table 6: F-measure of the cross-topic transfer.


when BN is the source domain, the capitalization information can be discovered by DAB.


Their kernel is defined on lexical dependency tree by the convolution of similarities between all possible subtrees.

if the convolution containing too many irrelevant subtrees, over-fitting may occur and decreases the performance of the classifier.


We observe that the three learning based methods(SVM-1, SVM-WTree, SVM-PTree) perform better than the Adjacent baseline in the first three domains.

in other domains, directly adjacent method is better than the learning based methods.


This latter work was then further improved by Huang (2008) to 91.7%, by utilizing the benefit of forest structure.

one of the limitations of these techniques is the huge number of features which makes the training very expensive and inefficient in space and memory usage.


For each tree, its constituent count is the sum of all the counts of its constituent.

as suggested in (Sagae and Lavie 2006), this feature favours precision over recall.


Also many works have discussed the issues, such as word segmentation, POS tagging etc, between English and Chinese (Wang et al., 2006;Wu et al., 2003).

to the best of our knowledge, no studies have been reported on discussing preprocessing techniques on Chinese document and sentence-level novelty mining, which is the focus of our paper.


The problem of re-ranking initial retrieval results exploring the intrinsic structure of documents is widely researched in information retrieval (IR) and has attracted a considerable amount of time and study.

one of the drawbacks is that those algorithms treat queries and documents separately.


In their work, LDA-based document model and language model-based document model were linearly combined to rank the entire corpus.

unlike this approach we only apply LDA to a small set of documents.


Given a query , a set of initial results ∈ of top documents are returned by a standard information retrieval model (initial ranker).

the initial ranker tends to be imperfect.


Prior to It is worth noting that the CLEF-2008 TEL data is actually multilingual: all collections to a greater or lesser extent contain records pointing to documents in other languages.

this is not a major problem because the majority of documents in the test collection are written in main languages of those test collections (BL-English, BNF-French).


The search ranges for these two parameters were:  : 0.1, 0.2, …, 0.9 k : 5, 10, 15, …, 45 As it turned out, for many instances, the optimal value of with respect to MAP was either 0.1 or 0.2, suggesting the initial retrieval scores have valuable information inside them.

the optimal value of was between 20 and 40.


As expected, the RWILM method bought improvements in many cases in CLEF-2008 test collections.

the performance over CLEF-2007 collection was somewhat disappointing.


The increased expressiveness of the model, combined with the more robust parameter estimates provided by the smoothing, results in a nice increase in parsing accuracy on a held-out set.

as reported by Petrov (2009) and Huang and Harper (2009), an additional 7th SM round actually hurts performance.


This specialization leads to greater diversity in their prediction preferences, especially in the presence of a small training set.

the self-labeled training set size is much larger, and so the specialization process is therefore slowed down.


They have provided evidence that syntactic consistency exists not only within coordinate structures, but also in a variety of other contexts, such as within sentences, between sentences, within documents, and between speaker turns in the Switchboard corpus.

their analysis rests on a selected number of constructions concerning the internal structure of noun phrases.


Sentiment analysis (Pang and Lee, 2008) offers the promise of automatically discerning how people feel about a product, person, organization, or issue based on what they write online, which is potentially of great value to businesses and other organizations.

the vast majority of sentiment resources and algorithms are limited to a single language, usually English (Wilson, 2008;Baccianella and Sebastiani, 2010).


The conditional topic distribution for SLDA (Blei and McAuliffe, 2007) replaces this term with the standard Multinomial-Dirichlet.

we believe this is the first published SLDA-style model using MCMC inference, as prior work has used variational inference (Blei and McAuliffe, 2007;Chang and Blei, 2009;Wang et al., 2009).


Supervised learning also suffers from its heavy dependence on training data.

unsupervised, knowledge-lean topic modeling approach has been shown to be effective in automatically identifying aspects and their representative words (Titov and McDonald, 2008;Brody and Elhadad, 2010).


(2007) propose to separate topic and sentiment words using a positive sentiment model and a negative sentiment model, but both models capture general opinion words only.

we model aspect-specific opinion words as well as general opinion words.


As for lexical features, words from a sentiment lexicon can also be helpful in discovering opinion words.

lexical features are more diverse so presumably we need more training data in order to detect useful lexical features.


Recognizing plans and goals depends on world knowledge and inference, and is beyond the scope of this paper.

we identified two cases where affect states often can be inferred based on syntactic properties.


Our simple heuristics for creating links work surprisingly well for xchar and a links when given perfect affect states.

these heuristics produce relatively low precision for m links, albeit with 100% recall.


There exists some work to remove noise from SMS (Choudhury et al., 2007) (Byun et al., 2008) (Aw et al., 2006) (Neef et al., 2007 (Kobus et al., 2008).

all of these techniques require an aligned corpus of SMS and conventional language for training.


We compute the score of each question in C using Score(Q) and the question with highest score is treated asQ h .

the naive approach suffers from high runtime cost.


If we consider the question Q 24 below as reference, question Q 26 will be deemed more useful than Q 25 when using cos or mcs because of the higher relative lexical and conceptual overlap with Q 24 .

this is contrary to the actual ordering Q 25 ≻ Q 26 |Q 24 , which reflects the fact that Q 25 , which expects the same answer type as Q 24 , should be deemed more useful than Q 26 , which has a different answer type.


retrieving with high precision, low recall).

current search engines are still facing the lexical ambiguity issue (Furnas et al., 1987) -i.e.


Users can then select the cluster(s) and the pages therein that best answer their information needs.

many Web clustering engines group search results on the basis of their lexical similarity.


Over the years, different methods for SIR have been proposed (Krovetz and Croft, 1992;Voorhees, 1993;Mandala et al., 1998;Gonzalo et al., 1999;Kim et al., 2004;Liu et al., 2005a, inter alia).

contrasting results have been reported on the benefits of these techniques: it has been shown that WSD has to be very accurate to benefit Information Retrieval (Sanderson, 1994) -a result that was later debated (Gonzalo et al., 1999;Stokoe et al., 2003).


Our triangular measure is the edge counterpart of the clustering coefficient (or curvature) for vertices, previously used to perform WSI (Widdows and Dorow, 2002).

it is our hunch that measuring the ratio of squares an edge participates in provides a stronger clue of how important that edge is within a meaning component.


We note that the ranking and optimality of clusters can be improved with more sophisticated techniques (Crabtree et al., 2005;Kurland, 2008;Kurland and Domshlak, 2008;Lee et al., 2008, inter alia).

this is outside the scope of this paper.


The above considerations might not seem intuitive at first glance, as the average polysemy of longer queries is lower (17.9 on AMBIENT vs. 6.7 on MORESQUE according to our gold standard).

we note that while the kind of ambiguity of 1-word queries is generally coarser (e.g., beagle as dog vs. lander vs. search tool), with longer queries we often encounter much finer sense distinctions (e.g., Across the Universe as song by The Beatles vs. a 2007 film based on the song vs. a Star Trek novel vs. a rock album by Trip Shakespeare, etc.).


This would clearly exclude tasks like translation of medical reports, business contracts, or literary works, where the validation of a qualified bilingual translator is absolutely necessary.

it does include a great many real-world scenarios, such as following news reports in another country, reading international comments about a product, or generating a decent first draft translation of a Wikipedia page for Wikipedia editors to improve.


As shown in , hierarchical phrase-based models significantly outperform tree-to-string models (Liu et al., 2006;Huang et al., 2006), even when attempts are made to alleviate parsing errors using either forest-based decoding  or forest-based rule extraction .

when properly used, syntactic constraints can provide invaluable benefits to improve translation quality.


In our example above, rule (2) can be extracted from rule (1) with the following sub phrase pair: The use of a unified X nonterminal makes hierarchical phrase-based models flexible at capturing non-local reordering of phrases.

such flexibility also comes at the cost that it is not able to differentiate between different syntactic usages of phrases.


The binarized decomposition forest compactly encodes the hierarchical structure among phrases and non-phrases.

the coarse abstraction of phrases with X and non-phrases with B provides little information on the constraints of the hierarchy.


Instead, we use the raw counts for the two models # m (f ,ē) and # w→m (f ,ē) directly as follows: For lexicalized translation probabilities, we would like to use simple interpolation.

we notice that when a phrase pair belongs to only one of the phrase tables, the corresponding lexicalized score for the other table would be zero.


Self-training improves over the baseline by about 0.6% on the de-velopment set.

the gains from self-training are more modest (0.2%) on the evaluation (test) set.


Most previous punctuation prediction techniques, developed mostly by the speech processing community, exploit both lexical and prosodic cues.

in order to fully exploit prosodic features such as pitch and pause duration, it is necessary to have access to the original raw speech waveforms.


The skip-chain CRF (Sutton and McCallum, 2004), another variant of linear-chain CRF, attaches additional edges on top of a linear-chain CRF for better modeling of long range dependencies between states with similar observations.

such a model usually requires known long range dependencies in advance and may not be readily applicable to our task where such clues are not explicit.


The majority of the sentences are declarative sentences.

question sentences are more frequent in the BTEC dataset compared to the CT dataset.


Thus, duplicating the ending punctuation symbol to the start of a sentence so that it is near these indicative words helps to improve the prediction accuracy.

chinese presents quite different syntactic structures for question sentences.


This simple modification of the HMM takes advantage of the dichotomy in natural language between content and function words.

a standard HMM draws all prior distributions once over all states and it is known to perform poorly in unsupervised and semisupervised POS tagging.


Similarly peaked distributions are observed for other function categories such as MD and CC.

the joint probability of any word occurring with NN is much less likely to be zero and the distribution is much less likely to be peaked.


In the past few years, several authors have studied the problem of gender classification in the natural language processing and linguistic communities.

most existing works deal with formal writings, e.g., essays of people, the Reuters news corpus and the British National Corpus (BNC).


(Houvardas and Stamatatos, 2006) even applied character (rather than word or tag) n-grams to capture stylistic features for authorship classification of news articles in Reuters.

these works use only one or a subset of the classes of features.


Moreover, having many feature classes is very useful as they provide features with varied granularities and diversities.

this also results in a huge number of features and many of them are redundant and may obscure classification.


POS n-grams can also be used as features.

since we mine all POS sequence patterns and use them as features, most discriminative POS ngrams are already covered.


A set of top ranked features are selected.

the wrapper model chooses features and adds to the current feature pool based on whether the new features improve the classification accuracy.


A fundamental assumption is that the training and test data are identically distributed.

this assumption may not hold in practice.


Since the hidden positives in U should have the same behaviors as the positives in P in terms of their similarities to pr, we set their minimum similarity as the threshold value ω which is the minimum similarity before a document is considered as a potential negative document: In a noiseless scenario, using the minimum similarity is acceptable.

most real-life applications contain outliers and noisy artifacts.


Hence, to obtain a paragraph's function label, we need to first label its sentences.

we are faced with the same problem: how can we obtain the sentence function labels?


If a paragraph contains Thesis, Prompt, or Background sentences, the paragraph is likely to be an Introduction.

if a paragraph contains Main Idea, Support, or Conclusion sentences, it is likely to be a Body paragraph.


Figures 3(a)   counts, respectively, for the 20 Newsgroups dataset.

without pre-processing the number of types scales from 107,211 to 892,983 and the number of tokens from 2,261,805 to 3,073,208.


The advantage of pre-translation is that MT systems tend to preserve the meaning of documents.

mT can be very slow (more than 1 second per document), preventing its use on large training sets.


Translating word-by-word operates at 274K words per second.

machine translation processes 50 words per second, approximately 3 orders of magnitude slower.


The tests show that OPCA is better than CCA, CL-LSI, plain word-by-word translation, and even full translation for Spanish documents.

if we post-process full translation by an LSI model trained on the English training set, full translation is the most accurate.


In practice, the decoder has to employ beam search to make it tractable (Koehn, 2004).

even beam search runs in quadratic-time in general (see Sec.


Strong indications of perspective can often come from collocations of arbitrary length; for example, someone writing get the government out of my X is typically expressing a conservative rather than progressive viewpoint.

going beyond unigram or bigram features in perspective classification gives rise to problems of data sparsity.


Indeed, it is not uncommon to see the feature space for sentiment analysis limited to unigrams.

important indicators of perspective can also be longer (get the government out of my).


Briefly, adaptor grammars allow nonterminals to be rewritten to entire subtrees.

a nonterminal in a PCFG rewrites only to a collection of grammar symbols; their subsequent productions are independent of each other.


It is easy to find clearer ambiguities online, such as compositional examples of typically noncompositional verbs (how to recover a couch, when to redress a wound, etc.).

in our data verbs like recover and redress always occur in their more dominant non-compositional sense.


Noncompositionality is the majority class on the examples that are in the dictionary.

one would expect verbs that are not in a comprehensive dictionary to be largely compositional, and indeed most of the / ∈ CELEX verbs are compositional.


This relation is obvious when reading the sentence, so it is omitted by the writer.

any semantic representation of text needs as much semantics as possible explicitly stated.


Information-extraction (IE) research typically focuses on clean-text inputs.

an IE engine serving real applications yields many false alarms due to less-well-formed input.


The mixed and gazetteer systems, having a variety of noisy data in their training set, perform much better on the noisy conditions, particularly on Latin-alphabet-non-English data because that is one of the conditions included in its training, while Transactions remains a condition not covered in the training set and so shows less improvement.

because the mixed classifier, and moreso the gazetteer classifier, are oriented to noisy data, on clean data they suffer in performance by 2.5 and 5 F -measure points, respectively.


Alternatively,  bootstrap a set of weighted support vectors from both labeled and unlabeled data using SVM and feed these instances into semi-supervised relation extraction.

their seed set is sequentially generated only to ensure that there are at least 5 instances for each relation class.


adopt a stratified sampling strategy to select the seed set.

their method needs a stratification variable such as the known distribution of the relation types, while our method uses clustering to divide relation instances into different strata.


Normally, this number is proportional to the size of that stratum in the whole data set.

in case this number is 0 due to the rounding of real numbers, it is set to 1 to ensure the existence of at least one seed from that stratum.


The method in (Haveliwala, 2002), is similar to TPR which also decompose PageRank into various topics.

the method in (Haveliwala, 2002) only considered to set the preference values using pr(w|z) (In the context of (Haveliwala, 2002), w indicates Web pages).


In general, the minimization of the normalized cut criterion is NP-complete.

the linearity constraint on text segmentation for monologue allows them to find an exact solution in polynomial time.


Email programs (e.g., Gmail, Yahoomail) group emails into threads using headers.

our annotations show that topics change at a finer level of granularity than emails.


In the second phase the annotators identify the most appropriate topic for each sentence.

if a sentence covers more than one topic, they were asked to label it with all the relevant topics according to their order of relevance.


Automatic detection of discourse relations in natural language text is important for numerous tasks in NLP, such as sentiment analysis (Somasundaran et al., 2009), text summarization (Marcu, 2000) and dialogue generation (Piwek et al., 2007).

most of the recent work employing discourse relation classifiers are based on fully-supervised machine learning approaches (duVerle and Prendinger, 2009;Pitler et al., 2009;Lin et al., 2009).


Some of the relations corresponding to these classes are relatively more frequent in the corpus, such as the ELAB- ORATION[N][S] relation (4441 instances), or the ATTRIBUTION[S][N] relation (1612 instances).

1 other relation types occur very rarely, such as TOPIC-COMMENT [S] [N] (2 instances), or EVALUATION [N][N] (3 instances).


For each measure, the authors note a performance increase when little training data is available, or when the feature representations are very sparse.

for our task, classification of discourse relations, we employ not only words but also other types of features such as parse tree production rules, and thus cannot compute semantic kernels using WordNet.


(2008b)-for each instance with n senses in the corpus, we create n identical feature vectors, each being labeled by one of the instance's senses.

in the RST framework, only one relation is allowed to hold between two EDUs.


As α increases, the utterances increase in complexity, as does the success rate.

when α approaches 1, the utterances are too complex and the success rate decreases.


In recent years, statistical MT systems have been easy to develop due to the rapid explosion in data availability, especially parallel data.

in reality there are still many language pairs which lack parallel data, such as Urdu-English, Chinese-Italian, where large amounts of speakers exist for both languages; of course, the problem is far worse for pairs such as Catalan-Irish.


In this case, the role of paraphrases in decoding becomes a little weaker.

it might become a kind of noise to interfere with the exact translation of the original source-side phrases when decoding.


These results show that 1) most of the paraphrases that are added in are lower-order n-grams; 2) the paraphrases can increase the coverage of the input by handling the unknown words to some extent.

we observed that most untranslated words in the "Para-Sub" and "Lattice" systems are still NEs, which shows that in our paraphrase table, there are few paraphrases for the NEs.


One common limitation of the above works is they only allow the one-to-one mapping between each non-terminal frontier node, and thus they suffer from the issue of rule coverage.

due to the data sparseness issue and model coverage issue, previous tree-to-tree (Zhang et al., 2008;Liu et al., 2009) decoder has to rely solely on the span information or source side information to combine the target syntactic structures, without checking the compatibility of the merging nodes, in order not to fail many translation paths.


At first glance, this seems only peripherally related to our work, since the specific/general distinction is made for features rather than instances.

for multinomial models like our LMs and TMs, there is a one to one correspondence between instances and features, eg the correspondence between a phrase pair (s, t) and its conditional multinomial probability p(s|t).


unlike text, which requires little or no preprocessing, audio files are typically first transcribed into text before applying standard NLP tools.


Note that for some pseudo-terms, the words match exactly, while for others, the phrases are distinct but phonetically similar.

even in this case, there is often substantial overlap in the spoken terms.


Unsupervised clustering methods are attractive since they require no human annotations.

obtaining a few labeled examples for a simple labeling task can be done quickly, especially with crowd sourcing systems such as CrowdFlower and Amazon's Mechanical Turk (Snow et al., 2008;Callison-Burch and Dredze, 2010).


Our results demonstrate that such use of eye gaze can potentially compensate for a conversational systems limited language processing and domain modeling capability (Prasov and Chai, 2008).

this work is conducted in a static visual environment and evaluated only on transcribed spoken utterances.


In this case, the system must first identify one long sword as a referring expression and then resolve it to the correct set of entities in the virtual world.

not until the twenty fifth ranked recognition hypothesis H 25 , do we see a referring expression closest to the actual uttered referring expression.


It should be noted that these weights are learned based on single sentences.

to have a fair comparison between all our summary types we use these weights to generate summaries using the A* search with the word limit as constraint.


This confirms that MERT is maximizing the metric for which it was trained.

this is not the case for regression results.


These results concur with the automatic evaluation results as described in section 6.1.

this is not the case for the grammaticality and redundancy criteria.


With single sentences we have only a guarantee for high content overlap between single training and model sentences.

when these sentences are combined into summaries it is not guaranteed that these summaries will also have high content overlap with the entire model ones.


Most state-of-the-art models will incorrectly link we to the israelis because of their proximity and compatibility of attributes (both we and the israelis are plural).

a more cautious approach is to first cluster the israelis with israel because the demonymy relation is highly precise.


To facilitate comparison with most of the recent previous work, we report results using gold mention boundaries.

our approach does not make any assumptions about the underlying mentions, so it is trivial to adapt it to predicted mention boundaries (e.g., see Haghighi and Klein (2010) for a simple mention detection model).


SUMTIME forecasts are intended to be read by trained meteorologists, and thus the text is quite abbreviated.

wEATHERGOV texts are intended to be read by the general public and thus is more English-like.


Here, we focus on a single sentence as this is most appropriate for title generation.

multi-sentence output can be easily generated by setting a summary length constraint.


If the same word appears twice in a sequence, then it may receive two different pronunciations, since the mapping is probabilistic.

a token's syllable/stress pattern is chosen independently of other tokens in the sequence; we look at relaxing this assumption later.


Indeed we find that this distribution is too sharp and overemphasises short phrases, so we use f C = 1.

it does allow us to rank target phrases as possible translations.


For example, the translations of a source sentence in MT08 are as follows 2 : • Src: • HPB: South Korean government late last month to start with 400,000 tons of rice aid to the DPRK • HPB+MER: Start at the end of last month, South Korean government plans to provide 400,000 tons of rice in aid to the DPRK The most obvious error that the baseline system makes is the order of the time expression " , the end of last month", which should be either at the beginning or the end on target side.

the baseline produced a monotone translation by using the rule " X 1 , South Korean government X 1 ".


We evaluated the realizations using seven automatic metrics, and analyzed correlations obtained between the human judgments and the automatic scores.

to previous NLG meta-evaluations, we find that several of the metrics correlate moderately well with human judgments of both adequacy and fluency, with the TER family performing best overall.


In contrast to previous NLG meta-evaluations, we found that several of the metrics correlate moderately well with human judgments of both adequacy and fluency, with the TER family performing best.

when looking at statistically significant system-level differences in human judgments, we found that some of the metrics get some of the rankings correct, but none get them all correct, with different metrics making different ranking errors.


Chance agreement for this data is calculated by the method discussed in Carletta's squib.

in previous work in MT meta-evaluation, Callison-Burch et al.


Human fluency judgments of outputs with only punctuation problems were generally high, and many realizations with commas inserted or removed were rated fully fluent by the annotators.

tERP penalizes such insertions or deletions.


The correlations of each of the metrics with the human judgments of fluency for the realizer systems indicate at least a moderate relationship, in contrast with the results reported in (Stent et al., 2005) for paraphrase data, which found an inverse correlation for fluency, and (Cahill, 2009) for the output of a surface realizer for German, which found only a weak correlation.

the former study employed a corpus-based paraphrase generation system rather than grammar-driven surface realizers, and the resulting paraphrases exhibited much broader variation.


For the targeted metrics, HNIST is correct for all five comparisons, while neither HBLEU nor HME-TEOR correctly rank all the OpenCCG models.

hTER and hGTM incorrectly rank the XLE-best system versus OpenCCG-based models.


These results suggest that the MT-evaluation metrics are useful for developing surface realizers.

the correlations are lower than those reported for MT data, suggesting that they should be used with caution, especially for cross-system evaluation, where consulting multiple metrics may yield more reliable comparisons.


This is the most commonly used metric across the literature as it is intuitive and creates a meaningful POS sequence out of the cluster identifiers.

it tends to yield higher scores as |C| increases, making comparisons difficult when |C| can vary.


Recall that crossval was proposed as a possible solution to this problem, and it does solve the extreme case of single, yielding 0% accuracy rather than 100%.

it patterns just like many-to-1 for up to 200 clusters, suggesting that there is very little difference 4 We used the Stanford Tagger trained on the WSJ corpus: http://nlp.stanford.edu/software/tagger.shtml.


We first present results for the same WSJ corpus used above.

because most of the systems were initially developed on this corpus, and often evaluated only on it, there is a question of whether their methods and/or hyperparameters are overly specific to the domain or to the English language.


One might expect that the two systems with morphological features (clark and feat) would show less difference between English and some of the other languages (all of which have complex morphology) than the other systems.

although clark and feat (along with Brown) are the best performing systems overall, they don't show any particular benefit for the morphologically complex languages.


Versions of this problem include adapting using only unlabeled target domain data Blitzer et al., 2007;Jiang and Zhai, 2007), adapting using a limited amount of target domain labeled data (Daumé, 2007;Finkel and Manning, 2009), and learning across multiple domains simultaneously in an online setting (Dredze and Crammer, 2008b).

in practical settings, we do not know if the data distribution will change, and certainly not when.


Since the A-distance assumes a stream of single values, we can apply an A-distance detector to each feature (e.g., unigram and bigram count) individually.

our extensive experiments with this approach (omitted here) show that it suffers from a number of flaws, such as a high false positive rate if all features are tracked, the difficult problem of identifying an informative subset of features for tracking, and deciding how many such features need to change before a shift has occurred, which turns out to be highly variable between shifts.


We have shown detection of sudden shifts between the source and target domains.

some shifts may happen gradually over time.


When CWPM is slow to detect a change (over 400 examples), the domain classifier is the clear winner.

in the majority of experiments, especially for ACE and spam data, both detectors register a change quickly.


It is always difficult to determine how many samples are enough for sampling algorithms.

both fertility models achieve better results than their baseline models using a small amount of samples.


An exact implementation of a new decoder for factorized grammars can make better use of all the templates.

the experiments will show that even an approximation like this can already provide significant improvement on small training data sets, i.e.


It helps to alleviate the pronoun dropping problem in Arabic.

we notice that most of the templates in the 200 lists are rather simple.


They also described a technique based on TF-IDF to de-emphasize sentences similar to those that have already been selected, thereby encouraging diversity.

this strategy is bootstrapped by random initial choices that do not necessarily favor sentences that are difficult to translate.


The top-ranked sentences are chosen for manual translation.

this approach requires that the pool have the same distributional characteristics as the development sets used to train the ranking model.


The translation results show that the system trained with this null element (*PRO*) translates verbs that follow the null element largely in such a manner.

it may not be always closest to the reference.


Indexing by suffix arrays is used to allow fast access to phrase instances in the corpus, and random sampling to avoid collecting the full set of examples has been shown to perform well.

these approaches consider all instances of a phrase as equivalent for the estimation of its translations.


The larger grammar results in more combinations of partial theories in decoding.

for computing reasons, we kept the beam size of the decoder constant despite the increase in grammar size, potentially pruning out good theories.


Thus they only find a complete fully connected parse at the very end.

both subjective and experimental evidence show that people understand a sentence word-to-word as they go along, or close to it.


Thus they only find a complete fully-connected parse at the very end.

human syntactic parsing must be fully connected (or close to it) as people are able to apply vast amounts of real-world knowledge to the process as it proceeds from word-toword (van Gompel and Pickering, 2007).


As is invariably the case, when combined the two models perform much better than either by itself (C/R Combined -88.8%).

we still achieve a 0.6% improvement over that result.


If the test data is also similarly composed, performance on any particular test instance might improve if training is done on a training subset coming from the same source.

even when the training and test data are from the same source, a ZL algorithm may capture fine differences between subsets.


One possibility for an initial model was to extract the word-to-lextype mappings from the grammar lexicon as Baldridge does, and make all starting probabilities uniform.

our lexicon maps between lextypes and lemmas, rather than inflected word forms, which is what we'd be tagging.


For example, it would be easy to enforce such constraints in the Eisner (1996) algorithm or using Integer Linear Programming approaches (Riedel and Clarke, 2006;Martins et al., 2009).

such richer modeling capacity comes with a much higher computational cost.


should have "stand" as the main head of the sentence, and "does" as its aux.

the WSJ model labeled "does" as the main head.


Most of the initial research in this literature focused on either recognizing negated terms or identifying speculative sentences, using some heuristic rules (Chapman et al., 2001;Light et al., 2004), and machine learning methods (Goldin and Chapman, 2003;Medlock and Briscoe, 2007).

scope learning has been largely ignored until the recent release of the BioScope corpus (Szarvas et al., 2008), where negation/speculation cues and their scopes are annotated explicitly.


Table 6 shows that: 1) For both negation and speculaiton scope identification, automatic syntactic parsing lowers the performance on the abstracts subcorpus (e.g., from 83.10% to 81.84% in accuracy for negation scope identification and from 86.41% to 83.74% in accuracy for speculaiton scope identification).

the performance drop shows that both negation and speculation scope identification are not as senstive to automatic syntactic parsing as common shallow semantic parsing, whose performance might decrease by about ~10 in F1measure (Toutanova et al., 2005).


It also shows that our final system significantly outperforms the state-of-the-art ones using a chunking approach, especially on the abstracts and full papers subcorpora.

the improvement on the clinical reports subcorpora for negation scope identification is much less apparent, partly due to the fact that the sentences in this subcorpus are much simpler (with average length of 6.6 words per sentence) and thus a chunking approach can achieve high performance.


In addition, we have also experimented on only these words, which happen to be a cue or inside a cue in the training data as cue candidates.

this experimental setting achieves a lower performance than that when all words are considered.


The SAMA analyzer has good coverage; for typical texts, the correct analysis of an orthographic word can be found somewhere in SAMA's list of alternatives about 95% of the time.

this broad coverage comes at a cost; the list of analytic alternatives must include a long Zipfian tail of rare or contextually-implausible analyses, which collectively are correct often enough to make a large contribution to the coverage statistics.


It is straightforward to read off the highest scoring parse from a packed chart, and similarly routine to generate an n-best list containing a highly-ranked subset of the parses.

a packed chart built on an atomic CFG does not make available all of the features that are important to many CFG-based SRL systems.


Finding the values of θ k using the MLE method is straightforward.

this is not the case for maximising the likelihood function over the space of all possible dendrograms.


The largest performance difference is 6.0% at p 1 = 45, p 2 = 10 and p 3 = 0.05.

this picture is not the same when considering a higher context similarity threshold (p 3 = 0.13) as Figure 7 shows.


The advantage of implicit modeling is that it is easy to implement based on the dependency structure.

its limitation is that the distance measure does not capture sufficient information of semantic relations between language constituents.


We predict that h j (x, y) is entailed if the distance between x * and y * is smaller than λ L .

as can be seen, this distance does not reflect whether the type of relationship between x * and y * is similar to the relationship that holds between x and y.


For fact, the most benefit of incorporating explicit modeling of long distance relationship appears at the alignment stage, but not much at the inference stage.

this situation is different for intent, where the benefit of explicitly modeling long distance relationship mostly happened at the inference stage.


Therefore, it is important to develop algorithms that find features which work across domains.

labeled adaptation frameworks are also required because we would like to take advantages of target labeled data.


As our example adaptation algorithms we selected: Labeled adaptation: FE framework One of the most popular adaptation frameworks that requires the use of labeled target data is the "Frustratingly Easy" (FE) adaptation framework (Daumé III, 2007).

why and when this framework works remains unclear in the NLP community.


Before showing the bound analysis, note that the framework proposed by (Evgeniou and Pontil, 2004;Finkel and Manning, 2009) is a generalization over these three frameworks (FE, S+T, and the baseline) 6 .

our goal in this paper is different: we try to provide a deep discussion on when and why one should use a particular framework.


When β is close to zero, z 1 and z 2 will be very similar.

when β is large, z 1 and z 2 can be very different.


In the previous experiment, we used only cosine as our task similarity measurement to decide what is the best framework.

task similarity should consider the difference in both P (X) and P (Y |X), and the cosine measurement is not sufficient for this.


features improves the Token-F1 by around 10%.

adding target labeled data also helps the results significantly.


In this case, two vocabularies need to be considered, corresponding respectively to the context vocabulary V c used to define the history; and the prediction vocabulary V p .

this method fails to deliver any probability estimate for words outside of the prediction vocabulary, meaning that a fall-back strategy needs to be defined for those words.


We can observe that the LBL model converges faster than the standard model: the latter needs 13 epochs to reach the stopping criteria, while the former only needs 6 epochs.

upon convergence, the standard model reaches a lower perplexity than the LBL model.


The unsupervised nature of the approach means good ability to deal with domain variation.

the approach did not show a segmentation performance as good as that of the supervised approach.


5 To make the results comparable, we use the same feature templates, that is feature template (a)-(c).

sVM-HMM takes interactions between nearby labels into the model, which means there is a label bigram feature template implicitly used in the sVM-HMM.


One concern to the bootstrapping approach in this paper is that it takes time to work with, which will make it difficult to be incorporated into language applications that need to responses in real time.

we believe that such an approach can be used in offline contexts.


Note that the LDC algorithm is deterministic.

the randomness in the sparse-matrix implementation of reduced-rank SVD used in the initialization step causes a small variability in performance (the standard deviation of the MTO score is 0.0004 for PTB17 and 0.003 for PTB45).


Both models use an iterative approach to minimize an objective function, and both initialize with frequent words.


Figure 4 clearly shows that hybrid-morfette linkers outperform hybrid-maxent linkers on unknown words.

figures 2-4 show that hybridmorfette's advantage on unknown words is counteracted by its lower performance on known words; therefore, it has slightly lower overall accuracy than hybrid-maxent.


The proposed criterion with an automatically determined threshold value produced slightly worse results than that of Jin and Tanaka-Ishii at the CHILDES corpus.

we found out that our approach achieves approximately 1% higher score when the best performing threshold value is selected from the candidate list.


Both our decoding algorithm and the decoding algorithm of Z&C08 run in linear time.

in order to generate possible candidates for each character, Z&C08 uses an extra loop to search for possible words that end with the current character.


A restriction to the maximum word length is applied to limit the number of iterations in this loop, without which the algorithm would have quadratic time complexity.

our decoder does not search backword for the possible starting character of any word.


(2009) is based on Nakagawa and Uchimoto (2007), which separates the processing of known words and unknown words, and uses a set of segmentation tags to represent the segmentation of characters.

our model is conceptually simpler, and does not differentiate known words and unknown words.


Dynamic programming is exact inference, for which the time complexity is decided by the locality of feature templates.

beam-search is approximate and can run in linear time.


By design, they readily capture regularities at the token-level.

these approaches are ill-equipped to directly represent type-based constraints such as sparsity.


In total there are O(K 2 ) parameters associated with the transition parameters.

to the Bayesian HMM, θ t is not drawn from a distribution which has support for each of the n word types.


BoW+P+P t ) helped to achieve higher accuracies, and the Author feature was also beneficial.

list decreased the performance, as the flow of dialogues can change, and when a larger history of dialogue acts is included, it tends to introduce noise.


Instead, Zhao and Ng (2007) proposed featurebased methods to zero anaphora resolution on the same corpus from Convese (2006).

they only considered zero anaphors with explicit noun phrase referents and discarded those with split an-tecedents or referring to events.


Isozaki and Hirao (2003) explored some ranking rules and a machine learning method on zero anaphora resolution.

they assumed that zero anaphors were already detected and each zero anaphor's grammatical case was already determined by a zero anaphor detector.


In fact, the ratio of positive and negative instances reaches about 1:12.

this ratio is much better than that (1:30) using the heuristic rule as described in Zhao and Ng (2007).


Actually, among the remaining 7% zero anaphors, about 5% are driven by a preposition phrase (PP) node, and 2% are driven by a noun phrase (NP) node.

our preliminary experiments show that simple inclusion of those PP-driven and NPdriven zero anaphors will largely increase the imbalance between positive and negative instances, which significantly decrease the performance.


This rate is almost the same as that of bunsetsu boundaries.

the commas inserted at the clause boundaries "topicalized element-wa" accounted for 8.84% (1,446/16,357) of all the inserted commas.


The baseline method was not able to insert commas right after the bunsetsu "MtVG5%> (are floated)" or " §¨B{5k= (not decided)" but inserted commas at unnatural positions such as between "©ª% (calling himself)" and "«¬-®> (the vice commander)."

our method was able to insert commas properly at such bunsetsu boundaries.


Unknown words are a hindrance to the performance of hand-crafted computational grammars of natural language.

words with incomplete and incorrect lexical entries pose an even bigger problem because they can be the cause of a parsing failure despite being listed in the lexicon of the grammar.


For example, the word afwater (to drain) is listed as a first person singular present verb in the Alpino lexicon.

the error miner identifies this word as the reason for the parsing failure of 9 sentences.


If there is no analysis spanning the whole sentence, the parser finds all parses for each substring and returns what it considers to be the best sequence of non-overlapping parses.

in the context of this experiment, a sentence will be considered successfully parsed only if it receives a full-span analysis.


That subcategorization frame requires a noun phrase ('the soup' in (5-c)) and a locative NP ('the bowl' in (5-c)).

in some cases, the entries generated by this lexical rule cannot account for other possible usages of the verbs in question.


Clearly, this baseline is expected to perform worse than both our model and the universal types one since those are able to cover most of the sentences and thus, they are likely to produce more correct dependency relations.

it gives us an idea how much extra quality is gained when coverage improves.


It is important to note that this paper should be viewed as a case study where we illustrate the results of the application of what we believe to be a good algorithm for dealing with incomplete or incorrect lexical entries-namely, the combination of error mining and LA.

our method is general enough to be applied to other large-scale grammars and languages.


TIMEX2 1 and TimeML 2 ), publicly available annotated corpora (ACE, 3 TimeBank 4 ) and a number of automatic taggers (see, for example, (Mani and Wilson, 2000;Schilder, 2004;Hacioglu et al., 2005;Negri and Marseglia, 2005;Saquete, 2005;Han et al., 2006;Ahn et al., 2007)).

existing corpora have their limitations.


These schemes were used to annotate corpora that are often used in research on temporal expression recognition and normalisation: the series of corpora used for training and evaluation in the Automatic Content Extraction (ACE) program 6 run in 2004, 2005 The ACE corpora were prepared for the development and evaluation of systems participating in the ACE program.

the evaluation corpora have never been publicly released, and thus are currently, for all practical purposes, unavailable.


This is not to say that Wikipedia content is necessarily of low quality; this is an encyclopedia with many people and bots controlling its quality, and there exist manuals of style for authors to help them avoid errors and ambiguity and to ensure maximum consistency.

10 given the large number of editors with various degrees of fluency and experience in writing and editing, it would not be surprising if some parts of the texts are not perfect.


Various evaluation metrics have been proposed, and BLEU is now used as the de facto standard metric.

when we consider translation between distant language pairs such as Japanese and English, most popular metrics (e.g., BLEU, NIST, PER, and TER) do not work well.


(2009) showed that the popular BLEU and NIST do not work well by using the system outputs of the NTCIR-7 PATMT (patent translation) JE task (Fujii et al., 2008).

rOUGE-L (Lin and Hovy, 2003), Word Error rate (WEr), and IMPACT (Echizen-ya and Araki, 2007) worked better.


Since Pearson's correlation metric assumes linearity, nonlinear monotonic functions can change its score.

spearman's ρ and Kendall's τ uses ranks instead of raw evaluation scores, and simple application of monotonic functions cannot change them (use of other operations such as averaging sentence scores can change them).


We can combine the above word order metrics with BP, e.g., NKT × BP and NSR × BP.

we cannot very much expect from this solution because BP scores do not correlate with human judgments well.


Perhaps, NSR penalized alternative good translations too much.

one of the NSR-based metrics, NSRP 1/4 , gave the best Spearman score of 0.947, and the difference between NSRP α and NKTP α was small.


We preferred NSR because it penalizes global word order mistakes much more than does NKT, and as discussed above, global word order mistakes often lead to incomprehensibility and misunderstanding.

on the other hand, they also tried Hamming distance, and summarized their experiments as follows: the Hamming distance seems to be more informative than Kendall's tau for small amounts of reordering.


LEXUS offers an interface permitting to the user to define formats of lexical bases with a perspective to enable the construction of lexical bases according to the LMF model.

it does not allow the verification of the compliance between the model and the norm.