The cosine distance between the word distribution for documents ¡ X and ¡ is computed as: This measure has been found to perform well and was used by all the TDT 2002 link detection systems (unpublished presentations at the TDT2002 workshop).

to the Euclidean distance based cosine measure, the Hellinger measure is a probabilistic measure.


That is, they use patterns to independently discover semantic relationships of words.

for infrequent words, these patterns do not match or, worse yet, generate incorrect relationships.


Given a typical annotation rate of 5,000 words per hour, we estimated that setting up a name finder for a new problem would take four person days of annotation work -a period we considered reasonable.

this user's problems were too dynamic for that much setup time.


Capitalization models presented in most previous approaches are monolingual because the models are estimated only from monolingual texts.

for capitalizing machine translation outputs, using only monolingual capitalization models is not enough.


As one can see, our features are "coarse-grained" (e.g., the language model feature).

kim and Woodland (2004) and Roark et al.


Unlike a real phrase aligner, the NPA need not wait for the training of the translation model to finish, making it possible for parallelization of translation model training and capitalization model training.

we believe that a real phrase aligner may make phrase alignment quality higher.


Given that storing all of these phrases leads to very large phrase tables, many research systems simply limit the phrases gathered to those that could possibly influence some test set.

this is not feasible for true production MT systems, since the data to be translated is unknown.


Table 5 gives the percentage of these which have translations in each of the three training corpora, if we do not use paraphrasing.

after expanding the phrase table using the translations of paraphrases, the coverage of the unique test set phrases goes up dramatically (shown in Table 6).


The computational cost of training the DTs on large quantities of data is comparable to that of training phrase tables on the same data -large but manageable -and increases linearly with the amount of training data.

currently there is a major problem with DT training: the low proportion of Chinese-English sentence pairs that can be fully segment-aligned and thus be used for DT training (about 27%).


The way to show that one graph element does not follow from another is to make the cost of aligning them high.

since we are embedded in a search for the lowest cost alignment, this will just cause the system to choose an alternate alignment rather than recognizing a non-entailment.


Such a sentence is not observed frequently in corpora, and will not be used as clues to generate rules in practice.

we frequently observe sentences of the second type in corpora, and our method generates the paraphrases from the verb-verb cooccurrences taken from such sentences.


S-V V (n, vcon, vpre, arg, arg ) = Parg(n, vcon)P arg (n, vpre)/P (n) 2 S-N V (n, vcon, vpre) = P coord (vcon, vpre) M I(n, vcon, vpre) = P coord (vcon, vpre)/(P (vcon)P (vpre)) Cond(n, vcon, vpre, arg, arg ) = P coord (vcon, vpre, arg, arg )Parg(n|vcon)P arg (n|vpre) /(P arg (n, vpre)P (n)) Rand(n, vcon, vpre, arg, arg ) = random number S-V V was obtained by approximating the probabilities of coordinated sentences, as in the case of BasicS.

we assumed the occurrences of two verbs were independent.


These methods were discussed in (Yang and Pedersen, 1997).

with any of the three feature ranking criteria, cross validation showed that selecting all features gave the best average validation performance.


In general, when the test data is similar to the training data, IG (or CHI) is advantageous over F (Yang and Pedersen, 1997).

in this case when the test domain is different from the training domains, F shows advantages for adaptation.


Moreover, it is often difficult to find experts in these languages both for the expensive annotation effort and even for language specific clues.

comparable multilingual data (such as multilingual news streams) are increasingly available (see section 4).


Generally, the ensemble of classifiers is generated by training on different subsets of data, rather than different features.

there is some literature within unstructured classified on combining models trained on feature subsets.


A recent attempt to combine outputs of different alignments views the combination problem as a classifier ensemble in the neural network framework (Ayan et al., 2005).

this method is subject to the unpredictability of random network initialization, whereas ACME is guaranteed to find the model that maximizes the likelihood of training data.


Also, when we want to combine two models for prediction, finding the Viterbi alignment argmax z p 1 (z | x)p 2 (z | x) is intractable for HMM models (by a reduction from quadratic assignment), and a hard intersection argmax z 1 p 1 (z 1 | x) ∩ argmax z 2 p 2 (z 2 | x) might be too sparse.

we can threshold the product of two edge posteriors quite easily: z = {z ij = 1 : We noticed a 5.8% relative reduction in AER (for our best model) by using posterior decoding with a validation-set optimized threshold δ instead of using hard intersection of Viterbi alignments.


Initializing the HMM with model 1 parameters alleviates this problem.

if we jointly train two HMMs starting from a uniform initialization, the HMMs converge to a surprisingly good solution.


One of the most useful features for the basic matching model is, of course, the set of predictions of IBM model 4.

computing these features is very expensive and we would like to build a competitive model that doesn't require them.


The example in Figure 4 shows another interesting phenomenon: the multi-fertile alignments for not and député are learned even without lexical fertility features (Figure 4b), because the Dice coefficients of those words with their two alignees are both high.

the surface association of aurait with have is much higher than with would.


We have avoided using expensive-to-compute features like IBM model 4 predictions up to this point.

if these are available, our model can improve further.


For example, the antecedent for the pronoun subject they in the first example of work in Table 3 should be ringers, an agent subject that is typical for Sense 1 (exert oneself in an activity).

the feature exaction module found the wrong antecedent changes that is an unlikely fit for the intended verb sense.


This may be due to the indiscriminately high confidence scores; or it could indicate that classifiers, which are geared at distinguishing between known classes rather than detecting objects that differ from all seen data, are not optimally suited to the task.

one further disadvantage of this approach is that, as mentioned above, it can only be applied to lemmas with more than one annotated sense.


To test this hypothesis fully would require suitable annotation tools and subjects skilled in CCG annotation, which we do not currently have access to.

there is some evidence that annotating category sequences can be done very efficiently.


Since they somehow use lexical information in the tagged corpus, they are called "lexicalized parsers".

unlexicalized parsers achieved an almost equivalent accuracy to such lexicalized parsers (Klein and Manning, 2003;.


In sentence (3) in Figure 3, the baseline method correctly recognized the head of "iin-wa" (commissioner-TM) as "hirakimasu" (open).

the proposed method incorrectly judged it as "oujite-imasuga" (offer).


An early exception to this was (Collins, 1997) itself, where Model 2 used function tags during the training process for heuristics to identify arguments (e.g., the TMP tag on the NP in Figure 1 disqualifies the NP-TMP from being treated as an argument).

after this use, the tags are ignored, not included in the models, and absent from the parser output.


We ported the WordNet similarity path length based measures to the Wikipedia category graph.

the category relations in Wikipedia cannot only be interpreted as corresponding to is-a links in a taxonomy since they denote meronymic relations as well.


In the NIST pilot study, it was apparent that human annotators often disagreed on whether a belief statement was or was not an opinion.

high annotator agreement was seen on judg-ment opinions.


We expanded those manually selected seed words of each sentiment class by collecting synonyms from WordNet.

we cannot simply assume that all the synonyms of positive words are positive since most words could have synonym relationships with all three sentiment classes.


Despite successes in identifying opinion expressions and subjective words/phrases (See Section 1), there has been less achievement on the factors closely related to subjectivity and polarity, such as identifying the opinion holder.

our research indicates that without this information, it is difficult, if not impossible, to define 'opinion' accurately enough to obtain reasonable interannotator agreement.


This means that classification modeling 3 can select many candidates as answers as long as they are marked as true, and does not select any candidate if every one is marked as false.

ranking always selects the most probable candidate as an answer, which suits our task better.


In the example, the complete path from Hhead to Ehead is "<H> NP S VP S S VP VBG <E>".

representing each complete path as a single feature produces so many different paths with low frequencies that the ME system would learn poorly.


This is seen in Figure 2 for English, Dutch, Afrikaans, and Italian.

a larger rule set does not mean that the average context width is greater.


Instead, the data shows little sign of saturation (Figure 4).

the average perplexity of the letter-to-phoneme distributions remains level with corpus size ( Figure 5).


The auxiliary has expects a VP-C, permitting the bare verb phrase demonstrate to be incorrectly used.

if we tag-annotate all VP-Cs, rule 6 would be relabeled as VP-C_VB in rule 6 and rule 7 as 7 in Figure 5.


(2003), and component weights are adjusted by minimum error rate training (Och, 2003).

to phrase-based SMT and to the above cited dependency-based SMT approaches, our system feeds dependency-structure snippets into a grammar-based generator, and determines target language ordering by applying n-gram and distortion models after grammar-based generation.


Alhough generic parameters are useful predictors of user satisfaction in other PARADISE applications, overall our parameters produce less useful user satisfaction models in our system.

generic and tutoring-specific parameters do produce useful models of student learning in our system.


Our results show that, although generic parameters were useful predictors of user satisfaction in other PARADISE applications, overall our parameters produce less useful user satisfaction models in our tutoring system.

generic and tutoringspecific parameters do produce useful models of student learning in our system.


The state with the plusses is the positive final state, and the one at the bottom is the negative final state.

we are most concerned with how the non-final states converge.


To compare the utility of each of the features, we use three metrics: 1  the largest number of differences: 10, followed by Frustration, and then Percent Correctness.

counting the number of differences does not completely describe the effect of the feature on the policy.


This means that Percent Correctness effects a smaller amount of change than this random baseline and thus is fairly useless as a feature to add since the random feature is probably capturing some aspect of the data that is more useful.

the Concept Repetition and Frustration cause more change in the policies than the random feature baseline so one can view them as fairly useful still.


For example, back channels usually have shorter sentences and are constant in discourse pattern over a DA.

questions and statements typically have longer, and more complex, discourse structures.


(2005) introduce additional chunking features to enhance the parse tree features.

the hierarchical structured information in the parse trees is not well preserved in their parse treerelated features.


State of the art extraction algorithms may be able to detect the son and sibling relations from local language clues.

the cousin relation is only implied by the text and requires additional knowledge to be extracted.


Our work restricts induced features to conjunctions of base features, rather than using first-order clauses.

the patterns we learn are based on information extracted from natural language.


In this case, we cluster articles by using their basic patterns as features.

each basic pattern is still connected to its entity so that we can extract the name from it.


When a link is found between two basic clusters that were already assigned to a metacluster, we try to put them into all the existing metaclusters it belongs to.

we allow a basic cluster to be added only if it can fill all the columns in that table.


From these results, we can see the large efficiency benefit of the Markov assumption, as the size of the non-terminal and production sets shrink.

the efficiency gains come at a cost, with the Markov order-0 factored grammar resulting in a loss of a full 8 percentage points of F-measure accuracy.


One way out of this dilemma could be to ignore the detailed morphological structure of the word and focus on determining only the major and minor parts of speech.

(Oflazer et al., 1999) observes that the modifier words in Turkish can have dependencies to any one of the inflectional groups of a derived word.


If this number is large, direct computation of (5) on the corpus might be more efficient.

if the corpus at hand is very large, one might opt for direct computation of (3).


Furthermore, the entries share the same values for the attributes YDS and TD (i.e., 237 and 1).

clusters 5 and 6 have no attributes in common.


This is achieved by limiting the number of entry pairs with positive labels for each document: x e i ,e j ≤ m (5) Notice that the number m is not known in advance.

we can estimate this parameter from our development data by considering documents of similar size (as measured by the number of corresponding entry pairs.)


The task of summarizing spontaneous spoken dialogue from meetings presents many challenges: information is sparse; speech is disfluent and fragmented; automatic speech recognition is imperfect.

there are numerous speech-specific characteristics to be explored and taken advantage of.


Combining the rankings of two such systems might create a third system which is comparable but not any better than either of the first two systems alone.

it is still possible that the combined system will be better in terms of balancing the two types of importance discussed above: utterances that contain a lot of informative content and keywords and utterances that relate to decision-making and meeting structure.


The solution, we believe, is to solicit judgments from multiple assessors and develop a more refined sense of nugget importance.

given finite resources, it is important to balance the amount of additional manual effort required with the gains derived from those efforts.


In Bradshaw's experiment, scientific documents are indexed by the text that refers to them in documents that cite them.

unlike in experiments with previous collections, we need both the citing and the cited article as full documents in our collection.


The majority were criticisms of the test collection paradigm itself and are not pertinent here.

the source-document principle (i.e., the use of queries created from documents in the collection) attracted particular criticisms.


Our average number of judged relevant documents per query is lower than for Cranfield, which had an average of 7.2 (Spärck Jones et al., 2000).

this is the final number for the Cranfield collection, arrived at after the second stage of relevance judging, which we have not yet carried out.


The "global MTF" has been shown to slightly outperform the local version in the aforementioned paper.

we believe that the global mode is merely for demonstration but unlikely practical of online judgment since it insists that all queries are judged simultaneously with a strict synchronisation among all assessors.


given a document d, the simplest way to estimate the document language model is to treat the document as a sample from the underlying multinomial word distribution and use the maximum likelihood estimator: P (w|Θ d ) = c (w,d) |d| , where c(w, d) is the count of word w in document d, and |d| is the length of d. as discussed in virtually all the existing work on using language models for retrieval, such an estimate is problematic and inaccurate; indeed, it would assign zero probability to any word not present in document d, causing problems in scoring a document with query likelihood or KLdivergence (Zhai and Lafferty, 2001b).


In the worst case, our runtime may be exponential in the number of constraints, since we are considering an intractable class of problems.

we show that in practice, the method is quite effective at rapid decoding under global hard constraints.


If not, then we reintroduce the constraints.

rather than include all at once, we introduce them only as they are violated by successive solutions to the relaxed problems: y * 0 , y * 1 , etc.


Recent work on semantic role labeling (SRL) has focused almost exclusively on the analysis of the predicate-argument structure of verbs, largely due to the lack of human-annotated resources for other types of predicates that can serve as training and test data for the semantic role labeling systems.

it is wellknown that verbs are not the only type of predicates that can take arguments.


We speculate that the lack of improvement is due to the fact that the constraint that core (numbered) arguments should not have the same semantic role label for Chinese nominalized predicates is not as rigid as it is for English verbs.

further error analysis is needed to substantiate this speculation.


Indeed, attempts have been made to directly apply machine translation systems to the problem of semantic parsing (Papineni et al., 1997;Macherey et al., 2001).

these systems make no use of the MRL grammar, thus allocating probability mass to MR translations that are not even syntactically well-formed.


Human efforts are preferred if the evaluation task is easily conducted and managed, and does not need to be performed repeatedly.

when resources are limited, automated evaluation methods become more desirable.


We chose English-Chinese MT parallel data because they are news-oriented which coincides with the task genre from DUC.

it is unknown how large a parallel corpus is sufficient in providing a paraphrase collection good enough to help the evaluation process.


While we experimented with several parameter settings for LSA and Brown methods, we do not claim that the selected settings are necessarily optimal.

these methods present sensible com-  parison points for understanding the relationship between paraphrase quality and its impact on automatic evaluation.


The algorithm here does some of that with the steps that normalize the strings.

the largest boost in performance is with CEQ, which expands the number of allowable cross-language matches for many characters.


Our ¡ -step random walk approach is similar to the one proposed by Harel and Koren (2001).

their algorithm is proposed for "spatial data" where the nodes of the graph are connected by undirected links that are determined by a (symmetric) similarity function.


The highest result (shown boldface) for each algorithm and corpus was achieved by using generation vectors.

unlike in the k-means experiments, ¡ ¢ £ ¤ ¥ ¢ was able to outperform I Q and I R in one or two cases.


The recognition approach taken for Turkish involves a static decoding network construction and optimization resulting in near real time decoding.

the memory requirements of network optimization becomes prohibitive for large lexicon and language models as presented in this paper.


Measures using WordNet taxonomy are state-ofthe-art in capturing semantic similarity, attaining r=.85 -.89 correlations with the MC dataset (Jiang and Conrath, 1997;Budanitsky and Hirst, 2006).

they fall short of measuring relatedness, as, operating within a single-POS taxonomy, they cannot meaningfully compare kill to death.


In a previous study (Chatain et al., 2006), linguistic model (LiM) adaptation using different types of word models has proved useful in order to improve summary quality.

sparsity of the data available for adaptation makes it difficult to obtain reliable estimates of word n-gram probabilities.


With smaller labeled dataset, the gap between LP and SVM is larger.

lP JS divergence consistently outperforms lP Cosine .


Over 712 questions, it replaced 14, two of which improved performance, the rest stayed the same.

random selection of paraphrases decreased performance to 0.156, clearly showing the importance of selecting a good paraphrase.


We believe that in face-to-face discourse, it is important to consider the possibility that non-verbal communication may offer features that are critical to language understanding.

due to the long-standing emphasis on text datasets, there has been relatively little work on non-textual features in unconstrained natural language (prosody being the most notable exception).


Instead, it can be described as a series of patterns at various levels of regularity.

compositionality is not a necessary assumption: finite-state models are well-suited for representing mappings from strings of meaning elements to strings of form elements without necessarily pairing them one-to-one.


A drawback of most finite-state models is their inability to generalize to novel items the way a human could.

the output of our finite-state model could potentially be used to generate training sets for connectionist or statistical models.


For example, a decision tree or SVM that builds a 3-way superclassifier using the posterior probabilities from the HMM and Maxent.

so far we have not found any gain from more complicated system combination than a simple linear interpolation.


al., 2003), which rely upon acoustic/prosodic cues.

none of these efforts allow for the context-dependence of extractive summarization, such that the inclusion of one word or sentence in a summary depends upon prior selection decisions.


A significant exception is the work of Conroy and O'Leary (2001), which employs an HMM model with pivoted QR decomposition for text summarization.

the structure of their model is constrained by identifying a fixed number of 'lead' sentences to be extracted for a summary.


One way to manipulate an extractor's precisionrecall tradeoff is to assign a confidence score to each extracted entity and then apply a global threshold to confidence level.

confidence thresholding of this sort cannot increase recall.


1 Accordingly, it is not possible to draw a straightforward quantitative comparison between our PropBank SSN parser and other PropBank parsers.

state-of-theart semantic role labelling systems (CoNLL, 2005) use parse trees output by state-of-the-art parsers (Collins, 1999;Charniak, 2000), both for training and testing, and return partial trees annotated with semantic role labels.


We are now building the environment for AAC users with cooperation with ISAAC-ISRAEL 2 , in order to make the system fully accessible and to be tested by AAC-users.


1 While videos contain a rich source of audiovisual information, text-based video search is still among the most effective and widely used approaches.

the quality of such text-based video search engines still lags behind the quality of those that search textual information like web pages.


In both corpora, we find positive priming effects.

pp priming is stronger, and Cp priming is much stronger in Map Task.


In the English and Mandarin systems, the lexical and acoustic feature sets perform similarly, and combine to yield improved results.

on the Arabic data, the acoustic feature set performs quite poorly, suggesting that the use of vocal cues to topic transitions may be fundamentally different in Arabic.


Weights for different types of constituents from each parser can be set in a similar way to configuration 3 in the dependency experiments.

instead of measuring accuracy for each part-of-speech tag of dependents, we measure precision for each non-terminal label.


The choice of using maximum likelihood estimation for estimating the intermediate language models for W (j) is motivated by the simplification in the entropy calculation which reduces the order from O(V ) to O(k).

maximum likelihood estimation of language models is poor when compared to smoothing based estimation.


Recently, the Web has been used as a corpus in the NLP community, where mainly counts of hit pages have been exploited (Kilgarriff and Grefenstette, 2003).

our proposal, Web-Based Language Modeling (Sarikaya, 2005), and Bootstrapping Large Sense-Tagged corpora (Mihalcea, 2002) use the content within the hit pages.


Our proposal is similar to previous studies in that both use machine learning.

previous methods used expensive resources, e.g., a corpus in which words are manually tagged according to their pronunciation.


The live system assigns non-default categories with 86.5% precision; a revised algorithm achieved 93.0% precision, both based on an evaluation of 982 topics.

our precision on identifying unambiguous topics with DMOZ was only 83%.


Note that we use the same notation to denote a matrix and its normalized matrix.

the affinity weight between two sentences in the affinity graph is currently computed simply based on their own content similarity and ignore the affinity diffusion process on the graph.


Since the dictionary-based approach is a well-known method, we skip its technical descriptions.

keep in mind that the dictionary-based approach can produce a higher R-iv rate.


This is a simple method and the quantization resolution can be adjusted based on the amount of data available for training.

the model does not perform as well when combined with the syntactic features.


Often, unwritten rules based on factors like social roles, personal assertiveness, and the current locus of control play a part in determining who will give away."

haller and Fossum did not further investigate how conversants efficiently resolve conflicts of dialogue initiative.


The urgency level of 10 seconds requires conversants to start the interruption game very quickly in order to complete it in time.

the urgency level of 40 seconds allows conversants ample time to wait for the best time to start the game (Heeman et al., 2005).


Moreover, if the initiator had an advantage, we would expect the system to have fought more strongly in the user-initiated segments in order to win.

we do not see that the relative volume of the system winning in user-initiated segments is statistically higher than in system-initiated segments in this small sample size (p = 0.9, ttest).


Automatically extracting these argument models is a challenging task.

researchers have begun to make progress towards this goal.


Usually, about 300 or more sentences are used to automatically rank MT systems (Koehn, 2004).

the quality of a sentence translated by an MT system is difficult to evaluate.


This is because conventional methods are based on the similarity between a translated sentence and its reference translation, and they give the translated sentence a high score when the two sentences are globally similar to each other in terms of lexical overlap.

in the case of the above example, the most important thing to maintain a high translation quality is to correctly translate "for" into the target language, and it would be difficult to detect the importance just by comparing an MT result and its reference translations even if the number of reference translations is increased.


These results indicate that we can reduce the development cost for constructing sub-goals.

there are still significant gaps between the correlation coefficients obtained using a fully automatic method and upper bounds.


Most stateof-the-art SMT systems treat grammatical elements in exactly the same way as content words, and rely on general-purpose phrasal translations and target language models to generate these elements (e.g., Och and Ney, 2002;Koehn et al., 2003;Quirk et al., 2005;Chiang, 2005;Galley et al., 2006).

since these grammatical elements in the target language often correspond to long-range dependencies and/or do not have any words corresponding in the source, they may be difficult to model, and the output of an SMT system is often ungrammatical.


The SMT system, trained on this domain, produces a natural lexical translation for the English word patch as correction program, and translates replace into passive voice, which is more appropriate in Japanese.

1 there is a problem in the case marker assignment: the accusative marker wo, which was output by the SMT system, is completely inappropriate when the main verb is passive.


These results show that the strategy of only including the new information as features in a standard n-best re-ranking scenario does not lead to an improvement over the baseline.

method 2 obtains notable improvements over the baseline.


While recent phrase-based statistical machine translation (SMT) systems achieve significant improvement over the original source-channel statistical translation models, they 1) use a large inventory of blocks which have significant overlap and 2) limit the use of training to just a few parameters (on the order of ten).

we show that our proposed minimalist system (DTM2) achieves equal or better performance by 1) recasting the translation problem in the traditional statistical modeling approach using blocks with no overlap and 2) relying on training most system parameters (on the order of millions or larger).


In the above text, the "she" in the last sentence is coreferent with both mentions of "Elizabeth".

when we consider "she" and "Elizabeth (1) " in isolation from the remaining coreference chain, it can be difficult for a machine learning method to determine whether the pair is coreferent or not.


A simple approach is to perform the transitive closure of the pairwise decisions.

as shown in recent work (McCallum and Wellner, 2003;Singla and Domingos, 2005), better performance can be obtained by performing relational inference to directly consider the dependence among a set of predictions.


Also, we attribute the gains from error-driven training to the fact that training examples are generated based on errors made on the training data.

(we should note that there are also small differences in the feature sets used for error-driven and standard training results.)


For such queries, typical field matching would retrieve no documents at all.

the SRM approach achieves a mean average precision of over twenty percent.


This task is not a typical IR task because the fielded structure of the query is a critical aspect of the processing, not one that is largely ignored in favor of pure content based retrieval.

the approach used is different from most DB work because cross-field dependencies are a key component of the technique.


It is easy to verify that the probability will be non-zero only if some training record w actually contained these words in their respective fields -an unlikely event.

the probability of 'elementary' and 'differential' co-occurring in the same title might be considerably higher.


Items at group centers have higher probabilities, and tighter groups have overall higher probabilities.

the stationary distribution does not address diversity at all.


In contrast, GRASSHOPPER does not involve clustering.

it is still able to automatically take advantage of cluster structures in the data.


This involves inverting an (n − |G|) × (n − |G|) matrix, which is expensive.

the Q matrix is reduced by one row and one column in every iteration, but is otherwise unchanged.


A node's prominence comes from its intrinsic stature, as well as the prominence of the nodes it touches.

to ensure that the topranked nodes are representative of the larger graph structure, it is important to make sure the results are not dominated by a small group of highly prominent nodes who are closely linked to one another.


We seek an actor ranking such that the top actors are prominent.

we also want the top actors to be diverse, so they represent comedians from around the world.


The initial high coverage comes from the random selection of actors.

these randomly selected actors are often not prominent, as we show next.


They show that a 2-parameter Markov process gives rise to a stationary distribution that exhibits the word frequency distribution and the letter frequency distribution characteristics of natural language.

the Markov process is initialized such that any state has exactly two successor states, which means that after each word, only two other following words are possible.


The deviation in single letter words can be attributed to the writing system being a transcription of phonemes and few phonemes being expressed with only one letter.

the slight quantitative differences do not oppose the similar distribution of word lengths in both samples, which is reflected in a curve of similar shape in figure 6 and fits well the gamma distribution variant of (Sigurd et al., 2004).


(2005) explored a large set of features that are potentially useful for relation extraction.

the feature space was defined and explored in a somewhat ad hoc manner.


Second, if we compare the best performance achieved in each feature subspace, we can see that for both classifiers, syntactic parse tree is the most effective feature space, while sequence and dependency tree are similar.

the difference in performance between the syntactic parse tree subspace and the other two subspaces is not very large.


The standard methods for inferring the parameters of probabilistic models in computational linguistics are based on the principle of maximum-likelihood estimation; for example, the parameters of Probabilistic Context-Free Grammars (PCFGs) are typically estimated from strings of terminals using the Inside-Outside (IO) algorithm, an instance of the Expectation Maximization (EM) procedure (Lari and Young, 1990).

much recent work in machine learning and statistics has turned away from maximum-likelihood in favor of Bayesian methods, and there is increasing interest in Bayesian methods in computational linguistics as well (Finkel et al., 2006).


We also experimented with a version of IO modified to perform Bayesian MAP estimation, where the Maximization step of the IO procedure is replaced with Bayesian inference using a Dirichlet prior, i.e., where the rule probabilities θ (k) at iteration k are estimated using: Clearly such an approach is very closely related to the Bayesian procedures presented in this article, and in some circumstances this may be a useful estimator.

in our experiments with the Sesotho data above we found that for the small values of α necessary to obtain a sparse solution,the expected rule count E[f r ] for many rules r was less than 1 − α.


Thus The spans counted by ℓ 1 and ℓ 2 correspond to continuous spans counted in computing the complexity of a chart parsing operation.

unlike in the diagrams in the earlier part of this paper, in our graph theoretic argument there is no requirement that T select only corresponding pairs of vertices from V 1 and V 2 .


Schone and Jurafsky's (2001) work represents one of the few attempts to address this inappropriate morpheme attachment problem, introducing a method that exploits the semantic relatedness between word pairs.

we propose two arguably simpler, yet effective techniques that rely on relative corpus frequency and suffix level similarity to solve the problem.


The random mode serves as an absolute baseline in this respect: it indicates how well a particular base generator performs on its own.

different base generators have different effects on the generation modes.


a statistical model that translates MRL to NL.

there has been little, if any, research on exploiting recent SMT methods for NLG.


players 2 3 7 5, or not learn any phrases for those predicates at all.

only 4.47% of the phrases generated during testing for GEOQUERY were discontiguous, so the advantage of WASP −1 ++ over PHARAOH++ was not as obvious.


The three next sentences display some advantages of our approach over the K&M model: here, the latter model performs deletion with too little lexicosyntactic information, and accidentally removes certain modifiers that are sometimes, but not always, good candidates for deletions (e.g., ADJP in Sentence 2, PP in sentences 3 and 4).

our model keeps these constituent intact.


Research on WSD has concentrated on using contextual information, which may be limited for infrequent unknown words.

chinese characters carry semantic information that is useful for predicting the semantic properties of the words containing them.


Rule-based models have not been used for this task before.

there are some regularities in the relationship between the semantic categories of unknown words and those of their component characters that can be captured in a more direct and effective way by linguistic rules than by statistical models.


Variant 2 performs better when only the best guess is considered, indicating that it is useful to model the effect of position on a character's contribution to word meaning in this case.

it is not helpful to be sensitive to character position when the best five guesses are considered.


These outputs could simply be replaced with the perfect SVM predictions (1 for the true class, -1 elsewhere) since the labels are known.

the second-pass learner might actually benefit from the information contained in the misclassifications.


The success of factoid QA systems, particularly in the NIST-sponsored TREC evaluations (Voorhees, 2003), has reinforced the perception about the superiority of QA systems over traditional IR engines.


that answers a particular question-and the development of such technology has been encouraged by the setup of the TREC QA tracks.

a study by  shows that users actually prefer answers embedded within some sort of context, e.g., the sentence or the paragraph that the answer was found in.


Previous work (Shapiro and Taksa, 2003) in the web environment attempted to convert a user's natural language query into one suited for use with web search engines.

the focus was on merging the results from using different sub-queries, and not selection of a single sub-query.


The modified Powell's method has been previously used in optimizing the weights of a standard feature-based MT decoder in (Och, 2003) where a more efficient algorithm for log-linear models was proposed.

this is specific to log-linear models and cannot be easily extended for more complicated functions.


One option would be to use the output from the system with the best performance on some development set.

it was found that this approach did not always yield better combination output compared to the best single system on all evaluation metrics.


The phrase-level combination yields fairly good gains on the tuning sets.

the performance does not seem to generalize to the test sets used in this work.


There is a large literature on document classification and automated text categorization (Sebastiani, 2002).

that work assumes that the categories of interest are already known, and tries to assign documents to categories.


However, that work assumes that the categories of interest are already known, and tries to assign documents to categories.

in this paper we focus on the problem of determining the categories of interest.


In unsupervised learning, where no training takes place, one simply hopes that the unsupervised learner will work well on any unlabeled test collection.

when the variability in the data is large, such hope may be unrealistic; a tuning of the unsupervised algorithm may then be necessary in order to perform well on new test collections.


9 One might obtain better performance (across all methods being compared) by choosing a separate θ * for each of the 9 training sets.

to simulate real limited-data training conditions, one should then find the θ * for each {i, ..., j} using a separate cross-validation within {i, ..., j} only; this would slow down the experiments considerably.


Side information is sometimes encoded as "virtual examples" like our contrast examples or pseudoexamples.

past work generates these by automatically transforming the training examples in ways that are expected to preserve or alter the classification (Abu-Mostafa, 1995).


(2005) asked annotators to identify globally "relevant" features.

our approach does not force the annotator to evaluate the importance of features individually, nor in a global context outside any specific document, nor even to know the learner's feature space.


Our previous results indicated that Concept Repetition was the best feature to add to the Baseline 2 state-space model, but also that Percent Correctness and Frustration (when added to Baseline 2) offered an improvement over the Baseline MDP's.

these conclusions were based on a very qualitative approach for determining if a policy is reliable or not.


For example, a model can have an optimal policy with a very high ECR value, but have very wide confidence bounds reaching even into negative rewards.

another model can have a relatively lower ECR but if its bounds are tighter (and the lower bound is not negative), one can know that that policy is less affected by poor parameter estimates stemming from data sparsity issues.


The Concept Repetition model (which is composed of the Concept Repetition, Certainty and Correctness features) was especially useful, as it led to a policy that outperformed the Baseline 1 model, even when we take into account the uncertainty in the model estimates caused by data sparsity.

for the Baseline 2, Percent Correctness, and Frustration models, the estimates for the expected cumulative reward are much less reliable, and no conclusion can be reliably drawn about the usefulness of these features.


Our initial findings indicate that, as more data becomes available the bounds tighten for most parameters in the transition matrix.

for some of the parameters the bounds can remain wide, and that is enough to keep the confidence interval for the expected cumulative reward from converging.


We also tried learning the priming weight with an EM algorithm.

we found out that the learned priming weight performed worse than the empirical one in our experiments.


In late application, the S-Bigram1 model performed better than the trigram model (t = 2.11, p < 0.02, one-tailed), so did the S-Bigram2 model (t = 1.99, p < 0.025, onetailed).

compared to the bigram model, the S-Bigram1 model did not change the recognition performance significantly (t = 0.38, N.S., two-tailed) in late application, neither did the S-Bigram2 model (t = 0.50, N.S., two-tailed).


In the bigram case, the path "<s> a tree" has a higher language score (summation of bigram probabilities along the path) than "<s> sheet", and "a floor" has a higher language score than "a full".

these correct paths "<s> a tree" and "a floor" (not exactly correct, but better than "a full") do not appear in the best hypothesis in the resulting n-best list.


In this example, the user was talking about a picture entity picture bamboo.

this entity was not salient, only entities bedroom and chandelier 1 were salient.


The four methods above are based on context information.

our method exploits the internal structure of the semantic orientations of phrases.


Since we cannot expect a large labeled dataset to be available for each adjective, we use not the approximated leave-one-out error rate, but the magnetization-like criterion.

the magnetization above is defined for the Ising model.


There are still 3320 (=12066-8746) word pairs which could not be classified, because there are no entries for those words in the dictionary.

the main cause of this problem is word segmenta-tion, since many compound nouns and exceedinglysubdivided morphemes are not in dictionaries.


As Table 2 shows, we outperform the PRANK baseline in both cases.

on the consensus instances we achieve a relative reduction in error of 21.8% compared to only a 1.1% reduction for the other set.


The agreement model, when considered on its own, achieves classification accuracy of 67% on the test set, compared to a majority baseline of 58%.

those instances with high confidence |a • x| exhibit substantially higher classification accuracy.


Each classifier in Section 3.3 is also used to learn a sentence classifier.

we then must make a document-level prediction.


In fact, the proposed method outperforms simple WordNet based approaches such as Edge-Counting and Information Content measures.

considering the high correlation between human subjects (0.9), there is still room for improvement.


In word sense disambiguation, choosing the most frequent sense for an ambiguous word is a powerful heuristic.

its usefulness is restricted by the availability of sense-annotated data.


Nonetheless, the task is still difficult, even for human judges, as we will see in Section 6.4.

because the solution allows only one correct answer the accuracies are underestimated.


Transliteration between languages which share similar alphabets and sound systems is usually not difficult, because the majority of letters remain the same.

the task is significantly more difficult when the language pairs are considerably different, for example, English-Arabic, English-Chinese, and English-Japanese.


The second solution is to add a null letter in the grapheme sequence.

the null letter not only confuses the phoneme prediction model, but also complicates the the phoneme generation phase.


Arabic words consist of a stem surrounded by prefixes and suffixes, which are fairly successfully segmented out by Morfessor.

arabic also has templatic morphology, i.e., the stem is formed through the insertion of a vowel pattern into a "consonantal skeleton".


The approach is more suitable for dependency parsing since trees do not have non-terminal nodes, therefore revisions do not require adding/removing nodes.

the method applies to any parser since it only analyzes output trees.


Similarly to re-ranking our method aims at improving the accuracy of the base parser with an additional learner.

as in transformationbased learning, it avoids generating multiple parses and applies revisions to arcs in the tree which it considers incorrect.


(2006) that a hierarchically split PCFG could exceed the accuracy of lexicalized PCFGs (Collins, 1999;Charniak and Johnson, 2005).

many questions about inference with such split PCFGs remain open.


Adapting unlexicalized parsers appears to be equally difficult: Levy and Manning (2003) adapt the unlexicalized parser of Klein and Manning (2003) to Chinese, but even after significant efforts on choosing category splits, only modest performance gains are reported.

automatically learned grammars like the one of Matsuzaki et al.


On the one hand, segmentation is done automatically, so it is realistic that given a "real world" document, we can compute segment boundaries to help our classification judgments.

testing on unsegmented input allows us to compare more directly to the numbers from our previous section.


On the PDTB test data, performance using the segment-trained classifiers improves in two of three cases, with a mean improvement of 1.2%.

because of the small size of this set, this margin is not statistically significant.


Whether or not we use topic segmentation to constrain our training instances, our patterns rely on sentence boundaries and cue phrase anchors to demarcate the extents of the text spans which form our RSR instances.

an instance which matches such a pattern often contains some amount of text which is not relevant to the relation in question.


as "a drop in oil prices" and "weakness in the automotive sector."

the output of our instance mining process simply splits the string around the cue phrase "because of" and extracts the entire first and second parts of the sentence as the constituents.


We evaluate the impact of our syntactic heuristics on classification over the Auto and PDTB test sets using the same instance set of 400,000 training instances per relation.

each applies different filters to the instances I before computing the frequencies F (all other parameters use the same values; these are set slightly differently than the optimized values discussed earlier because of the smaller training sets).


Even with this reduced training set, the syntactic heuristic improves performance in two out of three cases on the PDTB test set, including a 2.7 percent improvement for the Cause vs NoRel classifier.

due to the small size of the PDTB test set, none of these differences is statistically significant.


Some local models also have trouble deciding which of a pair of related sentences ought to come first.

the global HMM model of Barzilay and Lee (2004) tries to track the predictable changes in topic between sentences.


The AIRPLANE documents have some advantages for coherence research: they are short (11.5 sentences on average) and quite formulaic, which makes it easy to find lexical and structural patterns.


Domains with more natural writing styles will make lexical prediction a much more difficult problem.

the wider variety of grammatical constructions used may motivate more complex syntactic features, for instance as proposed by (Siddharthan et al., 2004) in sentence clustering.


Being heuristic in nature, this algorithm is not guaranteed to find an optimal solution.

its simplicity and time efficiency make it a decoding algorithm of choice for a wide range of NLP applications.


We constrain demand variables d v by: The right hand side sums the number of selected edges entering v, and will therefore be either 0 or 1.

next we add variables a v and b v constrained by the equations: note that a v sums over f values on all edges entering v. by the previous constraints those f -values can only be nonzero on the (at most one) selected edge entering v. So, a v is simply the fvalue on the selected edge entering v, if one exists, and 0 otherwise.


A static linear interpolation of predictions using the two approaches led to only minimal reductions of prediction error, likely because predictions from the poorer performing grammar-based classifier were always given the same weight.

with the confidence measures, predictions from the grammar-based classifier could be given more weight when the confidence measure was high, and less weight when the measure was low and the predictions were likely to be inaccurate.


The trends were similar for both sets of grammatical features.

the first set of features that included complex syntactic constructs led to better performance than the second set, which included only verb tenses, part of speech labels, and sentence length.


ated reference translation to explicitly bias the language model.

one has to take care not to bias towards the correct response so strongly that the student is allowed to make mistakes with impunity.


The transcriber was instructed to transcribe spontaneous speech effects, such as false starts and filled pauses.

tonal mispronunciations are completely ignored, and segmental errors are largely ignored to the extent that they do not result in a different syllable.


The use of the Europarl corpus is not an accurate simulation of the intended situation because it enables us to use a relatively large parallel corpus for direct training.

it is necessary to evaluate the performance of the pivot strategies against that of Direct SMT systems under controlled experiments in order to determine how much the pivot strategies can be improved.


of phrases in PhraseTrans system Recall was reasonably high.

the upper bound of recall was 100 percent because we used a multilingual corpus whose sentences were aligned to each other across all four languages, as described in Section 4.1.


They showed that this direct training method is superior to the sentence translation strategy (SntTrans1) in translating Catalan into English but is inferior to it in the opposite translation direction (in terms of the BLEU score).

we have shown that the phrase translation strategy consistently outperformed the sentence translation strategy in the controlled experiments.


MT users typically try to reduce the post-editing load by customizing their MT systems.

in Rule-based Machine Translation (RBMT), which still constitutes the bulk of the current commercial offering, customization is usually restricted to the development of "user dictionaries".


For how-questions, this corresponds to a numerical type.

retrieving all numerical entities will provide lower answer identification precision than a system that only provides those specified with the expected answer units.


The queryspecific oracle is able to achieve the highest performance because of perfect knowledge of the units appropriate to a given question.


Its precision runs at just under 5%, about a quarter of the lowest precision of any of our unit-list approaches at any recall value.

fixed-category typing does achieve high recall, roughly 96%, missing only numerical entities unrecognized by Minipar.


In response to questions about particular events of interest that can be enumerated in advance, it is possible to perform a deeper semantic analysis focusing on the entities, relations, and sub-events of interest.

the deeper analysis may be errorful and will also not always provide complete coverage of the information relevant to the query.


The collected Discovery video is a small but "long" OCR document set, which results the estimation of DF value unreliable.

a passage is more coherent than a long document, thus we replace the DF estimation with PF score.


This approach works surprisingly well, mainly because an explicit effort was made to use arguments Arg0 and Arg1 consistently across different verbs; and because those two argument labels account for 85% of all arguments.

this approach causes the system to conflate different argument types, especially with the highly overloaded arguments Arg2-Arg5.


For example, the Arg1 (underlined) in "John broke the window" is the same window that is annotated as the Arg1 in "The window broke", even though it is the syntactic subject in one sentence and the syntactic object in the other.

there is no guarantee that an argument label will be used consistently across different verbs.


Generally, the arguments are simply listed in the order of their prominence for each verb.

an explicit effort was made when PropBank was created to use Arg0 for arguments that fulfill Dowty's criteria for "prototypical agent," and Arg1 for arguments that fulfill the criteria for "prototypical patient."


From this figure, we can see that Arg0 maps to agent-like roles, such as "agent" and "experiencer," over 94% of the time; and Arg1 maps to patientlike roles, including "theme," "topic," and "patient," over 82% of the time.

arguments Arg2-5 get mapped to a much broader variety of roles.


For a system trained on WSJ, there is a fairly small drop in performance of the ID task when tested on Brown vs tested on WSJ (Fscore 92.7 vs 95.3).

in this same condition, the Classification task has a very large drop in performance (72.0% vs 86.1%).


We investigated this technique but discarded it due to subtle yet critical issues with pattern canonicalization that resulted in rejecting nearly all inferences.

we are investigating other ways of using Web corpora for this task.


There were 225 valid abbreviations already identified by the expertise taxonomy team.

we found many abbreviations that appeared in the skill statements but were not listed there.


The assumption for this shortest path approach is that the links in the taxonomy represent uniform distances.

in most taxonomies, sibling concepts deep in the taxonomy are usually more closely related than those higher up.


Since the system strategies are evaluated and adapted based on the analysis of these simulated dialog behaviors, we would expect that these behaviors are what we are going to see in the test phase when the systems interact with human users.

for automatically learning dialog strategies, it is not clear how realistic versus how exploratory (Singh et al., 2002) the training corpus should be.


EM1 measures exactly what RF1 tries to optimize; while EM2 measures exactly what RF2 tries to optimize.

we show the results evaluated by both EM1 and EM2 for all configurations since the two evaluation measures have their own practical values and can be deployed under different design requirements.


It does explore some states that are at the end of the paths, but since these are the infrequent states in the test phase, exploring these states does not actually improve the model's performance much.

while the student correctness rate in the real corpus is 60%, the RRM prevents itself from being trapped in the less-likely states on incorrect answer paths by keeping its correctness rate to be 50%.


To find the k-best matches for a particular substring s, we do what we would normally do for standard suffix arrays on lexicographic splits.

on popularity splits, we search the more popular half first and then we search the less popular half, if necessary.


In order to make the method computationally feasible, potential cognates were filtered based on word order, location in the document, frequency, and length.

we found that a faster and simpler procedure, which is described below, performed extremely well, eliminating the need for a more sophisticated approach.


Students then hear a segment of a lecture concerning the same topic.

the lecture contradicts and complements the information contained in the reading.


Vergyri and Kirchhoff (2004) follow an approach similar to ours in that they choose from the diacritizations proposed by BAMA.

they train a single tagger using unannotated data and EM, which necessarily leads to a lower performance.


It is possible to construct a combined system which uses a lexicon, but backs off to a Zitouni-style system for unknown words.

a large portion of the unknown words are in fact foreign words and names, and it is not clear whether the models learned handle such words well.


The bound is the same as for the reported algorithms for checking wellnestedness (Möhl, 2006).

the following theorem allows wellnestedness checking to be linear for projective trees, to be faster for random input, and to remain O(n 2 ).


These systems use supervised learning methods which only utilize annotated NL sentences.

it requires considerable human effort to annotate sentences.


However, it requires considerable human effort to annotate sentences.

unannotated NL sentences are usually easily available.


Recently some alignment evaluation metrics have been proposed which are more informative when the alignments are used to extract translation units (Fraser and Marcu, 2006;Ayan and Dorr, 2006).

these metrics assess translation quality very indirectly.


For the classification task, training with MT data is less effective than with non-native data.

for the ranking task, models trained on publicly available MT data generalize well, performing as well as those trained with a non-native corpus of size 10000.


Finally, we fuse the extracted binary relations into a ternary relation.

with our discriminative model, a statistical parsing based generative model (Shi et al., 2007) has been proposed for a related task on this data set where the NEs and their relations are extracted together and used to identify which NEs are relevant in a particular sentence.


In this paper we explored the use of rich syntactic features for the relation extraction task.

with the previously used set of syntactic features for this task, we use a large number of features originally proposed for the Semantic Role Labeling task.


To our knowledge, no research has yet been conducted on soundbite speaker name identification in Mandarin BN domain.

this work is related to some extent to speaker role identification, speaker diarization, and named entity recognition.


IceTagger makes 11.5% less errors than TnT.

when tag profile gaps in the lexicon, used by TnT, are filled with tags produced by IceMorphy, our morphological analyser, TnT's tagging accuracy increases to 91.18%.


New assignments of part-ofspeech (POS) to words is found by optimising the product of lexical probabilities (p(w i |t j )) and contextual probabilities (p(t i |t i−1 , t i−2 )) (where w i and t i are the i th word and tag, respectively).

to DDMs, LRBMs are developed with the purpose of tagging a specific language using a particular tagset.


The average tagging accuracy of Ice-Tagger is 91.54%, compared to 90.44% obtained by the TnT tagger.

we were able to obtain 91.18% accuracy using TnT along with the tag profile gap filling mechanism of IceMorphy.


The algorithm they propose to compute this term is O(n 2 s 3 ) as it requires an extra nested loop in forward/backward.

the above form of the gradient is not the only possibility.


(2006), there are two terms, and the second is easily computable given the feature expectations obtained by forward/backward and the entropy for the sequence.

unlike the previous method, here the first term can be efficiently calculated as well.


2006) suggest this may no longer be the case.

the "additional" goals still favor symbolic parsing.


A syntactic variant of this method (denoted as GI-AMTI) has been used in this work in order to infer the models from training samples as it is summarized in section 3.

speech translation has been already carried out by integrating acoustic models into a SFST .


So far, the capability of the systems have been assessed in terms of time and spatial costs.

the quality of the translations they provide is, doubtless, the most relevant evaluation criterion.


This analysis reveals that both systems produce the same kind of errors in general.


Missing words is also an important problem.

most of them (approximately two thirds for both systems) are filler words (i.e.


The N -gram based system seems to be able to produce more accurate translations (reflected by a lower percentage of translation errors).

it generates too many additional (and incorrect words) in the process.


Given an N -best list of the phrase-based (N -grambased) system, we compute the cost of each target sentence of this N -best list for the N -gram-based (phrase-based) system.

this computation is not possible in all cases.


All the participants successfully detected "summary" as an error.


Extraction using Semi-supervised Parsing: Experiments with purely supervised learning show that our generative model requires a large curated set to minimize the sparse data problem, but domainspecific annotated corpora are always rare and expensive.

there is a huge source of unlabeled MEDLINE articles available that may meet our needs, by assuming that any sentence containing BACTERIUM, PROTEIN and LOCATION NEs has the BPL relation.


Moreover, speech recognition systems can benefit from being trained on hand-transcribed data where all the appropriate word level segmentations (i.e., the exact time of the word boundaries) are known.

with increasing amounts of raw speech data being made available, it is both time consuming and expensive to accurately segment every word for every given sentence.


As the TIMIT corpus provides phone level segmentations, P t is observed during training.

for reasons that will become clear in the next section, we treat P t as hidden but make it the parent of a rv C t , with, p(c t = 1|p t ) = δ lt=pt where l t is obtained from the transcriptions (l t ∈ D Pt ).


From the perspective of a transcriber, this simulates the task of going through an utterance and identifying only one frame that belongs to  Figure 3: Virtual Evidence Results each particular phone without having to identify the phone boundary.

to the task of determining the phone boundary, identifying one frame per word unit is much simpler, less prone to error or disagreement, and less costly (Greenberg, 1995).


Most high-tech AAC devices provide the user with an electronic letter and word board to input messages which are output via speech synthesis.

even with substantial user interface optimization, communication rate is often less than 10 words per minute (Newell et al., 1998) as compared to about 150-200 words per minute for unimpaired speech.


However, we have shown that despite significant cognitive load, the reduction in keystroke savings dominates the effect on output rate.

to earlier studies, our basic method showed a significantly improved communication rate over no prediction.


We focus on pitch resets in the ROIs and thus obtain pitch contours for the left and right tonal syllables for each ROIs.

the corpus transcription does not provide time annotations for those tonal syllables.


Considering the difficulty in obtaining high quality transcriptions, some researchers proposed speech summarization systems with non-lexical features (Inoue et al., 2004;Koumpis and Renals, 2005;Maskey and Hirschberg, 2003;Maskey and Hirschberg, 2006).

there does not exist any empirical study on speech summarization without lexical features for Mandarin Chinese sources.


Most commonly, we use sentences to model individual pieces of information.

more NLP applications require us to define text units smaller than sentences, essentially decomposing sentences into a collection of phrases.


In the PYRAMID scheme for manual evaluation of summaries (Nenkova and Passonneau, 2004), machine-generated summaries were compared with human-written ones at the nugget level.

automatic creation of the nuggets is not trivial.


The process of creating nuggets has been automated and we can assume a certain level of consistency based on the usage of the syntactic parser.


The first round initially appears successful because the two annotators had 100% agreement on nugget groups and their corresponding scores.

c2, the novice nuggetizer, was much more conservative than c1, because only 10 nugget groups were created.


Esuli and Sebastiani (2006) determine the polarity (positive/negative/objective) of word senses in WordNet.

there is no evaluation as to the accuracy of their approach.


Not all WordNet relations we use are subjectivitypreserving to the same degree: for example, hyponyms (such as simpleton) of objective senses (such as person) do not have to be objective.

we aim for high graph connectivity and we can assign different weights to different relations to reflect the degree to which they are subjectivitypreserving.


Many approaches to opinion, sentiment, and subjectivity analysis rely on lexicons of words that may be used to express subjectivity.

words may have both subjective and objective senses, which is a source of ambiguity in subjectivity and sentiment analysis.


These methods acquire contextual information directly from unannotated raw text, and senses can be induced from text using some similarity measure (Lin, 1997).

automatically acquired information is often noisy or even erroneous.


Comparing with tagging or chunking, parsing is relatively expensive and time-consuming.

in our method parsing is not performed in real time when we disambiguate words.


names (Chen Yao)  Shijiazhuang-city Branch, the second sales department) and (Science and Technology Commission of China, National Institution on Scientific Information Analysis).

it sometimes over-generalized to long words.


To borrow broad terminology from the Optimality Theory literature (Prince and Smolensky, 1993), such models incorporated faithfulness features, capturing the ways in which successive forms remained similar to one another.

each language has certain regular phonotactic patterns which con-strain these changes.


Only the words in the language at the root of the tree, if any, are explicitly encouraged to be well-formed.

we incorporate constraints on markedness for each language with both general and branch-specific constraints on faithfulness.


Our system's reconstruction had an edit distance of 3.02 to the truth against 3.10 for BCLKG.

this difference was not significant (p = 0.15).


Correlation among θ i and θ j , i = j, cannot be modeled directly, only through the normalization in step 2.

lN distributions (Aitchison, 1986) provide a natural way to model such correlation.


In MCTAG the derivation trees are often drawn with identifiers of entire tree sets as the nodes of the tree because the lexical locality constraints require that each elementary tree set be the derivational child of only one other tree set.

if we elaborate the derivation tree to include a node for each tree in the grammar rather than only for each tree set we can see a stark contrast in the derivational Figure 3: An example SL-MCTAG grammar that generates the language ww and associated derivation tree that demonstrating an arbitrarily long derivational distance between the trees of a given tree set and their nearest common ancestor.


The same complexity analysis applies for re-stricted V-TAG.

we can provide a somewhat tighter bound by noting that the rank, r, of the grammar-how many tree sets adjoin in a single tree-and the fan out, f of the grammar-how many trees may be in a single tree set-are limited by t. That is, a complete derivation containing |D| tree sets can contain no more than t |D| individual trees and also no more than rf |D| individual trees.


DMV was the first unsupervised dependency grammar induction system to achieve accuracy above a right-branching baseline.

dMV is not able to capture some of the more complex aspects of language.


The primary difference between EVG and DMV is that DMV uses valence information to determine the number of arguments a head takes but not their categories.

eVG allows different distributions over arguments for different valence slots.


Maximum likelihood estimation provides a point estimate ofθ.

often we want to incorporate information aboutθ by modeling its prior distribution.


Transductive graph-based regularization has been applied to large-margin learning on structured data (Altun et al., 2005).

scalability quickly becomes a problem with these approaches; we solve that issue by working on transitive closures as opposed to entire graphs.


Matches can be exact or fuzzy; the latter is similar to the identification of graph neighborhoods in our approach.

our GBL scheme propagates similarity scores not just from known to unknown sentences but also indirectly, via connections through other unknown sentences.


Various filtering techniques, such as (Johnson et al., 2007) and (Chen et al., 2008), have been applied to eliminate a large portion of the translation rules that were judged unlikely to be of value for the current translation.

these approaches were only able to improve the translation quality slightly.


The phrase pairs "ihre kinder nicht → their children are not" and "ihre kinder nicht → their children" are both likely also to appear in the phrase table and the former has greater estimated probability.

the language model would preferred the latter in this example because the sentence "They love their children are not."


It is difficult to adjust the system to work differently.

as the triangulated filtering procedure does not consider probability distributions in the models, it is possible to further filter the tables according to the probabilities.


Methodology for eliciting judgments The obvious way to evaluate the precision of our algorithm is to have human annotators judge each output item as to whether it is a DE operator or not.

there are some methodological issues that arise.


We therefore encouraged the judges to try to construct sentences wherein the arguments for candidate DE operators were drawn from a set of phrases and restricted replacements we specified (example: 'singing' vs 'singing loudly').

improvisation was still required in a number of cases; for example, the candidate 'act', as either a noun or a verb, cannot take 'singing' as an argument.


One final remark regarding the annotation: some decisions still seem uncertain, since various factors such as context, Gricean maxims, what should be presupposed 8 and so on come into play.

we take comfort in a comment by Eugene Charniak (personal communication) to the effect that if a word causes a native speaker to pause, that word is interesting enough to be included.


We will refer predicates such as word as observed because they are known in advance.

is-Predicate is hidden because we need to infer it at test time.


This type of algorithm could also be realised for an ILP formulation of SRL.

it would require us to write a dedicated separation routine for each type of constraint we want to add.


33% of the test items in Collins and Singer (1999) were people, as opposed to 21% of ours.

even without the pronoun features, that is, using the same feature set, our system scores equivalently to the EM model, at 83% (this score is   on dev, 25% people).


An interesting avenue of research is to construct the vocabulary tree based on WordNet, as a way to inject independent prior knowledge into the model.

wordNet has a low coverage problem, i.e.


Because the CMQD model can easily hypothesize implausible degradations, we see the MAP increases modestly with a few degradations, but then MAP decreases.

the MAP of the phrase-based system (PBQD-Fac) increases through to 500 query degradations using multigrams.


These features provide the means to link messages that may not have sufficient lexical overlap but are nevertheless likely to be topically related.

our work is different from them in several aspects: (1) They treat individual messages as the basic elements for clustering, and ignore the social and temporal contexts of the messages.


The lack of supervised labels makes it even more important to leverage rich features and global dependencies.

existing systems use directed generative models (Creutz and Lagus, 2007;Snyder and Barzilay, 2008b), making it difficult to extend them with arbitrary overlapping dependencies that are potentially helpful to segmentation.


In this way, we can simultaneously emphasize that a lexicon should contain few unique morphemes, and that those morphemes should be short.

the lexicon prior alone incorrectly favors the trivial segmentation that shatters each word into characters, which results in the smallest lexicon possible (single characters).


In this regard, DELORTRANS1 is suitable for POS tagging since deleting a word often results in an ungrammatical sentence.

in morphology, a word less a character is often a legitimate word too.


This makes sense: if a rule has a variable that can be filled by any English preposition, there is a risk that an incorrect preposition will fill it.

splitting at a period is a safe bet, and frees the model to use rules that dig deeper into NP and VP trees when constructing a top-level S. Table 5 shows weights for generated English nonterminals: SBAR-C nodes are rewarded and commas are punished.


The work in this phase is cubic in sentence length.

lexical rules in LNF can be applied without binarization, because they only apply to particular spans that contain the appropriate lexical items.


We can have a simple rule to achieve this.

in reality, there are many possible children for a verb.


Therefore, they would need to rely on reorder units that are likely not violating "phrase" boundaries.

since we reorder both training and test data, our system operates in a matched condition.


Although we use manually written rules in this study, it is possible to learn our rules automatically from alignments, similarly to Habash, 2007.

unlike Habash, 2007, our manually written rules handle unseen children and their order naturally because we have a default precedence weight and order type, and we do not need to match an often too specific condition, but rather just treat all children independently.


Intuitively, one solution is to extend the feature set by considering both boundary words, forming a more complete boundary description.

this method is still based on lexicalized features, which causes data sparseness problem and fails to generalize.


such features would be treated as unseen by the current ME model, since the training data can not possibly cover all such similar cases.


For example, in phrase-based SMT systems (Koehn et al., 2003;Koehn, 2004), distortion model is used, in which reordering probabilities depend on relative positions of target side phrases between adjacent blocks.

distortion model can not model long-distance reordering, due to the lack of context information, thus is difficult to predict correct orders under different circumstances.


Boundary POS is considered in LABTG only when source phrases are not syntactic phrases.

to the previous works, we present a reordering model for BTG that uses bilingual information including class-level features of POS and word classes.


Moreover, compared with the Treebank Chinese tagset, the CKIP tagset pro-vides more fine-grained tags, including many tags with semantic information (e.g., Nc for place nouns, Nd for time nouns), and verb transitivity and subcategorization (e.g., VA for intransitive verbs, VC for transitive verbs, VK for verbs that take a clause as object).

using the POS features in combination with the lexical features in target language will cause another sparseness problem in the phrase table, since one source phrase would map to multiple target ones with different POS sequences.


The first sentence contains positive opinion, the second negative opinion.

wishful statements like the third sentence are often annotated as non-opinion-bearing in sentiment analysis corpora (Hu and Liu, 2004;Ding et al., 2008), even though they clearly contain important information.


The 11 topics in Section 2.1 were manually predefined based on domain knowledge.

in this section we applied Latent Dirichlet Allocation (LDA) (Blei et al., 2003) to identify the latent topics in the full set of 89,574 English wishes in an unsupervised fashion.


In terms of average AUC across folds (Table 5), [Words + Templates] is also the best.

due to the small size of this corpus, the AUC values have high variance, and the difference between [Words + Templates] and [Words] is not statistically significant under a paired t-test (p = 0.16).


Many researchers have focused the related problem of predicting sentiment and opinion in text (Pang et al., 2002;Wiebe and Riloff, 2005), sometimes connected to extrinsic values like prediction markets (Lerman et al., 2008).

to text regression, text classification comprises a widely studied set of problems involving the prediction of categorial variables related to text.


In order to minimize the efforts required in the domain transfer, we often expect to use p s (x, y) to approximate p t (x, y).

data distribution are often varied with the domains.


FMM outperforms the SIM method by an average of 4% increase in performance (13% improvement after 10 iterations).

both the FMM and the SIM method are able to outperform the baseline method.


For example, voiceless plosive sounds such as p, t in English, tend to map to both voiced (such as b, d) and voiceless sounds in Chinese.

if the sound is voiceless in Chinese, its backtrack English sound must be voiceless.


Unfortunately, we could not find a published Chinese dataset.

our system achieved similar results to other systems, over a different dataset with similar number of training examples.


Consequently, in a word like vintage [vÁntÁ], we can rule out a syllabification like [vÁ-ntÁ] because [n] is more sonorant than [t].

ssP does not tell us whether to prefer [vÁn-tÁ] or [vÁnt-Á].


Both the Legality Principle and SSP tell us which onsets and codas are permitted in legal syllables, and which are not.

neither theory gives us any guidance when deciding between legal onsets.


The results of the competitive approaches that have been quoted so far (with the exception of tsylb) are not directly comparable, because neither the respective implementations, nor the actual train-test splits are publicly available.

we managed to obtain the English and German data sets used by Goldwater and Johnson (2005) in their study, which focused primarily on unsupervised syllabification.


Ideally, a named entity should correspond to a phrase in the constituency tree.

parse trees will occasionally lack some explicit structure, such as with right branching NPs.


Because we only allow named entity derivations which we have seen in the data, nested entities are impossible.

there is clear benefit in a representation allowing nested entities.


Because the underlying grammar (ignoring the additional named entity information) was the same for both the joint and baseline parsers, it is the case that whenever a sentence is unparseable by either the baseline or joint parser it is in fact unparsable by both of them, and would affect the parse scores of both models equally.

the CRF is able to named entity tag any sentence, so these unparsable sentences had an effect on the named entity score.


Previous work on linguistic annotation pipelines (Finkel et al., 2006;Hollingshead and Roark, 2007) has enforced consistency from one stage to the next.

these models are only used at test time; training of the components is still independent.


As pointed out by Gildea and Temperley (2007), however, finding the unconstrained minimal-length linearization is a well-studied problem with an O(n 1.6 ) solution (Chung, 1984).

this approach does not take into account constraints of projectivity or mild context-sensitivity.


In the unsupervised setting, a variety of successful systems have leveraged lexical cohesion (Halliday and Hasan, 1976) -the idea that topically-coherent segments display consistent lexical distributions (Hearst, 1994;Utiyama and Isahara, 2001; Eisenstein and Barzilay, 2008).

such systems almost invariably focus on linear segmentation, while it is widely believed that discourse displays a hierarchical structure (Grosz and Sidner, 1986).


Ideally we would like to choose the segmentation y = argmax y p(w|y)p(y).

we must deal with the hidden language models Θ and scale-level assignments z.


Given language models Θ, each w t can be thought of as a draw from a Bayesian mixture model, with z t as the index of the component that generates w t .

as we are marginalizing the language models, standard mixture model inference techniques do not apply.


Baseline systems As noted in Section 2, there is little related work on unsupervised hierarchical segmentation.

a straightforward baseline is a greedy approach: first segment at the top level, and then recursively feed each top-level segment to the segmenter again.


When BACKGROUND or DOCSPECIFIC topics are chosen, the model works exactly as in TOPICSUM.

when the CONTENT topic is drawn, we must decide whether to emit a general content word (from φ C 0 ) or from one of the specific content distributions (from one of φ C i for i = 1, .


However, in LDA each word's topic assignment is conditionally independent, following the bag of words view of documents.

our constraints on how topics are assigned let us connect word distributional patterns to document-level topic structure.


The normalization constant of this distribution is unknown, meaning that we cannot directly compute and invert the cumulative distribution function to sample from this distribution.

the distribution itself is univariate and unimodal, so we can expect that an MCMC technique such as slice sampling (Neal, 2003) should perform well.


Thus, we expect to reduce the edit overhead in proportion with ∆.

allowing the use of a right context leads to the current hypothesis lagging behind the gold standard.


This gives the user more flexibility to "say anything at any time".

in recent evaluations of one-exchange LBVS we have found that locations are recognized with much higher accuracy than listing names 1 .


designed to produce a minimal number of local area LMs.

if the user is near the edge of the pre-defined local area, the selected LM may exclude businesses close to the user and include businesses far away from the user.


This approach has the advantage that listings included in the language model will certainly be close to the user.

onthe-fly computation of geo-centric language models for large numbers of users is too computationally demanding given current database and processing technology.


When we combine local and nationwide LMs using LM union, we get small increases in sentence accuracy for nationwide queries compared to local LMs alone.

sentence accuracy for local listings decreases.


For languages with complex letter-to-sound mappings, such dictionaries are typically written by hand.

for morphologically rich languages, such as MSA, 1 pronunciation dictionaries are difficult to create by hand, because of the large number of word forms, each of which has a large number of possible pronunciations.


Therefore, entries in the decoding pronunciation dictionary consist of undiacritized words that are mapped to a set of phoneticallyrepresented diacritizations.

every entry in the training pronunciation dictionary is a fully diacritized word mapped to a set of possible contextdependent pronunciations.


As noted in Section 1, pronunciation dictionaries for ASR systems are usually written by hand.

arabic's morphological richness makes it difficult to create a pronunciation dictionary by hand since there are a very large number of word forms, each of which has a large number of possible pronunciations.


Figure 1 shows two lattices that encode the most linguistically plausible ways of segmenting two prototypical German compounds with compositional meanings.

while these words are structurally quite similar, translating them into English would seem to require different amounts of segmentation.


For each translation, we have access to the phrases used by the decoder to produce that output.

there may be islands of out-of-vocabulary (OOV) words that were not in the phrase table and not translated by the decoder as a phrase.


It has more than 200 million speakers around the world.

bangla has few available language resources, and lacks resources for machine translation.


We gain 35.6%, 5.2%, and 9.4% relative improvements, respectively.

the results tend to be worse when 20% and 80% training data were used initially, with 11.6% and 3.0% minimal relative loss.


With the backpointer (N ′ , x ′ , y ′ ) = BP (N, x, y, r, i), these special arcs are introduced as: are a mix of target language words and lattice pointers (Figure 4,  top).

each still represents the entire search space of all translation hypotheses covering the span.


For non-sparse models, there is significant variation in average discounts, but because 1 D F i=1 |λ i | is low, the overall error is low.

sparse models are dominated by singlecount n-grams with features whose average discount is quite close to γ = 0.938.


These techniques include the hold-out method; leave-one-out and k-fold cross-validation; and bootstrapping (Allen, 1974;Stone, 1974;Geisser, 1975;Craven and Wahba, 1979;Efron, 1983).

unlike non-datasplitting methods, these methods do not lend themselves well to providing insight into model design as discussed in Section 6.


In ML, we maximize the probability of a given sequence of observations O, belonging to a given class, given the HMM λ of the class, with respect to the parameters of the model λ.

this probability is the total likelihood of the observations and can be expressed mathematically as: there is no known way to analytically solve for the model λ = {A, B, π} , which maximize the quantity L tot , where A is the transition probabilities, B is the observation probabilities, and π is the initial state distribution.


The basic idea of the proposed modeling is to create a separate model for each word of the language and use the language model corpus to estimate the parameters of the model.

one could argue that the basic model could be improved by taking the contexts of the morphemes into account.


For a vocabulary size of V , the number of tri-morpheme could be as high as V 3 .

most of the tri-morphemes are either rare or will not be observed in the training data altogether.


Most of the probabilities lie along the diagonal line.

some trigram probabilities are modulated making TMLM probabilities quite different than the corresponding trigram probabilities.


This improvement is not statistically significant.

interpolating TMLM with Word-3gr improves the WER to 31.9%, which is 1.0% better than that of the Word-3gr.


As the distribution this model describes does not change, neither will its training performance.

the (unscaled) size F i=1 |λ i | of the model has been reduced from 3+4=7 to 0+1+3=4, and consequently by eq.


This algorithm has working storage that it can use to store parts of the input or other intermediate computations.

(and this is a critical constraint), this working storage space is significantly smaller than the input stream length.


This number is not significantly worse than the 36% of stream counts obtained from 4, 018k true counts for the smallest value of ǫ = 5e-8.

if we look at the other two metrics, the ranking correlation ρ of stream counts compared with true counts on ǫ = 50e-8 and 20e-8 is low compared to other ǫ values.


Performance starts to degrade as we get to 2, 000k (over 50% of all 5-grams), a result that is not too surprising.

even here we note that the MSE is low, suggesting that the frequencies of stream counts (found in top K true counts) are very close to the true counts.


As expected, the recall with respect to true counts is maximum for unigrams, bigrams, trigrams and 5-grams.

the amount of space required to store all true counts in comparison to stream counts is extremely high: we need 4.8GB of compressed space to store all the true counts for 5-grams.


For 5-grams, the best recall value is .020 (1.2k out of 60k 5-gram stream counts are found in the test set).

compared with the true counts we only loss a recall of 0.05 (4.3k out of 60k) points but memory savings of 150 times.


(2007) reported that translation quality continued to improve with increasing corpus size for training language models at even size of 2 trillion tokens, the increase became small at the corpus size of larger than 30 billion tokens.

for more complex NLP tasks, such as case structure analysis and zero anaphora resolution, it is necessary to obtain more structured knowledge, such as semantic case frames, which describe the cases each predicate has and the types of nouns that can fill a case slot.


For examples, Kawahara and Kurohashi proposed a method for constructing wide-coverage case frames from large corpora (Kawahara and Kurohashi, 2006b), and a model for syntactic and case structure analysis of Japanese that based upon case frames (Kawahara and Kurohashi, 2006a).

they did not demonstrate whether the coverage of case frames was wide enough for these tasks and how dependent the performance of the model was on the corpus size for case frame construction.


(Nakov and Hearst, 2005;Gledson and Keane, 2008)).

search engines are not designed for NLP research and the reported hit counts are subject to uncontrolled variations and approximations.


In English, overt pronouns such as she and definite noun phrases such as the company are anaphors that refer to preceding entities (antecedents).

in Japanese, anaphors are often omitted; these omissions are called zero pronouns.


P (n j |CF l , s j , A (s j ) = 1) is similar to P (n j |CF l , s j , A(s j ) = 1) and estimated from the frequencies of case slot examples in case frames.

while A (s j ) = 1 means s j is not filled with an overt argument but filled with an antecedent of zero pronoun, case frames are constructed from overt predicate argument pairs.


For noun with copula, the coverage was only 54.5%.

most predicate argument relations concerning nouns with copula were easily recognized from syntactic preference, and thus the low coverage would not quite affect the performance of discourse analysis.


We considered that generalized examples can benefit for the accuracy of syntactic analysis, and tried several models that utilize these examples.


In the worst case, the fan-out of p 2 can be as large as ϕ(B r−1 ) + ϕ(B r ).

1: Function NAIVE-BINARIZATION(p) 2: result ← ∅; 3: currentProd ← p; 4: while ρ(currentProd) > 2 do 5: currentProd ← p 1 ; 8: return result ∪ currentProd; We have defined reductions only for the last two occurrences of nonterminals in the right-hand side of a production p. it is easy to see that we can also define the concept for two arbitrary (not necessarily adjacent) occurrences of nonterminals, at the cost of making the notation more complicated.


Note that we have expressed the algorithm as a decision function that will return true if there exists a binarization of p with fan-out not greater than f , and false otherwise.

the algorithm can easily be modified to return a reduction producing such a binarization, by adding to each endpoint set ∆ ∈ workingSet two pointers to the adjacent endpoint sets that were used to obtain it.


This representation has size certainly smaller than 2f × q, where q is the size of the set workingSet.

both membership and insertion operations take now an amount of time O(2f ).


This will result into a more efficiently parsable LCFRS, since rank exponentially affects parsing complexity.

we must take into account that parsing complexity is also influenced by fan-out.


Even in the restricted case of f = ϕ(p), that is, when no increase in the fan-out of the input production is allowed, we do not know whether p can be binarized using only deterministic polynomial time in the value of p's fan-out.

our bounded binarization algorithm shows that the latter problem can be solved in polynomial time when the fan-out of the input LCFRS is bounded by some constant.


Many successful models of syntax are based on Probabilistic Context Free Grammars (PCFGs) (e.g., Collins (1999)).

directly learning a PCFG from a treebank results in poor parsing performance, due largely to the unrealistic independence assumptions imposed by the context-free assumption.


(2003)), a method which uses as productions all subtrees of the training corpus.

many of the DOP estimation methods have serious shortcomings (Johnson, 2002), namely inconsistency for DOP1 (Bod, 2003) and overfitting of the maximum likelihood estimate (Prescher et al., 2004).


This far surpasses the ML-PCFG (F1 of 70.7%), and is similar to Zuidema's (2007) DOP result of 83.8%.

it still well below state-of-the art parsers (e.g., the Berkeley parser trained using the same data representation scores 87.7%).


It is linguistically plausible that such structures are determined at least in part on the basis of the meaning of the related chunks of texts, and of the rhetorical intentions of their authors.

such knowledge is extremely difficult to capture.


al., 2008) focus primarily on news articles.

for us the development of the discourse parser is parasitic on our ultimate goal: developing resources and algorithms for language in-s1e1-s5e2 general:specific dd dd dd dd dd dd dd dd dd dd dd dd dd dd dd dd dd dd d  Z  Z  Z  Z  Z  Z  Z  Z  Z  Z  Z  Z  Z  Z  Z  Z  Z  Z  Z   s1e1 s2e1-s5e2  preparation:   1terfaces to instructional applications.


In many applications with just two classes, this is sufficient.

we are faced with a multi-classification problem.


Informally, our goal is to maximize each object's net happiness, which is computed by subtracting its membership score of the class it is not assigned to from its membership score of the class it is assigned to.

at the same time, we want to avoid assigning similar objects to different classes.


Citation texts have also been used to create summaries of single scientific articles in Qazvinian and Radev (2008) and Mei and Zhai (2008).

there is no previous work that uses the text of the citations to produce a multi-document survey of scientific articles.


This is similar to the concept of prestige in social networks, where the prestige of a person is dependent on the prestige of the people he/she knows.

since random walk may get caught in cycles or in disconnected components, we reserve a low probability to jump to random nodes instead of neighbors (a technique suggested by Langville and Meyer (2006)).


Our area includes the requisite French/English/German/Norwegian group, as well as the somewhat surprising Irish.

in addition to being intuitively plausible, it is not hard to find evidence in the literature for the contact relationship between English and Irish (Sommerfelt, 1960).


Our hierarchical Bayesian domain adaptation model is directed, as illustrated in Figure 1.

somewhat counterintuitively, the underlying (original) model of the data can be either directed or undirected, and for our experiments we use undi-rected, conditional random field-based models.


Probabilistic modeling has emerged as a dominant paradigm for these problems, and the EM algorithm has been a driving force for learning models in a simple and intuitive manner.


Online algorithms have the potential to speed up learning by making updates more frequently.

these updates can be seen as noisy approximations to the full batch update, and this noise can in fact impede learning.


For iEM, the time required to update µ with s i depends only on the number of nonzero components of s i .

the sEM update is µ ← (1−η k )µ+η k s i , and a naïve implementation would take time proportional to the total number of components.


Empirically, online methods are of-ten faster by an order of magnitude (Collins et al., 2008), and it has been argued on theoretical grounds that the fast, approximate nature of online methods is a good fit given that we are interested in test performance, not the training objective (Bottou and Bousquet, 2008;Shalev-Shwartz and Srebro, 2008).

in the unsupervised NLP literature, online methods are rarely seen, 5 and when they are, incremental EM is the dominant variant (Gildea and Hofmann, 1999;Kuo et al., 2008).


They can also be used for a variety of language processing tasks such as text categorization and information retrieval.