This process produces a hierarchical clustering of the word types in the corpus , and these clusterings have been found useful in many applications ( Ratinov and Roth , 2009 ; Koo et al. , 2008 ; Miller et al. , 2004 ) .
design challenges and misconceptions in named entity recognition
Lev Ratinov Dan Roth
The Brown clustering algorithm works by starting with an initial assignment of word types to classes ( which is usually either one unique class per type or a small number of seed classes corresponding to the most frequent types in the corpus ) , and then iteratively selecting the pair of classes to merge that would lead to the highest post-merge log-likelihood , doing so until all classes have been merged . This process produces a hierarchical clustering of the word types in the corpus , and these clusterings have been found useful in many applications ( Ratinov and Roth , 2009 ; Koo et al. , 2008 ; Miller et al. , 2004 ) . There are other similar models of distributional clustering of English words which can be similarly effective ( Pereira et al. , 1993 ) .
This process produces a hierarchical clustering of the word types in the corpus , and these clusterings have been found useful in many applications ( @@CITATION ; Koo et al. , 2008 ; Miller et al. , 2004 ) .