Variable Word Rate N-grams
<|reference_start|>Variable Word Rate N-grams: The rate of occurrence of words is not uniform but varies from document to document. Despite this observation, parameters for conventional n-gram language models are usually derived using the assumption of a constant word rate. In this paper we investigate the use of variable word rate assumption, modelled by a Poisson distribution or a continuous mixture of Poissons. We present an approach to estimating the relative frequencies of words or n-grams taking prior information of their occurrences into account. Discounting and smoothing schemes are also considered. Using the Broadcast News task, the approach demonstrates a reduction of perplexity up to 10%.<|reference_end|>
@article{gotoh2000variable,
    title={Variable Word Rate N-grams},
    author={Yoshihiko Gotoh and Steve Renals},
    journal={arXiv preprint arXiv:cs/0003081},
    year={2000},
    archivePrefix={arXiv},
    eprint={cs/0003081},
    primaryClass={cs.CL}
}