Corpus structure, language models, and ad hoc information retrieval
<|reference_start|>Corpus structure, language models, and ad hoc information retrieval: Most previous work on the recently developed language-modeling approach to information retrieval focuses on document-specific characteristics, and therefore does not take into account the structure of the surrounding corpus. We propose a novel algorithmic framework in which information provided by document-based language models is enhanced by the incorporation of information drawn from clusters of similar documents. Using this framework, we develop a suite of new algorithms. Even the simplest typically outperforms the standard language-modeling approach in precision and recall, and our new interpolation algorithm posts statistically significant improvements for both metrics over all three corpora tested.<|reference_end|>
@article{kurland2004corpus,
    title={Corpus structure, language models, and ad hoc information retrieval},
    author={Oren Kurland and Lillian Lee},
    journal={arXiv preprint arXiv:cs/0405044},
    year={2004},
    archivePrefix={arXiv},
    eprint={cs/0405044},
    primaryClass={cs.IR cs.CL}
}