LumberChunker: Long-Form Narrative Document Segmentation
<|reference_start|>LumberChunker: Long-Form Narrative Document Segmentation: Modern NLP tasks increasingly rely on dense retrieval methods to access up-to-date and relevant contextual information. We are motivated by the premise that retrieval benefits from segments that can vary in size such that a content's semantic independence is better captured. We propose LumberChunker, a method leveraging an LLM to dynamically segment documents, which iteratively prompts the LLM to identify the point within a group of sequential passages where the content begins to shift. To evaluate our method, we introduce GutenQA, a benchmark with 3000 "needle in a haystack" type of question-answer pairs derived from 100 public domain narrative books available on Project Gutenberg. Our experiments show that LumberChunker not only outperforms the most competitive baseline by 7.37% in retrieval performance (DCG@20) but also that, when integrated into a RAG pipeline, LumberChunker proves to be more effective than other chunking methods and competitive baselines, such as the Gemini 1.5M Pro. Our Code and Data are available at https://github.com/joaodsmarques/LumberChunker<|reference_end|>
@article{duarte2024lumberchunker:,
    title={LumberChunker: Long-Form Narrative Document Segmentation},
    author={Andr'e V. Duarte, Jo~ao Marques, Miguel Grac{c}a, Miguel Freire,
  Lei Li, Arlindo L. Oliveira},
    journal={arXiv preprint arXiv:2406.17526},
    year={2024},
    archivePrefix={arXiv},
    eprint={2406.17526},
    primaryClass={cs.CL cs.IR}
}