title: MoC: Mixtures of Text Chunking Learners for Retrieval-Augmented
    Generation System
summary: Retrieval-Augmented Generation (RAG), while serving as a viable complement to
  large language models (LLMs), often overlooks the crucial aspect of text
  chunking within its pipeline. This paper initially introduces a dual-metric
  evaluation method, comprising Boundary Clarity and Chunk Stickiness, to enable
  the direct quantification of chunking quality. Leveraging this assessment
  method, we highlight the inherent limitations of traditional and semantic
  chunking in handling complex contextual nuances, thereby substantiating the
  necessity of integrating LLMs into chunking process. To address the inherent
  trade-off between computational efficiency and chunking precision in LLM-based
  approaches, we devise the granularity-aware Mixture-of-Chunkers (MoC)
  framework, which consists of a three-stage processing mechanism. Notably, our
  objective is to guide the chunker towards generating a structured list of
  chunking regular expressions, which are subsequently employed to extract chunks
  from the original text. Extensive experiments demonstrate that both our
  proposed metrics and the MoC framework effectively settle challenges of the
  chunking task, revealing the chunking kernel while enhancing the performance of
  the RAG system.
paper_authors._id: 67d270d88f3def6bcbb87b6b
paper_authors.name: Jihao Zhao
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-03-13T08:23:58.864Z
paper_authors._id: 658e85bb5b7553ca5c29ba89
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/658e85bb5b7553ca5c29ba89/KK6UpS9agtrxevvBoup5N.jpeg
paper_authors.fullname: Jihao Zhao
paper_authors.type: user
paper_authors.user: Robot2050
paper_authors._id: 67d270d88f3def6bcbb87b6c
paper_authors.name: Zhiyuan Ji
paper_authors._id: 67d270d88f3def6bcbb87b6d
paper_authors.name: Zhaoxin Fan
paper_authors._id: 67d270d88f3def6bcbb87b6e
paper_authors.name: Hanyu Wang
paper_authors._id: 67d270d88f3def6bcbb87b6f
paper_authors.name: Simin Niu
paper_authors._id: 67d270d88f3def6bcbb87b70
paper_authors.name: Bo Tang
paper_authors._id: 67d270d88f3def6bcbb87b71
paper_authors.name: Feiyu Xiong
paper_authors._id: 67d270d88f3def6bcbb87b72
paper_authors.name: Zhiyu Li
paper_title: MoC: Mixtures of Text Chunking Learners for Retrieval-Augmented
    Generation System
paper_summary: Retrieval-Augmented Generation (RAG), while serving as a viable complement to
  large language models (LLMs), often overlooks the crucial aspect of text
  chunking within its pipeline. This paper initially introduces a dual-metric
  evaluation method, comprising Boundary Clarity and Chunk Stickiness, to enable
  the direct quantification of chunking quality. Leveraging this assessment
  method, we highlight the inherent limitations of traditional and semantic
  chunking in handling complex contextual nuances, thereby substantiating the
  necessity of integrating LLMs into chunking process. To address the inherent
  trade-off between computational efficiency and chunking precision in LLM-based
  approaches, we devise the granularity-aware Mixture-of-Chunkers (MoC)
  framework, which consists of a three-stage processing mechanism. Notably, our
  objective is to guide the chunker towards generating a structured list of
  chunking regular expressions, which are subsequently employed to extract chunks
  from the original text. Extensive experiments demonstrate that both our
  proposed metrics and the MoC framework effectively settle challenges of the
  chunking task, revealing the chunking kernel while enhancing the performance of
  the RAG system.
paper_ai_keywords: Retrieval-Augmented Generation (RAG) Large language models (LLMs) text chunking dual-metric evaluation method Boundary Clarity Chunk Stickiness chunking quality traditional chunking semantic chunking contextual nuances granularity-aware Mixture-of-Chunkers (MoC) three-stage processing mechanism chunking regular expressions chunk extraction chunking task chunking kernel