title: Meta-Chunking: Learning Efficient Text Segmentation via Logical
    Perception
summary: Retrieval-Augmented Generation (RAG), while serving as a viable complement to
  large language models (LLMs), often overlooks the crucial aspect of text
  chunking within its pipeline, which impacts the quality of knowledge-intensive
  tasks. This paper introduces the concept of Meta-Chunking, which refers to a
  granularity between sentences and paragraphs, consisting of a collection of
  sentences within a paragraph that have deep linguistic logical connections. To
  implement Meta-Chunking, we designed two strategies based on LLMs: Margin
  Sampling Chunking and Perplexity Chunking. The former employs LLMs to perform
  binary classification on whether consecutive sentences need to be segmented,
  making decisions based on the probability difference obtained from margin
  sampling. The latter precisely identifies text chunk boundaries by analyzing
  the characteristics of perplexity distribution. Additionally, considering the
  inherent complexity of different texts, we propose a strategy that combines
  Meta-Chunking with dynamic merging to achieve a balance between fine-grained
  and coarse-grained text chunking. Experiments conducted on eleven datasets
  demonstrate that Meta-Chunking can more efficiently improve the performance of
  single-hop and multi-hop question answering based on RAG. For instance, on the
  2WikiMultihopQA dataset, it outperforms similarity chunking by 1.32 while only
  consuming 45.8% of the time. Our code is available at
  https://github.com/IAAR-Shanghai/Meta-Chunking.
paper_authors._id: 6717157f69532dd147c7579e
paper_authors.name: Jihao Zhao
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-10-22T08:00:47.204Z
paper_authors._id: 658e85bb5b7553ca5c29ba89
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/658e85bb5b7553ca5c29ba89/KK6UpS9agtrxevvBoup5N.jpeg
paper_authors.fullname: Jihao Zhao
paper_authors.type: user
paper_authors.user: Robot2050
paper_authors._id: 6717157f69532dd147c7579f
paper_authors.name: Zhiyuan Ji
paper_authors._id: 6717157f69532dd147c757a0
paper_authors.name: Pengnian Qi
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-10-22T09:15:31.720Z
paper_authors._id: 64be260cf8f28a19b0e02f35
paper_authors.avatarUrl: /avatars/cc55af21b31589f87dbdf9fed65be6bc.svg
paper_authors.fullname: pengnian qi
paper_authors.type: user
paper_authors.user: MoCun
paper_authors._id: 6717157f69532dd147c757a1
paper_authors.name: Simin Niu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-10-22T09:15:37.051Z
paper_authors._id: 66daea8776dbaaa372eabec5
paper_authors.avatarUrl: /avatars/1e5fbe4ff06bb6121c7029253b76b79f.svg
paper_authors.fullname: siminniu
paper_authors.type: user
paper_authors.user: siminniu
paper_authors._id: 6717157f69532dd147c757a2
paper_authors.name: Bo Tang
paper_authors._id: 6717157f69532dd147c757a3
paper_authors.name: Feiyu Xiong
paper_authors._id: 6717157f69532dd147c757a4
paper_authors.name: Zhiyu Li
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-10-22T09:16:33.336Z
paper_authors._id: 62a155e615eeab266b2f2243
paper_authors.avatarUrl: /avatars/e89ef156e73af028e3ce3664e6cb4e62.svg
paper_authors.fullname: Zhiyu Li
paper_authors.type: user
paper_authors.user: jimi888
paper_title: Meta-Chunking: Learning Efficient Text Segmentation via Logical
    Perception
paper_summary: Retrieval-Augmented Generation (RAG), while serving as a viable complement to
  large language models (LLMs), often overlooks the crucial aspect of text
  chunking within its pipeline, which impacts the quality of knowledge-intensive
  tasks. This paper introduces the concept of Meta-Chunking, which refers to a
  granularity between sentences and paragraphs, consisting of a collection of
  sentences within a paragraph that have deep linguistic logical connections. To
  implement Meta-Chunking, we designed two strategies based on LLMs: Margin
  Sampling Chunking and Perplexity Chunking. The former employs LLMs to perform
  binary classification on whether consecutive sentences need to be segmented,
  making decisions based on the probability difference obtained from margin
  sampling. The latter precisely identifies text chunk boundaries by analyzing
  the characteristics of perplexity distribution. Additionally, considering the
  inherent complexity of different texts, we propose a strategy that combines
  Meta-Chunking with dynamic merging to achieve a balance between fine-grained
  and coarse-grained text chunking. Experiments conducted on eleven datasets
  demonstrate that Meta-Chunking can more efficiently improve the performance of
  single-hop and multi-hop question answering based on RAG. For instance, on the
  2WikiMultihopQA dataset, it outperforms similarity chunking by 1.32 while only
  consuming 45.8% of the time. Our code is available at
  https://github.com/IAAR-Shanghai/Meta-Chunking.
paper_ai_keywords: Retrieval-Augmented Generation (RAG) large language models (LLMs) text chunking Meta-Chunking linguistic logical connections Margin Sampling Chunking Perplexity Chunking binary classification margin sampling perplexity distribution dynamic merging fine-grained coarse-grained single-hop question answering multi-hop question answering 2WikiMultihopQA dataset similarity chunking