{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "70e9b619",
   "metadata": {},
   "source": [
    "# How to split Markdown by Headers\n",
    "\n",
    "### Motivation\n",
    "\n",
    "Many chat or Q+A applications involve chunking input documents prior to embedding and vector storage.\n",
    "\n",
    "[These notes](https://www.pinecone.io/learn/chunking-strategies/) from Pinecone provide some useful tips:\n",
    "\n",
    "```\n",
    "When a full paragraph or document is embedded, the embedding process considers both the overall context and the relationships between the sentences and phrases within the text. This can result in a more comprehensive vector representation that captures the broader meaning and themes of the text.\n",
    "```\n",
    " \n",
    "As mentioned, chunking often aims to keep text with common context together. With this in mind, we might want to specifically honor the structure of the document itself. For example, a markdown file is organized by headers. Creating chunks within specific header groups is an intuitive idea. To address this challenge, we can use [MarkdownHeaderTextSplitter](https://api.python.langchain.com/en/latest/markdown/langchain_text_splitters.markdown.MarkdownHeaderTextSplitter.html). This will split a markdown file by a specified set of headers. \n",
    "\n",
    "For example, if we want to split this markdown:\n",
    "```\n",
    "md = '# Foo\\n\\n ## Bar\\n\\nHi this is Jim  \\nHi this is Joe\\n\\n ## Baz\\n\\n Hi this is Molly' \n",
    "```\n",
    " \n",
    "We can specify the headers to split on:\n",
    "```\n",
    "[(\"#\", \"Header 1\"),(\"##\", \"Header 2\")]\n",
    "```\n",
    "\n",
    "And content is grouped or split by common headers:\n",
    "```\n",
    "{'content': 'Hi this is Jim  \\nHi this is Joe', 'metadata': {'Header 1': 'Foo', 'Header 2': 'Bar'}}\n",
    "{'content': 'Hi this is Molly', 'metadata': {'Header 1': 'Foo', 'Header 2': 'Baz'}}\n",
    "```\n",
    "\n",
    "Let's have a look at some examples below.\n",
    "\n",
    "### Basic usage:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "0cd11819-4d4e-4fc1-aa85-faf69d24db89",
   "metadata": {},
   "outputs": [],
   "source": [
    "%pip install -qU langchain-text-splitters"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "ceb3c1fb",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2023-09-25T19:12:27.243781300Z",
     "start_time": "2023-09-25T19:12:24.943559400Z"
    }
   },
   "outputs": [],
   "source": [
    "from langchain_text_splitters import MarkdownHeaderTextSplitter"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "2ae3649b",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2023-09-25T19:12:31.917013600Z",
     "start_time": "2023-09-25T19:12:31.905694500Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[Document(page_content='Hi this is Jim  \\nHi this is Joe', metadata={'Header 1': 'Foo', 'Header 2': 'Bar'}),\n",
       " Document(page_content='Hi this is Lance', metadata={'Header 1': 'Foo', 'Header 2': 'Bar', 'Header 3': 'Boo'}),\n",
       " Document(page_content='Hi this is Molly', metadata={'Header 1': 'Foo', 'Header 2': 'Baz'})]"
      ]
     },
     "execution_count": 2,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "markdown_document = \"# Foo\\n\\n    ## Bar\\n\\nHi this is Jim\\n\\nHi this is Joe\\n\\n ### Boo \\n\\n Hi this is Lance \\n\\n ## Baz\\n\\n Hi this is Molly\"\n",
    "\n",
    "headers_to_split_on = [\n",
    "    (\"#\", \"Header 1\"),\n",
    "    (\"##\", \"Header 2\"),\n",
    "    (\"###\", \"Header 3\"),\n",
    "]\n",
    "\n",
    "markdown_splitter = MarkdownHeaderTextSplitter(headers_to_split_on)\n",
    "md_header_splits = markdown_splitter.split_text(markdown_document)\n",
    "md_header_splits"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "aac1738c",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2023-09-25T19:12:35.672077100Z",
     "start_time": "2023-09-25T19:12:35.666731400Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "langchain_core.documents.base.Document"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "type(md_header_splits[0])"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "102aad57-7bef-42d3-ab4e-b50d6dc11718",
   "metadata": {},
   "source": [
    "By default, `MarkdownHeaderTextSplitter` strips headers being split on from the output chunk's content. This can be disabled by setting `strip_headers = False`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "9fce45ba-a4be-4a69-ad27-f5ff195c4fd7",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[Document(page_content='# Foo  \\n## Bar  \\nHi this is Jim  \\nHi this is Joe', metadata={'Header 1': 'Foo', 'Header 2': 'Bar'}),\n",
       " Document(page_content='### Boo  \\nHi this is Lance', metadata={'Header 1': 'Foo', 'Header 2': 'Bar', 'Header 3': 'Boo'}),\n",
       " Document(page_content='## Baz  \\nHi this is Molly', metadata={'Header 1': 'Foo', 'Header 2': 'Baz'})]"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "markdown_splitter = MarkdownHeaderTextSplitter(headers_to_split_on, strip_headers=False)\n",
    "md_header_splits = markdown_splitter.split_text(markdown_document)\n",
    "md_header_splits"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "aa67e0cc-d721-4536-9c7a-9fa3a7a69cbe",
   "metadata": {},
   "source": [
    "### How to return Markdown lines as separate documents\n",
    "\n",
    "By default, `MarkdownHeaderTextSplitter` aggregates lines based on the headers specified in `headers_to_split_on`. We can disable this by specifying `return_each_line`:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "940bb609-c9c3-4593-ac2d-d825c80ceb44",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[Document(page_content='Hi this is Jim', metadata={'Header 1': 'Foo', 'Header 2': 'Bar'}),\n",
       " Document(page_content='Hi this is Joe', metadata={'Header 1': 'Foo', 'Header 2': 'Bar'}),\n",
       " Document(page_content='Hi this is Lance', metadata={'Header 1': 'Foo', 'Header 2': 'Bar', 'Header 3': 'Boo'}),\n",
       " Document(page_content='Hi this is Molly', metadata={'Header 1': 'Foo', 'Header 2': 'Baz'})]"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "markdown_splitter = MarkdownHeaderTextSplitter(\n",
    "    headers_to_split_on,\n",
    "    return_each_line=True,\n",
    ")\n",
    "md_header_splits = markdown_splitter.split_text(markdown_document)\n",
    "md_header_splits"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9bd8977a",
   "metadata": {},
   "source": [
    "Note that here header information is retained in the `metadata` for each document.\n",
    "\n",
    "### How to constrain chunk size:\n",
    "\n",
    "Within each markdown group we can then apply any text splitter we want, such as `RecursiveCharacterTextSplitter`, which allows for further control of the chunk size."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "6f1f62bf-2653-4361-9bb0-964d86cb14db",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[Document(page_content='# Intro  \\n## History  \\nMarkdown[9] is a lightweight markup language for creating formatted text using a plain-text editor. John Gruber created Markdown in 2004 as a markup language that is appealing to human readers in its source code form.[9]', metadata={'Header 1': 'Intro', 'Header 2': 'History'}),\n",
       " Document(page_content='Markdown is widely used in blogging, instant messaging, online forums, collaborative software, documentation pages, and readme files.', metadata={'Header 1': 'Intro', 'Header 2': 'History'}),\n",
       " Document(page_content='## Rise and divergence  \\nAs Markdown popularity grew rapidly, many Markdown implementations appeared, driven mostly by the need for  \\nadditional features such as tables, footnotes, definition lists,[note 1] and Markdown inside HTML blocks.', metadata={'Header 1': 'Intro', 'Header 2': 'Rise and divergence'}),\n",
       " Document(page_content='#### Standardization  \\nFrom 2012, a group of people, including Jeff Atwood and John MacFarlane, launched what Atwood characterised as a standardisation effort.', metadata={'Header 1': 'Intro', 'Header 2': 'Rise and divergence'}),\n",
       " Document(page_content='## Implementations  \\nImplementations of Markdown are available for over a dozen programming languages.', metadata={'Header 1': 'Intro', 'Header 2': 'Implementations'})]"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "markdown_document = \"# Intro \\n\\n    ## History \\n\\n Markdown[9] is a lightweight markup language for creating formatted text using a plain-text editor. John Gruber created Markdown in 2004 as a markup language that is appealing to human readers in its source code form.[9] \\n\\n Markdown is widely used in blogging, instant messaging, online forums, collaborative software, documentation pages, and readme files. \\n\\n ## Rise and divergence \\n\\n As Markdown popularity grew rapidly, many Markdown implementations appeared, driven mostly by the need for \\n\\n additional features such as tables, footnotes, definition lists,[note 1] and Markdown inside HTML blocks. \\n\\n #### Standardization \\n\\n From 2012, a group of people, including Jeff Atwood and John MacFarlane, launched what Atwood characterised as a standardisation effort. \\n\\n ## Implementations \\n\\n Implementations of Markdown are available for over a dozen programming languages.\"\n",
    "\n",
    "headers_to_split_on = [\n",
    "    (\"#\", \"Header 1\"),\n",
    "    (\"##\", \"Header 2\"),\n",
    "]\n",
    "\n",
    "# MD splits\n",
    "markdown_splitter = MarkdownHeaderTextSplitter(\n",
    "    headers_to_split_on=headers_to_split_on, strip_headers=False\n",
    ")\n",
    "md_header_splits = markdown_splitter.split_text(markdown_document)\n",
    "\n",
    "# Char-level splits\n",
    "from langchain_text_splitters import RecursiveCharacterTextSplitter\n",
    "\n",
    "chunk_size = 250\n",
    "chunk_overlap = 30\n",
    "text_splitter = RecursiveCharacterTextSplitter(\n",
    "    chunk_size=chunk_size, chunk_overlap=chunk_overlap\n",
    ")\n",
    "\n",
    "# Split\n",
    "splits = text_splitter.split_documents(md_header_splits)\n",
    "splits"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.4"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}
