Skip to main contentü¶úÔ∏èüîó LangChainDocsUse casesIntegrationsAPICommunityChat our docsLangSmithJS/TS DocsSearchCTRLKQA over structured dataSQLRetrieval-augmented generation (RAG)RAG over codeUsing a RetrieverRemembering chat historyAnalyze a single long documentAgent with retrieval toolText splitting by headerRAG over in-memory documentsRAG using local modelsDynamically select from multiple retrieversRetrieving from multiple sourcesCiting retrieval sourcesRetrieve from vector stores directlyInteracting with APIsChatbotsExtractionSummarizationTaggingWeb scrapingSynthetic data generationGraph queryingRetrieval-augmented generation (RAG)Text splitting by headerText splitting by headerText splitting for vector storage often uses sentences or other delimiters to keep related text together. But many documents (such as Markdown files) have structure (headers) that can be explicitly used in splitting. The MarkdownHeaderTextSplitter lets a user split Markdown files files based on specified headers. This results in chunks that retain the header(s) that it came from in the metadata.This works nicely w/ SelfQueryRetriever.First, tell the retriever about our splits.Then, query based on the doc structure (e.g., "summarize the doc introduction"). Chunks only from that section of the Document will be filtered and used in chat / Q+A.Let's test this out on an example Notion page!First, I download the page to Markdown as explained here.# Load Notion page as a markdownfile filefrom langchain.document_loaders import NotionDirectoryLoaderpath = "../Notion_DB/"loader = NotionDirectoryLoader(path)docs = loader.load()md_file = docs[0].page_content# Let's create groups based on the section headers in our pagefrom langchain.text_splitter import MarkdownHeaderTextSplitterheaders_to_split_on = [    ("###", "Section"),]markdown_splitter = MarkdownHeaderTextSplitter(headers_to_split_on=headers_to_split_on)md_header_splits = markdown_splitter.split_text(md_file)Now, perform text splitting on
Text splitting for vector storage often uses sentences or other delimiters to keep related text together.
Text splitting for vector storage often uses sentences or other delimiters to keep related text together. ->: Skip to main contentü¶úÔ∏èüîó LangChainDocsUse casesIntegrationsAPICommunityChat our docsLangSmithJS/TS DocsSearchCTRLKQA over structured dataSQLRetrieval-augmented generation (RAG)RAG over codeUsing a RetrieverRemembering chat historyAnalyze a single long documentAgent with retrieval toolText splitting by headerRAG over in-memory documentsRAG using local modelsDynamically select from multiple retrieversRetrieving from multiple sourcesCiting retrieval sourcesRetrieve from vector stores directlyInteracting with APIsChatbotsExtractionSummarizationTaggingWeb scrapingSynthetic data generationGraph queryingRetrieval-augmented generation (RAG)Text splitting by headerText splitting by headerText splitting for vector storage often uses sentences or other delimiters to keep related text together. But many documents (such as Markdown files) have structure (headers) that can be explicitly used in splitting. The MarkdownHeaderTextSplitter lets a user split Markdown files files based on specified headers. This results in chunks that retain the header(s) that it came from in the metadata.This works nicely w/ SelfQueryRetriever.First, tell the retriever about our splits.Then, query based on the doc structure (e.g., "summarize the doc introduction"). Chunks only from that section of the Document will be filtered and used in chat / Q+A.Let's test this out on an example Notion page!First, I download the page to Markdown as explained here.# Load Notion page as a markdownfile filefrom langchain.document_loaders import NotionDirectoryLoaderpath = "../Notion_DB/"loader = NotionDirectoryLoader(path)docs = loader.load()md_file = docs[0].page_content# Let's create groups based on the section headers in our pagefrom langchain.text_splitter import MarkdownHeaderTextSplitterheaders_to_split_on = [    ("###", "Section"),]markdown_splitter = MarkdownHeaderTextSplitter(headers_to_split_on=headers_to_split_on)md_header_splits = markdown_splitter.split_text(md_file)Now, perform text splitting on