Skip to main contentü¶úÔ∏èüîó LangChainDocsUse casesIntegrationsAPICommunityChat our docsLangSmithJS/TS DocsSearchCTRLKGet startedIntroductionInstallationQuickstartLangChain Expression LanguageInterfaceHow toCookbookLangChain Expression Language (LCEL)Why use LCEL?ModulesModel I/‚ÄãORetrievalDocument loadersDocument transformersText splittersHTMLHeaderTextSplitterSplit by characterSplit codeMarkdownHeaderTextSplitterRecursively split by characterSplit by tokensPost retrievalText embedding modelsVector storesRetrieversIndexingChainsMemoryAgentsCallbacksModulesSecurityGuidesMoreModulesRetrievalDocument transformersText splittersMarkdownHeaderTextSplitterOn this pageMarkdownHeaderTextSplitterMotivation‚ÄãMany chat or Q+A applications involve chunking input documents prior to embedding and vector storage.These notes from Pinecone provide some useful tips:When a full paragraph or document is embedded, the embedding process considers both the overall context and the relationships between the sentences and phrases within the text. This can result in a more comprehensive vector representation that captures the broader meaning and themes of the text.As mentioned, chunking often aims to keep text with common context together. With this in mind, we might want to specifically honor the structure of the document itself. For example, a markdown file is organized by headers. Creating chunks within specific header groups is an intuitive idea. To address this challenge, we can use MarkdownHeaderTextSplitter. This will split a markdown file by a specified set of headers. For example, if we want to split this markdown:md = '# Foo\n\n ## Bar\n\nHi this is Jim  \nHi this is Joe\n\n ## Baz\n\n Hi this is Molly' We can specify the headers to split on:[("#", "Header 1"),("##", "Header 2")]And content is grouped or split by common headers:{'content': 'Hi this is Jim  \nHi this is Joe', 'metadata': {'Header 1': 'Foo', 'Header 2': 'Bar'}}{'content': 'Hi this is Molly', 'metadata': {'Header
Motivation
Motivation ->: Skip to main contentü¶úÔ∏èüîó LangChainDocsUse casesIntegrationsAPICommunityChat our docsLangSmithJS/TS DocsSearchCTRLKGet startedIntroductionInstallationQuickstartLangChain Expression LanguageInterfaceHow toCookbookLangChain Expression Language (LCEL)Why use LCEL?ModulesModel I/‚ÄãORetrievalDocument loadersDocument transformersText splittersHTMLHeaderTextSplitterSplit by characterSplit codeMarkdownHeaderTextSplitterRecursively split by characterSplit by tokensPost retrievalText embedding modelsVector storesRetrieversIndexingChainsMemoryAgentsCallbacksModulesSecurityGuidesMoreModulesRetrievalDocument transformersText splittersMarkdownHeaderTextSplitterOn this pageMarkdownHeaderTextSplitterMotivation‚ÄãMany chat or Q+A applications involve chunking input documents prior to embedding and vector storage.These notes from Pinecone provide some useful tips:When a full paragraph or document is embedded, the embedding process considers both the overall context and the relationships between the sentences and phrases within the text. This can result in a more comprehensive vector representation that captures the broader meaning and themes of the text.As mentioned, chunking often aims to keep text with common context together. With this in mind, we might want to specifically honor the structure of the document itself. For example, a markdown file is organized by headers. Creating chunks within specific header groups is an intuitive idea. To address this challenge, we can use MarkdownHeaderTextSplitter. This will split a markdown file by a specified set of headers. For example, if we want to split this markdown:md = '# Foo\n\n ## Bar\n\nHi this is Jim  \nHi this is Joe\n\n ## Baz\n\n Hi this is Molly' We can specify the headers to split on:[("#", "Header 1"),("##", "Header 2")]And content is grouped or split by common headers:{'content': 'Hi this is Jim  \nHi this is Joe', 'metadata': {'Header 1': 'Foo', 'Header 2': 'Bar'}}{'content': 'Hi this is Molly', 'metadata': {'Header