This notebook showcases several ways to do that.At a high level, text splitters work as following:Split the text up into small, semantically meaningful chunks (often sentences).Start combining these small chunks into a larger chunk until you reach a certain size (as measured by some function).Once you reach that size, make that chunk its own piece of text and then start creating a new chunk of text with some overlap (to keep context between chunks).That means there are two different axes along which you can customize your text splitter:How the text is splitHow the chunk size is measuredGet started with text splitters‚ÄãThe default recommended text splitter is the RecursiveCharacterTextSplitter. This text splitter takes a list of characters. It tries to create chunks based on splitting on the first character, but if any chunks are too large it then moves onto the next character, and so forth. By default the characters it tries to split on are ["\n\n", "\n", " ", ""]In addition to controlling which characters you can split on, you can also control a few other things:length_function: how the length of chunks is calculated. Defaults to just counting number of characters, but it's pretty common to pass a token counter here.chunk_size: the maximum size of your chunks (as measured by the length function).chunk_overlap: the maximum overlap between chunks. It can be nice to have some overlap to maintain some continuity between chunks (e.g. do a sliding window).add_start_index: whether to include the starting position of each chunk within the original document in the metadata.# This is a long document we can split up.with open('../../state_of_the_union.txt') as f:    state_of_the_union = f.read()from langchain.text_splitter import RecursiveCharacterTextSplittertext_splitter = RecursiveCharacterTextSplitter(    # Set a really small chunk size, just to show.    chunk_size = 100,    chunk_overlap  = 20,    length_function = len,    add_start_index = True,)texts =
Head to Integrations for documentation on built-in document transformer integrations with 3rd-party tools.
Head to Integrations for documentation on built-in document transformer integrations with 3rd-party tools. ->: This notebook showcases several ways to do that.At a high level, text splitters work as following:Split the text up into small, semantically meaningful chunks (often sentences).Start combining these small chunks into a larger chunk until you reach a certain size (as measured by some function).Once you reach that size, make that chunk its own piece of text and then start creating a new chunk of text with some overlap (to keep context between chunks).That means there are two different axes along which you can customize your text splitter:How the text is splitHow the chunk size is measuredGet started with text splitters‚ÄãThe default recommended text splitter is the RecursiveCharacterTextSplitter. This text splitter takes a list of characters. It tries to create chunks based on splitting on the first character, but if any chunks are too large it then moves onto the next character, and so forth. By default the characters it tries to split on are ["\n\n", "\n", " ", ""]In addition to controlling which characters you can split on, you can also control a few other things:length_function: how the length of chunks is calculated. Defaults to just counting number of characters, but it's pretty common to pass a token counter here.chunk_size: the maximum size of your chunks (as measured by the length function).chunk_overlap: the maximum overlap between chunks. It can be nice to have some overlap to maintain some continuity between chunks (e.g. do a sliding window).add_start_index: whether to include the starting position of each chunk within the original document in the metadata.# This is a long document we can split up.with open('../../state_of_the_union.txt') as f:    state_of_the_union = f.read()from langchain.text_splitter import RecursiveCharacterTextSplittertext_splitter = RecursiveCharacterTextSplitter(    # Set a really small chunk size, just to show.    chunk_size = 100,    chunk_overlap  = 20,    length_function = len,    add_start_index = True,)texts =