With a duty to one another to the American people to the Constitution.Note that if we use CharacterTextSplitter.from_tiktoken_encoder, text is only split by CharacterTextSplitter and tiktoken tokenizer is used to merge splits. It means that split can be larger than chunk size measured by tiktoken tokenizer. We can use RecursiveCharacterTextSplitter.from_tiktoken_encoder to make sure splits are not larger than chunk size of tokens allowed by the language model, where each split will be recursively split if it has a larger size.We can also load a tiktoken splitter directly, which ensure each split is smaller than chunk size.from langchain.text_splitter import TokenTextSplittertext_splitter = TokenTextSplitter(chunk_size=10, chunk_overlap=0)texts = text_splitter.split_text(state_of_the_union)print(texts[0])spaCy‚ÄãspaCy is an open-source software library for advanced natural language processing, written in the programming languages Python and Cython.Another alternative to NLTK is to use spaCy tokenizer.How the text is split: by spaCy tokenizer.How the chunk size is measured: by number of characters.#!pip install spacy# This is a long document we can split up.with open("../../../state_of_the_union.txt") as f:    state_of_the_union = f.read()from langchain.text_splitter import SpacyTextSplittertext_splitter = SpacyTextSplitter(chunk_size=1000)texts = text_splitter.split_text(state_of_the_union)print(texts[0])    Madam Speaker, Madam Vice President, our First Lady and Second Gentleman.        Members of Congress and the Cabinet.        Justices of the Supreme Court.        My fellow Americans.                  Last year COVID-19 kept us apart.        This year we are finally together again.                 Tonight, we meet as Democrats Republicans and Independents.        But most importantly as Americans.                 With a duty to one another to the American people to the Constitution.                 And with an unwavering resolve that freedom will always
Language models have a token limit. You should not exceed the token limit. When you split your text into chunks it is therefore a good idea to count the number of tokens. There are many tokenizers. When you count tokens in your text you should use the same tokenizer as used in the language model.
Language models have a token limit. You should not exceed the token limit. When you split your text into chunks it is therefore a good idea to count the number of tokens. There are many tokenizers. When you count tokens in your text you should use the same tokenizer as used in the language model. ->: With a duty to one another to the American people to the Constitution.Note that if we use CharacterTextSplitter.from_tiktoken_encoder, text is only split by CharacterTextSplitter and tiktoken tokenizer is used to merge splits. It means that split can be larger than chunk size measured by tiktoken tokenizer. We can use RecursiveCharacterTextSplitter.from_tiktoken_encoder to make sure splits are not larger than chunk size of tokens allowed by the language model, where each split will be recursively split if it has a larger size.We can also load a tiktoken splitter directly, which ensure each split is smaller than chunk size.from langchain.text_splitter import TokenTextSplittertext_splitter = TokenTextSplitter(chunk_size=10, chunk_overlap=0)texts = text_splitter.split_text(state_of_the_union)print(texts[0])spaCy‚ÄãspaCy is an open-source software library for advanced natural language processing, written in the programming languages Python and Cython.Another alternative to NLTK is to use spaCy tokenizer.How the text is split: by spaCy tokenizer.How the chunk size is measured: by number of characters.#!pip install spacy# This is a long document we can split up.with open("../../../state_of_the_union.txt") as f:    state_of_the_union = f.read()from langchain.text_splitter import SpacyTextSplittertext_splitter = SpacyTextSplitter(chunk_size=1000)texts = text_splitter.split_text(state_of_the_union)print(texts[0])    Madam Speaker, Madam Vice President, our First Lady and Second Gentleman.        Members of Congress and the Cabinet.        Justices of the Supreme Court.        My fellow Americans.                  Last year COVID-19 kept us apart.        This year we are finally together again.                 Tonight, we meet as Democrats Republicans and Independents.        But most importantly as Americans.                 With a duty to one another to the American people to the Constitution.                 And with an unwavering resolve that freedom will always