langchain.storage import InMemoryStorefrom langchain.document_loaders import TextLoaderloaders = [    TextLoader('../../paul_graham_essay.txt'),    TextLoader('../../state_of_the_union.txt'),]docs = []for l in loaders:    docs.extend(l.load())text_splitter = RecursiveCharacterTextSplitter(chunk_size=10000)docs = text_splitter.split_documents(docs)Smaller chunks‚ÄãOften times it can be useful to retrieve larger chunks of information, but embed smaller chunks. This allows for embeddings to capture the semantic meaning as closely as possible, but for as much context as possible to be passed downstream. Note that this is what the ParentDocumentRetriever does. Here we show what is going on under the hood.# The vectorstore to use to index the child chunksvectorstore = Chroma(    collection_name="full_documents",    embedding_function=OpenAIEmbeddings())# The storage layer for the parent documentsstore = InMemoryStore()id_key = "doc_id"# The retriever (empty to start)retriever = MultiVectorRetriever(    vectorstore=vectorstore,     docstore=store,     id_key=id_key,)import uuiddoc_ids = [str(uuid.uuid4()) for _ in docs]# The splitter to use to create smaller chunkschild_text_splitter = RecursiveCharacterTextSplitter(chunk_size=400)sub_docs = []for i, doc in enumerate(docs):    _id = doc_ids[i]    _sub_docs = child_text_splitter.split_documents([doc])    for _doc in _sub_docs:        _doc.metadata[id_key] = _id    sub_docs.extend(_sub_docs)retriever.vectorstore.add_documents(sub_docs)retriever.docstore.mset(list(zip(doc_ids, docs)))# Vectorstore alone retrieves the small chunksretriever.vectorstore.similarity_search("justice breyer")[0]    Document(page_content='Tonight, I‚Äôd like to honor someone who has dedicated his life to serve this country: Justice Stephen Breyer‚Äîan Army veteran, Constitutional scholar, and retiring Justice of the United States Supreme Court. Justice Breyer, thank you for your service. \n\nOne of the most serious constitutional responsibilities a
It can often be beneficial to store multiple vectors per document. There are multiple use cases where this is beneficial. LangChain has a base MultiVectorRetriever which makes querying this type of setup easy. A lot of the complexity lies in how to create the multiple vectors per document. This notebook covers some of the common ways to create those vectors and use the MultiVectorRetriever.
It can often be beneficial to store multiple vectors per document. There are multiple use cases where this is beneficial. LangChain has a base MultiVectorRetriever which makes querying this type of setup easy. A lot of the complexity lies in how to create the multiple vectors per document. This notebook covers some of the common ways to create those vectors and use the MultiVectorRetriever. ->: langchain.storage import InMemoryStorefrom langchain.document_loaders import TextLoaderloaders = [    TextLoader('../../paul_graham_essay.txt'),    TextLoader('../../state_of_the_union.txt'),]docs = []for l in loaders:    docs.extend(l.load())text_splitter = RecursiveCharacterTextSplitter(chunk_size=10000)docs = text_splitter.split_documents(docs)Smaller chunks‚ÄãOften times it can be useful to retrieve larger chunks of information, but embed smaller chunks. This allows for embeddings to capture the semantic meaning as closely as possible, but for as much context as possible to be passed downstream. Note that this is what the ParentDocumentRetriever does. Here we show what is going on under the hood.# The vectorstore to use to index the child chunksvectorstore = Chroma(    collection_name="full_documents",    embedding_function=OpenAIEmbeddings())# The storage layer for the parent documentsstore = InMemoryStore()id_key = "doc_id"# The retriever (empty to start)retriever = MultiVectorRetriever(    vectorstore=vectorstore,     docstore=store,     id_key=id_key,)import uuiddoc_ids = [str(uuid.uuid4()) for _ in docs]# The splitter to use to create smaller chunkschild_text_splitter = RecursiveCharacterTextSplitter(chunk_size=400)sub_docs = []for i, doc in enumerate(docs):    _id = doc_ids[i]    _sub_docs = child_text_splitter.split_documents([doc])    for _doc in _sub_docs:        _doc.metadata[id_key] = _id    sub_docs.extend(_sub_docs)retriever.vectorstore.add_documents(sub_docs)retriever.docstore.mset(list(zip(doc_ids, docs)))# Vectorstore alone retrieves the small chunksretriever.vectorstore.similarity_search("justice breyer")[0]    Document(page_content='Tonight, I‚Äôd like to honor someone who has dedicated his life to serve this country: Justice Stephen Breyer‚Äîan Army veteran, Constitutional scholar, and retiring Justice of the United States Supreme Court. Justice Breyer, thank you for your service. \n\nOne of the most serious constitutional responsibilities a