chunk.from langchain.retrievers import ParentDocumentRetrieverfrom langchain.vectorstores import Chromafrom langchain.embeddings import OpenAIEmbeddingsfrom langchain.text_splitter import RecursiveCharacterTextSplitterfrom langchain.storage import InMemoryStorefrom langchain.document_loaders import TextLoaderloaders = [    TextLoader('../../paul_graham_essay.txt'),    TextLoader('../../state_of_the_union.txt'),]docs = []for l in loaders:    docs.extend(l.load())Retrieving full documents‚ÄãIn this mode, we want to retrieve the full documents. Therefore, we only specify a child splitter.# This text splitter is used to create the child documentschild_splitter = RecursiveCharacterTextSplitter(chunk_size=400)# The vectorstore to use to index the child chunksvectorstore = Chroma(    collection_name="full_documents",    embedding_function=OpenAIEmbeddings())# The storage layer for the parent documentsstore = InMemoryStore()retriever = ParentDocumentRetriever(    vectorstore=vectorstore,     docstore=store,     child_splitter=child_splitter,)retriever.add_documents(docs, ids=None)This should yield two keys, because we added two documents.list(store.yield_keys())    ['05fe8d8a-bf60-4f87-b576-4351b23df266',     '571cc9e5-9ef7-4f6c-b800-835c83a1858b']Let's now call the vector store search functionality - we should see that it returns small chunks (since we're storing the small chunks).sub_docs = vectorstore.similarity_search("justice breyer")print(sub_docs[0].page_content)    Tonight, I‚Äôd like to honor someone who has dedicated his life to serve this country: Justice Stephen Breyer‚Äîan Army veteran, Constitutional scholar, and retiring Justice of the United States Supreme Court. Justice Breyer, thank you for your service.         One of the most serious constitutional responsibilities a President has is nominating someone to serve on the United States Supreme Court.Let's now retrieve from the overall retriever. This should return large documents - since it returns the
When splitting documents for retrieval, there are often conflicting desires:
When splitting documents for retrieval, there are often conflicting desires: ->: chunk.from langchain.retrievers import ParentDocumentRetrieverfrom langchain.vectorstores import Chromafrom langchain.embeddings import OpenAIEmbeddingsfrom langchain.text_splitter import RecursiveCharacterTextSplitterfrom langchain.storage import InMemoryStorefrom langchain.document_loaders import TextLoaderloaders = [    TextLoader('../../paul_graham_essay.txt'),    TextLoader('../../state_of_the_union.txt'),]docs = []for l in loaders:    docs.extend(l.load())Retrieving full documents‚ÄãIn this mode, we want to retrieve the full documents. Therefore, we only specify a child splitter.# This text splitter is used to create the child documentschild_splitter = RecursiveCharacterTextSplitter(chunk_size=400)# The vectorstore to use to index the child chunksvectorstore = Chroma(    collection_name="full_documents",    embedding_function=OpenAIEmbeddings())# The storage layer for the parent documentsstore = InMemoryStore()retriever = ParentDocumentRetriever(    vectorstore=vectorstore,     docstore=store,     child_splitter=child_splitter,)retriever.add_documents(docs, ids=None)This should yield two keys, because we added two documents.list(store.yield_keys())    ['05fe8d8a-bf60-4f87-b576-4351b23df266',     '571cc9e5-9ef7-4f6c-b800-835c83a1858b']Let's now call the vector store search functionality - we should see that it returns small chunks (since we're storing the small chunks).sub_docs = vectorstore.similarity_search("justice breyer")print(sub_docs[0].page_content)    Tonight, I‚Äôd like to honor someone who has dedicated his life to serve this country: Justice Stephen Breyer‚Äîan Army veteran, Constitutional scholar, and retiring Justice of the United States Supreme Court. Justice Breyer, thank you for your service.         One of the most serious constitutional responsibilities a President has is nominating someone to serve on the United States Supreme Court.Let's now retrieve from the overall retriever. This should return large documents - since it returns the