return large documents - since it returns the documents where the smaller chunks are located.retrieved_docs = retriever.get_relevant_documents("justice breyer")len(retrieved_docs[0].page_content)    38539Retrieving larger chunks‚ÄãSometimes, the full documents can be too big to want to retrieve them as is. In that case, what we really want to do is to first split the raw documents into larger chunks, and then split it into smaller chunks. We then index the smaller chunks, but on retrieval we retrieve the larger chunks (but still not the full documents).# This text splitter is used to create the parent documentsparent_splitter = RecursiveCharacterTextSplitter(chunk_size=2000)# This text splitter is used to create the child documents# It should create documents smaller than the parentchild_splitter = RecursiveCharacterTextSplitter(chunk_size=400)# The vectorstore to use to index the child chunksvectorstore = Chroma(collection_name="split_parents", embedding_function=OpenAIEmbeddings())# The storage layer for the parent documentsstore = InMemoryStore()retriever = ParentDocumentRetriever(    vectorstore=vectorstore,     docstore=store,     child_splitter=child_splitter,    parent_splitter=parent_splitter,)retriever.add_documents(docs)We can see that there are much more than two documents now - these are the larger chunks.len(list(store.yield_keys()))    66Let's make sure the underlying vector store still retrieves the small chunks.sub_docs = vectorstore.similarity_search("justice breyer")print(sub_docs[0].page_content)    Tonight, I‚Äôd like to honor someone who has dedicated his life to serve this country: Justice Stephen Breyer‚Äîan Army veteran, Constitutional scholar, and retiring Justice of the United States Supreme Court. Justice Breyer, thank you for your service.         One of the most serious constitutional responsibilities a President has is nominating someone to serve on the United States Supreme Court.retrieved_docs =
When splitting documents for retrieval, there are often conflicting desires:
When splitting documents for retrieval, there are often conflicting desires: ->: return large documents - since it returns the documents where the smaller chunks are located.retrieved_docs = retriever.get_relevant_documents("justice breyer")len(retrieved_docs[0].page_content)    38539Retrieving larger chunks‚ÄãSometimes, the full documents can be too big to want to retrieve them as is. In that case, what we really want to do is to first split the raw documents into larger chunks, and then split it into smaller chunks. We then index the smaller chunks, but on retrieval we retrieve the larger chunks (but still not the full documents).# This text splitter is used to create the parent documentsparent_splitter = RecursiveCharacterTextSplitter(chunk_size=2000)# This text splitter is used to create the child documents# It should create documents smaller than the parentchild_splitter = RecursiveCharacterTextSplitter(chunk_size=400)# The vectorstore to use to index the child chunksvectorstore = Chroma(collection_name="split_parents", embedding_function=OpenAIEmbeddings())# The storage layer for the parent documentsstore = InMemoryStore()retriever = ParentDocumentRetriever(    vectorstore=vectorstore,     docstore=store,     child_splitter=child_splitter,    parent_splitter=parent_splitter,)retriever.add_documents(docs)We can see that there are much more than two documents now - these are the larger chunks.len(list(store.yield_keys()))    66Let's make sure the underlying vector store still retrieves the small chunks.sub_docs = vectorstore.similarity_search("justice breyer")print(sub_docs[0].page_content)    Tonight, I‚Äôd like to honor someone who has dedicated his life to serve this country: Justice Stephen Breyer‚Äîan Army veteran, Constitutional scholar, and retiring Justice of the United States Supreme Court. Justice Breyer, thank you for your service.         One of the most serious constitutional responsibilities a President has is nominating someone to serve on the United States Supreme Court.retrieved_docs =