* 100}\n".join([f"Document {i+1}:\n\n" + d.page_content for i, d in enumerate(docs)]))Using a vanilla vector store retriever‚ÄãLet's start by initializing a simple vector store retriever and storing the 2023 State of the Union speech (in chunks). We can see that given an example question our retriever returns one or two relevant docs and a few irrelevant docs. And even the relevant docs have a lot of irrelevant information in them.from langchain.text_splitter import CharacterTextSplitterfrom langchain.embeddings import OpenAIEmbeddingsfrom langchain.document_loaders import TextLoaderfrom langchain.vectorstores import FAISSdocuments = TextLoader('../../../state_of_the_union.txt').load()text_splitter = CharacterTextSplitter(chunk_size=1000, chunk_overlap=0)texts = text_splitter.split_documents(documents)retriever = FAISS.from_documents(texts, OpenAIEmbeddings()).as_retriever()docs = retriever.get_relevant_documents("What did the president say about Ketanji Brown Jackson")pretty_print_docs(docs)    Document 1:    Tonight. I call on the Senate to: Pass the Freedom to Vote Act. Pass the John Lewis Voting Rights Act. And while you‚Äôre at it, pass the Disclose Act so Americans can know who is funding our elections.    Tonight, I‚Äôd like to honor someone who has dedicated his life to serve this country: Justice Stephen Breyer‚Äîan Army veteran, Constitutional scholar, and retiring Justice of the United States Supreme Court. Justice Breyer, thank you for your service.    One of the most serious constitutional responsibilities a President has is nominating someone to serve on the United States Supreme Court.    And I did that 4 days ago, when I nominated Circuit Court of Appeals Judge Ketanji Brown Jackson. One of our nation‚Äôs top legal minds, who will continue Justice Breyer‚Äôs legacy of excellence.    ----------------------------------------------------------------------------------------------------    Document 2:    A former top litigator in private practice. A
One challenge with retrieval is that usually you don't know the specific queries your document storage system will face when you ingest data into the system. This means that the information most relevant to a query may be buried in a document with a lot of irrelevant text. Passing that full document through your application can lead to more expensive LLM calls and poorer responses.
One challenge with retrieval is that usually you don't know the specific queries your document storage system will face when you ingest data into the system. This means that the information most relevant to a query may be buried in a document with a lot of irrelevant text. Passing that full document through your application can lead to more expensive LLM calls and poorer responses. ->: * 100}\n".join([f"Document {i+1}:\n\n" + d.page_content for i, d in enumerate(docs)]))Using a vanilla vector store retriever‚ÄãLet's start by initializing a simple vector store retriever and storing the 2023 State of the Union speech (in chunks). We can see that given an example question our retriever returns one or two relevant docs and a few irrelevant docs. And even the relevant docs have a lot of irrelevant information in them.from langchain.text_splitter import CharacterTextSplitterfrom langchain.embeddings import OpenAIEmbeddingsfrom langchain.document_loaders import TextLoaderfrom langchain.vectorstores import FAISSdocuments = TextLoader('../../../state_of_the_union.txt').load()text_splitter = CharacterTextSplitter(chunk_size=1000, chunk_overlap=0)texts = text_splitter.split_documents(documents)retriever = FAISS.from_documents(texts, OpenAIEmbeddings()).as_retriever()docs = retriever.get_relevant_documents("What did the president say about Ketanji Brown Jackson")pretty_print_docs(docs)    Document 1:    Tonight. I call on the Senate to: Pass the Freedom to Vote Act. Pass the John Lewis Voting Rights Act. And while you‚Äôre at it, pass the Disclose Act so Americans can know who is funding our elections.    Tonight, I‚Äôd like to honor someone who has dedicated his life to serve this country: Justice Stephen Breyer‚Äîan Army veteran, Constitutional scholar, and retiring Justice of the United States Supreme Court. Justice Breyer, thank you for your service.    One of the most serious constitutional responsibilities a President has is nominating someone to serve on the United States Supreme Court.    And I did that 4 days ago, when I nominated Circuit Court of Appeals Judge Ketanji Brown Jackson. One of our nation‚Äôs top legal minds, who will continue Justice Breyer‚Äôs legacy of excellence.    ----------------------------------------------------------------------------------------------------    Document 2:    A former top litigator in private practice. A