= RecursiveCharacterTextSplitter(chunk_size = 500, chunk_overlap = 0)all_splits = text_splitter.split_documents(data)Go deeper‚ÄãDocumentSplitters are just one type of the more generic DocumentTransformers.See further documentation on transformers here.Context-aware splitters keep the location ("context") of each split in the original Document:Markdown filesCode (py or js)DocumentsStep 3. Store‚ÄãTo be able to look up our document splits, we first need to store them where we can later look them up.The most common way to do this is to embed the contents of each document split.We store the embedding and splits in a vectorstore.from langchain.embeddings import OpenAIEmbeddingsfrom langchain.vectorstores import Chromavectorstore = Chroma.from_documents(documents=all_splits, embedding=OpenAIEmbeddings())Go deeper‚ÄãBrowse the > 40 vectorstores integrations here.See further documentation on vectorstores here.Browse the > 30 text embedding integrations here.See further documentation on embedding models here.Here are Steps 1-3:Step 4. Retrieve‚ÄãRetrieve relevant splits for any question using similarity search.This is simply "top K" retrieval where we select documents based on embedding similarity to the query.question = "What are the approaches to Task Decomposition?"docs = vectorstore.similarity_search(question)len(docs)    4Go deeper‚ÄãVectorstores are commonly used for retrieval, but they are not the only option. For example, SVMs (see thread here) can also be used.LangChain has many retrievers including, but not limited to, vectorstores. All retrievers implement a common method get_relevant_documents() (and its asynchronous variant aget_relevant_documents()).from langchain.retrievers import SVMRetrieversvm_retriever = SVMRetriever.from_documents(all_splits,OpenAIEmbeddings())docs_svm=svm_retriever.get_relevant_documents(question)len(docs_svm)    4Some common ways to improve on vector similarity search include:MultiQueryRetriever generates variants of the input
Open In Colab
Open In Colab ->: = RecursiveCharacterTextSplitter(chunk_size = 500, chunk_overlap = 0)all_splits = text_splitter.split_documents(data)Go deeper‚ÄãDocumentSplitters are just one type of the more generic DocumentTransformers.See further documentation on transformers here.Context-aware splitters keep the location ("context") of each split in the original Document:Markdown filesCode (py or js)DocumentsStep 3. Store‚ÄãTo be able to look up our document splits, we first need to store them where we can later look them up.The most common way to do this is to embed the contents of each document split.We store the embedding and splits in a vectorstore.from langchain.embeddings import OpenAIEmbeddingsfrom langchain.vectorstores import Chromavectorstore = Chroma.from_documents(documents=all_splits, embedding=OpenAIEmbeddings())Go deeper‚ÄãBrowse the > 40 vectorstores integrations here.See further documentation on vectorstores here.Browse the > 30 text embedding integrations here.See further documentation on embedding models here.Here are Steps 1-3:Step 4. Retrieve‚ÄãRetrieve relevant splits for any question using similarity search.This is simply "top K" retrieval where we select documents based on embedding similarity to the query.question = "What are the approaches to Task Decomposition?"docs = vectorstore.similarity_search(question)len(docs)    4Go deeper‚ÄãVectorstores are commonly used for retrieval, but they are not the only option. For example, SVMs (see thread here) can also be used.LangChain has many retrievers including, but not limited to, vectorstores. All retrievers implement a common method get_relevant_documents() (and its asynchronous variant aget_relevant_documents()).from langchain.retrievers import SVMRetrieversvm_retriever = SVMRetriever.from_documents(all_splits,OpenAIEmbeddings())docs_svm=svm_retriever.get_relevant_documents(question)len(docs_svm)    4Some common ways to improve on vector similarity search include:MultiQueryRetriever generates variants of the input