import LanguageParser# Clonerepo_path = "/Users/rlm/Desktop/test_repo"# repo = Repo.clone_from("https://github.com/langchain-ai/langchain", to_path=repo_path)We load the py code using LanguageParser, which will:Keep top-level functions and classes together (into a single document)Put remaining code into a separate documentRetains metadata about where each split comes from# Loadloader = GenericLoader.from_filesystem(    repo_path+"/libs/langchain/langchain",    glob="**/*",    suffixes=[".py"],    parser=LanguageParser(language=Language.PYTHON, parser_threshold=500))documents = loader.load()len(documents)    1293Splitting‚ÄãSplit the Document into chunks for embedding and vector storage.We can use RecursiveCharacterTextSplitter w/ language specified.from langchain.text_splitter import RecursiveCharacterTextSplitterpython_splitter = RecursiveCharacterTextSplitter.from_language(language=Language.PYTHON,                                                                chunk_size=2000,                                                                chunk_overlap=200)texts = python_splitter.split_documents(documents)len(texts)    3748RetrievalQA‚ÄãWe need to store the documents in a way we can semantically search for their content. The most common approach is to embed the contents of each document then store the embedding and document in a vector store. When setting up the vectorstore retriever:We test max marginal relevance for retrievalAnd 8 documents returnedGo deeper‚ÄãBrowse the > 40 vectorstores integrations here.See further documentation on vectorstores here.Browse the > 30 text embedding integrations here.See further documentation on embedding models here.from langchain.vectorstores import Chromafrom langchain.embeddings.openai import OpenAIEmbeddingsdb = Chroma.from_documents(texts, OpenAIEmbeddings(disallowed_special=()))retriever = db.as_retriever(    search_type="mmr", # Also test "similarity"    search_kwargs={"k": 8},)Chat‚ÄãTest chat, just as we do for
Open In Collab
Open In Collab ->: import LanguageParser# Clonerepo_path = "/Users/rlm/Desktop/test_repo"# repo = Repo.clone_from("https://github.com/langchain-ai/langchain", to_path=repo_path)We load the py code using LanguageParser, which will:Keep top-level functions and classes together (into a single document)Put remaining code into a separate documentRetains metadata about where each split comes from# Loadloader = GenericLoader.from_filesystem(    repo_path+"/libs/langchain/langchain",    glob="**/*",    suffixes=[".py"],    parser=LanguageParser(language=Language.PYTHON, parser_threshold=500))documents = loader.load()len(documents)    1293Splitting‚ÄãSplit the Document into chunks for embedding and vector storage.We can use RecursiveCharacterTextSplitter w/ language specified.from langchain.text_splitter import RecursiveCharacterTextSplitterpython_splitter = RecursiveCharacterTextSplitter.from_language(language=Language.PYTHON,                                                                chunk_size=2000,                                                                chunk_overlap=200)texts = python_splitter.split_documents(documents)len(texts)    3748RetrievalQA‚ÄãWe need to store the documents in a way we can semantically search for their content. The most common approach is to embed the contents of each document then store the embedding and document in a vector store. When setting up the vectorstore retriever:We test max marginal relevance for retrievalAnd 8 documents returnedGo deeper‚ÄãBrowse the > 40 vectorstores integrations here.See further documentation on vectorstores here.Browse the > 30 text embedding integrations here.See further documentation on embedding models here.from langchain.vectorstores import Chromafrom langchain.embeddings.openai import OpenAIEmbeddingsdb = Chroma.from_documents(texts, OpenAIEmbeddings(disallowed_special=()))retriever = db.as_retriever(    search_type="mmr", # Also test "similarity"    search_kwargs={"k": 8},)Chat‚ÄãTest chat, just as we do for