and extract text content from the web pages.In the example below, we will deeply crawl the Python documentation of LangChain's Chat LLM models and answer a question over it.First, install the requirements
pip install apify-client openai langchain chromadb tiktokenNext, set OPENAI_API_KEY and APIFY_API_TOKEN in your environment variables.The full code follows:from langchain.docstore.document import Documentfrom langchain.indexes import VectorstoreIndexCreatorfrom langchain.utilities import ApifyWrapperapify = ApifyWrapper()# Call the Actor to obtain text from the crawled webpagesloader = apify.call_actor(    actor_id="apify/website-content-crawler",    run_input={"startUrls": [{"url": "https://python.langchain.com/docs/integrations/chat/"}]},    dataset_mapping_function=lambda item: Document(        page_content=item["text"] or "", metadata={"source": item["url"]}    ),)# Create a vector store based on the crawled dataindex = VectorstoreIndexCreator().from_loaders([loader])# Query the vector storequery = "Are any OpenAI chat models integrated in LangChain?"result = index.query(query)print(result)     Yes, LangChain offers integration with OpenAI chat models. You can use the ChatOpenAI class to interact with OpenAI models.PreviousTaggingNextSynthetic data generationUse caseOverviewQuickstartLoaderAsyncHtmlLoaderAsyncChromiumLoaderTransformerHTML2TextBeautiful SoupScraping with extractionLLM with function callingDefine a schemaRun the web scraper w/ BeautifulSoupResearch automationGoing deeperQuestion answering over a websiteCommunityDiscordTwitterGitHubPythonJS/TSMoreHomepageBlogCopyright ¬© 2023 LangChain, Inc.
Open In Collab
Open In Collab ->: and extract text content from the web pages.In the example below, we will deeply crawl the Python documentation of LangChain's Chat LLM models and answer a question over it.First, install the requirements
pip install apify-client openai langchain chromadb tiktokenNext, set OPENAI_API_KEY and APIFY_API_TOKEN in your environment variables.The full code follows:from langchain.docstore.document import Documentfrom langchain.indexes import VectorstoreIndexCreatorfrom langchain.utilities import ApifyWrapperapify = ApifyWrapper()# Call the Actor to obtain text from the crawled webpagesloader = apify.call_actor(    actor_id="apify/website-content-crawler",    run_input={"startUrls": [{"url": "https://python.langchain.com/docs/integrations/chat/"}]},    dataset_mapping_function=lambda item: Document(        page_content=item["text"] or "", metadata={"source": item["url"]}    ),)# Create a vector store based on the crawled dataindex = VectorstoreIndexCreator().from_loaders([loader])# Query the vector storequery = "Are any OpenAI chat models integrated in LangChain?"result = index.query(query)print(result)     Yes, LangChain offers integration with OpenAI chat models. You can use the ChatOpenAI class to interact with OpenAI models.PreviousTaggingNextSynthetic data generationUse caseOverviewQuickstartLoaderAsyncHtmlLoaderAsyncChromiumLoaderTransformerHTML2TextBeautiful SoupScraping with extractionLLM with function callingDefine a schemaRun the web scraper w/ BeautifulSoupResearch automationGoing deeperQuestion answering over a websiteCommunityDiscordTwitterGitHubPythonJS/TSMoreHomepageBlogCopyright ¬© 2023 LangChain, Inc.