Skip to main contentü¶úÔ∏èüîó LangChainDocsUse casesIntegrationsAPICommunityChat our docsLangSmithJS/TS DocsSearchCTRLKQA over structured dataSQLRetrieval-augmented generation (RAG)Interacting with APIsChatbotsExtractionSummarizationTaggingWeb scrapingSynthetic data generationGraph queryingWeb scrapingOn this pageWeb scrapingUse case‚ÄãWeb research is one of the killer LLM applications:Users have highlighted it as one of his top desired AI tools. OSS repos like gpt-researcher are growing in popularity. Overview‚ÄãGathering content from the web has a few components:Search: Query to url (e.g., using GoogleSearchAPIWrapper).Loading: Url to HTML  (e.g., using AsyncHtmlLoader, AsyncChromiumLoader, etc).Transforming: HTML to formatted text (e.g., using HTML2Text or Beautiful Soup).Quickstart‚Äãpip install -q openai langchain playwright beautifulsoup4playwright install# Set env var OPENAI_API_KEY or load from a .env file:# import dotenv# dotenv.load_dotenv()Scraping HTML content using a headless instance of Chromium.The async nature of the scraping process is handled using Python's asyncio library.The actual interaction with the web pages is handled by Playwright.from langchain.document_loaders import AsyncChromiumLoaderfrom langchain.document_transformers import BeautifulSoupTransformer# Load HTMLloader = AsyncChromiumLoader(["https://www.wsj.com"])html = loader.load()Scrape text content tags such as <p>, <li>, <div>, and <a> tags from the HTML content:<p>: The paragraph tag. It defines a paragraph in HTML and is used to group together related sentences and/or phrases.<li>: The list item tag. It is used within ordered (<ol>) and unordered (<ul>) lists to define individual items within the list.<div>: The division tag. It is a block-level element used to group other inline or block-level elements.<a>: The anchor tag. It is used to define hyperlinks.<span>:  an inline container used to mark up a part of a text, or a part of a document. For many news websites (e.g., WSJ,
Open In Collab
Open In Collab ->: Skip to main contentü¶úÔ∏èüîó LangChainDocsUse casesIntegrationsAPICommunityChat our docsLangSmithJS/TS DocsSearchCTRLKQA over structured dataSQLRetrieval-augmented generation (RAG)Interacting with APIsChatbotsExtractionSummarizationTaggingWeb scrapingSynthetic data generationGraph queryingWeb scrapingOn this pageWeb scrapingUse case‚ÄãWeb research is one of the killer LLM applications:Users have highlighted it as one of his top desired AI tools. OSS repos like gpt-researcher are growing in popularity. Overview‚ÄãGathering content from the web has a few components:Search: Query to url (e.g., using GoogleSearchAPIWrapper).Loading: Url to HTML  (e.g., using AsyncHtmlLoader, AsyncChromiumLoader, etc).Transforming: HTML to formatted text (e.g., using HTML2Text or Beautiful Soup).Quickstart‚Äãpip install -q openai langchain playwright beautifulsoup4playwright install# Set env var OPENAI_API_KEY or load from a .env file:# import dotenv# dotenv.load_dotenv()Scraping HTML content using a headless instance of Chromium.The async nature of the scraping process is handled using Python's asyncio library.The actual interaction with the web pages is handled by Playwright.from langchain.document_loaders import AsyncChromiumLoaderfrom langchain.document_transformers import BeautifulSoupTransformer# Load HTMLloader = AsyncChromiumLoader(["https://www.wsj.com"])html = loader.load()Scrape text content tags such as <p>, <li>, <div>, and <a> tags from the HTML content:<p>: The paragraph tag. It defines a paragraph in HTML and is used to group together related sentences and/or phrases.<li>: The list item tag. It is used within ordered (<ol>) and unordered (<ul>) lists to define individual items within the list.<div>: The division tag. It is a block-level element used to group other inline or block-level elements.<a>: The anchor tag. It is used to define hyperlinks.<span>:  an inline container used to mark up a part of a text, or a part of a document. For many news websites (e.g., WSJ,