of a document. For many news websites (e.g., WSJ, CNN), headlines and summaries are all in <span> tags.# Transformbs_transformer = BeautifulSoupTransformer()docs_transformed = bs_transformer.transform_documents(html,tags_to_extract=["span"])# Resultdocs_transformed[0].page_content[0:500]    'English EditionEnglish‰∏≠Êñá (Chinese)Êó•Êú¨Ë™û (Japanese) More Other Products from WSJBuy Side from WSJWSJ ShopWSJ Wine Other Products from WSJ Search Quotes and Companies Search Quotes and Companies 0.15% 0.03% 0.12% -0.42% 4.102% -0.69% -0.25% -0.15% -1.82% 0.24% 0.19% -1.10% About Evan His Family Reflects His Reporting How You Can Help Write a Message Life in Detention Latest News Get Email Updates Four Americans Released From Iranian Prison The Americans will remain under house arrest until they are 'These Documents now are staged for downstream usage in various LLM apps, as discussed below.Loader‚ÄãAsyncHtmlLoader‚ÄãThe AsyncHtmlLoader uses the aiohttp library to make asynchronous HTTP requests, suitable for simpler and lightweight scraping.AsyncChromiumLoader‚ÄãThe AsyncChromiumLoader uses Playwright to launch a Chromium instance, which can handle JavaScript rendering and more complex web interactions.Chromium is one of the browsers supported by Playwright, a library used to control browser automation. Headless mode means that the browser is running without a graphical user interface, which is commonly used for web scraping.from langchain.document_loaders import AsyncHtmlLoaderurls = ["https://www.espn.com","https://lilianweng.github.io/posts/2023-06-23-agent/"]loader = AsyncHtmlLoader(urls)docs = loader.load()Transformer‚ÄãHTML2Text‚ÄãHTML2Text provides a straightforward conversion of HTML content into plain text (with markdown-like formatting) without any specific tag manipulation. It's best suited for scenarios where the goal is to extract human-readable text without needing to manipulate specific HTML elements.Beautiful Soup‚ÄãBeautiful Soup offers more fine-grained
Open In Collab
Open In Collab ->: of a document. For many news websites (e.g., WSJ, CNN), headlines and summaries are all in <span> tags.# Transformbs_transformer = BeautifulSoupTransformer()docs_transformed = bs_transformer.transform_documents(html,tags_to_extract=["span"])# Resultdocs_transformed[0].page_content[0:500]    'English EditionEnglish‰∏≠Êñá (Chinese)Êó•Êú¨Ë™û (Japanese) More Other Products from WSJBuy Side from WSJWSJ ShopWSJ Wine Other Products from WSJ Search Quotes and Companies Search Quotes and Companies 0.15% 0.03% 0.12% -0.42% 4.102% -0.69% -0.25% -0.15% -1.82% 0.24% 0.19% -1.10% About Evan His Family Reflects His Reporting How You Can Help Write a Message Life in Detention Latest News Get Email Updates Four Americans Released From Iranian Prison The Americans will remain under house arrest until they are 'These Documents now are staged for downstream usage in various LLM apps, as discussed below.Loader‚ÄãAsyncHtmlLoader‚ÄãThe AsyncHtmlLoader uses the aiohttp library to make asynchronous HTTP requests, suitable for simpler and lightweight scraping.AsyncChromiumLoader‚ÄãThe AsyncChromiumLoader uses Playwright to launch a Chromium instance, which can handle JavaScript rendering and more complex web interactions.Chromium is one of the browsers supported by Playwright, a library used to control browser automation. Headless mode means that the browser is running without a graphical user interface, which is commonly used for web scraping.from langchain.document_loaders import AsyncHtmlLoaderurls = ["https://www.espn.com","https://lilianweng.github.io/posts/2023-06-23-agent/"]loader = AsyncHtmlLoader(urls)docs = loader.load()Transformer‚ÄãHTML2Text‚ÄãHTML2Text provides a straightforward conversion of HTML content into plain text (with markdown-like formatting) without any specific tag manipulation. It's best suited for scenarios where the goal is to extract human-readable text without needing to manipulate specific HTML elements.Beautiful Soup‚ÄãBeautiful Soup offers more fine-grained