match one of the patterns will be loaded.loader = SitemapLoader(    web_path="https://langchain.readthedocs.io/sitemap.xml",    filter_urls=["https://api.python.langchain.com/en/latest"],)documents = loader.load()    Fetching pages: 100%|##########| 1/1 [00:00<00:00, 16.39it/s]documents[0]    Document(page_content='\n\n\n\n\n\n\n\n\n\nLangChain Python API Reference Documentation.\n\n\n\n\n\n\n\n\n\nYou will be automatically redirected to the new location of this page.\n\n', metadata={'source': 'https://api.python.langchain.com/en/latest/', 'loc': 'https://api.python.langchain.com/en/latest/', 'lastmod': '2023-10-13T18:09:58.478681+00:00', 'changefreq': 'daily', 'priority': '0.9'})Add custom scraping rules‚ÄãThe SitemapLoader uses beautifulsoup4 for the scraping process, and it scrapes every element on the page by default. The SitemapLoader constructor accepts a custom scraping function. This feature can be helpful to tailor the scraping process to your specific needs; for example, you might want to avoid scraping headers or navigation elements. The following example shows how to develop and use a custom function to avoid navigation and header elements.Import the beautifulsoup4 library and define the custom function.pip install beautifulsoup4from bs4 import BeautifulSoupdef remove_nav_and_header_elements(content: BeautifulSoup) -> str:    # Find all 'nav' and 'header' elements in the BeautifulSoup object    nav_elements = content.find_all("nav")    header_elements = content.find_all("header")    # Remove each 'nav' and 'header' element from the BeautifulSoup object    for element in nav_elements + header_elements:        element.decompose()    return str(content.get_text())Add your custom function to the SitemapLoader object.loader = SitemapLoader(    "https://langchain.readthedocs.io/sitemap.xml",    filter_urls=["https://api.python.langchain.com/en/latest/"],    parsing_function=remove_nav_and_header_elements,)Local Sitemap‚ÄãThe sitemap loader can also be used to
Extends from the WebBaseLoader, SitemapLoader loads a sitemap from a given URL, and then scrape and load all pages in the sitemap, returning each page as a Document.
Extends from the WebBaseLoader, SitemapLoader loads a sitemap from a given URL, and then scrape and load all pages in the sitemap, returning each page as a Document. ->: match one of the patterns will be loaded.loader = SitemapLoader(    web_path="https://langchain.readthedocs.io/sitemap.xml",    filter_urls=["https://api.python.langchain.com/en/latest"],)documents = loader.load()    Fetching pages: 100%|##########| 1/1 [00:00<00:00, 16.39it/s]documents[0]    Document(page_content='\n\n\n\n\n\n\n\n\n\nLangChain Python API Reference Documentation.\n\n\n\n\n\n\n\n\n\nYou will be automatically redirected to the new location of this page.\n\n', metadata={'source': 'https://api.python.langchain.com/en/latest/', 'loc': 'https://api.python.langchain.com/en/latest/', 'lastmod': '2023-10-13T18:09:58.478681+00:00', 'changefreq': 'daily', 'priority': '0.9'})Add custom scraping rules‚ÄãThe SitemapLoader uses beautifulsoup4 for the scraping process, and it scrapes every element on the page by default. The SitemapLoader constructor accepts a custom scraping function. This feature can be helpful to tailor the scraping process to your specific needs; for example, you might want to avoid scraping headers or navigation elements. The following example shows how to develop and use a custom function to avoid navigation and header elements.Import the beautifulsoup4 library and define the custom function.pip install beautifulsoup4from bs4 import BeautifulSoupdef remove_nav_and_header_elements(content: BeautifulSoup) -> str:    # Find all 'nav' and 'header' elements in the BeautifulSoup object    nav_elements = content.find_all("nav")    header_elements = content.find_all("header")    # Remove each 'nav' and 'header' element from the BeautifulSoup object    for element in nav_elements + header_elements:        element.decompose()    return str(content.get_text())Add your custom function to the SitemapLoader object.loader = SitemapLoader(    "https://langchain.readthedocs.io/sitemap.xml",    filter_urls=["https://api.python.langchain.com/en/latest/"],    parsing_function=remove_nav_and_header_elements,)Local Sitemap‚ÄãThe sitemap loader can also be used to