of the LLM down.from langchain.chat_models import ChatOpenAIllm = ChatOpenAI(temperature=0, model="gpt-3.5-turbo-0613")Define a schema‚ÄãNext, you define a schema to specify what kind of data you want to extract. Here, the key names matter as they tell the LLM what kind of information they want. So, be as detailed as possible. In this example, we want to scrape only news article's name and summary from The Wall Street Journal website.from langchain.chains import create_extraction_chainschema = {    "properties": {        "news_article_title": {"type": "string"},        "news_article_summary": {"type": "string"},    },    "required": ["news_article_title", "news_article_summary"],}def extract(content: str, schema: dict):    return create_extraction_chain(schema=schema, llm=llm).run(content)Run the web scraper w/ BeautifulSoup‚ÄãAs shown above, we'll be using BeautifulSoupTransformer.import pprintfrom langchain.text_splitter import RecursiveCharacterTextSplitterdef scrape_with_playwright(urls, schema):        loader = AsyncChromiumLoader(urls)    docs = loader.load()    bs_transformer = BeautifulSoupTransformer()    docs_transformed = bs_transformer.transform_documents(docs,tags_to_extract=["span"])    print("Extracting content with LLM")        # Grab the first 1000 tokens of the site    splitter = RecursiveCharacterTextSplitter.from_tiktoken_encoder(chunk_size=1000,                                                                     chunk_overlap=0)    splits = splitter.split_documents(docs_transformed)        # Process the first split     extracted_content = extract(        schema=schema, content=splits[0].page_content    )    pprint.pprint(extracted_content)    return extracted_contenturls = ["https://www.wsj.com"]extracted_content = scrape_with_playwright(urls, schema=schema)    Extracting content with LLM    [{'news_article_summary': 'The Americans will remain under house arrest until '                              'they are allowed to return to the U.S. in
Open In Collab
Open In Collab ->: of the LLM down.from langchain.chat_models import ChatOpenAIllm = ChatOpenAI(temperature=0, model="gpt-3.5-turbo-0613")Define a schema‚ÄãNext, you define a schema to specify what kind of data you want to extract. Here, the key names matter as they tell the LLM what kind of information they want. So, be as detailed as possible. In this example, we want to scrape only news article's name and summary from The Wall Street Journal website.from langchain.chains import create_extraction_chainschema = {    "properties": {        "news_article_title": {"type": "string"},        "news_article_summary": {"type": "string"},    },    "required": ["news_article_title", "news_article_summary"],}def extract(content: str, schema: dict):    return create_extraction_chain(schema=schema, llm=llm).run(content)Run the web scraper w/ BeautifulSoup‚ÄãAs shown above, we'll be using BeautifulSoupTransformer.import pprintfrom langchain.text_splitter import RecursiveCharacterTextSplitterdef scrape_with_playwright(urls, schema):        loader = AsyncChromiumLoader(urls)    docs = loader.load()    bs_transformer = BeautifulSoupTransformer()    docs_transformed = bs_transformer.transform_documents(docs,tags_to_extract=["span"])    print("Extracting content with LLM")        # Grab the first 1000 tokens of the site    splitter = RecursiveCharacterTextSplitter.from_tiktoken_encoder(chunk_size=1000,                                                                     chunk_overlap=0)    splits = splitter.split_documents(docs_transformed)        # Process the first split     extracted_content = extract(        schema=schema, content=splits[0].page_content    )    pprint.pprint(extracted_content)    return extracted_contenturls = ["https://www.wsj.com"]extracted_content = scrape_with_playwright(urls, schema=schema)    Extracting content with LLM    [{'news_article_summary': 'The Americans will remain under house arrest until '                              'they are allowed to return to the U.S. in