},    },    "required": ["movie_title", "critic", "tone"],}# Must be an OpenAI model that supports functionsllm = ChatOpenAI(temperature=0, model="gpt-3.5-turbo-0613")document_transformer = create_metadata_tagger(metadata_schema=schema, llm=llm)You can then simply pass the document transformer a list of documents, and it will extract metadata from the contents:original_documents = [    Document(        page_content="Review of The Bee Movie\nBy Roger Ebert\n\nThis is the greatest movie ever made. 4 out of 5 stars."    ),    Document(        page_content="Review of The Godfather\nBy Anonymous\n\nThis movie was super boring. 1 out of 5 stars.",        metadata={"reliable": False},    ),]enhanced_documents = document_transformer.transform_documents(original_documents)import jsonprint(    *[d.page_content + "\n\n" + json.dumps(d.metadata) for d in enhanced_documents],    sep="\n\n---------------\n\n")    Review of The Bee Movie    By Roger Ebert        This is the greatest movie ever made. 4 out of 5 stars.        {"movie_title": "The Bee Movie", "critic": "Roger Ebert", "tone": "positive", "rating": 4}        ---------------        Review of The Godfather    By Anonymous        This movie was super boring. 1 out of 5 stars.        {"movie_title": "The Godfather", "critic": "Anonymous", "tone": "negative", "rating": 1, "reliable": false}The new documents can then be further processed by a text splitter before being loaded into a vector store. Extracted fields will not overwrite existing metadata.You can also initialize the document transformer with a Pydantic schema:from typing import Literalfrom pydantic import BaseModel, Fieldclass Properties(BaseModel):    movie_title: str    critic: str    tone: Literal["positive", "negative"]    rating: int = Field(description="Rating out of 5 stars")document_transformer = create_metadata_tagger(Properties, llm)enhanced_documents = document_transformer.transform_documents(original_documents)print(    *[d.page_content +
It can often be useful to tag ingested documents with structured metadata, such as the title, tone, or length of a document, to allow for a more targeted similarity search later. However, for large numbers of documents, performing this labelling process manually can be tedious.
It can often be useful to tag ingested documents with structured metadata, such as the title, tone, or length of a document, to allow for a more targeted similarity search later. However, for large numbers of documents, performing this labelling process manually can be tedious. ->: },    },    "required": ["movie_title", "critic", "tone"],}# Must be an OpenAI model that supports functionsllm = ChatOpenAI(temperature=0, model="gpt-3.5-turbo-0613")document_transformer = create_metadata_tagger(metadata_schema=schema, llm=llm)You can then simply pass the document transformer a list of documents, and it will extract metadata from the contents:original_documents = [    Document(        page_content="Review of The Bee Movie\nBy Roger Ebert\n\nThis is the greatest movie ever made. 4 out of 5 stars."    ),    Document(        page_content="Review of The Godfather\nBy Anonymous\n\nThis movie was super boring. 1 out of 5 stars.",        metadata={"reliable": False},    ),]enhanced_documents = document_transformer.transform_documents(original_documents)import jsonprint(    *[d.page_content + "\n\n" + json.dumps(d.metadata) for d in enhanced_documents],    sep="\n\n---------------\n\n")    Review of The Bee Movie    By Roger Ebert        This is the greatest movie ever made. 4 out of 5 stars.        {"movie_title": "The Bee Movie", "critic": "Roger Ebert", "tone": "positive", "rating": 4}        ---------------        Review of The Godfather    By Anonymous        This movie was super boring. 1 out of 5 stars.        {"movie_title": "The Godfather", "critic": "Anonymous", "tone": "negative", "rating": 1, "reliable": false}The new documents can then be further processed by a text splitter before being loaded into a vector store. Extracted fields will not overwrite existing metadata.You can also initialize the document transformer with a Pydantic schema:from typing import Literalfrom pydantic import BaseModel, Fieldclass Properties(BaseModel):    movie_title: str    critic: str    tone: Literal["positive", "negative"]    rating: int = Field(description="Rating out of 5 stars")document_transformer = create_metadata_tagger(Properties, llm)enhanced_documents = document_transformer.transform_documents(original_documents)print(    *[d.page_content +