later.Specifically, let's look at the additional metadata that is returned on the documents returned by docugami, in the form of some simple key/value pairs on all the text chunks:loader = DocugamiLoader(docset_id="wh2kned25uqm")documents = loader.load()documents[0].metadata    {'xpath': '/docset:OFFICELEASEAGREEMENT-section/docset:OFFICELEASEAGREEMENT/docset:LeaseParties',     'id': 'v1bvgaozfkak',     'source': 'TruTone Lane 2.docx',     'structure': 'p',     'tag': 'LeaseParties',     'Lease Date': 'April  24  \n\n ,',     'Landlord': 'BUBBA CENTER PARTNERSHIP',     'Tenant': 'Truetone Lane LLC',     'Lease Parties': 'This  OFFICE LEASE AGREEMENT  (this "Lease") is made and entered into by and between  BUBBA CENTER PARTNERSHIP  (" Landlord "), and  Truetone Lane LLC , a  Delaware  limited liability company  (" Tenant ").'}We can use a self-querying retriever to improve our query accuracy, using this additional metadata:from langchain.chains.query_constructor.schema import AttributeInfofrom langchain.retrievers.self_query.base import SelfQueryRetrieverEXCLUDE_KEYS = ["id", "xpath", "structure"]metadata_field_info = [    AttributeInfo(        name=key,        description=f"The {key} for this chunk",        type="string",    )    for key in documents[0].metadata    if key.lower() not in EXCLUDE_KEYS]document_content_description = "Contents of this chunk"llm = OpenAI(temperature=0)vectordb = Chroma.from_documents(documents=documents, embedding=embedding)retriever = SelfQueryRetriever.from_llm(    llm, vectordb, document_content_description, metadata_field_info, verbose=True)qa_chain = RetrievalQA.from_chain_type(    llm=OpenAI(), chain_type="stuff", retriever=retriever, return_source_documents=True)Let's run the same question again. It returns the correct result since all the chunks have metadata key/value pairs on them carrying key information about the document even if this information is physically very far away from the source chunk used to generate the
This notebook covers how to load documents from Docugami. It provides the advantages of using this system over alternative data loaders.
This notebook covers how to load documents from Docugami. It provides the advantages of using this system over alternative data loaders. ->: later.Specifically, let's look at the additional metadata that is returned on the documents returned by docugami, in the form of some simple key/value pairs on all the text chunks:loader = DocugamiLoader(docset_id="wh2kned25uqm")documents = loader.load()documents[0].metadata    {'xpath': '/docset:OFFICELEASEAGREEMENT-section/docset:OFFICELEASEAGREEMENT/docset:LeaseParties',     'id': 'v1bvgaozfkak',     'source': 'TruTone Lane 2.docx',     'structure': 'p',     'tag': 'LeaseParties',     'Lease Date': 'April  24  \n\n ,',     'Landlord': 'BUBBA CENTER PARTNERSHIP',     'Tenant': 'Truetone Lane LLC',     'Lease Parties': 'This  OFFICE LEASE AGREEMENT  (this "Lease") is made and entered into by and between  BUBBA CENTER PARTNERSHIP  (" Landlord "), and  Truetone Lane LLC , a  Delaware  limited liability company  (" Tenant ").'}We can use a self-querying retriever to improve our query accuracy, using this additional metadata:from langchain.chains.query_constructor.schema import AttributeInfofrom langchain.retrievers.self_query.base import SelfQueryRetrieverEXCLUDE_KEYS = ["id", "xpath", "structure"]metadata_field_info = [    AttributeInfo(        name=key,        description=f"The {key} for this chunk",        type="string",    )    for key in documents[0].metadata    if key.lower() not in EXCLUDE_KEYS]document_content_description = "Contents of this chunk"llm = OpenAI(temperature=0)vectordb = Chroma.from_documents(documents=documents, embedding=embedding)retriever = SelfQueryRetriever.from_llm(    llm, vectordb, document_content_description, metadata_field_info, verbose=True)qa_chain = RetrievalQA.from_chain_type(    llm=OpenAI(), chain_type="stuff", retriever=retriever, return_source_documents=True)Let's run the same question again. It returns the correct result since all the chunks have metadata key/value pairs on them carrying key information about the document even if this information is physically very far away from the source chunk used to generate the