it may be possible to stuff a lot of tokens, perhaps even entire documents, inside the context but this will still hit limits at some point with very long documents, or a lot of documents.For example, if we ask a more complex question that requires the LLM to draw on chunks from different parts of the document, even OpenAI's powerful LLM is unable to answer correctly.chain_response = qa_chain("What is rentable area for the property owned by DHA Group?")chain_response["result"]  # correct answer should be 13,500 sq ft    ' 9,753 square feet.'At first glance the answer may seem reasonable, but if you review the source chunks carefully for this answer, you will see that the chunking of the document did not end up putting the Landlord name and the rentable area in the same context, since they are far apart in the document. The retriever therefore ends up finding unrelated chunks from other documents not even related to the DHA Group landlord. That landlord happens to be mentioned on the first page of the file Shorebucks LLC_NJ.pdf file, and while one of the source chunks used by the chain is indeed from that doc that contains the correct answer (13,500), other source chunks from different docs are included, and the answer is therefore incorrect.chain_response["source_documents"]    [Document(page_content='1.1 Landlord . DHA Group , a  Delaware  limited liability company  authorized to transact business in  New Jersey .', metadata={'Landlord': 'DHA Group', 'Lease Date': 'March  29th ,  2019', 'Lease Parties': 'THIS OFFICE LEASE  (the "Lease") is made and entered into as of  March  29th ,  2019 , by and between  Landlord  and  Tenant . "Date of this Lease" shall mean the date on which the last one of the  Landlord  and  Tenant  has signed this  Lease .', 'Tenant': 'Shorebucks LLC', 'id': 'md8rieecquyv', 'source': 'Shorebucks LLC_NJ.pdf', 'structure': 'div', 'tag': 'DhaGroup', 'xpath':
This notebook covers how to load documents from Docugami. It provides the advantages of using this system over alternative data loaders.
This notebook covers how to load documents from Docugami. It provides the advantages of using this system over alternative data loaders. ->: it may be possible to stuff a lot of tokens, perhaps even entire documents, inside the context but this will still hit limits at some point with very long documents, or a lot of documents.For example, if we ask a more complex question that requires the LLM to draw on chunks from different parts of the document, even OpenAI's powerful LLM is unable to answer correctly.chain_response = qa_chain("What is rentable area for the property owned by DHA Group?")chain_response["result"]  # correct answer should be 13,500 sq ft    ' 9,753 square feet.'At first glance the answer may seem reasonable, but if you review the source chunks carefully for this answer, you will see that the chunking of the document did not end up putting the Landlord name and the rentable area in the same context, since they are far apart in the document. The retriever therefore ends up finding unrelated chunks from other documents not even related to the DHA Group landlord. That landlord happens to be mentioned on the first page of the file Shorebucks LLC_NJ.pdf file, and while one of the source chunks used by the chain is indeed from that doc that contains the correct answer (13,500), other source chunks from different docs are included, and the answer is therefore incorrect.chain_response["source_documents"]    [Document(page_content='1.1 Landlord . DHA Group , a  Delaware  limited liability company  authorized to transact business in  New Jersey .', metadata={'Landlord': 'DHA Group', 'Lease Date': 'March  29th ,  2019', 'Lease Parties': 'THIS OFFICE LEASE  (the "Lease") is made and entered into as of  March  29th ,  2019 , by and between  Landlord  and  Tenant . "Date of this Lease" shall mean the date on which the last one of the  Landlord  and  Tenant  has signed this  Lease .', 'Tenant': 'Shorebucks LLC', 'id': 'md8rieecquyv', 'source': 'Shorebucks LLC_NJ.pdf', 'structure': 'div', 'tag': 'DhaGroup', 'xpath':