Skip to main contentü¶úÔ∏èüîó LangChainDocsUse casesIntegrationsAPICommunityChat our docsLangSmithJS/TS DocsSearchCTRLKProvidersAnthropicAWSGoogleMicrosoftOpenAIMoreComponentsLLMsChat modelsDocument loadersDocument transformersBeautiful SoupGoogle Cloud Document AIDoctran: extract propertiesDoctran: interrogate documentsDoctran: language translationHTML to textNucliaOpenAI metadata taggerText embedding modelsVector storesRetrieversToolsAgents and toolkitsMemoryCallbacksChat loadersComponentsDocument transformersDoctran: interrogate documentsOn this pageDoctran: interrogate documentsDocuments used in a vector store knowledge base are typically stored in a narrative or conversational format. However, most user queries are in question format. If we convert documents into Q&A format before vectorizing them, we can increase the likelihood of retrieving relevant documents, and decrease the likelihood of retrieving irrelevant documents.We can accomplish this using the Doctran library, which uses OpenAI's function calling feature to "interrogate" documents.See this notebook for benchmarks on vector similarity scores for various queries based on raw documents versus interrogated documents.pip install doctranimport jsonfrom langchain.schema import Documentfrom langchain.document_transformers import DoctranQATransformerfrom dotenv import load_dotenvload_dotenv()    TrueInput‚ÄãThis is the document we'll interrogatesample_text = """[Generated with ChatGPT]Confidential Document - For Internal Use OnlyDate: July 1, 2023Subject: Updates and Discussions on Various TopicsDear Team,I hope this email finds you well. In this document, I would like to provide you with some important updates and discuss various topics that require our attention. Please treat the information contained herein as highly confidential.Security and Privacy MeasuresAs part of our ongoing commitment to ensure the security and privacy of our customers' data, we have implemented robust measures across all our
Documents used in a vector store knowledge base are typically stored in a narrative or conversational format. However, most user queries are in question format. If we convert documents into Q&A format before vectorizing them, we can increase the likelihood of retrieving relevant documents, and decrease the likelihood of retrieving irrelevant documents.
Documents used in a vector store knowledge base are typically stored in a narrative or conversational format. However, most user queries are in question format. If we convert documents into Q&A format before vectorizing them, we can increase the likelihood of retrieving relevant documents, and decrease the likelihood of retrieving irrelevant documents. ->: Skip to main contentü¶úÔ∏èüîó LangChainDocsUse casesIntegrationsAPICommunityChat our docsLangSmithJS/TS DocsSearchCTRLKProvidersAnthropicAWSGoogleMicrosoftOpenAIMoreComponentsLLMsChat modelsDocument loadersDocument transformersBeautiful SoupGoogle Cloud Document AIDoctran: extract propertiesDoctran: interrogate documentsDoctran: language translationHTML to textNucliaOpenAI metadata taggerText embedding modelsVector storesRetrieversToolsAgents and toolkitsMemoryCallbacksChat loadersComponentsDocument transformersDoctran: interrogate documentsOn this pageDoctran: interrogate documentsDocuments used in a vector store knowledge base are typically stored in a narrative or conversational format. However, most user queries are in question format. If we convert documents into Q&A format before vectorizing them, we can increase the likelihood of retrieving relevant documents, and decrease the likelihood of retrieving irrelevant documents.We can accomplish this using the Doctran library, which uses OpenAI's function calling feature to "interrogate" documents.See this notebook for benchmarks on vector similarity scores for various queries based on raw documents versus interrogated documents.pip install doctranimport jsonfrom langchain.schema import Documentfrom langchain.document_transformers import DoctranQATransformerfrom dotenv import load_dotenvload_dotenv()    TrueInput‚ÄãThis is the document we'll interrogatesample_text = """[Generated with ChatGPT]Confidential Document - For Internal Use OnlyDate: July 1, 2023Subject: Updates and Discussions on Various TopicsDear Team,I hope this email finds you well. In this document, I would like to provide you with some important updates and discuss various topics that require our attention. Please treat the information contained herein as highly confidential.Security and Privacy MeasuresAs part of our ongoing commitment to ensure the security and privacy of our customers' data, we have implemented robust measures across all our