transformation steps (e.g., via text chunking) with respect to the original source documents.How it works‚ÄãLangChain indexing makes use of a record manager (RecordManager) that keeps track of document writes into the vector store.When indexing content, hashes are computed for each document, and the following information is stored in the record manager: the document hash (hash of both page content and metadata)write timethe source id -- each document should include information in its metadata to allow us to determine the ultimate source of this documentDeletion modes‚ÄãWhen indexing documents into a vector store, it's possible that some existing documents in the vector store should be deleted. In certain situations you may want to remove any existing documents that are derived from the same sources as the new documents being indexed. In others you may want to delete all existing documents wholesale. The indexing API deletion modes let you pick the behavior you want:Cleanup ModeDe-Duplicates ContentParallelizableCleans Up Deleted Source DocsCleans Up Mutations of Source Docs and/or Derived DocsClean Up TimingNone‚úÖ‚úÖ‚ùå‚ùå-Incremental‚úÖ‚úÖ‚ùå‚úÖContinuouslyFull‚úÖ‚ùå‚úÖ‚úÖAt end of indexingNone does not do any automatic clean up, allowing the user to manually do clean up of old content. incremental and full offer the following automated clean up:If the content of the source document or derived documents has changed, both incremental or full modes will clean up (delete) previous versions of the content.If the source document has been deleted (meaning it is not included in the documents currently being indexed), the full cleanup mode will delete it from the vector store correctly, but the incremental mode will not.When content is mutated (e.g., the source PDF file was revised) there will be a period of time during indexing when both the new and old versions may be returned to the user. This happens after the new content was written, but before the old version was
Here, we will look at a basic indexing workflow using the LangChain indexing API.
Here, we will look at a basic indexing workflow using the LangChain indexing API. ->: transformation steps (e.g., via text chunking) with respect to the original source documents.How it works‚ÄãLangChain indexing makes use of a record manager (RecordManager) that keeps track of document writes into the vector store.When indexing content, hashes are computed for each document, and the following information is stored in the record manager: the document hash (hash of both page content and metadata)write timethe source id -- each document should include information in its metadata to allow us to determine the ultimate source of this documentDeletion modes‚ÄãWhen indexing documents into a vector store, it's possible that some existing documents in the vector store should be deleted. In certain situations you may want to remove any existing documents that are derived from the same sources as the new documents being indexed. In others you may want to delete all existing documents wholesale. The indexing API deletion modes let you pick the behavior you want:Cleanup ModeDe-Duplicates ContentParallelizableCleans Up Deleted Source DocsCleans Up Mutations of Source Docs and/or Derived DocsClean Up TimingNone‚úÖ‚úÖ‚ùå‚ùå-Incremental‚úÖ‚úÖ‚ùå‚úÖContinuouslyFull‚úÖ‚ùå‚úÖ‚úÖAt end of indexingNone does not do any automatic clean up, allowing the user to manually do clean up of old content. incremental and full offer the following automated clean up:If the content of the source document or derived documents has changed, both incremental or full modes will clean up (delete) previous versions of the content.If the source document has been deleted (meaning it is not included in the documents currently being indexed), the full cleanup mode will delete it from the vector store correctly, but the incremental mode will not.When content is mutated (e.g., the source PDF file was revised) there will be a period of time during indexing when both the new and old versions may be returned to the user. This happens after the new content was written, but before the old version was