source should always be specified. Only use a None, if you never intend to use incremental mode, and for some reason can't specify the source field correctly.from langchain.text_splitter import CharacterTextSplitterdoc1 = Document(    page_content="kitty kitty kitty kitty kitty", metadata={"source": "kitty.txt"})doc2 = Document(page_content="doggy doggy the doggy", metadata={"source": "doggy.txt"})new_docs = CharacterTextSplitter(    separator="t", keep_separator=True, chunk_size=12, chunk_overlap=2).split_documents([doc1, doc2])new_docs    [Document(page_content='kitty kit', metadata={'source': 'kitty.txt'}),     Document(page_content='tty kitty ki', metadata={'source': 'kitty.txt'}),     Document(page_content='tty kitty', metadata={'source': 'kitty.txt'}),     Document(page_content='doggy doggy', metadata={'source': 'doggy.txt'}),     Document(page_content='the doggy', metadata={'source': 'doggy.txt'})]_clear()index(    new_docs,    record_manager,    vectorstore,    cleanup="incremental",    source_id_key="source",)    {'num_added': 5, 'num_updated': 0, 'num_skipped': 0, 'num_deleted': 0}changed_doggy_docs = [    Document(page_content="woof woof", metadata={"source": "doggy.txt"}),    Document(page_content="woof woof woof", metadata={"source": "doggy.txt"}),]This should delete the old versions of documents associated with doggy.txt source and replace them with the new versions.index(    changed_doggy_docs,    record_manager,    vectorstore,    cleanup="incremental",    source_id_key="source",)    {'num_added': 0, 'num_updated': 0, 'num_skipped': 2, 'num_deleted': 2}vectorstore.similarity_search("dog", k=30)    [Document(page_content='tty kitty', metadata={'source': 'kitty.txt'}),     Document(page_content='tty kitty ki', metadata={'source': 'kitty.txt'}),     Document(page_content='kitty kit', metadata={'source': 'kitty.txt'})]Using with loaders‚ÄãIndexing can accept either an iterable of documents or else any loader.Attention: The loader must set source keys
Here, we will look at a basic indexing workflow using the LangChain indexing API.
Here, we will look at a basic indexing workflow using the LangChain indexing API. ->: source should always be specified. Only use a None, if you never intend to use incremental mode, and for some reason can't specify the source field correctly.from langchain.text_splitter import CharacterTextSplitterdoc1 = Document(    page_content="kitty kitty kitty kitty kitty", metadata={"source": "kitty.txt"})doc2 = Document(page_content="doggy doggy the doggy", metadata={"source": "doggy.txt"})new_docs = CharacterTextSplitter(    separator="t", keep_separator=True, chunk_size=12, chunk_overlap=2).split_documents([doc1, doc2])new_docs    [Document(page_content='kitty kit', metadata={'source': 'kitty.txt'}),     Document(page_content='tty kitty ki', metadata={'source': 'kitty.txt'}),     Document(page_content='tty kitty', metadata={'source': 'kitty.txt'}),     Document(page_content='doggy doggy', metadata={'source': 'doggy.txt'}),     Document(page_content='the doggy', metadata={'source': 'doggy.txt'})]_clear()index(    new_docs,    record_manager,    vectorstore,    cleanup="incremental",    source_id_key="source",)    {'num_added': 5, 'num_updated': 0, 'num_skipped': 0, 'num_deleted': 0}changed_doggy_docs = [    Document(page_content="woof woof", metadata={"source": "doggy.txt"}),    Document(page_content="woof woof woof", metadata={"source": "doggy.txt"}),]This should delete the old versions of documents associated with doggy.txt source and replace them with the new versions.index(    changed_doggy_docs,    record_manager,    vectorstore,    cleanup="incremental",    source_id_key="source",)    {'num_added': 0, 'num_updated': 0, 'num_skipped': 2, 'num_deleted': 2}vectorstore.similarity_search("dog", k=30)    [Document(page_content='tty kitty', metadata={'source': 'kitty.txt'}),     Document(page_content='tty kitty ki', metadata={'source': 'kitty.txt'}),     Document(page_content='kitty kit', metadata={'source': 'kitty.txt'})]Using with loaders‚ÄãIndexing can accept either an iterable of documents or else any loader.Attention: The loader must set source keys