In the second section, we'll select one of these backends and illustrate how to use it through a basic example.Document Index Backends‚Äãfrom langchain.retrievers import DocArrayRetrieverfrom docarray import BaseDocfrom docarray.typing import NdArrayimport numpy as npfrom langchain.embeddings import FakeEmbeddingsimport randomembeddings = FakeEmbeddings(size=32)Before you start building the index, it's important to define your document schema. This determines what fields your documents will have and what type of data each field will hold.For this demonstration, we'll create a somewhat random schema containing 'title' (str), 'title_embedding' (numpy array), 'year' (int), and 'color' (str)class MyDoc(BaseDoc):    title: str    title_embedding: NdArray[32]    year: int    color: strInMemoryExactNNIndex‚ÄãInMemoryExactNNIndex stores all Documents in memory. It is a great starting point for small datasets, where you may not want to launch a database server.Learn more here: https://docs.docarray.org/user_guide/storing/index_in_memory/from docarray.index import InMemoryExactNNIndex# initialize the indexdb = InMemoryExactNNIndex[MyDoc]()# index datadb.index(    [        MyDoc(            title=f"My document {i}",            title_embedding=embeddings.embed_query(f"query {i}"),            year=i,            color=random.choice(["red", "green", "blue"]),        )        for i in range(100)    ])# optionally, you can create a filter queryfilter_query = {"year": {"$lte": 90}}# create a retrieverretriever = DocArrayRetriever(    index=db,    embeddings=embeddings,    search_field="title_embedding",    content_field="title",    filters=filter_query,)# find the relevant documentdoc = retriever.get_relevant_documents("some query")print(doc)    [Document(page_content='My document 56', metadata={'id': '1f33e58b6468ab722f3786b96b20afe6', 'year': 56, 'color': 'red'})]HnswDocumentIndex‚ÄãHnswDocumentIndex is a lightweight Document Index implementation that runs fully locally and is
DocArray is a versatile, open-source tool for managing your multi-modal data. It lets you shape your data however you want, and offers the flexibility to store and search it using various document index backends. Plus, it gets even better - you can utilize your DocArray document index to create a DocArrayRetriever, and build awesome Langchain apps!
DocArray is a versatile, open-source tool for managing your multi-modal data. It lets you shape your data however you want, and offers the flexibility to store and search it using various document index backends. Plus, it gets even better - you can utilize your DocArray document index to create a DocArrayRetriever, and build awesome Langchain apps! ->: In the second section, we'll select one of these backends and illustrate how to use it through a basic example.Document Index Backends‚Äãfrom langchain.retrievers import DocArrayRetrieverfrom docarray import BaseDocfrom docarray.typing import NdArrayimport numpy as npfrom langchain.embeddings import FakeEmbeddingsimport randomembeddings = FakeEmbeddings(size=32)Before you start building the index, it's important to define your document schema. This determines what fields your documents will have and what type of data each field will hold.For this demonstration, we'll create a somewhat random schema containing 'title' (str), 'title_embedding' (numpy array), 'year' (int), and 'color' (str)class MyDoc(BaseDoc):    title: str    title_embedding: NdArray[32]    year: int    color: strInMemoryExactNNIndex‚ÄãInMemoryExactNNIndex stores all Documents in memory. It is a great starting point for small datasets, where you may not want to launch a database server.Learn more here: https://docs.docarray.org/user_guide/storing/index_in_memory/from docarray.index import InMemoryExactNNIndex# initialize the indexdb = InMemoryExactNNIndex[MyDoc]()# index datadb.index(    [        MyDoc(            title=f"My document {i}",            title_embedding=embeddings.embed_query(f"query {i}"),            year=i,            color=random.choice(["red", "green", "blue"]),        )        for i in range(100)    ])# optionally, you can create a filter queryfilter_query = {"year": {"$lte": 90}}# create a retrieverretriever = DocArrayRetriever(    index=db,    embeddings=embeddings,    search_field="title_embedding",    content_field="title",    filters=filter_query,)# find the relevant documentdoc = retriever.get_relevant_documents("some query")print(doc)    [Document(page_content='My document 56', metadata={'id': '1f33e58b6468ab722f3786b96b20afe6', 'year': 56, 'color': 'red'})]HnswDocumentIndex‚ÄãHnswDocumentIndex is a lightweight Document Index implementation that runs fully locally and is