Get 3 diff embeddings.all_mini = HuggingFaceEmbeddings(model_name="all-MiniLM-L6-v2")multi_qa_mini = HuggingFaceEmbeddings(model_name="multi-qa-MiniLM-L6-dot-v1")filter_embeddings = OpenAIEmbeddings()ABS_PATH = os.path.dirname(os.path.abspath(__file__))DB_DIR = os.path.join(ABS_PATH, "db")# Instantiate 2 diff cromadb indexs, each one with a diff embedding.client_settings = chromadb.config.Settings(    is_persistent=True,    persist_directory=DB_DIR,    anonymized_telemetry=False,)db_all = Chroma(    collection_name="project_store_all",    persist_directory=DB_DIR,    client_settings=client_settings,    embedding_function=all_mini,)db_multi_qa = Chroma(    collection_name="project_store_multi",    persist_directory=DB_DIR,    client_settings=client_settings,    embedding_function=multi_qa_mini,)# Define 2 diff retrievers with 2 diff embeddings and diff search type.retriever_all = db_all.as_retriever(    search_type="similarity", search_kwargs={"k": 5, "include_metadata": True})retriever_multi_qa = db_multi_qa.as_retriever(    search_type="mmr", search_kwargs={"k": 5, "include_metadata": True})# The Lord of the Retrievers will hold the ouput of boths retrievers and can be used as any other# retriever on different types of chains.lotr = MergerRetriever(retrievers=[retriever_all, retriever_multi_qa])Remove redundant results from the merged retrievers.‚Äã# We can remove redundant results from both retrievers using yet another embedding.# Using multiples embeddings in diff steps could help reduce biases.filter = EmbeddingsRedundantFilter(embeddings=filter_embeddings)pipeline = DocumentCompressorPipeline(transformers=[filter])compression_retriever = ContextualCompressionRetriever(    base_compressor=pipeline, base_retriever=lotr)Pick a representative sample of documents from the merged retrievers.‚Äã# This filter will divide the documents vectors into clusters or "centers" of meaning.# Then it will pick the closest document to that center for the final results.# By
Lord of the Retrievers, also known as MergerRetriever, takes a list of retrievers as input and merges the results of their getrelevantdocuments() methods into a single list. The merged results will be a list of documents that are relevant to the query and that have been ranked by the different retrievers.
Lord of the Retrievers, also known as MergerRetriever, takes a list of retrievers as input and merges the results of their getrelevantdocuments() methods into a single list. The merged results will be a list of documents that are relevant to the query and that have been ranked by the different retrievers. ->: Get 3 diff embeddings.all_mini = HuggingFaceEmbeddings(model_name="all-MiniLM-L6-v2")multi_qa_mini = HuggingFaceEmbeddings(model_name="multi-qa-MiniLM-L6-dot-v1")filter_embeddings = OpenAIEmbeddings()ABS_PATH = os.path.dirname(os.path.abspath(__file__))DB_DIR = os.path.join(ABS_PATH, "db")# Instantiate 2 diff cromadb indexs, each one with a diff embedding.client_settings = chromadb.config.Settings(    is_persistent=True,    persist_directory=DB_DIR,    anonymized_telemetry=False,)db_all = Chroma(    collection_name="project_store_all",    persist_directory=DB_DIR,    client_settings=client_settings,    embedding_function=all_mini,)db_multi_qa = Chroma(    collection_name="project_store_multi",    persist_directory=DB_DIR,    client_settings=client_settings,    embedding_function=multi_qa_mini,)# Define 2 diff retrievers with 2 diff embeddings and diff search type.retriever_all = db_all.as_retriever(    search_type="similarity", search_kwargs={"k": 5, "include_metadata": True})retriever_multi_qa = db_multi_qa.as_retriever(    search_type="mmr", search_kwargs={"k": 5, "include_metadata": True})# The Lord of the Retrievers will hold the ouput of boths retrievers and can be used as any other# retriever on different types of chains.lotr = MergerRetriever(retrievers=[retriever_all, retriever_multi_qa])Remove redundant results from the merged retrievers.‚Äã# We can remove redundant results from both retrievers using yet another embedding.# Using multiples embeddings in diff steps could help reduce biases.filter = EmbeddingsRedundantFilter(embeddings=filter_embeddings)pipeline = DocumentCompressorPipeline(transformers=[filter])compression_retriever = ContextualCompressionRetriever(    base_compressor=pipeline, base_retriever=lotr)Pick a representative sample of documents from the merged retrievers.‚Äã# This filter will divide the documents vectors into clusters or "centers" of meaning.# Then it will pick the closest document to that center for the final results.# By