pinecone_text.sparse import SpladeEncoder if you wish to work with SPLADE# use default tf-idf valuesbm25_encoder = BM25Encoder().default()The above code is using default tfids values. It's highly recommended to fit the tf-idf values to your own corpus. You can do it as follow:corpus = ["foo", "bar", "world", "hello"]# fit tf-idf values on your corpusbm25_encoder.fit(corpus)# store the values to a json filebm25_encoder.dump("bm25_values.json")# load to your BM25Encoder objectbm25_encoder = BM25Encoder().load("bm25_values.json")Load Retriever‚ÄãWe can now construct the retriever!retriever = PineconeHybridSearchRetriever(    embeddings=embeddings, sparse_encoder=bm25_encoder, index=index)Add texts (if necessary)‚ÄãWe can optionally add texts to the retriever (if they aren't already in there)retriever.add_texts(["foo", "bar", "world", "hello"])    100%|‚ñà‚ñà‚ñà‚ñà‚ñà‚ñà‚ñà‚ñà‚ñà‚ñà| 1/1 [00:02<00:00,  2.27s/it]Use Retriever‚ÄãWe can now use the retriever!result = retriever.get_relevant_documents("foo")result[0]    Document(page_content='foo', metadata={})PreviousMetalNextPubMedSetup PineconeGet embeddings and sparse encodersLoad RetrieverAdd texts (if necessary)Use RetrieverCommunityDiscordTwitterGitHubPythonJS/TSMoreHomepageBlogCopyright ¬© 2023 LangChain, Inc.
Pinecone is a vector database with broad functionality.
Pinecone is a vector database with broad functionality. ->: pinecone_text.sparse import SpladeEncoder if you wish to work with SPLADE# use default tf-idf valuesbm25_encoder = BM25Encoder().default()The above code is using default tfids values. It's highly recommended to fit the tf-idf values to your own corpus. You can do it as follow:corpus = ["foo", "bar", "world", "hello"]# fit tf-idf values on your corpusbm25_encoder.fit(corpus)# store the values to a json filebm25_encoder.dump("bm25_values.json")# load to your BM25Encoder objectbm25_encoder = BM25Encoder().load("bm25_values.json")Load Retriever‚ÄãWe can now construct the retriever!retriever = PineconeHybridSearchRetriever(    embeddings=embeddings, sparse_encoder=bm25_encoder, index=index)Add texts (if necessary)‚ÄãWe can optionally add texts to the retriever (if they aren't already in there)retriever.add_texts(["foo", "bar", "world", "hello"])    100%|‚ñà‚ñà‚ñà‚ñà‚ñà‚ñà‚ñà‚ñà‚ñà‚ñà| 1/1 [00:02<00:00,  2.27s/it]Use Retriever‚ÄãWe can now use the retriever!result = retriever.get_relevant_documents("foo")result[0]    Document(page_content='foo', metadata={})PreviousMetalNextPubMedSetup PineconeGet embeddings and sparse encodersLoad RetrieverAdd texts (if necessary)Use RetrieverCommunityDiscordTwitterGitHubPythonJS/TSMoreHomepageBlogCopyright ¬© 2023 LangChain, Inc.