we can do together.    First, beat the opioid epidemic.Stringing compressors and document transformers togetherUsing the DocumentCompressorPipeline we can also easily combine multiple compressors in sequence. Along with compressors we can add BaseDocumentTransformers to our pipeline, which don't perform any contextual compression but simply perform some transformation on a set of documents. For example TextSplitters can be used as document transformers to split documents into smaller pieces, and the EmbeddingsRedundantFilter can be used to filter out redundant documents based on embedding similarity between documents.Below we create a compressor pipeline by first splitting our docs into smaller chunks, then removing redundant documents, and then filtering based on relevance to the query.from langchain.document_transformers import EmbeddingsRedundantFilterfrom langchain.retrievers.document_compressors import DocumentCompressorPipelinefrom langchain.text_splitter import CharacterTextSplittersplitter = CharacterTextSplitter(chunk_size=300, chunk_overlap=0, separator=". ")redundant_filter = EmbeddingsRedundantFilter(embeddings=embeddings)relevant_filter = EmbeddingsFilter(embeddings=embeddings, similarity_threshold=0.76)pipeline_compressor = DocumentCompressorPipeline(    transformers=[splitter, redundant_filter, relevant_filter])compression_retriever = ContextualCompressionRetriever(base_compressor=pipeline_compressor, base_retriever=retriever)compressed_docs = compression_retriever.get_relevant_documents("What did the president say about Ketanji Jackson Brown")pretty_print_docs(compressed_docs)    Document 1:    One of the most serious constitutional responsibilities a President has is nominating someone to serve on the United States Supreme Court.    And I did that 4 days ago, when I nominated Circuit Court of Appeals Judge Ketanji Brown Jackson    ----------------------------------------------------------------------------------------------------    Document 2:
One challenge with retrieval is that usually you don't know the specific queries your document storage system will face when you ingest data into the system. This means that the information most relevant to a query may be buried in a document with a lot of irrelevant text. Passing that full document through your application can lead to more expensive LLM calls and poorer responses.
One challenge with retrieval is that usually you don't know the specific queries your document storage system will face when you ingest data into the system. This means that the information most relevant to a query may be buried in a document with a lot of irrelevant text. Passing that full document through your application can lead to more expensive LLM calls and poorer responses. ->: we can do together.    First, beat the opioid epidemic.Stringing compressors and document transformers togetherUsing the DocumentCompressorPipeline we can also easily combine multiple compressors in sequence. Along with compressors we can add BaseDocumentTransformers to our pipeline, which don't perform any contextual compression but simply perform some transformation on a set of documents. For example TextSplitters can be used as document transformers to split documents into smaller pieces, and the EmbeddingsRedundantFilter can be used to filter out redundant documents based on embedding similarity between documents.Below we create a compressor pipeline by first splitting our docs into smaller chunks, then removing redundant documents, and then filtering based on relevance to the query.from langchain.document_transformers import EmbeddingsRedundantFilterfrom langchain.retrievers.document_compressors import DocumentCompressorPipelinefrom langchain.text_splitter import CharacterTextSplittersplitter = CharacterTextSplitter(chunk_size=300, chunk_overlap=0, separator=". ")redundant_filter = EmbeddingsRedundantFilter(embeddings=embeddings)relevant_filter = EmbeddingsFilter(embeddings=embeddings, similarity_threshold=0.76)pipeline_compressor = DocumentCompressorPipeline(    transformers=[splitter, redundant_filter, relevant_filter])compression_retriever = ContextualCompressionRetriever(base_compressor=pipeline_compressor, base_retriever=retriever)compressed_docs = compression_retriever.get_relevant_documents("What did the president say about Ketanji Jackson Brown")pretty_print_docs(compressed_docs)    Document 1:    One of the most serious constitutional responsibilities a President has is nominating someone to serve on the United States Supreme Court.    And I did that 4 days ago, when I nominated Circuit Court of Appeals Judge Ketanji Brown Jackson    ----------------------------------------------------------------------------------------------------    Document 2: