title: Contextual Document Embeddings
summary: Dense document embeddings are central to neural retrieval. The dominant
  paradigm is to train and construct embeddings by running encoders directly on
  individual documents. In this work, we argue that these embeddings, while
  effective, are implicitly out-of-context for targeted use cases of retrieval,
  and that a contextualized document embedding should take into account both the
  document and neighboring documents in context - analogous to contextualized
  word embeddings. We propose two complementary methods for contextualized
  document embeddings: first, an alternative contrastive learning objective that
  explicitly incorporates the document neighbors into the intra-batch contextual
  loss; second, a new contextual architecture that explicitly encodes neighbor
  document information into the encoded representation. Results show that both
  methods achieve better performance than biencoders in several settings, with
  differences especially pronounced out-of-domain. We achieve state-of-the-art
  results on the MTEB benchmark with no hard negative mining, score distillation,
  dataset-specific instructions, intra-GPU example-sharing, or extremely large
  batch sizes. Our method can be applied to improve performance on any
  contrastive learning dataset and any biencoder.
paper_authors._id: 6700315f223c62ec884f43a4
paper_authors.name: John X. Morris
paper_authors.status: extracted_pending
paper_authors.statusLastChangedAt: 2024-10-04T18:18:09.128Z
paper_authors._id: 5edc0500968f6028e0559ff8
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1652114014657-5edc0500968f6028e0559ff8.jpeg
paper_authors.fullname: Jack Morris
paper_authors.type: user
paper_authors.user: jxm
paper_authors._id: 6700315f223c62ec884f43a5
paper_authors.name: Alexander M. Rush
paper_title: Contextual Document Embeddings
paper_summary: Dense document embeddings are central to neural retrieval. The dominant
  paradigm is to train and construct embeddings by running encoders directly on
  individual documents. In this work, we argue that these embeddings, while
  effective, are implicitly out-of-context for targeted use cases of retrieval,
  and that a contextualized document embedding should take into account both the
  document and neighboring documents in context - analogous to contextualized
  word embeddings. We propose two complementary methods for contextualized
  document embeddings: first, an alternative contrastive learning objective that
  explicitly incorporates the document neighbors into the intra-batch contextual
  loss; second, a new contextual architecture that explicitly encodes neighbor
  document information into the encoded representation. Results show that both
  methods achieve better performance than biencoders in several settings, with
  differences especially pronounced out-of-domain. We achieve state-of-the-art
  results on the MTEB benchmark with no hard negative mining, score distillation,
  dataset-specific instructions, intra-GPU example-sharing, or extremely large
  batch sizes. Our method can be applied to improve performance on any
  contrastive learning dataset and any biencoder.
paper_ai_keywords: Feder retrieval dense document embeddings encoders contextualized document embedding contextualized word embeddings contrastive learning objective intra-batch contextual loss contextual architecture biencoders MTEB benchmark hard negative mining score distillation dataset-specific instructions intra-GPU example-sharing extremely large batch sizes