title: Context is Gold to find the Gold Passage: Evaluating and Training
    Contextual Document Embeddings
summary: A limitation of modern document retrieval embedding methods is that they
  typically encode passages (chunks) from the same documents independently, often
  overlooking crucial contextual information from the rest of the document that
  could greatly improve individual chunk representations.
    In this work, we introduce ConTEB (Context-aware Text Embedding Benchmark), a
  benchmark designed to evaluate retrieval models on their ability to leverage
  document-wide context. Our results show that state-of-the-art embedding models
  struggle in retrieval scenarios where context is required. To address this
  limitation, we propose InSeNT (In-sequence Negative Training), a novel
  contrastive post-training approach which combined with late chunking pooling
  enhances contextual representation learning while preserving computational
  efficiency. Our method significantly improves retrieval quality on ConTEB
  without sacrificing base model performance. We further find chunks embedded
  with our method are more robust to suboptimal chunking strategies and larger
  retrieval corpus sizes. We open-source all artifacts at
  https://github.com/illuin-tech/contextual-embeddings.
paper_authors._id: 683d53ace2a7d8d977a00dab
paper_authors.name: Max Conti
paper_authors._id: 683d53ace2a7d8d977a00dac
paper_authors.name: Manuel Faysse
paper_authors.status: extracted_confirmed
paper_authors.statusLastChangedAt: 2025-06-02T15:07:26.977Z
paper_authors._id: 60f2e021adf471cbdf8bb660
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1654090481550-60f2e021adf471cbdf8bb660.jpeg
paper_authors.fullname: Manuel Faysse
paper_authors.type: user
paper_authors.user: manu
paper_authors._id: 683d53ace2a7d8d977a00dad
paper_authors.name: Gautier Viaud
paper_authors._id: 683d53ace2a7d8d977a00dae
paper_authors.name: Antoine Bosselut
paper_authors._id: 683d53ace2a7d8d977a00daf
paper_authors.name: Céline Hudelot
paper_authors._id: 683d53ace2a7d8d977a00db0
paper_authors.name: Pierre Colombo
paper_title: Context is Gold to find the Gold Passage: Evaluating and Training
    Contextual Document Embeddings
paper_summary: A limitation of modern document retrieval embedding methods is that they
  typically encode passages (chunks) from the same documents independently, often
  overlooking crucial contextual information from the rest of the document that
  could greatly improve individual chunk representations.
    In this work, we introduce ConTEB (Context-aware Text Embedding Benchmark), a
  benchmark designed to evaluate retrieval models on their ability to leverage
  document-wide context. Our results show that state-of-the-art embedding models
  struggle in retrieval scenarios where context is required. To address this
  limitation, we propose InSeNT (In-sequence Negative Training), a novel
  contrastive post-training approach which combined with late chunking pooling
  enhances contextual representation learning while preserving computational
  efficiency. Our method significantly improves retrieval quality on ConTEB
  without sacrificing base model performance. We further find chunks embedded
  with our method are more robust to suboptimal chunking strategies and larger
  retrieval corpus sizes. We open-source all artifacts at
  https://github.com/illuin-tech/contextual-embeddings.
paper_ai_summary: A context-aware benchmark and contrastive training method improve document retrieval quality by leveraging full-document context and maintaining computational efficiency.
paper_ai_keywords: ConTEB Context-aware Text Embedding Benchmark InSeNT In-sequence Negative Training contrastive post-training contextual representation learning chunking strategies retrieval corpus size