title: Making Text Embedders Few-Shot Learners
summary: Large language models (LLMs) with decoder-only architectures demonstrate
  remarkable in-context learning (ICL) capabilities. This feature enables them to
  effectively handle both familiar and novel tasks by utilizing examples provided
  within their input context. Recognizing the potential of this capability, we
  propose leveraging the ICL feature in LLMs to enhance the process of text
  embedding generation. To this end, we introduce a novel model bge-en-icl, which
  employs few-shot examples to produce high-quality text embeddings. Our approach
  integrates task-related examples directly into the query side, resulting in
  significant improvements across various tasks. Additionally, we have
  investigated how to effectively utilize LLMs as embedding models, including
  various attention mechanisms, pooling methods, etc. Our findings suggest that
  retaining the original framework often yields the best results, underscoring
  that simplicity is best. Experimental results on the MTEB and AIR-Bench
  benchmarks demonstrate that our approach sets new state-of-the-art (SOTA)
  performance. Our model, code and dataset are freely available at
  https://github.com/FlagOpen/FlagEmbedding .
paper_authors._id: 66f37ef96a5bb2a469ed1424
paper_authors.name: Chaofan Li
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-09-25T07:32:25.853Z
paper_authors._id: 65ccbcfde75fc480714d1d04
paper_authors.avatarUrl: /avatars/2def06230512892928537cf2d8499563.svg
paper_authors.fullname: Chaofan Li
paper_authors.type: user
paper_authors.user: cfli
paper_authors._id: 66f37ef96a5bb2a469ed1425
paper_authors.name: MingHao Qin
paper_authors._id: 66f37ef96a5bb2a469ed1426
paper_authors.name: Shitao Xiao
paper_authors._id: 66f37ef96a5bb2a469ed1427
paper_authors.name: Jianlyu Chen
paper_authors._id: 66f37ef96a5bb2a469ed1428
paper_authors.name: Kun Luo
paper_authors._id: 66f37ef96a5bb2a469ed1429
paper_authors.name: Yingxia Shao
paper_authors._id: 66f37ef96a5bb2a469ed142a
paper_authors.name: Defu Lian
paper_authors._id: 66f37ef96a5bb2a469ed142b
paper_authors.name: Zheng Liu
paper_title: Making Text Embedders Few-Shot Learners
paper_summary: Large language models (LLMs) with decoder-only architectures demonstrate
  remarkable in-context learning (ICL) capabilities. This feature enables them to
  effectively handle both familiar and novel tasks by utilizing examples provided
  within their input context. Recognizing the potential of this capability, we
  propose leveraging the ICL feature in LLMs to enhance the process of text
  embedding generation. To this end, we introduce a novel model bge-en-icl, which
  employs few-shot examples to produce high-quality text embeddings. Our approach
  integrates task-related examples directly into the query side, resulting in
  significant improvements across various tasks. Additionally, we have
  investigated how to effectively utilize LLMs as embedding models, including
  various attention mechanisms, pooling methods, etc. Our findings suggest that
  retaining the original framework often yields the best results, underscoring
  that simplicity is best. Experimental results on the MTEB and AIR-Bench
  benchmarks demonstrate that our approach sets new state-of-the-art (SOTA)
  performance. Our model, code and dataset are freely available at
  https://github.com/FlagOpen/FlagEmbedding .
paper_ai_keywords: Large language models (LLMs) Decoder-only architectures In-context learning (ICL) Embedding generation Query side Attention prompts Pooling methods State-of-the-art (SOTA) MTEB benchmark AIR-Bench benchmark Transformer-based detector