Making Text Embedders Few-Shot Learners
Large language models (LLMs) with decoder-only architectures demonstrate
remarkable in-context learning (ICL) capabilities. This feature enables them to
effectively handle both familiar and novel tasks by utilizing examples provided
within their input context. Recognizing the potential of this capability, we
propose leveraging the ICL feature in LLMs to enhance the process of text
embedding generation. To this end, we introduce a novel model bge-en-icl, which
employs few-shot examples to produce high-quality text embeddings. Our approach
integrates task-related examples directly into the query side, resulting in
significant improvements across various tasks. Additionally, we have
investigated how to effectively utilize LLMs as embedding models, including
various attention mechanisms, pooling methods, etc. Our findings suggest that
retaining the original framework often yields the best results, underscoring
that simplicity is best. Experimental results on the MTEB and AIR-Bench
benchmarks demonstrate that our approach sets new state-of-the-art (SOTA)
performance. Our model, code and dataset are freely available at
https://github.com/FlagOpen/FlagEmbedding .
66f37ef96a5bb2a469ed1424
Chaofan Li
claimed_verified
2024-09-25T07:32:25.853Z
65ccbcfde75fc480714d1d04
/avatars/2def06230512892928537cf2d8499563.svg
Chaofan Li
user
cfli
66f37ef96a5bb2a469ed1425
MingHao Qin
66f37ef96a5bb2a469ed1426
Shitao Xiao
66f37ef96a5bb2a469ed1427
Jianlyu Chen
66f37ef96a5bb2a469ed1428
Kun Luo
66f37ef96a5bb2a469ed1429
Yingxia Shao
66f37ef96a5bb2a469ed142a
Defu Lian
66f37ef96a5bb2a469ed142b
Zheng Liu
Making Text Embedders Few-Shot Learners
Large language models (LLMs) with decoder-only architectures demonstrate
remarkable in-context learning (ICL) capabilities. This feature enables them to
effectively handle both familiar and novel tasks by utilizing examples provided
within their input context. Recognizing the potential of this capability, we
propose leveraging the ICL feature in LLMs to enhance the process of text
embedding generation. To this end, we introduce a novel model bge-en-icl, which
employs few-shot examples to produce high-quality text embeddings. Our approach
integrates task-related examples directly into the query side, resulting in
significant improvements across various tasks. Additionally, we have
investigated how to effectively utilize LLMs as embedding models, including
various attention mechanisms, pooling methods, etc. Our findings suggest that
retaining the original framework often yields the best results, underscoring
that simplicity is best. Experimental results on the MTEB and AIR-Bench
benchmarks demonstrate that our approach sets new state-of-the-art (SOTA)
performance. Our model, code and dataset are freely available at
https://github.com/FlagOpen/FlagEmbedding .
A novel model leveraging in-context learning and few-shot examples in large language models improves text embedding generation, achieving state-of-the-art performance on MTEB and AIR-Bench benchmarks.
Large language models decoder-only architectures in-context learning text embeddings bge-en-icl few-shot examples attention mechanisms pooling methods