title: LLM2Vec: Large Language Models Are Secretly Powerful Text Encoders
summary: Large decoder-only language models (LLMs) are the state-of-the-art models on
  most of today's NLP tasks and benchmarks. Yet, the community is only slowly
  adopting these models for text embedding tasks, which require rich
  contextualized representations. In this work, we introduce LLM2Vec, a simple
  unsupervised approach that can transform any decoder-only LLM into a strong
  text encoder. LLM2Vec consists of three simple steps: 1) enabling bidirectional
  attention, 2) masked next token prediction, and 3) unsupervised contrastive
  learning. We demonstrate the effectiveness of LLM2Vec by applying it to 3
  popular LLMs ranging from 1.3B to 7B parameters and evaluate the transformed
  models on English word- and sequence-level tasks. We outperform encoder-only
  models by a large margin on word-level tasks and reach a new unsupervised
  state-of-the-art performance on the Massive Text Embeddings Benchmark (MTEB).
  Moreover, when combining LLM2Vec with supervised contrastive learning, we
  achieve state-of-the-art performance on MTEB among models that train only on
  publicly available data. Our strong empirical results and extensive analysis
  demonstrate that LLMs can be effectively transformed into universal text
  encoders in a parameter-efficient manner without the need for expensive
  adaptation or synthetic GPT-4 generated data.
paper_authors._id: 6615e08f362219047e082f98
paper_authors.name: Parishad BehnamGhader
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-04-18T14:29:15.707Z
paper_authors._id: 627d5ead401f42c57b6ce54c
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/627d5ead401f42c57b6ce54c/GajmN5G_MRUFRZs6ens0t.jpeg
paper_authors.fullname: Parishad BehnamGhader
paper_authors.type: user
paper_authors.user: parishadbehnam
paper_authors._id: 6615e08f362219047e082f99
paper_authors.name: Vaibhav Adlakha
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-04-18T14:29:24.085Z
paper_authors._id: 61981af5d420757268e195ac
paper_authors.avatarUrl: /avatars/8b59aaf33447224f83d497425fd7ea8f.svg
paper_authors.fullname: Vaibhav Adlakha
paper_authors.type: user
paper_authors.user: vaibhavad
paper_authors._id: 6615e08f362219047e082f9a
paper_authors.name: Marius Mosbach
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-04-18T14:29:31.079Z
paper_authors._id: 608865d2511e863acdb20bae
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/608865d2511e863acdb20bae/WSJQHZ4PC9hjLMCWv9mSb.png
paper_authors.fullname: Marius Mosbach
paper_authors.type: user
paper_authors.user: mmosbach
paper_authors._id: 6615e08f362219047e082f9b
paper_authors.name: Dzmitry Bahdanau
paper_authors._id: 6615e08f362219047e082f9c
paper_authors.name: Nicolas Chapados
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-04-18T14:29:41.309Z
paper_authors._id: 631f54aa5ba8c026340b13cf
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/631f54aa5ba8c026340b13cf/2jI0VUDG5cKkdf2C5KJuy.png
paper_authors.fullname: Nicolas Chapados
paper_authors.type: user
paper_authors.user: nicolaschapados
paper_authors._id: 6615e08f362219047e082f9d
paper_authors.name: Siva Reddy
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-04-10T10:57:34.527Z
paper_authors._id: 624734dc4c731bb6bfab8af7
paper_authors.avatarUrl: /avatars/6b250b58710a3287b85e4733c1824558.svg
paper_authors.fullname: Siva Reddy
paper_authors.type: user
paper_authors.user: sivareddyg
paper_title: LLM2Vec: Large Language Models Are Secretly Powerful Text Encoders
paper_summary: Large decoder-only language models (LLMs) are the state-of-the-art models on
  most of today's NLP tasks and benchmarks. Yet, the community is only slowly
  adopting these models for text embedding tasks, which require rich
  contextualized representations. In this work, we introduce LLM2Vec, a simple
  unsupervised approach that can transform any decoder-only LLM into a strong
  text encoder. LLM2Vec consists of three simple steps: 1) enabling bidirectional
  attention, 2) masked next token prediction, and 3) unsupervised contrastive
  learning. We demonstrate the effectiveness of LLM2Vec by applying it to 3
  popular LLMs ranging from 1.3B to 7B parameters and evaluate the transformed
  models on English word- and sequence-level tasks. We outperform encoder-only
  models by a large margin on word-level tasks and reach a new unsupervised
  state-of-the-art performance on the Massive Text Embeddings Benchmark (MTEB).
  Moreover, when combining LLM2Vec with supervised contrastive learning, we
  achieve state-of-the-art performance on MTEB among models that train only on
  publicly available data. Our strong empirical results and extensive analysis
  demonstrate that LLMs can be effectively transformed into universal text
  encoders in a parameter-efficient manner without the need for expensive
  adaptation or synthetic GPT-4 generated data.
paper_ai_keywords: decoder-only language models (LLMs) text embedding tasks bidirectional attention masked next token prediction unsupervised contrastive learning encoder-only models Massive Text Embeddings Benchmark (MTEB) supervised contrastive learning universal text encoders parameter-efficient manner