title: NV-Embed: Improved Techniques for Training LLMs as Generalist Embedding
    Models
summary: Decoder-only large language model (LLM)-based embedding models are beginning
  to outperform BERT or T5-based embedding models in general-purpose text
  embedding tasks, including dense vector-based retrieval. In this work, we
  introduce the NV-Embed model with a variety of architectural designs and
  training procedures to significantly enhance the performance of LLM as a
  versatile embedding model, while maintaining its simplicity and
  reproducibility. For model architecture, we propose a latent attention layer to
  obtain pooled embeddings, which consistently improves retrieval and downstream
  task accuracy compared to mean pooling or using the last <EOS> token embedding
  from LLMs. To enhance representation learning, we remove the causal attention
  mask of LLMs during contrastive training. For model training, we introduce a
  two-stage contrastive instruction-tuning method. It first applies contrastive
  training with instructions on retrieval datasets, utilizing in-batch negatives
  and curated hard negative examples. At stage-2, it blends various non-retrieval
  datasets into instruction tuning, which not only enhances non-retrieval task
  accuracy but also improves retrieval performance. Combining these techniques,
  our NV-Embed model, using only publicly available data, has achieved a
  record-high score of 69.32, ranking No. 1 on the Massive Text Embedding
  Benchmark (MTEB) (as of May 24, 2024), with 56 tasks, encompassing retrieval,
  reranking, classification, clustering, and semantic textual similarity tasks.
  Notably, our model also attains the highest score of 59.36 on 15 retrieval
  tasks in the MTEB benchmark (also known as BEIR). We will open-source the model
  at: https://huggingface.co/nvidia/NV-Embed-v1.
paper_authors._id: 66554f26965ea394ee8e102b
paper_authors.name: Chankyu Lee
paper_authors.status: extracted_pending
paper_authors.statusLastChangedAt: 2024-05-28T03:27:34.715Z
paper_authors._id: 6403c5e656038547952520b7
paper_authors.avatarUrl: /avatars/f30ea8f1cca3355b4f0d70694fadd184.svg
paper_authors.fullname: Chankyu Lee
paper_authors.type: user
paper_authors.user: nada5
paper_authors._id: 66554f26965ea394ee8e102c
paper_authors.name: Rajarshi Roy
paper_authors._id: 66554f26965ea394ee8e102d
paper_authors.name: Mengyao Xu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-05-28T10:19:17.294Z
paper_authors._id: 64e90e91fb77a3eaa7cb8062
paper_authors.avatarUrl: /avatars/9a5787939ecc7dfa0cd56b1d6108d466.svg
paper_authors.fullname: Mengyao Xu
paper_authors.type: user
paper_authors.user: Mengyao00
paper_authors._id: 66554f26965ea394ee8e102e
paper_authors.name: Jonathan Raiman
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-05-28T10:19:30.057Z
paper_authors._id: 664d33ab4f959a203df54bb3
paper_authors.avatarUrl: /avatars/8ba48cf818c6ca0d311b46185488f350.svg
paper_authors.fullname: Jonathan Raiman
paper_authors.type: user
paper_authors.user: jraimannvidia
paper_authors._id: 66554f26965ea394ee8e102f
paper_authors.name: Mohammad Shoeybi
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-05-28T07:18:43.953Z
paper_authors._id: 6641544c695975af2cbd0da6
paper_authors.avatarUrl: /avatars/0ad3c18dcba585259b064fe9b00a07ce.svg
paper_authors.fullname: Mohammad Shoeybi
paper_authors.type: user
paper_authors.user: shoeybi
paper_authors._id: 66554f26965ea394ee8e1030
paper_authors.name: Bryan Catanzaro
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-05-28T10:19:36.527Z
paper_authors._id: 6311021788942700629e6247
paper_authors.avatarUrl: /avatars/e7adc1632b76e80e7e4a590033d1c20a.svg
paper_authors.fullname: Bryan Catanzaro
paper_authors.type: user
paper_authors.user: ctnzr
paper_authors._id: 66554f26965ea394ee8e1031
paper_authors.name: Wei Ping
paper_authors.status: extracted_pending
paper_authors.statusLastChangedAt: 2024-05-28T03:27:34.715Z
paper_authors._id: 663ee43bfeeb49803537da98
paper_authors.avatarUrl: /avatars/17c3e9c435cc36fb04b4589e6176a243.svg
paper_authors.fullname: Wei Ping
paper_authors.type: user
paper_authors.user: wping
paper_title: NV-Embed: Improved Techniques for Training LLMs as Generalist Embedding
    Models
paper_summary: Decoder-only large language model (LLM)-based embedding models are beginning
  to outperform BERT or T5-based embedding models in general-purpose text
  embedding tasks, including dense vector-based retrieval. In this work, we
  introduce the NV-Embed model with a variety of architectural designs and
  training procedures to significantly enhance the performance of LLM as a
  versatile embedding model, while maintaining its simplicity and
  reproducibility. For model architecture, we propose a latent attention layer to
  obtain pooled embeddings, which consistently improves retrieval and downstream
  task accuracy compared to mean pooling or using the last <EOS> token embedding
  from LLMs. To enhance representation learning, we remove the causal attention
  mask of LLMs during contrastive training. For model training, we introduce a
  two-stage contrastive instruction-tuning method. It first applies contrastive
  training with instructions on retrieval datasets, utilizing in-batch negatives
  and curated hard negative examples. At stage-2, it blends various non-retrieval
  datasets into instruction tuning, which not only enhances non-retrieval task
  accuracy but also improves retrieval performance. Combining these techniques,
  our NV-Embed model, using only publicly available data, has achieved a
  record-high score of 69.32, ranking No. 1 on the Massive Text Embedding
  Benchmark (MTEB) (as of May 24, 2024), with 56 tasks, encompassing retrieval,
  reranking, classification, clustering, and semantic textual similarity tasks.
  Notably, our model also attains the highest score of 59.36 on 15 retrieval
  tasks in the MTEB benchmark (also known as BEIR). We will open-source the model
  at: https://huggingface.co/nvidia/NV-Embed-v1.
paper_ai_keywords: Decoder-only large language model embedding models BERT T5 dense vector-based retrieval NV-Embed latent attention layer pooled embeddings retrieval and downstream task accuracy causal attention mask contrastive training instructions in-batch negatives hard negative examples instruction-tuning non-retrieval datasets Massive Text Embedding Benchmark (MTEB) retrieval tasks reranking classification clustering semantic textual similarity tasks BEIR