NV-Embed: Improved Techniques for Training LLMs as Generalist Embedding
  Models
Decoder-only large language model (LLM)-based embedding models are beginning
to outperform BERT or T5-based embedding models in general-purpose text
embedding tasks, including dense vector-based retrieval. In this work, we
introduce the NV-Embed model with a variety of architectural designs and
training procedures to significantly enhance the performance of LLM as a
versatile embedding model, while maintaining its simplicity and
reproducibility. For model architecture, we propose a latent attention layer to
obtain pooled embeddings, which consistently improves retrieval and downstream
task accuracy compared to mean pooling or using the last <EOS> token embedding
from LLMs. To enhance representation learning, we remove the causal attention
mask of LLMs during contrastive training. For model training, we introduce a
two-stage contrastive instruction-tuning method. It first applies contrastive
training with instructions on retrieval datasets, utilizing in-batch negatives
and curated hard negative examples. At stage-2, it blends various non-retrieval
datasets into instruction tuning, which not only enhances non-retrieval task
accuracy but also improves retrieval performance. Combining these techniques,
our NV-Embed model, using only publicly available data, has achieved a
record-high score of 69.32, ranking No. 1 on the Massive Text Embedding
Benchmark (MTEB) (as of May 24, 2024), with 56 tasks, encompassing retrieval,
reranking, classification, clustering, and semantic textual similarity tasks.
Notably, our model also attains the highest score of 59.36 on 15 retrieval
tasks in the MTEB benchmark (also known as BEIR). We will open-source the model
at: https://huggingface.co/nvidia/NV-Embed-v1.
66554f26965ea394ee8e102b
Chankyu Lee
extracted_pending
2024-05-28T03:27:34.715Z
6403c5e656038547952520b7
/avatars/f30ea8f1cca3355b4f0d70694fadd184.svg
Chankyu Lee
user
nada5
66554f26965ea394ee8e102c
Rajarshi Roy
66554f26965ea394ee8e102d
Mengyao Xu
admin_assigned
2024-05-28T10:19:17.294Z
64e90e91fb77a3eaa7cb8062
/avatars/9a5787939ecc7dfa0cd56b1d6108d466.svg
Mengyao Xu
user
Mengyao00
66554f26965ea394ee8e102e
Jonathan Raiman
admin_assigned
2024-05-28T10:19:30.057Z
664d33ab4f959a203df54bb3
/avatars/8ba48cf818c6ca0d311b46185488f350.svg
Jonathan Raiman
user
jraimannvidia
66554f26965ea394ee8e102f
Mohammad Shoeybi
claimed_verified
2024-05-28T07:18:43.953Z
6641544c695975af2cbd0da6
/avatars/0ad3c18dcba585259b064fe9b00a07ce.svg
Mohammad Shoeybi
user
shoeybi
66554f26965ea394ee8e1030
Bryan Catanzaro
admin_assigned
2024-05-28T10:19:36.527Z
6311021788942700629e6247
/avatars/e7adc1632b76e80e7e4a590033d1c20a.svg
Bryan Catanzaro
user
ctnzr
66554f26965ea394ee8e1031
Wei Ping
extracted_pending
2024-05-28T03:27:34.715Z
663ee43bfeeb49803537da98
/avatars/17c3e9c435cc36fb04b4589e6176a243.svg
Wei Ping
user
wping
NV-Embed: Improved Techniques for Training LLMs as Generalist Embedding
  Models
Decoder-only large language model (LLM)-based embedding models are beginning
to outperform BERT or T5-based embedding models in general-purpose text
embedding tasks, including dense vector-based retrieval. In this work, we
introduce the NV-Embed model with a variety of architectural designs and
training procedures to significantly enhance the performance of LLM as a
versatile embedding model, while maintaining its simplicity and
reproducibility. For model architecture, we propose a latent attention layer to
obtain pooled embeddings, which consistently improves retrieval and downstream
task accuracy compared to mean pooling or using the last <EOS> token embedding
from LLMs. To enhance representation learning, we remove the causal attention
mask of LLMs during contrastive training. For model training, we introduce a
two-stage contrastive instruction-tuning method. It first applies contrastive
training with instructions on retrieval datasets, utilizing in-batch negatives
and curated hard negative examples. At stage-2, it blends various non-retrieval
datasets into instruction tuning, which not only enhances non-retrieval task
accuracy but also improves retrieval performance. Combining these techniques,
our NV-Embed model, using only publicly available data, has achieved a
record-high score of 69.32, ranking No. 1 on the Massive Text Embedding
Benchmark (MTEB) (as of May 24, 2024), with 56 tasks, encompassing retrieval,
reranking, classification, clustering, and semantic textual similarity tasks.
Notably, our model also attains the highest score of 59.36 on 15 retrieval
tasks in the MTEB benchmark (also known as BEIR). We will open-source the model
at: https://huggingface.co/nvidia/NV-Embed-v1.
NV-Embed, a decoder-only large language model with latent attention and instruction-tuned contrastive training, achieves state-of-the-art performance across various text embedding and retrieval tasks.
decoder-only large language model LLM embedding models BERT T5 latent attention layer contrastive training causal attention mask in-batch negatives hard negative examples two-stage contrastive instruction-tuning Massive Text Embedding Benchmark MTEB BEIR