Diffusion vs. Autoregressive Language Models: A Text Embedding
  Perspective
Large language model (LLM)-based embedding models, benefiting from large
scale pre-training and post-training, have begun to surpass BERT and T5-based
models on general-purpose text embedding tasks such as document retrieval.
However, a fundamental limitation of LLM embeddings lies in the unidirectional
attention used during autoregressive pre-training, which misaligns with the
bidirectional nature of text embedding tasks. To this end, We propose adopting
diffusion language models for text embeddings, motivated by their inherent
bidirectional architecture and recent success in matching or surpassing LLMs
especially on reasoning tasks. We present the first systematic study of the
diffusion language embedding model, which outperforms the LLM-based embedding
model by 20% on long-document retrieval, 8% on reasoning-intensive retrieval,
2% on instruction-following retrieval, and achieve competitive performance on
traditional text embedding benchmarks. Our analysis verifies that bidirectional
attention is crucial for encoding global context in long and complex text.
682e9672d28d9650c90db133
Siyue Zhang
claimed_verified
2025-05-22T07:16:04.722Z
638f1803c67af472d317a922
https://cdn-avatars.huggingface.co/v1/production/uploads/638f1803c67af472d317a922/9BMVXqHa-AsdZPmBprcbd.jpeg
siyue zhang
user
siyue
682e9672d28d9650c90db134
Yilun Zhao
claimed_verified
2025-05-22T07:16:02.079Z
62f662bcc58915315c4eccea
https://cdn-avatars.huggingface.co/v1/production/uploads/62f662bcc58915315c4eccea/zOAQLONfMP88zr70sxHK-.jpeg
Yilun Zhao
user
yilunzhao
682e9672d28d9650c90db135
Liyuan Geng
claimed_verified
2025-05-22T07:15:59.203Z
65457e29bd25cef7d118122c
/avatars/67777b3f4584b8ba92f12e95dbf93482.svg
Liyuan Geng
user
LYGeng
682e9672d28d9650c90db136
Arman Cohan
682e9672d28d9650c90db137
Anh Tuan Luu
682e9672d28d9650c90db138
Chen Zhao
Diffusion vs. Autoregressive Language Models: A Text Embedding
  Perspective
Large language model (LLM)-based embedding models, benefiting from large
scale pre-training and post-training, have begun to surpass BERT and T5-based
models on general-purpose text embedding tasks such as document retrieval.
However, a fundamental limitation of LLM embeddings lies in the unidirectional
attention used during autoregressive pre-training, which misaligns with the
bidirectional nature of text embedding tasks. To this end, We propose adopting
diffusion language models for text embeddings, motivated by their inherent
bidirectional architecture and recent success in matching or surpassing LLMs
especially on reasoning tasks. We present the first systematic study of the
diffusion language embedding model, which outperforms the LLM-based embedding
model by 20% on long-document retrieval, 8% on reasoning-intensive retrieval,
2% on instruction-following retrieval, and achieve competitive performance on
traditional text embedding benchmarks. Our analysis verifies that bidirectional
attention is crucial for encoding global context in long and complex text.
Diffusion language models outperform large language model embeddings in text retrieval tasks due to their bidirectional architecture.
large language model (LLM) diffusion language models unidirectional attention bidirectional attention document retrieval reasoning-intensive retrieval instruction-following retrieval text embedding benchmarks