title: Diffusion vs. Autoregressive Language Models: A Text Embedding
    Perspective
summary: Large language model (LLM)-based embedding models, benefiting from large
  scale pre-training and post-training, have begun to surpass BERT and T5-based
  models on general-purpose text embedding tasks such as document retrieval.
  However, a fundamental limitation of LLM embeddings lies in the unidirectional
  attention used during autoregressive pre-training, which misaligns with the
  bidirectional nature of text embedding tasks. To this end, We propose adopting
  diffusion language models for text embeddings, motivated by their inherent
  bidirectional architecture and recent success in matching or surpassing LLMs
  especially on reasoning tasks. We present the first systematic study of the
  diffusion language embedding model, which outperforms the LLM-based embedding
  model by 20% on long-document retrieval, 8% on reasoning-intensive retrieval,
  2% on instruction-following retrieval, and achieve competitive performance on
  traditional text embedding benchmarks. Our analysis verifies that bidirectional
  attention is crucial for encoding global context in long and complex text.
paper_authors._id: 682e9672d28d9650c90db133
paper_authors.name: Siyue Zhang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-05-22T07:16:04.722Z
paper_authors._id: 638f1803c67af472d317a922
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/638f1803c67af472d317a922/9BMVXqHa-AsdZPmBprcbd.jpeg
paper_authors.fullname: siyue zhang
paper_authors.type: user
paper_authors.user: siyue
paper_authors._id: 682e9672d28d9650c90db134
paper_authors.name: Yilun Zhao
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-05-22T07:16:02.079Z
paper_authors._id: 62f662bcc58915315c4eccea
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/62f662bcc58915315c4eccea/zOAQLONfMP88zr70sxHK-.jpeg
paper_authors.fullname: Yilun Zhao
paper_authors.type: user
paper_authors.user: yilunzhao
paper_authors._id: 682e9672d28d9650c90db135
paper_authors.name: Liyuan Geng
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-05-22T07:15:59.203Z
paper_authors._id: 65457e29bd25cef7d118122c
paper_authors.avatarUrl: /avatars/67777b3f4584b8ba92f12e95dbf93482.svg
paper_authors.fullname: Liyuan Geng
paper_authors.type: user
paper_authors.user: LYGeng
paper_authors._id: 682e9672d28d9650c90db136
paper_authors.name: Arman Cohan
paper_authors._id: 682e9672d28d9650c90db137
paper_authors.name: Anh Tuan Luu
paper_authors._id: 682e9672d28d9650c90db138
paper_authors.name: Chen Zhao
paper_title: Diffusion vs. Autoregressive Language Models: A Text Embedding
    Perspective
paper_summary: Large language model (LLM)-based embedding models, benefiting from large
  scale pre-training and post-training, have begun to surpass BERT and T5-based
  models on general-purpose text embedding tasks such as document retrieval.
  However, a fundamental limitation of LLM embeddings lies in the unidirectional
  attention used during autoregressive pre-training, which misaligns with the
  bidirectional nature of text embedding tasks. To this end, We propose adopting
  diffusion language models for text embeddings, motivated by their inherent
  bidirectional architecture and recent success in matching or surpassing LLMs
  especially on reasoning tasks. We present the first systematic study of the
  diffusion language embedding model, which outperforms the LLM-based embedding
  model by 20% on long-document retrieval, 8% on reasoning-intensive retrieval,
  2% on instruction-following retrieval, and achieve competitive performance on
  traditional text embedding benchmarks. Our analysis verifies that bidirectional
  attention is crucial for encoding global context in long and complex text.
paper_ai_summary: Diffusion language models outperform large language model embeddings in text retrieval tasks due to their bidirectional architecture.
paper_ai_keywords: large language model (LLM) diffusion language models unidirectional attention bidirectional attention document retrieval reasoning-intensive retrieval instruction-following retrieval text embedding benchmarks