title: LLaDA 1.5: Variance-Reduced Preference Optimization for Large Language
    Diffusion Models
summary: While Masked Diffusion Models (MDMs), such as LLaDA, present a promising
  paradigm for language modeling, there has been relatively little effort in
  aligning these models with human preferences via reinforcement learning. The
  challenge primarily arises from the high variance in Evidence Lower Bound
  (ELBO)-based likelihood estimates required for preference optimization. To
  address this issue, we propose Variance-Reduced Preference Optimization (VRPO),
  a framework that formally analyzes the variance of ELBO estimators and derives
  bounds on both the bias and variance of preference optimization gradients.
  Building on this theoretical foundation, we introduce unbiased variance
  reduction strategies, including optimal Monte Carlo budget allocation and
  antithetic sampling, that significantly improve the performance of MDM
  alignment. We demonstrate the effectiveness of VRPO by applying it to LLaDA,
  and the resulting model, LLaDA 1.5, outperforms its SFT-only predecessor
  consistently and significantly across mathematical (GSM8K +4.7), code
  (HumanEval +3.0, MBPP +1.8), and alignment benchmarks (IFEval +4.0, Arena-Hard
  +4.3). Furthermore, LLaDA 1.5 demonstrates a highly competitive mathematical
  performance compared to strong language MDMs and ARMs. Project page:
  https://ml-gsai.github.io/LLaDA-1.5-Demo/.
paper_authors._id: 68357a21d0fbc64a8e829088
paper_authors.name: Fengqi Zhu
paper_authors._id: 68357a21d0fbc64a8e829089
paper_authors.name: Rongzhen Wang
paper_authors._id: 68357a21d0fbc64a8e82908a
paper_authors.name: Shen Nie
paper_authors._id: 68357a21d0fbc64a8e82908b
paper_authors.name: Xiaolu Zhang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-05-27T08:45:40.970Z
paper_authors._id: 67513d6d3b8586521cda5d76
paper_authors.avatarUrl: /avatars/0f95cc5c23a0a1da289aa785bd33b616.svg
paper_authors.fullname: Xiaolu  Zhang
paper_authors.type: user
paper_authors.user: xiaolu0714
paper_authors._id: 68357a21d0fbc64a8e82908c
paper_authors.name: Chunwei Wu
paper_authors._id: 68357a21d0fbc64a8e82908d
paper_authors.name: Jun Hu
paper_authors._id: 68357a21d0fbc64a8e82908e
paper_authors.name: Jun Zhou
paper_authors._id: 68357a21d0fbc64a8e82908f
paper_authors.name: Jianfei Chen
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-05-27T08:45:00.594Z
paper_authors._id: 65fcad0ba0d7adc40b54fac2
paper_authors.avatarUrl: /avatars/7564b5642378fddb46ec3b5ae57c0402.svg
paper_authors.fullname: Jianfei Chen
paper_authors.type: user
paper_authors.user: surfingtomchen
paper_authors._id: 68357a21d0fbc64a8e829090
paper_authors.name: Yankai Lin
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-05-27T08:44:53.835Z
paper_authors._id: 657a651e1433ea7d44de6397
paper_authors.avatarUrl: /avatars/ccfc76f94595a38ff4a80f77c911eabf.svg
paper_authors.fullname: Yankai Lin
paper_authors.type: user
paper_authors.user: lyk423
paper_authors._id: 68357a21d0fbc64a8e829091
paper_authors.name: Ji-Rong Wen
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-05-27T08:44:47.347Z
paper_authors._id: 64b8c89052b7353d8c6a1013
paper_authors.avatarUrl: /avatars/cd59fffe81f6b07b4519540b8ff3d95f.svg
paper_authors.fullname: Ji-Rong Wen
paper_authors.type: user
paper_authors.user: jrwen
paper_authors._id: 68357a21d0fbc64a8e829092
paper_authors.name: Chongxuan Li
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-05-27T08:44:37.114Z
paper_authors._id: 64c07b488e2612254361153b
paper_authors.avatarUrl: /avatars/ade0f783cc4c2d3e73f402637f595471.svg
paper_authors.fullname: chongxuan li
paper_authors.type: user
paper_authors.user: zhenxuan00
paper_title: LLaDA 1.5: Variance-Reduced Preference Optimization for Large Language
    Diffusion Models
paper_summary: While Masked Diffusion Models (MDMs), such as LLaDA, present a promising
  paradigm for language modeling, there has been relatively little effort in
  aligning these models with human preferences via reinforcement learning. The
  challenge primarily arises from the high variance in Evidence Lower Bound
  (ELBO)-based likelihood estimates required for preference optimization. To
  address this issue, we propose Variance-Reduced Preference Optimization (VRPO),
  a framework that formally analyzes the variance of ELBO estimators and derives
  bounds on both the bias and variance of preference optimization gradients.
  Building on this theoretical foundation, we introduce unbiased variance
  reduction strategies, including optimal Monte Carlo budget allocation and
  antithetic sampling, that significantly improve the performance of MDM
  alignment. We demonstrate the effectiveness of VRPO by applying it to LLaDA,
  and the resulting model, LLaDA 1.5, outperforms its SFT-only predecessor
  consistently and significantly across mathematical (GSM8K +4.7), code
  (HumanEval +3.0, MBPP +1.8), and alignment benchmarks (IFEval +4.0, Arena-Hard
  +4.3). Furthermore, LLaDA 1.5 demonstrates a highly competitive mathematical
  performance compared to strong language MDMs and ARMs. Project page:
  https://ml-gsai.github.io/LLaDA-1.5-Demo/.
paper_ai_summary: VRPO is a variance-reduced preference optimization framework for Masked Diffusion Models that significantly enhances their alignment with human preferences and performance across various benchmarks.
paper_ai_keywords: Masked Diffusion Models LLaDA Variance-Reduced Preference Optimization VRPO Evidence Lower Bound ELBO bias variance preference optimization unbiased variance reduction optimal Monte Carlo budget allocation antithetic sampling GSM8K HumanEval MBPP IFEval Arena-Hard ARMs