LLaDA 1.5: Variance-Reduced Preference Optimization for Large Language
  Diffusion Models
While Masked Diffusion Models (MDMs), such as LLaDA, present a promising
paradigm for language modeling, there has been relatively little effort in
aligning these models with human preferences via reinforcement learning. The
challenge primarily arises from the high variance in Evidence Lower Bound
(ELBO)-based likelihood estimates required for preference optimization. To
address this issue, we propose Variance-Reduced Preference Optimization (VRPO),
a framework that formally analyzes the variance of ELBO estimators and derives
bounds on both the bias and variance of preference optimization gradients.
Building on this theoretical foundation, we introduce unbiased variance
reduction strategies, including optimal Monte Carlo budget allocation and
antithetic sampling, that significantly improve the performance of MDM
alignment. We demonstrate the effectiveness of VRPO by applying it to LLaDA,
and the resulting model, LLaDA 1.5, outperforms its SFT-only predecessor
consistently and significantly across mathematical (GSM8K +4.7), code
(HumanEval +3.0, MBPP +1.8), and alignment benchmarks (IFEval +4.0, Arena-Hard
+4.3). Furthermore, LLaDA 1.5 demonstrates a highly competitive mathematical
performance compared to strong language MDMs and ARMs. Project page:
https://ml-gsai.github.io/LLaDA-1.5-Demo/.
68357a21d0fbc64a8e829088
Fengqi Zhu
68357a21d0fbc64a8e829089
Rongzhen Wang
68357a21d0fbc64a8e82908a
Shen Nie
68357a21d0fbc64a8e82908b
Xiaolu Zhang
admin_assigned
2025-05-27T08:45:40.970Z
67513d6d3b8586521cda5d76
/avatars/0f95cc5c23a0a1da289aa785bd33b616.svg
Xiaolu  Zhang
user
xiaolu0714
68357a21d0fbc64a8e82908c
Chunwei Wu
68357a21d0fbc64a8e82908d
Jun Hu
68357a21d0fbc64a8e82908e
Jun Zhou
68357a21d0fbc64a8e82908f
Jianfei Chen
admin_assigned
2025-05-27T08:45:00.594Z
65fcad0ba0d7adc40b54fac2
/avatars/7564b5642378fddb46ec3b5ae57c0402.svg
Jianfei Chen
user
surfingtomchen
68357a21d0fbc64a8e829090
Yankai Lin
admin_assigned
2025-05-27T08:44:53.835Z
657a651e1433ea7d44de6397
/avatars/ccfc76f94595a38ff4a80f77c911eabf.svg
Yankai Lin
user
lyk423
68357a21d0fbc64a8e829091
Ji-Rong Wen
admin_assigned
2025-05-27T08:44:47.347Z
64b8c89052b7353d8c6a1013
/avatars/cd59fffe81f6b07b4519540b8ff3d95f.svg
Ji-Rong Wen
user
jrwen
68357a21d0fbc64a8e829092
Chongxuan Li
admin_assigned
2025-05-27T08:44:37.114Z
64c07b488e2612254361153b
/avatars/ade0f783cc4c2d3e73f402637f595471.svg
chongxuan li
user
zhenxuan00
LLaDA 1.5: Variance-Reduced Preference Optimization for Large Language
  Diffusion Models
While Masked Diffusion Models (MDMs), such as LLaDA, present a promising
paradigm for language modeling, there has been relatively little effort in
aligning these models with human preferences via reinforcement learning. The
challenge primarily arises from the high variance in Evidence Lower Bound
(ELBO)-based likelihood estimates required for preference optimization. To
address this issue, we propose Variance-Reduced Preference Optimization (VRPO),
a framework that formally analyzes the variance of ELBO estimators and derives
bounds on both the bias and variance of preference optimization gradients.
Building on this theoretical foundation, we introduce unbiased variance
reduction strategies, including optimal Monte Carlo budget allocation and
antithetic sampling, that significantly improve the performance of MDM
alignment. We demonstrate the effectiveness of VRPO by applying it to LLaDA,
and the resulting model, LLaDA 1.5, outperforms its SFT-only predecessor
consistently and significantly across mathematical (GSM8K +4.7), code
(HumanEval +3.0, MBPP +1.8), and alignment benchmarks (IFEval +4.0, Arena-Hard
+4.3). Furthermore, LLaDA 1.5 demonstrates a highly competitive mathematical
performance compared to strong language MDMs and ARMs. Project page:
https://ml-gsai.github.io/LLaDA-1.5-Demo/.
VRPO is a variance-reduced preference optimization framework for Masked Diffusion Models that significantly enhances their alignment with human preferences and performance across various benchmarks.
Masked Diffusion Models LLaDA Variance-Reduced Preference Optimization VRPO Evidence Lower Bound ELBO bias variance preference optimization unbiased variance reduction optimal Monte Carlo budget allocation antithetic sampling GSM8K HumanEval MBPP IFEval Arena-Hard ARMs