and the KL-divergence penalty greatly restricts any appreciable adaptation.

S6. Further SFT Discussions
We also partially attribute this difference in effectiveness of SFT to the gap in pretraining vs. downstream task considered in
the original DPO paper [33] vs. our work. On two of the DPO LLM tasks (sentiment generation, single-turn dialogue),
generic off-the-shelf autoregressive language models are tuned on specific tasks in the SFT stage. In the final setting, sum-
marization, the SFT model has been pretrained on a similar task/dataset. In this case, finetuning on the “preferred” dataset
(preferred-FT) baseline performs comparably to the SFT initialization.
DiffusionModelAlignmentUsing Direct Preference Optimization