Well Begun is Half Done: Low-resource Preference Alignment by
  Weak-to-Strong Decoding
Large Language Models (LLMs) require alignment with human preferences to
avoid generating offensive, false, or meaningless content. Recently,
low-resource methods for LLM alignment have been popular, while still facing
challenges in obtaining both high-quality and aligned content. Motivated by the
observation that the difficulty of generating aligned responses is concentrated
at the beginning of decoding, we propose a novel framework, Weak-to-Strong
Decoding (WSD), to enhance the alignment ability of base models by the guidance
of a small aligned model. The small model first drafts well-aligned beginnings,
followed by the large base model to continue the rest, controlled by a
well-designed auto-switch mechanism. We also collect a new dataset, GenerAlign,
to fine-tune a small-sized Pilot-3B as the draft model, which effectively
enhances different base models under the WSD framework to outperform all
baseline methods, while avoiding degradation on downstream tasks, termed as the
alignment tax. Extensive experiments are further conducted to examine the
impact of different settings and time efficiency, as well as analyses on the
intrinsic mechanisms of WSD in depth.
684797f33ec10bdd8ab4de7a
Feifan Song
claimed_verified
2025-06-10T08:43:29.497Z
6447ca6ca478b20f1755b294
/avatars/5049856b5ed1b74533fff902e14b4c7c.svg
Feifan Song
user
songff
684797f33ec10bdd8ab4de7b
Shaohang Wei
claimed_verified
2025-06-10T08:43:27.498Z
67244a81aa8556c561925ab6
https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/w-vZ0uwYACagrNq-H1oyO.jpeg
Shaohang Wei
user
SylvainWei
684797f33ec10bdd8ab4de7c
Wen Luo
684797f33ec10bdd8ab4de7d
Yuxuan Fan
684797f33ec10bdd8ab4de7e
Tianyu Liu
684797f33ec10bdd8ab4de7f
Guoyin Wang
684797f33ec10bdd8ab4de80
Houfeng Wang
Well Begun is Half Done: Low-resource Preference Alignment by
  Weak-to-Strong Decoding
Large Language Models (LLMs) require alignment with human preferences to
avoid generating offensive, false, or meaningless content. Recently,
low-resource methods for LLM alignment have been popular, while still facing
challenges in obtaining both high-quality and aligned content. Motivated by the
observation that the difficulty of generating aligned responses is concentrated
at the beginning of decoding, we propose a novel framework, Weak-to-Strong
Decoding (WSD), to enhance the alignment ability of base models by the guidance
of a small aligned model. The small model first drafts well-aligned beginnings,
followed by the large base model to continue the rest, controlled by a
well-designed auto-switch mechanism. We also collect a new dataset, GenerAlign,
to fine-tune a small-sized Pilot-3B as the draft model, which effectively
enhances different base models under the WSD framework to outperform all
baseline methods, while avoiding degradation on downstream tasks, termed as the
alignment tax. Extensive experiments are further conducted to examine the
impact of different settings and time efficiency, as well as analyses on the
intrinsic mechanisms of WSD in depth.
A new decoding framework (Weak-to-Strong Decoding, WSD) enhances the alignment of large language models by using a small aligned model to draft responses, followed by the base model, with a design to prevent degradation in performance on downstream tasks.
Large Language Models (LLMs) LLM alignment human preferences low-resource methods decoding small aligned model auto-switch mechanism GenerAlign Pilot-3B draft model alignment tax intrinsic mechanisms