GenARM: Reward Guided Generation with Autoregressive Reward Model for
  Test-time Alignment
Large Language Models (LLMs) exhibit impressive capabilities but require
careful alignment with human preferences. Traditional training-time methods
finetune LLMs using human preference datasets but incur significant training
costs and require repeated training to handle diverse user preferences.
Test-time alignment methods address this by using reward models (RMs) to guide
frozen LLMs without retraining. However, existing test-time approaches rely on
trajectory-level RMs which are designed to evaluate complete responses, making
them unsuitable for autoregressive text generation that requires computing
next-token rewards from partial responses. To address this, we introduce
GenARM, a test-time alignment approach that leverages the Autoregressive Reward
Model--a novel reward parametrization designed to predict next-token rewards
for efficient and effective autoregressive generation. Theoretically, we
demonstrate that this parametrization can provably guide frozen LLMs toward any
distribution achievable by traditional RMs within the KL-regularized
reinforcement learning framework. Experimental results show that GenARM
significantly outperforms prior test-time alignment baselines and matches the
performance of training-time methods. Additionally, GenARM enables efficient
weak-to-strong guidance, aligning larger LLMs with smaller RMs without the high
costs of training larger models. Furthermore, GenARM supports multi-objective
alignment, allowing real-time trade-offs between preference dimensions and
catering to diverse user preferences without retraining.
6708bba1d2dd526a476139ad
Yuancheng Xu
claimed_verified
2024-10-15T09:11:25.869Z
6505e2cdad3134ed7e63c8d0
/avatars/f0e5dcf5a3deb96d2c9cb50b4433cc13.svg
Yuancheng Xu
user
YuanchengXu
6708bba1d2dd526a476139ae
Udari Madhushani Sehwag
6708bba1d2dd526a476139af
Alec Koppel
6708bba1d2dd526a476139b0
Sicheng Zhu
6708bba1d2dd526a476139b1
Bang An
extracted_confirmed
2024-10-11T14:15:10.381Z
645071555fdcff143ad8e07f
/avatars/a59f4e518a500e1c58b3a6d688644432.svg
Bang An
user
bangan
6708bba1d2dd526a476139b2
Furong Huang
extracted_pending
2024-10-11T05:46:10.318Z
64cbc3e2a257a3212c00a115
/avatars/836e61be4aeda2080ddf2db9f2626cc6.svg
Furong Huang Lab at UMD
user
furongh-lab
6708bba1d2dd526a476139b3
Sumitra Ganesh
GenARM: Reward Guided Generation with Autoregressive Reward Model for
  Test-time Alignment
Large Language Models (LLMs) exhibit impressive capabilities but require
careful alignment with human preferences. Traditional training-time methods
finetune LLMs using human preference datasets but incur significant training
costs and require repeated training to handle diverse user preferences.
Test-time alignment methods address this by using reward models (RMs) to guide
frozen LLMs without retraining. However, existing test-time approaches rely on
trajectory-level RMs which are designed to evaluate complete responses, making
them unsuitable for autoregressive text generation that requires computing
next-token rewards from partial responses. To address this, we introduce
GenARM, a test-time alignment approach that leverages the Autoregressive Reward
Model--a novel reward parametrization designed to predict next-token rewards
for efficient and effective autoregressive generation. Theoretically, we
demonstrate that this parametrization can provably guide frozen LLMs toward any
distribution achievable by traditional RMs within the KL-regularized
reinforcement learning framework. Experimental results show that GenARM
significantly outperforms prior test-time alignment baselines and matches the
performance of training-time methods. Additionally, GenARM enables efficient
weak-to-strong guidance, aligning larger LLMs with smaller RMs without the high
costs of training larger models. Furthermore, GenARM supports multi-objective
alignment, allowing real-time trade-offs between preference dimensions and
catering to diverse user preferences without retraining.
GenARM enables efficient test-time alignment of LLMs using an autoregressive reward model, outperforming previous methods and handling diverse preferences without retraining.
Large Language Models LLMS human preference datasets reward models RMs autoregressive text generation GenARM Autoregressive Reward Model trajectory-level RMs KL-regularized reinforcement learning weak-to-strong guidance multi-objective alignment