title: GenARM: Reward Guided Generation with Autoregressive Reward Model for
    Test-time Alignment
summary: Large Language Models (LLMs) exhibit impressive capabilities but require
  careful alignment with human preferences. Traditional training-time methods
  finetune LLMs using human preference datasets but incur significant training
  costs and require repeated training to handle diverse user preferences.
  Test-time alignment methods address this by using reward models (RMs) to guide
  frozen LLMs without retraining. However, existing test-time approaches rely on
  trajectory-level RMs which are designed to evaluate complete responses, making
  them unsuitable for autoregressive text generation that requires computing
  next-token rewards from partial responses. To address this, we introduce
  GenARM, a test-time alignment approach that leverages the Autoregressive Reward
  Model--a novel reward parametrization designed to predict next-token rewards
  for efficient and effective autoregressive generation. Theoretically, we
  demonstrate that this parametrization can provably guide frozen LLMs toward any
  distribution achievable by traditional RMs within the KL-regularized
  reinforcement learning framework. Experimental results show that GenARM
  significantly outperforms prior test-time alignment baselines and matches the
  performance of training-time methods. Additionally, GenARM enables efficient
  weak-to-strong guidance, aligning larger LLMs with smaller RMs without the high
  costs of training larger models. Furthermore, GenARM supports multi-objective
  alignment, allowing real-time trade-offs between preference dimensions and
  catering to diverse user preferences without retraining.
paper_authors._id: 6708bba1d2dd526a476139ad
paper_authors.name: Yuancheng Xu
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-10-15T09:11:25.869Z
paper_authors._id: 6505e2cdad3134ed7e63c8d0
paper_authors.avatarUrl: /avatars/f0e5dcf5a3deb96d2c9cb50b4433cc13.svg
paper_authors.fullname: Yuancheng Xu
paper_authors.type: user
paper_authors.user: YuanchengXu
paper_authors._id: 6708bba1d2dd526a476139ae
paper_authors.name: Udari Madhushani Sehwag
paper_authors._id: 6708bba1d2dd526a476139af
paper_authors.name: Alec Koppel
paper_authors._id: 6708bba1d2dd526a476139b0
paper_authors.name: Sicheng Zhu
paper_authors._id: 6708bba1d2dd526a476139b1
paper_authors.name: Bang An
paper_authors.status: extracted_confirmed
paper_authors.statusLastChangedAt: 2024-10-11T14:15:10.381Z
paper_authors._id: 645071555fdcff143ad8e07f
paper_authors.avatarUrl: /avatars/a59f4e518a500e1c58b3a6d688644432.svg
paper_authors.fullname: Bang An
paper_authors.type: user
paper_authors.user: bangan
paper_authors._id: 6708bba1d2dd526a476139b2
paper_authors.name: Furong Huang
paper_authors.status: extracted_pending
paper_authors.statusLastChangedAt: 2024-10-11T05:46:10.318Z
paper_authors._id: 64cbc3e2a257a3212c00a115
paper_authors.avatarUrl: /avatars/836e61be4aeda2080ddf2db9f2626cc6.svg
paper_authors.fullname: Furong Huang Lab at UMD
paper_authors.type: user
paper_authors.user: furongh-lab
paper_authors._id: 6708bba1d2dd526a476139b3
paper_authors.name: Sumitra Ganesh
paper_title: GenARM: Reward Guided Generation with Autoregressive Reward Model for
    Test-time Alignment
paper_summary: Large Language Models (LLMs) exhibit impressive capabilities but require
  careful alignment with human preferences. Traditional training-time methods
  finetune LLMs using human preference datasets but incur significant training
  costs and require repeated training to handle diverse user preferences.
  Test-time alignment methods address this by using reward models (RMs) to guide
  frozen LLMs without retraining. However, existing test-time approaches rely on
  trajectory-level RMs which are designed to evaluate complete responses, making
  them unsuitable for autoregressive text generation that requires computing
  next-token rewards from partial responses. To address this, we introduce
  GenARM, a test-time alignment approach that leverages the Autoregressive Reward
  Model--a novel reward parametrization designed to predict next-token rewards
  for efficient and effective autoregressive generation. Theoretically, we
  demonstrate that this parametrization can provably guide frozen LLMs toward any
  distribution achievable by traditional RMs within the KL-regularized
  reinforcement learning framework. Experimental results show that GenARM
  significantly outperforms prior test-time alignment baselines and matches the
  performance of training-time methods. Additionally, GenARM enables efficient
  weak-to-strong guidance, aligning larger LLMs with smaller RMs without the high
  costs of training larger models. Furthermore, GenARM supports multi-objective
  alignment, allowing real-time trade-offs between preference dimensions and
  catering to diverse user preferences without retraining.
paper_ai_keywords: Large Language Models (LLMs) human preference datasets reward models (RMs) Autoregressive Reward Model next-token rewards autoregressive generation KL-regularized reinforcement learning multi-objective alignment