Test-Time Preference Optimization: On-the-Fly Alignment via Iterative
  Textual Feedback
Large language models (LLMs) demonstrate impressive performance but lack the
flexibility to adapt to human preferences quickly without retraining. In this
work, we introduce Test-time Preference Optimization (TPO), a framework that
aligns LLM outputs with human preferences during inference, removing the need
to update model parameters. Rather than relying on purely numerical rewards,
TPO translates reward signals into textual critiques and uses them as textual
rewards to iteratively refine its response. Evaluations on benchmarks covering
instruction following, preference alignment, safety, and mathematics reveal
that TPO progressively improves alignment with human preferences. Notably,
after only a few TPO steps, the initially unaligned Llama-3.1-70B-SFT model can
surpass the aligned counterpart, Llama-3.1-70B-Instruct. Furthermore, TPO
scales efficiently with both the search width and depth during inference.
Through case studies, we illustrate how TPO exploits the innate capacity of LLM
to interpret and act upon reward signals. Our findings establish TPO as a
practical, lightweight alternative for test-time preference optimization,
achieving alignment on the fly. Our code is publicly available at
https://github.com/yafuly/TPO.
6791c63e9e215712a7d4bac8
Yafu Li
admin_assigned
2025-01-23T09:37:06.794Z
63f3502a520c14618925825a
/avatars/e986a2a6625e7be6890616a417f908d2.svg
Yafu Li
user
yaful
6791c63e9e215712a7d4bac9
Xuyang Hu
claimed_verified
2025-01-23T09:56:39.062Z
6498fde776d49ee00f79cbfe
/avatars/4c284a71080150e6cb3b9632dfccef60.svg
Xuyang Hu
user
huxy912
6791c63e9e215712a7d4baca
Xiaoye Qu
admin_assigned
2025-01-23T09:37:33.373Z
64cb54da1af278541d663708
/avatars/c44507cc92bb2e83154bad31b90ce6dd.svg
Xiaoye Qu
user
Xiaoye08
6791c63e9e215712a7d4bacb
Linjie Li
admin_assigned
2025-01-23T09:37:47.571Z
63db16fff03c3d71ef397206
/avatars/bfb7e0d730b7d03302799d5d2828d97d.svg
Linjie Li
user
linjieli222
6791c63e9e215712a7d4bacc
Yu Cheng
claimed_verified
2025-01-23T15:05:05.310Z
67017abfe4d49b157ac534d9
/avatars/997e1b9f54b27a7728a9d4abfee4ba91.svg
Yu Cheng
user
ych133
Test-Time Preference Optimization: On-the-Fly Alignment via Iterative
  Textual Feedback
Large language models (LLMs) demonstrate impressive performance but lack the
flexibility to adapt to human preferences quickly without retraining. In this
work, we introduce Test-time Preference Optimization (TPO), a framework that
aligns LLM outputs with human preferences during inference, removing the need
to update model parameters. Rather than relying on purely numerical rewards,
TPO translates reward signals into textual critiques and uses them as textual
rewards to iteratively refine its response. Evaluations on benchmarks covering
instruction following, preference alignment, safety, and mathematics reveal
that TPO progressively improves alignment with human preferences. Notably,
after only a few TPO steps, the initially unaligned Llama-3.1-70B-SFT model can
surpass the aligned counterpart, Llama-3.1-70B-Instruct. Furthermore, TPO
scales efficiently with both the search width and depth during inference.
Through case studies, we illustrate how TPO exploits the innate capacity of LLM
to interpret and act upon reward signals. Our findings establish TPO as a
practical, lightweight alternative for test-time preference optimization,
achieving alignment on the fly. Our code is publicly available at
https://github.com/yafuly/TPO.
TPO refines LLM outputs in real-time using textual critiques, achieving preference alignment without retraining.
Test-time Preference Optimization TPO Large language models LLMs instruction following preference alignment safety mathematics search width search depth