title: Test-Time Preference Optimization: On-the-Fly Alignment via Iterative
    Textual Feedback
summary: Large language models (LLMs) demonstrate impressive performance but lack the
  flexibility to adapt to human preferences quickly without retraining. In this
  work, we introduce Test-time Preference Optimization (TPO), a framework that
  aligns LLM outputs with human preferences during inference, removing the need
  to update model parameters. Rather than relying on purely numerical rewards,
  TPO translates reward signals into textual critiques and uses them as textual
  rewards to iteratively refine its response. Evaluations on benchmarks covering
  instruction following, preference alignment, safety, and mathematics reveal
  that TPO progressively improves alignment with human preferences. Notably,
  after only a few TPO steps, the initially unaligned Llama-3.1-70B-SFT model can
  surpass the aligned counterpart, Llama-3.1-70B-Instruct. Furthermore, TPO
  scales efficiently with both the search width and depth during inference.
  Through case studies, we illustrate how TPO exploits the innate capacity of LLM
  to interpret and act upon reward signals. Our findings establish TPO as a
  practical, lightweight alternative for test-time preference optimization,
  achieving alignment on the fly. Our code is publicly available at
  https://github.com/yafuly/TPO.
paper_authors._id: 6791c63e9e215712a7d4bac8
paper_authors.name: Yafu Li
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-01-23T09:37:06.794Z
paper_authors._id: 63f3502a520c14618925825a
paper_authors.avatarUrl: /avatars/e986a2a6625e7be6890616a417f908d2.svg
paper_authors.fullname: Yafu Li
paper_authors.type: user
paper_authors.user: yaful
paper_authors._id: 6791c63e9e215712a7d4bac9
paper_authors.name: Xuyang Hu
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-01-23T09:56:39.062Z
paper_authors._id: 6498fde776d49ee00f79cbfe
paper_authors.avatarUrl: /avatars/4c284a71080150e6cb3b9632dfccef60.svg
paper_authors.fullname: Xuyang Hu
paper_authors.type: user
paper_authors.user: huxy912
paper_authors._id: 6791c63e9e215712a7d4baca
paper_authors.name: Xiaoye Qu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-01-23T09:37:33.373Z
paper_authors._id: 64cb54da1af278541d663708
paper_authors.avatarUrl: /avatars/c44507cc92bb2e83154bad31b90ce6dd.svg
paper_authors.fullname: Xiaoye Qu
paper_authors.type: user
paper_authors.user: Xiaoye08
paper_authors._id: 6791c63e9e215712a7d4bacb
paper_authors.name: Linjie Li
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-01-23T09:37:47.571Z
paper_authors._id: 63db16fff03c3d71ef397206
paper_authors.avatarUrl: /avatars/bfb7e0d730b7d03302799d5d2828d97d.svg
paper_authors.fullname: Linjie Li
paper_authors.type: user
paper_authors.user: linjieli222
paper_authors._id: 6791c63e9e215712a7d4bacc
paper_authors.name: Yu Cheng
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-01-23T15:05:05.310Z
paper_authors._id: 67017abfe4d49b157ac534d9
paper_authors.avatarUrl: /avatars/997e1b9f54b27a7728a9d4abfee4ba91.svg
paper_authors.fullname: Yu Cheng
paper_authors.type: user
paper_authors.user: ych133
paper_title: Test-Time Preference Optimization: On-the-Fly Alignment via Iterative
    Textual Feedback
paper_summary: Large language models (LLMs) demonstrate impressive performance but lack the
  flexibility to adapt to human preferences quickly without retraining. In this
  work, we introduce Test-time Preference Optimization (TPO), a framework that
  aligns LLM outputs with human preferences during inference, removing the need
  to update model parameters. Rather than relying on purely numerical rewards,
  TPO translates reward signals into textual critiques and uses them as textual
  rewards to iteratively refine its response. Evaluations on benchmarks covering
  instruction following, preference alignment, safety, and mathematics reveal
  that TPO progressively improves alignment with human preferences. Notably,
  after only a few TPO steps, the initially unaligned Llama-3.1-70B-SFT model can
  surpass the aligned counterpart, Llama-3.1-70B-Instruct. Furthermore, TPO
  scales efficiently with both the search width and depth during inference.
  Through case studies, we illustrate how TPO exploits the innate capacity of LLM
  to interpret and act upon reward signals. Our findings establish TPO as a
  practical, lightweight alternative for test-time preference optimization,
  achieving alignment on the fly. Our code is publicly available at
  https://github.com/yafuly/TPO.
paper_ai_keywords: Test-time Preference Optimization (TPO) sequence-to-sequence textual critiques textual rewards instruction following preference alignment safety mathematics search width search depth interpret act upon reward signals