title: Direct Preference Optimization Using Sparse Feature-Level Constraints
summary: The alignment of large language models (LLMs) with human preferences remains
  a key challenge. While post-training techniques like Reinforcement Learning
  from Human Feedback (RLHF) and Direct Preference Optimization (DPO) have
  achieved notable success, they often introduce computational inefficiencies and
  training instability. In this paper, we propose Feature-level constrained
  Preference Optimization (FPO), a novel method designed to simplify the
  alignment process while ensuring stability. FPO leverages pre-trained Sparse
  Autoencoders (SAEs) and introduces feature-level constraints, allowing for
  efficient, sparsity-enforced alignment. Our approach enjoys efficiency by using
  sparse features activated in a well-trained sparse autoencoder and the quality
  of sequential KL divergence by using the feature-level offline reference.
  Experimental results on benchmark datasets demonstrate that FPO achieves a
  5.08% absolute improvement in win rate with much lower computational cost
  compared to state-of-the-art baselines, making it a promising solution for
  efficient and controllable LLM alignments.
paper_authors._id: 673575eee12f354dfc1ea6f8
paper_authors.name: Qingyu Yin
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-11-15T14:16:41.130Z
paper_authors._id: 6453cb22908e259483c0a061
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/6453cb22908e259483c0a061/hMgdwZUsUbgquGalzPGzV.jpeg
paper_authors.fullname: Qingyu_Yin
paper_authors.type: user
paper_authors.user: MikaStars39
paper_authors._id: 673575eee12f354dfc1ea6f9
paper_authors.name: Chak Tou Leong
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-11-15T09:22:14.403Z
paper_authors._id: 631326d6289cf15634c52369
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/631326d6289cf15634c52369/lmPWGHLsQ36H556cqcXjT.jpeg
paper_authors.fullname: Cooper Leong
paper_authors.type: user
paper_authors.user: cooperleong00
paper_authors._id: 673575eee12f354dfc1ea6fa
paper_authors.name: Hongbo Zhang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-11-15T14:17:14.035Z
paper_authors._id: 660687a8cd1b5614c65e257b
paper_authors.avatarUrl: /avatars/64ef0ff1d5bff7ff5aadd9070abbff96.svg
paper_authors.fullname: zhanghongbo
paper_authors.type: user
paper_authors.user: loewen-hob
paper_authors._id: 673575eee12f354dfc1ea6fb
paper_authors.name: Minjun Zhu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-11-15T14:17:27.675Z
paper_authors._id: 622ee9f3165ba2c1bcbc7706
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/622ee9f3165ba2c1bcbc7706/4BIp_juKHqXgnctnNIFJf.jpeg
paper_authors.fullname: MinjunZhu
paper_authors.type: user
paper_authors.user: MinjunZhu
paper_authors._id: 673575eee12f354dfc1ea6fc
paper_authors.name: Hanqi Yan
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-11-15T14:17:20.271Z
paper_authors._id: 654d3672f0f195070248d80b
paper_authors.avatarUrl: /avatars/8f54607d3189690b90e4fe5b6fba94f1.svg
paper_authors.fullname: hanqi yan
paper_authors.type: user
paper_authors.user: hanqiyan
paper_authors._id: 673575eee12f354dfc1ea6fd
paper_authors.name: Qiang Zhang
paper_authors._id: 673575eee12f354dfc1ea6fe
paper_authors.name: Yulan He
paper_authors._id: 673575eee12f354dfc1ea6ff
paper_authors.name: Wenjie Li
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-11-15T14:18:04.907Z
paper_authors._id: 66a3710a4ee2a4c936315a5a
paper_authors.avatarUrl: /avatars/ef8da8fb1031695d77d34a5d365aa177.svg
paper_authors.fullname: Li
paper_authors.type: user
paper_authors.user: WenjieLi
paper_authors._id: 673575eee12f354dfc1ea700
paper_authors.name: Jun Wang
paper_authors._id: 673575eee12f354dfc1ea701
paper_authors.name: Yue Zhang
paper_authors._id: 673575eee12f354dfc1ea702
paper_authors.name: Linyi Yang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-11-15T14:18:24.971Z
paper_authors._id: 64895683f534abe18eec264b
paper_authors.avatarUrl: /avatars/73cc9e6db6db86793787750776b57c63.svg
paper_authors.fullname: Linyi Yang
paper_authors.type: user
paper_authors.user: linyiyang2023
paper_title: Direct Preference Optimization Using Sparse Feature-Level Constraints
paper_summary: The alignment of large language models (LLMs) with human preferences remains
  a key challenge. While post-training techniques like Reinforcement Learning
  from Human Feedback (RLHF) and Direct Preference Optimization (DPO) have
  achieved notable success, they often introduce computational inefficiencies and
  training instability. In this paper, we propose Feature-level constrained
  Preference Optimization (FPO), a novel method designed to simplify the
  alignment process while ensuring stability. FPO leverages pre-trained Sparse
  Autoencoders (SAEs) and introduces feature-level constraints, allowing for
  efficient, sparsity-enforced alignment. Our approach enjoys efficiency by using
  sparse features activated in a well-trained sparse autoencoder and the quality
  of sequential KL divergence by using the feature-level offline reference.
  Experimental results on benchmark datasets demonstrate that FPO achieves a
  5.08% absolute improvement in win rate with much lower computational cost
  compared to state-of-the-art baselines, making it a promising solution for
  efficient and controllable LLM alignments.
paper_ai_summary: A novel method, Feature-level constrained Preference Optimization (FPO), improves the alignment of large language models with human preferences by using sparse features from Sparse Autoencoders and feature-level offline reference, achieving better efficiency and win rate.
paper_ai_keywords: Large language models Reinforcement Learning from Human Feedback (RLHF) Direct Preference Optimization (DPO) Feature-level constrained Preference Optimization (FPO) Sparse Autoencoders (SAEs) feature-level constraints sequential KL divergence feature-level offline reference