title: Conditioned Language Policy: A General Framework for Steerable
    Multi-Objective Finetuning
summary: Reward-based finetuning is crucial for aligning language policies with
  intended behaviors (e.g., creativity and safety). A key challenge here is to
  develop steerable language models that trade-off multiple (conflicting)
  objectives in a flexible and efficient manner. This paper presents Conditioned
  Language Policy (CLP), a general framework for finetuning language models on
  multiple objectives. Building on techniques from multi-task training and
  parameter-efficient finetuning, CLP can learn steerable models that effectively
  trade-off conflicting objectives at inference time. Notably, this does not
  require training or maintaining multiple models to achieve different trade-offs
  between the objectives. Through an extensive set of experiments and ablations,
  we show that the CLP framework learns steerable models that outperform and
  Pareto-dominate the current state-of-the-art approaches for multi-objective
  finetuning.
paper_authors._id: 669f2406ab749e803fe0638e
paper_authors.name: Kaiwen Wang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-07-24T07:27:21.381Z
paper_authors._id: 655b7d6ffb74ac37d3e0f43b
paper_authors.avatarUrl: /avatars/fc525bd38e660134f8a9cd4c9b0ba197.svg
paper_authors.fullname: Kaiwen Wang
paper_authors.type: user
paper_authors.user: kaiwenw
paper_authors._id: 669f2406ab749e803fe0638f
paper_authors.name: Rahul Kidambi
paper_authors._id: 669f2406ab749e803fe06390
paper_authors.name: Ryan Sullivan
paper_authors._id: 669f2406ab749e803fe06391
paper_authors.name: Alekh Agarwal
paper_authors._id: 669f2406ab749e803fe06392
paper_authors.name: Christoph Dann
paper_authors._id: 669f2406ab749e803fe06393
paper_authors.name: Andrea Michi
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-07-23T11:44:56.153Z
paper_authors._id: 66466ef3b080e0f563dd2372
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/66466ef3b080e0f563dd2372/wOMZgsd8r28iwAvqmYQRI.jpeg
paper_authors.fullname: Andrea Michi
paper_authors.type: user
paper_authors.user: andreamichi
paper_authors._id: 669f2406ab749e803fe06394
paper_authors.name: Marco Gelmi
paper_authors._id: 669f2406ab749e803fe06395
paper_authors.name: Yunxuan Li
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-07-23T11:45:17.871Z
paper_authors._id: 63531848fc9b7ad0a53fa564
paper_authors.avatarUrl: /avatars/d8df470349aa976741604921cb408440.svg
paper_authors.fullname: Yunxuan Li
paper_authors.type: user
paper_authors.user: eric008
paper_authors._id: 669f2406ab749e803fe06396
paper_authors.name: Raghav Gupta
paper_authors._id: 669f2406ab749e803fe06397
paper_authors.name: Avinava Dubey
paper_authors._id: 669f2406ab749e803fe06398
paper_authors.name: Alexandre Ramé
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-07-23T11:45:57.166Z
paper_authors._id: 63c94ede00104ea998de19a6
paper_authors.avatarUrl: /avatars/273959d87f0c67747588cf0700d64039.svg
paper_authors.fullname: Alexandre Rame
paper_authors.type: user
paper_authors.user: alexrame
paper_authors._id: 669f2406ab749e803fe06399
paper_authors.name: Johan Ferret
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-07-23T11:46:12.206Z
paper_authors._id: 65afb7dbdd6bdfd73cd8e609
paper_authors.avatarUrl: /avatars/b21069bc2d7ee4cc1508008e3c8ade64.svg
paper_authors.fullname: Johan Ferret
paper_authors.type: user
paper_authors.user: ferretj
paper_authors._id: 669f2406ab749e803fe0639a
paper_authors.name: Geoffrey Cideron
paper_authors._id: 669f2406ab749e803fe0639b
paper_authors.name: Le Hou
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-07-23T11:46:30.636Z
paper_authors._id: 626b2402c804c432c1b5004c
paper_authors.avatarUrl: /avatars/a119644422335e260af0ca4ef6399870.svg
paper_authors.fullname: Le Hou
paper_authors.type: user
paper_authors.user: solooobear
paper_authors._id: 669f2406ab749e803fe0639c
paper_authors.name: Hongkun Yu
paper_authors._id: 669f2406ab749e803fe0639d
paper_authors.name: Amr Ahmed
paper_authors._id: 669f2406ab749e803fe0639e
paper_authors.name: Aranyak Mehta
paper_authors._id: 669f2406ab749e803fe0639f
paper_authors.name: Léonard Hussenot
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-07-23T11:47:18.736Z
paper_authors._id: 667a7b2e1764931987803b46
paper_authors.avatarUrl: /avatars/da2ee162f33c681e3d1c4d1e6d44dcbd.svg
paper_authors.fullname: Léonard Hussenot
paper_authors.type: user
paper_authors.user: leonardhussenot
paper_authors._id: 669f2406ab749e803fe063a0
paper_authors.name: Olivier Bachem
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-07-23T11:47:25.324Z
paper_authors._id: 667aac871bffb68706b4f62c
paper_authors.avatarUrl: /avatars/2e2f2431f03a368f604c992d0d7ca57a.svg
paper_authors.fullname: Olivier Bachem
paper_authors.type: user
paper_authors.user: bachem
paper_authors._id: 669f2406ab749e803fe063a1
paper_authors.name: Edouard Leurent
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-07-23T11:47:32.283Z
paper_authors._id: 633e093315033639f991760f
paper_authors.avatarUrl: /avatars/0339ec428a8d1fb961cb0b095e0839bd.svg
paper_authors.fullname: Edouard Leurent
paper_authors.type: user
paper_authors.user: eleurent
paper_title: Conditioned Language Policy: A General Framework for Steerable
    Multi-Objective Finetuning
paper_summary: Reward-based finetuning is crucial for aligning language policies with
  intended behaviors (e.g., creativity and safety). A key challenge here is to
  develop steerable language models that trade-off multiple (conflicting)
  objectives in a flexible and efficient manner. This paper presents Conditioned
  Language Policy (CLP), a general framework for finetuning language models on
  multiple objectives. Building on techniques from multi-task training and
  parameter-efficient finetuning, CLP can learn steerable models that effectively
  trade-off conflicting objectives at inference time. Notably, this does not
  require training or maintaining multiple models to achieve different trade-offs
  between the objectives. Through an extensive set of experiments and ablations,
  we show that the CLP framework learns steerable models that outperform and
  Pareto-dominate the current state-of-the-art approaches for multi-objective
  finetuning.
paper_ai_keywords: Conditioned Language Policy (CLP) multi-task training parameter-efficient finetuning steerable models trade-off conflicting objectives inference time Pareto-dominate multi-objective finetuning