Conditioned Language Policy: A General Framework for Steerable
  Multi-Objective Finetuning
Reward-based finetuning is crucial for aligning language policies with
intended behaviors (e.g., creativity and safety). A key challenge here is to
develop steerable language models that trade-off multiple (conflicting)
objectives in a flexible and efficient manner. This paper presents Conditioned
Language Policy (CLP), a general framework for finetuning language models on
multiple objectives. Building on techniques from multi-task training and
parameter-efficient finetuning, CLP can learn steerable models that effectively
trade-off conflicting objectives at inference time. Notably, this does not
require training or maintaining multiple models to achieve different trade-offs
between the objectives. Through an extensive set of experiments and ablations,
we show that the CLP framework learns steerable models that outperform and
Pareto-dominate the current state-of-the-art approaches for multi-objective
finetuning.
669f2406ab749e803fe0638e
Kaiwen Wang
claimed_verified
2024-07-24T07:27:21.381Z
655b7d6ffb74ac37d3e0f43b
/avatars/fc525bd38e660134f8a9cd4c9b0ba197.svg
Kaiwen Wang
user
kaiwenw
669f2406ab749e803fe0638f
Rahul Kidambi
669f2406ab749e803fe06390
Ryan Sullivan
669f2406ab749e803fe06391
Alekh Agarwal
669f2406ab749e803fe06392
Christoph Dann
669f2406ab749e803fe06393
Andrea Michi
admin_assigned
2024-07-23T11:44:56.153Z
66466ef3b080e0f563dd2372
https://cdn-avatars.huggingface.co/v1/production/uploads/66466ef3b080e0f563dd2372/wOMZgsd8r28iwAvqmYQRI.jpeg
Andrea Michi
user
andreamichi
669f2406ab749e803fe06394
Marco Gelmi
669f2406ab749e803fe06395
Yunxuan Li
admin_assigned
2024-07-23T11:45:17.871Z
63531848fc9b7ad0a53fa564
/avatars/d8df470349aa976741604921cb408440.svg
Yunxuan Li
user
eric008
669f2406ab749e803fe06396
Raghav Gupta
669f2406ab749e803fe06397
Avinava Dubey
669f2406ab749e803fe06398
Alexandre Ramé
admin_assigned
2024-07-23T11:45:57.166Z
63c94ede00104ea998de19a6
/avatars/273959d87f0c67747588cf0700d64039.svg
Alexandre Rame
user
alexrame
669f2406ab749e803fe06399
Johan Ferret
admin_assigned
2024-07-23T11:46:12.206Z
65afb7dbdd6bdfd73cd8e609
/avatars/b21069bc2d7ee4cc1508008e3c8ade64.svg
Johan Ferret
user
ferretj
669f2406ab749e803fe0639a
Geoffrey Cideron
669f2406ab749e803fe0639b
Le Hou
admin_assigned
2024-07-23T11:46:30.636Z
626b2402c804c432c1b5004c
/avatars/a119644422335e260af0ca4ef6399870.svg
Le Hou
user
solooobear
669f2406ab749e803fe0639c
Hongkun Yu
669f2406ab749e803fe0639d
Amr Ahmed
669f2406ab749e803fe0639e
Aranyak Mehta
669f2406ab749e803fe0639f
Léonard Hussenot
admin_assigned
2024-07-23T11:47:18.736Z
667a7b2e1764931987803b46
/avatars/da2ee162f33c681e3d1c4d1e6d44dcbd.svg
Léonard Hussenot
user
leonardhussenot
669f2406ab749e803fe063a0
Olivier Bachem
admin_assigned
2024-07-23T11:47:25.324Z
667aac871bffb68706b4f62c
/avatars/2e2f2431f03a368f604c992d0d7ca57a.svg
Olivier Bachem
user
bachem
669f2406ab749e803fe063a1
Edouard Leurent
admin_assigned
2024-07-23T11:47:32.283Z
633e093315033639f991760f
/avatars/0339ec428a8d1fb961cb0b095e0839bd.svg
Edouard Leurent
user
eleurent
Conditioned Language Policy: A General Framework for Steerable
  Multi-Objective Finetuning
Reward-based finetuning is crucial for aligning language policies with
intended behaviors (e.g., creativity and safety). A key challenge here is to
develop steerable language models that trade-off multiple (conflicting)
objectives in a flexible and efficient manner. This paper presents Conditioned
Language Policy (CLP), a general framework for finetuning language models on
multiple objectives. Building on techniques from multi-task training and
parameter-efficient finetuning, CLP can learn steerable models that effectively
trade-off conflicting objectives at inference time. Notably, this does not
require training or maintaining multiple models to achieve different trade-offs
between the objectives. Through an extensive set of experiments and ablations,
we show that the CLP framework learns steerable models that outperform and
Pareto-dominate the current state-of-the-art approaches for multi-objective
finetuning.
A Conditioned Language Policy framework enables efficient multi-objective finetuning of language models, achieving superior performance in balancing conflicting objectives.
steerable language models multi-objective finetuning Conditioned Language Policy (CLP) multi-task training parameter-efficient finetuning Pareto-dominate