Adaptive Decoding via Latent Preference Optimization
During language model decoding, it is known that using higher temperature
sampling gives more creative responses, while lower temperatures are more
factually accurate. However, such models are commonly applied to general
instruction following, which involves both creative and fact seeking tasks,
using a single fixed temperature across all examples and tokens. In this work,
we introduce Adaptive Decoding, a layer added to the model to select the
sampling temperature dynamically at inference time, at either the token or
example level, in order to optimize performance. To learn its parameters we
introduce Latent Preference Optimization (LPO) a general approach to train
discrete latent variables such as choices of temperature. Our method
outperforms all fixed decoding temperatures across a range of tasks that
require different temperatures, including UltraFeedback, Creative Story
Writing, and GSM8K.
673890125ca44c7491b36202
Shehzaad Dhuliawala
admin_assigned
2024-11-19T16:29:29.350Z
6325db12ddb9ff68f4698292
/avatars/88e9f57a17bc9b41ddad508e7b1d2640.svg
Shehzaad Dhuliawala
user
shehzaadzd
673890125ca44c7491b36203
Ilia Kulikov
admin_assigned
2024-11-19T16:29:34.928Z
65d645913bc37244c55c39d5
/avatars/508372876a654957650ee9429886d10c.svg
ilia kulikov
user
uralik
673890125ca44c7491b36204
Ping Yu
admin_assigned
2024-11-19T16:29:53.452Z
646c004e31968a60a022590e
/avatars/349e2ed4feef1be050fd9903dccec1e1.svg
Ping Yu
user
pingyu
673890125ca44c7491b36205
Asli Celikyilmaz
673890125ca44c7491b36206
Jason Weston
admin_assigned
2024-11-19T16:30:07.574Z
62f023a36a027498eaa2f9cc
/avatars/8ac1c5c74d0957e3c6cc94b3a7795c37.svg
Jason Weston
user
spermwhale
673890125ca44c7491b36207
Sainbayar Sukhbaatar
admin_assigned
2024-11-19T16:30:15.503Z
66a8611eb51510d82ed54231
/avatars/ad559e774fee4914091b82c9831ae2a2.svg
Sainbayar Sukhbaatar
user
sainbar
673890125ca44c7491b36208
Jack Lanchantin
claimed_verified
2024-11-20T10:09:12.355Z
65dd30a6ea1a202d3cb469f7
/avatars/14029def026380e6e20822d916b25a72.svg
Jack Lanchantin
user
jcklcn
Adaptive Decoding via Latent Preference Optimization
During language model decoding, it is known that using higher temperature
sampling gives more creative responses, while lower temperatures are more
factually accurate. However, such models are commonly applied to general
instruction following, which involves both creative and fact seeking tasks,
using a single fixed temperature across all examples and tokens. In this work,
we introduce Adaptive Decoding, a layer added to the model to select the
sampling temperature dynamically at inference time, at either the token or
example level, in order to optimize performance. To learn its parameters we
introduce Latent Preference Optimization (LPO) a general approach to train
discrete latent variables such as choices of temperature. Our method
outperforms all fixed decoding temperatures across a range of tasks that
require different temperatures, including UltraFeedback, Creative Story
Writing, and GSM8K.
Adaptive Decoding with Latent Preference Optimization dynamically adjusts sampling temperature during model inference, improving performance across tasks requiring varied temperature settings.
Adaptive Decoding Latent Preference Optimization sampling temperature inference time discrete latent variables UltraFeedback Creative Story Writing GSM8K