Provably Learning from Language Feedback
Interactively learning from observation and language feedback is an
increasingly studied area driven by the emergence of large language model (LLM)
agents. While impressive empirical demonstrations have been shown, so far a
principled framing of these decision problems remains lacking. In this paper,
we formalize the Learning from Language Feedback (LLF) problem, assert
sufficient assumptions to enable learning despite latent rewards, and introduce
transfer eluder dimension as a complexity measure to characterize
the hardness of LLF problems. We show that transfer eluder dimension captures
the intuition that information in the feedback changes the learning complexity
of the LLF problem. We demonstrate cases where learning from rich language
feedback can be exponentially faster than learning from reward. We develop a
no-regret algorithm, called HELiX, that provably solves LLF problems
through sequential interactions, with performance guarantees that scale with
the transfer eluder dimension of the problem. Across several empirical domains,
we show that HELiX performs well even when repeatedly prompting LLMs
does not work reliably. Our contributions mark a first step towards designing
principled interactive learning algorithms from generic language feedback.
684c9be23b733ba3336872f5
Wanqiao Xu
684c9be23b733ba3336872f6
Allen Nie
684c9be23b733ba3336872f7
Ruijie Zheng
684c9be23b733ba3336872f8
Aditya Modi
684c9be23b733ba3336872f9
Adith Swaminathan
684c9be23b733ba3336872fa
Ching-An Cheng
Provably Learning from Language Feedback
Interactively learning from observation and language feedback is an
increasingly studied area driven by the emergence of large language model (LLM)
agents. While impressive empirical demonstrations have been shown, so far a
principled framing of these decision problems remains lacking. In this paper,
we formalize the Learning from Language Feedback (LLF) problem, assert
sufficient assumptions to enable learning despite latent rewards, and introduce
transfer eluder dimension as a complexity measure to characterize
the hardness of LLF problems. We show that transfer eluder dimension captures
the intuition that information in the feedback changes the learning complexity
of the LLF problem. We demonstrate cases where learning from rich language
feedback can be exponentially faster than learning from reward. We develop a
no-regret algorithm, called HELiX, that provably solves LLF problems
through sequential interactions, with performance guarantees that scale with
the transfer eluder dimension of the problem. Across several empirical domains,
we show that HELiX performs well even when repeatedly prompting LLMs
does not work reliably. Our contributions mark a first step towards designing
principled interactive learning algorithms from generic language feedback.
A formal framework and no-regret algorithm are introduced for learning from language feedback, addressing challenges in interactive learning with large language models.
Learning from Language Feedback (LLF) large language model (LLM) agents latent rewards transfer eluder dimension no-regret algorithm HELiX