Bootstrapping LLM-based Task-Oriented Dialogue Agents via Self-Talk
Large language models (LLMs) are powerful dialogue agents, but specializing
them towards fulfilling a specific function can be challenging. Instructing
tuning, i.e. tuning models on instruction and sample responses generated by
humans (Ouyang et al., 2022), has proven as an effective method to do so, yet
requires a number of data samples that a) might not be available or b) costly
to generate. Furthermore, this cost increases when the goal is to make the LLM
follow a specific workflow within a dialogue instead of single instructions.
Inspired by the self-play technique in reinforcement learning and the use of
LLMs to simulate human agents, we propose a more effective method for data
collection through LLMs engaging in a conversation in various roles. This
approach generates a training data via "self-talk" of LLMs that can be refined
and utilized for supervised fine-tuning. We introduce an automated way to
measure the (partial) success of a dialogue. This metric is used to filter the
generated conversational data that is fed back in LLM for training. Based on
our automated and human evaluations of conversation quality, we demonstrate
that such self-talk data improves results. In addition, we examine the various
characteristics that showcase the quality of generated dialogues and how they
can be connected to their potential utility as training data.
659f5931210bd4a54e7fb38a
Dennis Ulmer
admin_assigned
2024-01-11T10:59:25.771Z
6389e86a92ad533329bf00fe
https://cdn-avatars.huggingface.co/v1/production/uploads/1669982276443-noauth.jpeg
Dennis Ulmer
user
kaleidophon
659f5931210bd4a54e7fb38b
Elman Mansimov
admin_assigned
2024-01-11T10:59:35.919Z
61f352fb4eba72baea21ea1e
https://cdn-avatars.huggingface.co/v1/production/uploads/1643336441806-noauth.png
Elman Mansimov
user
mansimov
659f5931210bd4a54e7fb38c
Kaixiang Lin
admin_assigned
2024-01-11T10:59:47.945Z
64f7f197b2c85cf9e5f4a1be
/avatars/645fd920492c16c90f86fffc0823a997.svg
Lin Kaixiang
user
LeoSuk
659f5931210bd4a54e7fb38d
Justin Sun
659f5931210bd4a54e7fb38e
Xibin Gao
659f5931210bd4a54e7fb38f
Yi Zhang
Bootstrapping LLM-based Task-Oriented Dialogue Agents via Self-Talk
Large language models (LLMs) are powerful dialogue agents, but specializing
them towards fulfilling a specific function can be challenging. Instructing
tuning, i.e. tuning models on instruction and sample responses generated by
humans (Ouyang et al., 2022), has proven as an effective method to do so, yet
requires a number of data samples that a) might not be available or b) costly
to generate. Furthermore, this cost increases when the goal is to make the LLM
follow a specific workflow within a dialogue instead of single instructions.
Inspired by the self-play technique in reinforcement learning and the use of
LLMs to simulate human agents, we propose a more effective method for data
collection through LLMs engaging in a conversation in various roles. This
approach generates a training data via "self-talk" of LLMs that can be refined
and utilized for supervised fine-tuning. We introduce an automated way to
measure the (partial) success of a dialogue. This metric is used to filter the
generated conversational data that is fed back in LLM for training. Based on
our automated and human evaluations of conversation quality, we demonstrate
that such self-talk data improves results. In addition, we examine the various
characteristics that showcase the quality of generated dialogues and how they
can be connected to their potential utility as training data.
LLMs generate high-quality dialogue training data through self-simulated conversations, improving fine-tuning results and evaluated by automated metrics.
large language models instructing tuning self-play technique reinforcement learning supervised fine-tuning dialogue quality automated evaluation