DialSim: A Real-Time Simulator for Evaluating Long-Term Dialogue
  Understanding of Conversational Agents
Recent advancements in Large Language Models (LLMs) have significantly
enhanced the capabilities of conversational agents, making them applicable to
various fields (e.g., education). Despite their progress, the evaluation of the
agents often overlooks the complexities of real-world conversations, such as
real-time interactions, multi-party dialogues, and extended contextual
dependencies. To bridge this gap, we introduce DialSim, a real-time dialogue
simulator. In this simulator, an agent is assigned the role of a character from
popular TV shows, requiring it to respond to spontaneous questions using past
dialogue information and to distinguish between known and unknown information.
Key features of DialSim include evaluating the agent's ability to respond
within a reasonable time limit, handling long-term multi-party dialogues, and
managing adversarial settings (e.g., swap character names) to challenge the
agent's reliance on pre-trained knowledge. We utilized this simulator to
evaluate the latest conversational agents and analyze their limitations. Our
experiments highlight both the strengths and weaknesses of these agents,
providing valuable insights for future improvements in the field of
conversational AI. DialSim is available at
https://github.com/jiho283/Simulator.
667ba21253401797def65940
Jiho Kim
claimed_verified
2024-06-26T08:03:09.335Z
6630eaa52d2b3451e7c0f61e
/avatars/d22517d4833f4255fd62fda5280bd8ca.svg
Jiho Kim
user
jiho283
667ba21253401797def65941
Woosog Chay
claimed_verified
2024-07-04T12:00:08.266Z
65e57ed5a5cb4780919f5d3c
/avatars/b0ada760adb6efc9e306a0a062d18551.svg
Woosog Chay
user
wbchay
667ba21253401797def65942
Hyeonji Hwang
admin_assigned
2024-06-26T13:21:02.798Z
65fbe95104769daf2192f836
/avatars/c2b5386a4c5c74c5dc365447141551ba.svg
hyeonji hwang
user
hyeonz
667ba21253401797def65943
Daeun Kyung
admin_assigned
2024-06-26T13:21:08.357Z
645cd00f5ebf379fd6d7a4c1
/avatars/be8375ed32ac234919e26a2450bf9d38.svg
Daeun Kyung
user
dek924
667ba21253401797def65944
Hyunseung Chung
667ba21253401797def65945
Eunbyeol Cho
667ba21253401797def65946
Yohan Jo
667ba21253401797def65947
Edward Choi
admin_assigned
2024-06-26T13:21:36.315Z
65e8fb912dcdb39d1934260c
/avatars/358504b57f2a600753e9bc459091061b.svg
Edward Choi
user
forgetnight
DialSim: A Real-Time Simulator for Evaluating Long-Term Dialogue
  Understanding of Conversational Agents
Recent advancements in Large Language Models (LLMs) have significantly
enhanced the capabilities of conversational agents, making them applicable to
various fields (e.g., education). Despite their progress, the evaluation of the
agents often overlooks the complexities of real-world conversations, such as
real-time interactions, multi-party dialogues, and extended contextual
dependencies. To bridge this gap, we introduce DialSim, a real-time dialogue
simulator. In this simulator, an agent is assigned the role of a character from
popular TV shows, requiring it to respond to spontaneous questions using past
dialogue information and to distinguish between known and unknown information.
Key features of DialSim include evaluating the agent's ability to respond
within a reasonable time limit, handling long-term multi-party dialogues, and
managing adversarial settings (e.g., swap character names) to challenge the
agent's reliance on pre-trained knowledge. We utilized this simulator to
evaluate the latest conversational agents and analyze their limitations. Our
experiments highlight both the strengths and weaknesses of these agents,
providing valuable insights for future improvements in the field of
conversational AI. DialSim is available at
https://github.com/jiho283/Simulator.
DialSim evaluates conversational agents through real-time, multi-party dialogues with adversarial challenges, offering insights into their strengths and weaknesses.
Large Language Models conversational agents real-time interactions multi-party dialogues extended contextual dependencies dialogue simulator adversarial settings