title: DialSim: A Real-Time Simulator for Evaluating Long-Term Dialogue
    Understanding of Conversational Agents
summary: Recent advancements in Large Language Models (LLMs) have significantly
  enhanced the capabilities of conversational agents, making them applicable to
  various fields (e.g., education). Despite their progress, the evaluation of the
  agents often overlooks the complexities of real-world conversations, such as
  real-time interactions, multi-party dialogues, and extended contextual
  dependencies. To bridge this gap, we introduce DialSim, a real-time dialogue
  simulator. In this simulator, an agent is assigned the role of a character from
  popular TV shows, requiring it to respond to spontaneous questions using past
  dialogue information and to distinguish between known and unknown information.
  Key features of DialSim include evaluating the agent's ability to respond
  within a reasonable time limit, handling long-term multi-party dialogues, and
  managing adversarial settings (e.g., swap character names) to challenge the
  agent's reliance on pre-trained knowledge. We utilized this simulator to
  evaluate the latest conversational agents and analyze their limitations. Our
  experiments highlight both the strengths and weaknesses of these agents,
  providing valuable insights for future improvements in the field of
  conversational AI. DialSim is available at
  https://github.com/jiho283/Simulator.
paper_authors._id: 667ba21253401797def65940
paper_authors.name: Jiho Kim
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-06-26T08:03:09.335Z
paper_authors._id: 6630eaa52d2b3451e7c0f61e
paper_authors.avatarUrl: /avatars/d22517d4833f4255fd62fda5280bd8ca.svg
paper_authors.fullname: Jiho Kim
paper_authors.type: user
paper_authors.user: jiho283
paper_authors._id: 667ba21253401797def65941
paper_authors.name: Woosog Chay
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-07-04T12:00:08.266Z
paper_authors._id: 65e57ed5a5cb4780919f5d3c
paper_authors.avatarUrl: /avatars/b0ada760adb6efc9e306a0a062d18551.svg
paper_authors.fullname: Woosog Chay
paper_authors.type: user
paper_authors.user: wbchay
paper_authors._id: 667ba21253401797def65942
paper_authors.name: Hyeonji Hwang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-06-26T13:21:02.798Z
paper_authors._id: 65fbe95104769daf2192f836
paper_authors.avatarUrl: /avatars/c2b5386a4c5c74c5dc365447141551ba.svg
paper_authors.fullname: hyeonji hwang
paper_authors.type: user
paper_authors.user: hyeonz
paper_authors._id: 667ba21253401797def65943
paper_authors.name: Daeun Kyung
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-06-26T13:21:08.357Z
paper_authors._id: 645cd00f5ebf379fd6d7a4c1
paper_authors.avatarUrl: /avatars/be8375ed32ac234919e26a2450bf9d38.svg
paper_authors.fullname: Daeun Kyung
paper_authors.type: user
paper_authors.user: dek924
paper_authors._id: 667ba21253401797def65944
paper_authors.name: Hyunseung Chung
paper_authors._id: 667ba21253401797def65945
paper_authors.name: Eunbyeol Cho
paper_authors._id: 667ba21253401797def65946
paper_authors.name: Yohan Jo
paper_authors._id: 667ba21253401797def65947
paper_authors.name: Edward Choi
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-06-26T13:21:36.315Z
paper_authors._id: 65e8fb912dcdb39d1934260c
paper_authors.avatarUrl: /avatars/358504b57f2a600753e9bc459091061b.svg
paper_authors.fullname: Edward Choi
paper_authors.type: user
paper_authors.user: forgetnight
paper_title: DialSim: A Real-Time Simulator for Evaluating Long-Term Dialogue
    Understanding of Conversational Agents
paper_summary: Recent advancements in Large Language Models (LLMs) have significantly
  enhanced the capabilities of conversational agents, making them applicable to
  various fields (e.g., education). Despite their progress, the evaluation of the
  agents often overlooks the complexities of real-world conversations, such as
  real-time interactions, multi-party dialogues, and extended contextual
  dependencies. To bridge this gap, we introduce DialSim, a real-time dialogue
  simulator. In this simulator, an agent is assigned the role of a character from
  popular TV shows, requiring it to respond to spontaneous questions using past
  dialogue information and to distinguish between known and unknown information.
  Key features of DialSim include evaluating the agent's ability to respond
  within a reasonable time limit, handling long-term multi-party dialogues, and
  managing adversarial settings (e.g., swap character names) to challenge the
  agent's reliance on pre-trained knowledge. We utilized this simulator to
  evaluate the latest conversational agents and analyze their limitations. Our
  experiments highlight both the strengths and weaknesses of these agents,
  providing valuable insights for future improvements in the field of
  conversational AI. DialSim is available at
  https://github.com/jiho283/Simulator.
paper_ai_keywords: Large Language Models (LLMs) conversational agents real-world conversations real-time interactions multi-party dialogues extended contextual dependencies AI-generated image simulator real-time dialogue dialogue information known information unknown information computational pre-trained knowledge conversational worlds adversarial settings long-term dialogues out-of-distribution generalization pre-trained knowledge dialogue simulator state-of-the-art conversational AI