C3: A Bilingual Benchmark for Spoken Dialogue Models Exploring
  Challenges in Complex Conversations
Spoken Dialogue Models (SDMs) have recently attracted significant attention
for their ability to generate voice responses directly to users' spoken
queries. Despite their increasing popularity, there exists a gap in research
focused on comprehensively understanding their practical effectiveness in
comprehending and emulating human conversations. This is especially true
compared to text-based Large Language Models (LLMs), which benefit from
extensive benchmarking. Human voice interactions are inherently more complex
than text due to characteristics unique to spoken dialogue. Ambiguity poses one
challenge, stemming from semantic factors like polysemy, as well as
phonological aspects such as heterograph, heteronyms, and stress patterns.
Additionally, context-dependency, like omission, coreference, and multi-turn
interaction, adds further complexity to human conversational dynamics. To
illuminate the current state of SDM development and to address these
challenges, we present a benchmark dataset in this paper, which comprises 1,079
instances in English and Chinese. Accompanied by an LLM-based evaluation method
that closely aligns with human judgment, this dataset facilitates a
comprehensive exploration of the performance of SDMs in tackling these
practical challenges.
688c13408c434640078cc335
Chengqian Ma
688c13408c434640078cc336
Wei Tao
688c13408c434640078cc337
Yiwen Guo
C3: A Bilingual Benchmark for Spoken Dialogue Models Exploring
  Challenges in Complex Conversations
Spoken Dialogue Models (SDMs) have recently attracted significant attention
for their ability to generate voice responses directly to users' spoken
queries. Despite their increasing popularity, there exists a gap in research
focused on comprehensively understanding their practical effectiveness in
comprehending and emulating human conversations. This is especially true
compared to text-based Large Language Models (LLMs), which benefit from
extensive benchmarking. Human voice interactions are inherently more complex
than text due to characteristics unique to spoken dialogue. Ambiguity poses one
challenge, stemming from semantic factors like polysemy, as well as
phonological aspects such as heterograph, heteronyms, and stress patterns.
Additionally, context-dependency, like omission, coreference, and multi-turn
interaction, adds further complexity to human conversational dynamics. To
illuminate the current state of SDM development and to address these
challenges, we present a benchmark dataset in this paper, which comprises 1,079
instances in English and Chinese. Accompanied by an LLM-based evaluation method
that closely aligns with human judgment, this dataset facilitates a
comprehensive exploration of the performance of SDMs in tackling these
practical challenges.
A benchmark dataset for Spoken Dialogue Models (SDMs) in English and Chinese is presented to evaluate their performance in understanding and emulating human conversations, addressing challenges like ambiguity and context-dependency.
Spoken Dialogue Models SDMs Large Language Models LLMs benchmark dataset human judgment ambiguity polysemy heterograph heteronyms stress patterns context-dependency omission coreference multi-turn interaction