TITLE: The Conversational Short-phrase Speaker Diarization (CSSD) Task: Dataset, Evaluation Metric and Baselines 

ABSTRACT: The conversation scenario is one of the most important and most challenging
scenarios for speech processing technologies because people in conversation
respond to each other in a casual style. Detecting the speech activities of
each person in a conversation is vital to downstream tasks, like natural
language processing, machine translation, etc. People refer to the detection
technology of "who speak when" as speaker diarization (SD). Traditionally,
diarization error rate (DER) has been used as the standard evaluation metric of
SD systems for a long time. However, DER fails to give enough importance to
short conversational phrases, which are short but important on the semantic
level. Also, a carefully and accurately manually-annotated testing dataset
suitable for evaluating the conversational SD technologies is still unavailable
in the speech community. In this paper, we design and describe the
Conversational Short-phrases Speaker Diarization (CSSD) task, which consists of
training and testing datasets, evaluation metric and baselines. In the dataset
aspect, despite the previously open-sourced 180-hour conversational
MagicData-RAMC dataset, we prepare an individual 20-hour conversational speech
test dataset with carefully and artificially verified speakers timestamps
annotations for the CSSD task. In the metric aspect, we design the new
conversational DER (CDER) evaluation metric, which calculates the SD accuracy
at the utterance level. In the baseline aspect, we adopt a commonly used
method: Variational Bayes HMM x-vector system, as the baseline of the CSSD
task. Our evaluation metric is publicly available at
https://github.com/SpeechClub/CDER_Metric.
The conversation scenario is one of the most important and most challenging
scenarios for speech processing technologies because people in conversation
respond to each other in a casual style. Detecting the speech activities of
each person in a conversation is vital to downstream tasks, like natural
language processing, machine translation, etc. People refer to the detection
technology of "who speak when" as speaker diarization (SD). Traditionally,
diarization error rate (DER) has been used as the standard evaluation metric of
SD systems for a long time. However, DER fails to give enough importance to
short conversational phrases, which are short but important on the semantic
level. Also, a carefully and accurately manually-annotated testing dataset
suitable for evaluating the conversational SD technologies is still unavailable
in the speech community. In this paper, we design and describe the
Conversational Short-phrases Speaker Diarization (CSSD) task, which consists of
training and testing datasets, evaluation metric and baselines. In the dataset
aspect, despite the previously open-sourced 180-hour conversational
MagicData-RAMC dataset, we prepare an individual 20-hour conversational speech
test dataset with carefully and artificially verified speakers timestamps
annotations for the CSSD task. In the metric aspect, we design the new
conversational DER (CDER) evaluation metric, which calculates the SD accuracy
at the utterance level. In the baseline aspect, we adopt a commonly used
method: Variational Bayes HMM x-vector system, as the baseline of the CSSD
task. Our evaluation metric is publicly available at
https://github.com/SpeechClub/CDER_Metric.
The Conversational Short-phrase Speaker Diarization (CSSD) Task: Dataset, Evaluation Metric and Baselines
http://arxiv.org/abs/2208.08042v1