title: MIKU-PAL: An Automated and Standardized Multi-Modal Method for Speech
    Paralinguistic and Affect Labeling
summary: Acquiring large-scale emotional speech data with strong consistency remains a
  challenge for speech synthesis. This paper presents MIKU-PAL, a fully automated
  multimodal pipeline for extracting high-consistency emotional speech from
  unlabeled video data. Leveraging face detection and tracking algorithms, we
  developed an automatic emotion analysis system using a multimodal large
  language model (MLLM). Our results demonstrate that MIKU-PAL can achieve
  human-level accuracy (68.5% on MELD) and superior consistency (0.93 Fleiss
  kappa score) while being much cheaper and faster than human annotation. With
  the high-quality, flexible, and consistent annotation from MIKU-PAL, we can
  annotate fine-grained speech emotion categories of up to 26 types, validated by
  human annotators with 83% rationality ratings. Based on our proposed system, we
  further released a fine-grained emotional speech dataset MIKU-EmoBench(131.2
  hours) as a new benchmark for emotional text-to-speech and visual voice
  cloning.
paper_authors._id: 683ec87a4246cd3c413046e1
paper_authors.name: Yifan Cheng
paper_authors._id: 683ec87a4246cd3c413046e2
paper_authors.name: Ruoyi Zhang
paper_authors._id: 683ec87a4246cd3c413046e3
paper_authors.name: Jiatong Shi
paper_title: MIKU-PAL: An Automated and Standardized Multi-Modal Method for Speech
    Paralinguistic and Affect Labeling
paper_summary: Acquiring large-scale emotional speech data with strong consistency remains a
  challenge for speech synthesis. This paper presents MIKU-PAL, a fully automated
  multimodal pipeline for extracting high-consistency emotional speech from
  unlabeled video data. Leveraging face detection and tracking algorithms, we
  developed an automatic emotion analysis system using a multimodal large
  language model (MLLM). Our results demonstrate that MIKU-PAL can achieve
  human-level accuracy (68.5% on MELD) and superior consistency (0.93 Fleiss
  kappa score) while being much cheaper and faster than human annotation. With
  the high-quality, flexible, and consistent annotation from MIKU-PAL, we can
  annotate fine-grained speech emotion categories of up to 26 types, validated by
  human annotators with 83% rationality ratings. Based on our proposed system, we
  further released a fine-grained emotional speech dataset MIKU-EmoBench(131.2
  hours) as a new benchmark for emotional text-to-speech and visual voice
  cloning.
paper_ai_summary: MIKU-PAL, an automated pipeline using multimodal large language models, extracts high-consistency emotional speech from video data, achieving human-level accuracy and consistency at lower cost, and releases a benchmark emotional speech dataset.
paper_ai_keywords: face detection face tracking multimodal large language model emotion analysis human-level accuracy Fleiss kappa score fine-grained speech emotion categories emotional text-to-speech visual voice cloning