title: Friends-MMC: A Dataset for Multi-modal Multi-party Conversation
    Understanding
summary: Multi-modal multi-party conversation (MMC) is a less studied yet important
  topic of research due to that it well fits real-world scenarios and thus
  potentially has more widely-used applications. Compared with the traditional
  multi-modal conversations, MMC requires stronger character-centered
  understanding abilities as there are many interlocutors appearing in both the
  visual and textual context. To facilitate the study of this problem, we present
  Friends-MMC in this paper, an MMC dataset that contains 24,000+ unique
  utterances paired with video context. To explore the character-centered
  understanding of the dialogue, we also annotate the speaker of each utterance,
  the names and bounding bboxes of faces that appear in the video. Based on this
  Friends-MMC dataset, we further study two fundamental MMC tasks: conversation
  speaker identification and conversation response prediction, both of which have
  the multi-party nature with the video or image as visual context. For
  conversation speaker identification, we demonstrate the inefficiencies of
  existing methods such as pre-trained models, and propose a simple yet effective
  baseline method that leverages an optimization solver to utilize the context of
  two modalities to achieve better performance. For conversation response
  prediction, we fine-tune generative dialogue models on Friend-MMC, and analyze
  the benefits of speaker information. The code and dataset is publicly available
  at https://github.com/yellow-binary-tree/Friends-MMC and thus we call for more
  attention on modeling speaker information when understanding conversations.
paper_authors._id: 676a3878eabbef01bb66f7f8
paper_authors.name: Yueqian Wang
paper_authors._id: 676a3878eabbef01bb66f7f9
paper_authors.name: Xiaojun Meng
paper_authors._id: 676a3878eabbef01bb66f7fa
paper_authors.name: Yuxuan Wang
paper_authors.status: extracted_confirmed
paper_authors.statusLastChangedAt: 2025-01-14T09:25:44.845Z
paper_authors._id: 60b9e6837946aff342f734ae
paper_authors.avatarUrl: /avatars/a711a6aa35757dfd7b78b26098a964fc.svg
paper_authors.fullname: Yuxuan Wang
paper_authors.type: user
paper_authors.user: ColorfulAI
paper_authors._id: 676a3878eabbef01bb66f7fb
paper_authors.name: Jianxin Liang
paper_authors._id: 676a3878eabbef01bb66f7fc
paper_authors.name: Qun Liu
paper_authors._id: 676a3878eabbef01bb66f7fd
paper_authors.name: Dongyan Zhao
paper_title: Friends-MMC: A Dataset for Multi-modal Multi-party Conversation
    Understanding
paper_summary: Multi-modal multi-party conversation (MMC) is a less studied yet important
  topic of research due to that it well fits real-world scenarios and thus
  potentially has more widely-used applications. Compared with the traditional
  multi-modal conversations, MMC requires stronger character-centered
  understanding abilities as there are many interlocutors appearing in both the
  visual and textual context. To facilitate the study of this problem, we present
  Friends-MMC in this paper, an MMC dataset that contains 24,000+ unique
  utterances paired with video context. To explore the character-centered
  understanding of the dialogue, we also annotate the speaker of each utterance,
  the names and bounding bboxes of faces that appear in the video. Based on this
  Friends-MMC dataset, we further study two fundamental MMC tasks: conversation
  speaker identification and conversation response prediction, both of which have
  the multi-party nature with the video or image as visual context. For
  conversation speaker identification, we demonstrate the inefficiencies of
  existing methods such as pre-trained models, and propose a simple yet effective
  baseline method that leverages an optimization solver to utilize the context of
  two modalities to achieve better performance. For conversation response
  prediction, we fine-tune generative dialogue models on Friend-MMC, and analyze
  the benefits of speaker information. The code and dataset is publicly available
  at https://github.com/yellow-binary-tree/Friends-MMC and thus we call for more
  attention on modeling speaker information when understanding conversations.
paper_ai_keywords: Friends-MMC Multi-modal multi-party conversation (MMC) character-centered understanding utterances video context bounding bboxes conversation speaker identification conversation response prediction pre-trained models optimization solver generative dialogue models speaker information