title: Stark: Social Long-Term Multi-Modal Conversation with Persona
    Commonsense Knowledge
summary: Humans share a wide variety of images related to their personal experiences
  within conversations via instant messaging tools. However, existing works focus
  on (1) image-sharing behavior in singular sessions, leading to limited
  long-term social interaction, and (2) a lack of personalized image-sharing
  behavior. In this work, we introduce Stark, a large-scale long-term multi-modal
  conversation dataset that covers a wide range of social personas in a
  multi-modality format, time intervals, and images. To construct Stark
  automatically, we propose a novel multi-modal contextualization framework, Mcu,
  that generates long-term multi-modal dialogue distilled from ChatGPT and our
  proposed Plan-and-Execute image aligner. Using our Stark, we train a
  multi-modal conversation model, Ultron 7B, which demonstrates impressive visual
  imagination ability. Furthermore, we demonstrate the effectiveness of our
  dataset in human evaluation. We make our source code and dataset publicly
  available.
paper_authors._id: 668ba1fcb424c825f7b2e9ee
paper_authors.name: Young-Jun Lee
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-07-08T21:07:32.847Z
paper_authors._id: 6434b6619bd5a84b5dcfa4de
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/6434b6619bd5a84b5dcfa4de/h8Q6kPNjFNc03wmdboHzq.jpeg
paper_authors.fullname: Young-Jun Lee
paper_authors.type: user
paper_authors.user: passing2961
paper_authors._id: 668ba1fcb424c825f7b2e9ef
paper_authors.name: Dokyong Lee
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-07-08T21:07:58.599Z
paper_authors._id: 661f60d1f959ccf1cb69943f
paper_authors.avatarUrl: /avatars/3db0b6ae8968b6d05584c1a29243a9ab.svg
paper_authors.fullname: dokyong
paper_authors.type: user
paper_authors.user: dklee2
paper_authors._id: 668ba1fcb424c825f7b2e9f0
paper_authors.name: Junyoung Youn
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-07-08T21:08:04.463Z
paper_authors._id: 65d695d46e43a738600bb594
paper_authors.avatarUrl: /avatars/341aee0c2a3bca619c49669926d00015.svg
paper_authors.fullname: junyoung youn
paper_authors.type: user
paper_authors.user: junyoungyoun
paper_authors._id: 668ba1fcb424c825f7b2e9f1
paper_authors.name: Kyeongjin Oh
paper_authors._id: 668ba1fcb424c825f7b2e9f2
paper_authors.name: Byungsoo Ko
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-07-08T21:08:16.049Z
paper_authors._id: 60ff7a87aa025227d344b735
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1627359121375-60ff7a87aa025227d344b735.jpeg
paper_authors.fullname: Byungsoo Ko
paper_authors.type: user
paper_authors.user: kobiso
paper_authors._id: 668ba1fcb424c825f7b2e9f3
paper_authors.name: Jonghwan Hyeon
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-07-08T20:00:43.556Z
paper_authors._id: 65a607fcc653a2c10cfd2c8e
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/65a607fcc653a2c10cfd2c8e/mOmR04rFyO_btblYukRAk.jpeg
paper_authors.fullname: Jonghwan Hyeon
paper_authors.type: user
paper_authors.user: jonghwanhyeon
paper_authors._id: 668ba1fcb424c825f7b2e9f4
paper_authors.name: Ho-Jin Choi
paper_title: Stark: Social Long-Term Multi-Modal Conversation with Persona
    Commonsense Knowledge
paper_summary: Humans share a wide variety of images related to their personal experiences
  within conversations via instant messaging tools. However, existing works focus
  on (1) image-sharing behavior in singular sessions, leading to limited
  long-term social interaction, and (2) a lack of personalized image-sharing
  behavior. In this work, we introduce Stark, a large-scale long-term multi-modal
  conversation dataset that covers a wide range of social personas in a
  multi-modality format, time intervals, and images. To construct Stark
  automatically, we propose a novel multi-modal contextualization framework, Mcu,
  that generates long-term multi-modal dialogue distilled from ChatGPT and our
  proposed Plan-and-Execute image aligner. Using our Stark, we train a
  multi-modal conversation model, Ultron 7B, which demonstrates impressive visual
  imagination ability. Furthermore, we demonstrate the effectiveness of our
  dataset in human evaluation. We make our source code and dataset publicly
  available.
paper_ai_keywords: Stark multi-modal conversation dataset multi-modality format multi-modal contextualization framework Mcu long-term multi-modal dialogue ChatGPT Plan-and-Execute image aligner multi-modal conversation model Ultron 7B visual imagination ability