Stark: Social Long-Term Multi-Modal Conversation with Persona
  Commonsense Knowledge
Humans share a wide variety of images related to their personal experiences
within conversations via instant messaging tools. However, existing works focus
on (1) image-sharing behavior in singular sessions, leading to limited
long-term social interaction, and (2) a lack of personalized image-sharing
behavior. In this work, we introduce Stark, a large-scale long-term multi-modal
conversation dataset that covers a wide range of social personas in a
multi-modality format, time intervals, and images. To construct Stark
automatically, we propose a novel multi-modal contextualization framework, Mcu,
that generates long-term multi-modal dialogue distilled from ChatGPT and our
proposed Plan-and-Execute image aligner. Using our Stark, we train a
multi-modal conversation model, Ultron 7B, which demonstrates impressive visual
imagination ability. Furthermore, we demonstrate the effectiveness of our
dataset in human evaluation. We make our source code and dataset publicly
available.
668ba1fcb424c825f7b2e9ee
Young-Jun Lee
admin_assigned
2024-07-08T21:07:32.847Z
6434b6619bd5a84b5dcfa4de
https://cdn-avatars.huggingface.co/v1/production/uploads/6434b6619bd5a84b5dcfa4de/h8Q6kPNjFNc03wmdboHzq.jpeg
Young-Jun Lee
user
passing2961
668ba1fcb424c825f7b2e9ef
Dokyong Lee
admin_assigned
2024-07-08T21:07:58.599Z
661f60d1f959ccf1cb69943f
/avatars/3db0b6ae8968b6d05584c1a29243a9ab.svg
dokyong
user
dklee2
668ba1fcb424c825f7b2e9f0
Junyoung Youn
admin_assigned
2024-07-08T21:08:04.463Z
65d695d46e43a738600bb594
/avatars/341aee0c2a3bca619c49669926d00015.svg
junyoung youn
user
junyoungyoun
668ba1fcb424c825f7b2e9f1
Kyeongjin Oh
668ba1fcb424c825f7b2e9f2
Byungsoo Ko
admin_assigned
2024-07-08T21:08:16.049Z
60ff7a87aa025227d344b735
https://cdn-avatars.huggingface.co/v1/production/uploads/1627359121375-60ff7a87aa025227d344b735.jpeg
Byungsoo Ko
user
kobiso
668ba1fcb424c825f7b2e9f3
Jonghwan Hyeon
claimed_verified
2024-07-08T20:00:43.556Z
65a607fcc653a2c10cfd2c8e
https://cdn-avatars.huggingface.co/v1/production/uploads/65a607fcc653a2c10cfd2c8e/mOmR04rFyO_btblYukRAk.jpeg
Jonghwan Hyeon
user
jonghwanhyeon
668ba1fcb424c825f7b2e9f4
Ho-Jin Choi
Stark: Social Long-Term Multi-Modal Conversation with Persona
  Commonsense Knowledge
Humans share a wide variety of images related to their personal experiences
within conversations via instant messaging tools. However, existing works focus
on (1) image-sharing behavior in singular sessions, leading to limited
long-term social interaction, and (2) a lack of personalized image-sharing
behavior. In this work, we introduce Stark, a large-scale long-term multi-modal
conversation dataset that covers a wide range of social personas in a
multi-modality format, time intervals, and images. To construct Stark
automatically, we propose a novel multi-modal contextualization framework, Mcu,
that generates long-term multi-modal dialogue distilled from ChatGPT and our
proposed Plan-and-Execute image aligner. Using our Stark, we train a
multi-modal conversation model, Ultron 7B, which demonstrates impressive visual
imagination ability. Furthermore, we demonstrate the effectiveness of our
dataset in human evaluation. We make our source code and dataset publicly
available.
A large-scale dataset and framework are introduced to capture long-term multi-modal conversations, enhancing a model's visual imagination ability.
multi-modal conversation dataset Stark multi-modal contextualization framework Mcu Plan-and-Execute image aligner ChatGPT multi-modal conversation model Ultron 7B visual imagination