title: SEED-Story: Multimodal Long Story Generation with Large Language Model
summary: With the remarkable advancements in image generation and open-form text
  generation, the creation of interleaved image-text content has become an
  increasingly intriguing field. Multimodal story generation, characterized by
  producing narrative texts and vivid images in an interleaved manner, has
  emerged as a valuable and practical task with broad applications. However, this
  task poses significant challenges, as it necessitates the comprehension of the
  complex interplay between texts and images, and the ability to generate long
  sequences of coherent, contextually relevant texts and visuals. In this work,
  we propose SEED-Story, a novel method that leverages a Multimodal Large
  Language Model (MLLM) to generate extended multimodal stories. Our model, built
  upon the powerful comprehension capability of MLLM, predicts text tokens as
  well as visual tokens, which are subsequently processed with an adapted visual
  de-tokenizer to produce images with consistent characters and styles. We
  further propose multimodal attention sink mechanism to enable the generation of
  stories with up to 25 sequences (only 10 for training) in a highly efficient
  autoregressive manner. Additionally, we present a large-scale and
  high-resolution dataset named StoryStream for training our model and
  quantitatively evaluating the task of multimodal story generation in various
  aspects.
paper_authors._id: 669091001e3f5b16ce22cce6
paper_authors.name: Shuai Yang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-07-12T16:03:30.318Z
paper_authors._id: 634ce90e741a5e37886a19e3
paper_authors.avatarUrl: /avatars/0d1579039136b37db5b67282b0a34c33.svg
paper_authors.fullname: Syang
paper_authors.type: user
paper_authors.user: Andyson
paper_authors._id: 669091001e3f5b16ce22cce7
paper_authors.name: Yuying Ge
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-07-12T10:22:50.689Z
paper_authors._id: 6455cc8f654d8bccae50e4d4
paper_authors.avatarUrl: /avatars/506a9992e5bf52e06d37cc22e4b307c0.svg
paper_authors.fullname: Yuying Ge
paper_authors.type: user
paper_authors.user: tttoaster
paper_authors._id: 669091001e3f5b16ce22cce8
paper_authors.name: Yang Li
paper_authors._id: 669091001e3f5b16ce22cce9
paper_authors.name: Yukang Chen
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-07-12T10:25:40.867Z
paper_authors._id: 65b31d77f32c79fea79ba4d1
paper_authors.avatarUrl: /avatars/8c51c1ff337ac372c7caafa4fa5019dc.svg
paper_authors.fullname: Yukang Chen
paper_authors.type: user
paper_authors.user: Yukang2
paper_authors._id: 669091001e3f5b16ce22ccea
paper_authors.name: Yixiao Ge
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-07-12T10:25:34.813Z
paper_authors._id: 640e9762b03f4cd29f58d982
paper_authors.avatarUrl: /avatars/81da37d628163fe3e094b247c7c3a3b5.svg
paper_authors.fullname: Yixiao Ge
paper_authors.type: user
paper_authors.user: yxgeee
paper_authors._id: 669091001e3f5b16ce22cceb
paper_authors.name: Ying Shan
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-07-12T16:09:04.359Z
paper_authors._id: 63ca3ddc04c979828310bfcb
paper_authors.avatarUrl: /avatars/615e0d8622950b4408b40d550f02a894.svg
paper_authors.fullname: Ying Shan
paper_authors.type: user
paper_authors.user: yshan2u
paper_authors._id: 669091001e3f5b16ce22ccec
paper_authors.name: Yingcong Chen
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-07-12T10:25:23.622Z
paper_authors._id: 655cba1d87b67834000590e8
paper_authors.avatarUrl: /avatars/3bd43b7c9351f65b8f38f4c8237a0146.svg
paper_authors.fullname: Yingcong Chen
paper_authors.type: user
paper_authors.user: yingcongchen
paper_title: SEED-Story: Multimodal Long Story Generation with Large Language Model
paper_summary: With the remarkable advancements in image generation and open-form text
  generation, the creation of interleaved image-text content has become an
  increasingly intriguing field. Multimodal story generation, characterized by
  producing narrative texts and vivid images in an interleaved manner, has
  emerged as a valuable and practical task with broad applications. However, this
  task poses significant challenges, as it necessitates the comprehension of the
  complex interplay between texts and images, and the ability to generate long
  sequences of coherent, contextually relevant texts and visuals. In this work,
  we propose SEED-Story, a novel method that leverages a Multimodal Large
  Language Model (MLLM) to generate extended multimodal stories. Our model, built
  upon the powerful comprehension capability of MLLM, predicts text tokens as
  well as visual tokens, which are subsequently processed with an adapted visual
  de-tokenizer to produce images with consistent characters and styles. We
  further propose multimodal attention sink mechanism to enable the generation of
  stories with up to 25 sequences (only 10 for training) in a highly efficient
  autoregressive manner. Additionally, we present a large-scale and
  high-resolution dataset named StoryStream for training our model and
  quantitatively evaluating the task of multimodal story generation in various
  aspects.
paper_ai_summary: SEED-Story uses a Multimodal Large Language Model with multimodal attention to generate extended multimodal stories from interleaved image-text sequences.
paper_ai_keywords: Multimodal Large Language Model MLLM text tokens visual tokens visual de-tokenizer multimodal attention sink mechanism autoregressive generation StoryStream dataset