title: DreamRunner: Fine-Grained Storytelling Video Generation with
    Retrieval-Augmented Motion Adaptation
summary: Storytelling video generation (SVG) has recently emerged as a task to create
  long, multi-motion, multi-scene videos that consistently represent the story
  described in the input text script. SVG holds great potential for diverse
  content creation in media and entertainment; however, it also presents
  significant challenges: (1) objects must exhibit a range of fine-grained,
  complex motions, (2) multiple objects need to appear consistently across
  scenes, and (3) subjects may require multiple motions with seamless transitions
  within a single scene. To address these challenges, we propose DreamRunner, a
  novel story-to-video generation method: First, we structure the input script
  using a large language model (LLM) to facilitate both coarse-grained scene
  planning as well as fine-grained object-level layout and motion planning. Next,
  DreamRunner presents retrieval-augmented test-time adaptation to capture target
  motion priors for objects in each scene, supporting diverse motion
  customization based on retrieved videos, thus facilitating the generation of
  new videos with complex, scripted motions. Lastly, we propose a novel
  spatial-temporal region-based 3D attention and prior injection module SR3AI for
  fine-grained object-motion binding and frame-by-frame semantic control. We
  compare DreamRunner with various SVG baselines, demonstrating state-of-the-art
  performance in character consistency, text alignment, and smooth transitions.
  Additionally, DreamRunner exhibits strong fine-grained condition-following
  ability in compositional text-to-video generation, significantly outperforming
  baselines on T2V-ComBench. Finally, we validate DreamRunner's robust ability to
  generate multi-object interactions with qualitative examples.
paper_authors._id: 674539fa5cb54d8d661272cc
paper_authors.name: Zun Wang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-11-26T22:43:26.509Z
paper_authors._id: 6622c710b0e5c5e3de8311c1
paper_authors.avatarUrl: /avatars/a824c150040731679bbd77762ca9d4eb.svg
paper_authors.fullname: Zun Wang
paper_authors.type: user
paper_authors.user: ZunWang
paper_authors._id: 674539fa5cb54d8d661272cd
paper_authors.name: Jialu Li
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-11-26T21:21:11.019Z
paper_authors._id: 65d69eb0bcf1fed72325a052
paper_authors.avatarUrl: /avatars/5ef735c3137555bb52e512ae820e437c.svg
paper_authors.fullname: Jialu Li
paper_authors.type: user
paper_authors.user: jialuliluka
paper_authors._id: 674539fa5cb54d8d661272ce
paper_authors.name: Han Lin
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-12-02T08:51:14.248Z
paper_authors._id: 646e86350867c99c2d3f2ecf
paper_authors.avatarUrl: /avatars/b89798ff623abffb169eacda2ac32fde.svg
paper_authors.fullname: Han Lin
paper_authors.type: user
paper_authors.user: hanlincs
paper_authors._id: 674539fa5cb54d8d661272cf
paper_authors.name: Jaehong Yoon
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-11-26T21:21:09.391Z
paper_authors._id: 652066649004117947e46ed6
paper_authors.avatarUrl: /avatars/972c97df6f26d2c3d6ce71ec579984bb.svg
paper_authors.fullname: Jaehong Yoon
paper_authors.type: user
paper_authors.user: jaehong31
paper_authors._id: 674539fa5cb54d8d661272d0
paper_authors.name: Mohit Bansal
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-11-26T22:43:42.862Z
paper_authors._id: 665d9d3a057f7c508f98c625
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/665d9d3a057f7c508f98c625/u1R9P9sJoAl4zEIcetbPy.jpeg
paper_authors.fullname: Mohit Bansal
paper_authors.type: user
paper_authors.user: mohitbansal
paper_title: DreamRunner: Fine-Grained Storytelling Video Generation with
    Retrieval-Augmented Motion Adaptation
paper_summary: Storytelling video generation (SVG) has recently emerged as a task to create
  long, multi-motion, multi-scene videos that consistently represent the story
  described in the input text script. SVG holds great potential for diverse
  content creation in media and entertainment; however, it also presents
  significant challenges: (1) objects must exhibit a range of fine-grained,
  complex motions, (2) multiple objects need to appear consistently across
  scenes, and (3) subjects may require multiple motions with seamless transitions
  within a single scene. To address these challenges, we propose DreamRunner, a
  novel story-to-video generation method: First, we structure the input script
  using a large language model (LLM) to facilitate both coarse-grained scene
  planning as well as fine-grained object-level layout and motion planning. Next,
  DreamRunner presents retrieval-augmented test-time adaptation to capture target
  motion priors for objects in each scene, supporting diverse motion
  customization based on retrieved videos, thus facilitating the generation of
  new videos with complex, scripted motions. Lastly, we propose a novel
  spatial-temporal region-based 3D attention and prior injection module SR3AI for
  fine-grained object-motion binding and frame-by-frame semantic control. We
  compare DreamRunner with various SVG baselines, demonstrating state-of-the-art
  performance in character consistency, text alignment, and smooth transitions.
  Additionally, DreamRunner exhibits strong fine-grained condition-following
  ability in compositional text-to-video generation, significantly outperforming
  baselines on T2V-ComBench. Finally, we validate DreamRunner's robust ability to
  generate multi-object interactions with qualitative examples.
paper_ai_summary: DreamRunner integrates scene structuring, motion retrieval, and spatial-temporal attention to generate coherent, motion-rich storytelling videos from text scripts.
paper_ai_keywords: storytelling video generation SVG large language model scene planning object-level layout motion planning retrieval-augmented test-time adaptation target motion priors spatial-temporal region-based 3D attention prior injection module SR3AI character consistency text alignment smooth transitions compositional text-to-video generation T2V-ComBench multi-object interactions