title: GPT4Motion: Scripting Physical Motions in Text-to-Video Generation via
    Blender-Oriented GPT Planning
summary: Recent advances in text-to-video generation have harnessed the power of
  diffusion models to create visually compelling content conditioned on text
  prompts. However, they usually encounter high computational costs and often
  struggle to produce videos with coherent physical motions. To tackle these
  issues, we propose GPT4Motion, a training-free framework that leverages the
  planning capability of large language models such as GPT, the physical
  simulation strength of Blender, and the excellent image generation ability of
  text-to-image diffusion models to enhance the quality of video synthesis.
  Specifically, GPT4Motion employs GPT-4 to generate a Blender script based on a
  user textual prompt, which commands Blender's built-in physics engine to craft
  fundamental scene components that encapsulate coherent physical motions across
  frames. Then these components are inputted into Stable Diffusion to generate a
  video aligned with the textual prompt. Experimental results on three basic
  physical motion scenarios, including rigid object drop and collision, cloth
  draping and swinging, and liquid flow, demonstrate that GPT4Motion can generate
  high-quality videos efficiently in maintaining motion coherency and entity
  consistency. GPT4Motion offers new insights in text-to-video research,
  enhancing its quality and broadening its horizon for future explorations.
paper_authors._id: 655d7a5e16ede91c780ce244
paper_authors.name: Jiaxi Lv
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2023-12-07T08:30:15.836Z
paper_authors._id: 63c15f0ef02ef5b95e102d1e
paper_authors.avatarUrl: /avatars/ea50adfdb91de8545dbbc3e06855a468.svg
paper_authors.fullname: JaX lyu
paper_authors.type: user
paper_authors.user: jax2000
paper_authors._id: 655d7a5e16ede91c780ce245
paper_authors.name: Yi Huang
paper_authors._id: 655d7a5e16ede91c780ce246
paper_authors.name: Mingfu Yan
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-11-22T13:44:28.238Z
paper_authors._id: 64fd99df66a284440bcdb46d
paper_authors.avatarUrl: /avatars/d02f5e3231a5f26c6894541103c27919.svg
paper_authors.fullname: Mingfu Yan
paper_authors.type: user
paper_authors.user: MingfuYAN
paper_authors._id: 655d7a5e16ede91c780ce247
paper_authors.name: Jiancheng Huang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-11-22T13:44:37.374Z
paper_authors._id: 6395e7c311074713ad1fc29e
paper_authors.avatarUrl: /avatars/6b1e61aaaed89af05daaf15f90747976.svg
paper_authors.fullname: jiancheng huang
paper_authors.type: user
paper_authors.user: jiancheng
paper_authors._id: 655d7a5e16ede91c780ce248
paper_authors.name: Jianzhuang Liu
paper_authors._id: 655d7a5e16ede91c780ce249
paper_authors.name: Yifan Liu
paper_authors._id: 655d7a5e16ede91c780ce24a
paper_authors.name: Yafei Wen
paper_authors._id: 655d7a5e16ede91c780ce24b
paper_authors.name: Xiaoxin Chen
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-11-22T13:55:11.798Z
paper_authors._id: 65389a669c474315d7425f96
paper_authors.avatarUrl: /avatars/2fa3828ca489cfe1948129a0eccf264f.svg
paper_authors.fullname: chenxiaoxin
paper_authors.type: user
paper_authors.user: steelozazala
paper_authors._id: 655d7a5e16ede91c780ce24c
paper_authors.name: Shifeng Chen
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-11-22T13:55:23.457Z
paper_authors._id: 6393446923aec2f4ee593925
paper_authors.avatarUrl: /avatars/6570a7ce6f6495ba47eda51e7c70c8ce.svg
paper_authors.fullname: chenshifeng
paper_authors.type: user
paper_authors.user: suifeng
paper_title: GPT4Motion: Scripting Physical Motions in Text-to-Video Generation via
    Blender-Oriented GPT Planning
paper_summary: Recent advances in text-to-video generation have harnessed the power of
  diffusion models to create visually compelling content conditioned on text
  prompts. However, they usually encounter high computational costs and often
  struggle to produce videos with coherent physical motions. To tackle these
  issues, we propose GPT4Motion, a training-free framework that leverages the
  planning capability of large language models such as GPT, the physical
  simulation strength of Blender, and the excellent image generation ability of
  text-to-image diffusion models to enhance the quality of video synthesis.
  Specifically, GPT4Motion employs GPT-4 to generate a Blender script based on a
  user textual prompt, which commands Blender's built-in physics engine to craft
  fundamental scene components that encapsulate coherent physical motions across
  frames. Then these components are inputted into Stable Diffusion to generate a
  video aligned with the textual prompt. Experimental results on three basic
  physical motion scenarios, including rigid object drop and collision, cloth
  draping and swinging, and liquid flow, demonstrate that GPT4Motion can generate
  high-quality videos efficiently in maintaining motion coherency and entity
  consistency. GPT4Motion offers new insights in text-to-video research,
  enhancing its quality and broadening its horizon for future explorations.
paper_ai_keywords: Diffusion models spec conditioning GPT Blender text-to-image diffusion models video synthesis stable diffusion GPT-4 Blender script physics engine fundamental scene components rigid object drop and collision cloth draping and swinging liquid flow motion coherency entity consistency text-to-video research