title: Anim-Director: A Large Multimodal Model Powered Agent for Controllable
    Animation Video Generation
summary: Traditional animation generation methods depend on training generative models
  with human-labelled data, entailing a sophisticated multi-stage pipeline that
  demands substantial human effort and incurs high training costs. Due to limited
  prompting plans, these methods typically produce brief, information-poor, and
  context-incoherent animations. To overcome these limitations and automate the
  animation process, we pioneer the introduction of large multimodal models
  (LMMs) as the core processor to build an autonomous animation-making agent,
  named Anim-Director. This agent mainly harnesses the advanced understanding and
  reasoning capabilities of LMMs and generative AI tools to create animated
  videos from concise narratives or simple instructions. Specifically, it
  operates in three main stages: Firstly, the Anim-Director generates a coherent
  storyline from user inputs, followed by a detailed director's script that
  encompasses settings of character profiles and interior/exterior descriptions,
  and context-coherent scene descriptions that include appearing characters,
  interiors or exteriors, and scene events. Secondly, we employ LMMs with the
  image generation tool to produce visual images of settings and scenes. These
  images are designed to maintain visual consistency across different scenes
  using a visual-language prompting method that combines scene descriptions and
  images of the appearing character and setting. Thirdly, scene images serve as
  the foundation for producing animated videos, with LMMs generating prompts to
  guide this process. The whole process is notably autonomous without manual
  intervention, as the LMMs interact seamlessly with generative tools to generate
  prompts, evaluate visual quality, and select the best one to optimize the final
  output.
paper_authors._id: 66c84b3a380e8fb44acee64d
paper_authors.name: Yunxin Li
paper_authors.status: extracted_pending
paper_authors.statusLastChangedAt: 2024-08-23T08:41:33.316Z
paper_authors._id: 62fdb01bc1588e1d4c6c1a7c
paper_authors.avatarUrl: /avatars/bd03085995b1c34e0ac8a845cf2c4e83.svg
paper_authors.fullname: Yunxin Li
paper_authors.type: user
paper_authors.user: YunxinLi
paper_authors._id: 66c84b3a380e8fb44acee64e
paper_authors.name: Haoyuan Shi
paper_authors._id: 66c84b3a380e8fb44acee64f
paper_authors.name: Baotian Hu
paper_authors._id: 66c84b3a380e8fb44acee650
paper_authors.name: Longyue Wang
paper_authors._id: 66c84b3a380e8fb44acee651
paper_authors.name: Jiashun Zhu
paper_authors._id: 66c84b3a380e8fb44acee652
paper_authors.name: Jinyi Xu
paper_authors._id: 66c84b3a380e8fb44acee653
paper_authors.name: Zhen Zhao
paper_authors._id: 66c84b3a380e8fb44acee654
paper_authors.name: Min Zhang
paper_title: Anim-Director: A Large Multimodal Model Powered Agent for Controllable
    Animation Video Generation
paper_summary: Traditional animation generation methods depend on training generative models
  with human-labelled data, entailing a sophisticated multi-stage pipeline that
  demands substantial human effort and incurs high training costs. Due to limited
  prompting plans, these methods typically produce brief, information-poor, and
  context-incoherent animations. To overcome these limitations and automate the
  animation process, we pioneer the introduction of large multimodal models
  (LMMs) as the core processor to build an autonomous animation-making agent,
  named Anim-Director. This agent mainly harnesses the advanced understanding and
  reasoning capabilities of LMMs and generative AI tools to create animated
  videos from concise narratives or simple instructions. Specifically, it
  operates in three main stages: Firstly, the Anim-Director generates a coherent
  storyline from user inputs, followed by a detailed director's script that
  encompasses settings of character profiles and interior/exterior descriptions,
  and context-coherent scene descriptions that include appearing characters,
  interiors or exteriors, and scene events. Secondly, we employ LMMs with the
  image generation tool to produce visual images of settings and scenes. These
  images are designed to maintain visual consistency across different scenes
  using a visual-language prompting method that combines scene descriptions and
  images of the appearing character and setting. Thirdly, scene images serve as
  the foundation for producing animated videos, with LMMs generating prompts to
  guide this process. The whole process is notably autonomous without manual
  intervention, as the LMMs interact seamlessly with generative tools to generate
  prompts, evaluate visual quality, and select the best one to optimize the final
  output.
paper_ai_summary: Anim-Director, an autonomous animation-making agent, uses large multimodal models to create consistent and coherent animations from concise narratives.
paper_ai_keywords: large multimodal models Anim-Director generative AI tools image generation visual-language prompting method animated videos scene descriptions visual consistency generative tools