AutoDirector: Online Auto-scheduling Agents for Multi-sensory Composition
<|reference_start|>AutoDirector: Online Auto-scheduling Agents for Multi-sensory Composition: With the advancement of generative models, the synthesis of different sensory elements such as music, visuals, and speech has achieved significant realism. However, the approach to generate multi-sensory outputs has not been fully explored, limiting the application on high-value scenarios such as of directing a film. Developing a movie director agent faces two major challenges: (1) Lack of parallelism and online scheduling with production steps: In the production of multi-sensory films, there are complex dependencies between different sensory elements, and the production time for each element varies. (2) Diverse needs and clear communication demands with users: Users often cannot clearly express their needs until they see a draft, which requires human-computer interaction and iteration to continually adjust and optimize the film content based on user feedback. To address these issues, we introduce AutoDirector, an interactive multi-sensory composition framework that supports long shots, special effects, music scoring, dubbing, and lip-syncing. This framework improves the efficiency of multi-sensory film production through automatic scheduling and supports the modification and improvement of interactive tasks to meet user needs. AutoDirector not only expands the application scope of human-machine collaboration but also demonstrates the potential of AI in collaborating with humans in the role of a film director to complete multi-sensory films.<|reference_end|>
@article{ni2024autodirector:,
    title={AutoDirector: Online Auto-scheduling Agents for Multi-sensory
  Composition},
    author={Minheng Ni, Chenfei Wu, Huaying Yuan, Zhengyuan Yang, Ming Gong,
  Lijuan Wang, Zicheng Liu, Wangmeng Zuo, Nan Duan},
    journal={arXiv preprint arXiv:2408.11564},
    year={2024},
    archivePrefix={arXiv},
    eprint={2408.11564},
    primaryClass={cs.CV}
}