title: TrackGo: A Flexible and Efficient Method for Controllable Video
    Generation
summary: Recent years have seen substantial progress in diffusion-based controllable
  video generation. However, achieving precise control in complex scenarios,
  including fine-grained object parts, sophisticated motion trajectories, and
  coherent background movement, remains a challenge. In this paper, we introduce
  TrackGo, a novel approach that leverages free-form masks and arrows for
  conditional video generation. This method offers users with a flexible and
  precise mechanism for manipulating video content. We also propose the
  TrackAdapter for control implementation, an efficient and lightweight adapter
  designed to be seamlessly integrated into the temporal self-attention layers of
  a pretrained video generation model. This design leverages our observation that
  the attention map of these layers can accurately activate regions corresponding
  to motion in videos. Our experimental results demonstrate that our new
  approach, enhanced by the TrackAdapter, achieves state-of-the-art performance
  on key metrics such as FVD, FID, and ObjMC scores. The project page of TrackGo
  can be found at: https://zhtjtcz.github.io/TrackGo-Page/
paper_authors._id: 66c6a4afc33122b03a2c1c27
paper_authors.name: Haitao Zhou
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-12-11T09:54:21.578Z
paper_authors._id: 6474352dd9e9598973cd8831
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/B_MnqQSvlkodzgf9krjF7.jpeg
paper_authors.fullname: Haitao Zhou
paper_authors.type: user
paper_authors.user: Marvolo
paper_authors._id: 66c6a4afc33122b03a2c1c28
paper_authors.name: Chuang Wang
paper_authors._id: 66c6a4afc33122b03a2c1c29
paper_authors.name: Rui Nie
paper_authors._id: 66c6a4afc33122b03a2c1c2a
paper_authors.name: Jinxiao Lin
paper_authors._id: 66c6a4afc33122b03a2c1c2b
paper_authors.name: Dongdong Yu
paper_authors._id: 66c6a4afc33122b03a2c1c2c
paper_authors.name: Qian Yu
paper_authors._id: 66c6a4afc33122b03a2c1c2d
paper_authors.name: Changhu Wang
paper_title: TrackGo: A Flexible and Efficient Method for Controllable Video
    Generation
paper_summary: Recent years have seen substantial progress in diffusion-based controllable
  video generation. However, achieving precise control in complex scenarios,
  including fine-grained object parts, sophisticated motion trajectories, and
  coherent background movement, remains a challenge. In this paper, we introduce
  TrackGo, a novel approach that leverages free-form masks and arrows for
  conditional video generation. This method offers users with a flexible and
  precise mechanism for manipulating video content. We also propose the
  TrackAdapter for control implementation, an efficient and lightweight adapter
  designed to be seamlessly integrated into the temporal self-attention layers of
  a pretrained video generation model. This design leverages our observation that
  the attention map of these layers can accurately activate regions corresponding
  to motion in videos. Our experimental results demonstrate that our new
  approach, enhanced by the TrackAdapter, achieves state-of-the-art performance
  on key metrics such as FVD, FID, and ObjMC scores. The project page of TrackGo
  can be found at: https://zhtjtcz.github.io/TrackGo-Page/
paper_ai_keywords: diffusion-based controllable video generation free-form masks arrows conditional video generation TrackGo TrackAdapter temporal self-attention layers attention map FVD FID ObjMC scores