Follow-Your-Click: Open-domain Regional Image Animation via Short
  Prompts
Despite recent advances in image-to-video generation, better controllability
and local animation are less explored. Most existing image-to-video methods are
not locally aware and tend to move the entire scene. However, human artists may
need to control the movement of different objects or regions. Additionally,
current I2V methods require users not only to describe the target motion but
also to provide redundant detailed descriptions of frame contents. These two
issues hinder the practical utilization of current I2V tools. In this paper, we
propose a practical framework, named Follow-Your-Click, to achieve image
animation with a simple user click (for specifying what to move) and a short
motion prompt (for specifying how to move). Technically, we propose the
first-frame masking strategy, which significantly improves the video generation
quality, and a motion-augmented module equipped with a short motion prompt
dataset to improve the short prompt following abilities of our model. To
further control the motion speed, we propose flow-based motion magnitude
control to control the speed of target movement more precisely. Our framework
has simpler yet precise user control and better generation performance than
previous methods. Extensive experiments compared with 7 baselines, including
both commercial tools and research methods on 8 metrics, suggest the
superiority of our approach. Project Page: https://follow-your-click.github.io/
65f2693333725f138b0f1719
Yue Ma
65f2693333725f138b0f171a
Yingqing He
admin_assigned
2024-03-14T08:34:16.010Z
630231bf7e137e3d6b3b0645
/avatars/dad52f8955110f0a2caeb613d6aa3ea2.svg
He
user
Yingqing
65f2693333725f138b0f171b
Hongfa Wang
65f2693333725f138b0f171c
Andong Wang
65f2693333725f138b0f171d
Chenyang Qi
admin_assigned
2024-03-14T08:34:37.856Z
642157a201c62c1e41fc526a
/avatars/73ae831a73947a04468b1e87d17da24d.svg
chenyangqi
user
chenyangqi
65f2693333725f138b0f171e
Chengfei Cai
65f2693333725f138b0f171f
Xiu Li
65f2693333725f138b0f1720
Zhifeng Li
65f2693333725f138b0f1721
Heung-Yeung Shum
65f2693333725f138b0f1722
Wei Liu
65f2693333725f138b0f1723
Qifeng Chen
admin_assigned
2024-03-14T08:35:20.475Z
6467b121e7a6a374fd19b44b
/avatars/3f2874d58986d651aef55e3408b05700.svg
Qifeng Chen
user
cqf
Follow-Your-Click: Open-domain Regional Image Animation via Short
  Prompts
Despite recent advances in image-to-video generation, better controllability
and local animation are less explored. Most existing image-to-video methods are
not locally aware and tend to move the entire scene. However, human artists may
need to control the movement of different objects or regions. Additionally,
current I2V methods require users not only to describe the target motion but
also to provide redundant detailed descriptions of frame contents. These two
issues hinder the practical utilization of current I2V tools. In this paper, we
propose a practical framework, named Follow-Your-Click, to achieve image
animation with a simple user click (for specifying what to move) and a short
motion prompt (for specifying how to move). Technically, we propose the
first-frame masking strategy, which significantly improves the video generation
quality, and a motion-augmented module equipped with a short motion prompt
dataset to improve the short prompt following abilities of our model. To
further control the motion speed, we propose flow-based motion magnitude
control to control the speed of target movement more precisely. Our framework
has simpler yet precise user control and better generation performance than
previous methods. Extensive experiments compared with 7 baselines, including
both commercial tools and research methods on 8 metrics, suggest the
superiority of our approach. Project Page: https://follow-your-click.github.io/
A framework named Follow-Your-Click allows precise image-to-video animation through user clicks and short motion prompts, enhancing controllability and local animation quality.
image-to-video generation local awareness first-frame masking motion-augmented module short motion prompt dataset flow-based motion magnitude control video generation quality