Fine-grained Controllable Video Generation via Object Appearance and
  Context
Text-to-video generation has shown promising results. However, by taking only
natural languages as input, users often face difficulties in providing detailed
information to precisely control the model's output. In this work, we propose
fine-grained controllable video generation (FACTOR) to achieve detailed
control. Specifically, FACTOR aims to control objects' appearances and context,
including their location and category, in conjunction with the text prompt. To
achieve detailed control, we propose a unified framework to jointly inject
control signals into the existing text-to-video model. Our model consists of a
joint encoder and adaptive cross-attention layers. By optimizing the encoder
and the inserted layer, we adapt the model to generate videos that are aligned
with both text prompts and fine-grained control. Compared to existing methods
relying on dense control signals such as edge maps, we provide a more intuitive
and user-friendly interface to allow object-level fine-grained control. Our
method achieves controllability of object appearances without finetuning, which
reduces the per-subject optimization efforts for the users. Extensive
experiments on standard benchmark datasets and user-provided inputs validate
that our model obtains a 70% improvement in controllability metrics over
competitive baselines.
656fda8234440bdf2ed8027d
Hsin-Ping Huang
claimed_verified
2024-12-19T18:06:31.675Z
667b2ee8e005e1dbcc76e2e2
/avatars/8b2f5f997f0ed5ae9f4f274941933c40.svg
Hsin-Ping Huang
user
hsinh
656fda8234440bdf2ed8027e
Yu-Chuan Su
656fda8234440bdf2ed8027f
Deqing Sun
656fda8234440bdf2ed80280
Lu Jiang
claimed_verified
2023-12-23T11:18:13.047Z
64a5cba3bea0116f8f7187a7
/avatars/97b3bad82e359c57d1ab7d6e3aa7748b.svg
Lu Jiang
user
roadjiang
656fda8234440bdf2ed80281
Xuhui Jia
admin_assigned
2023-12-06T13:46:17.668Z
634ce7ce05f736dff37aae5f
/avatars/1405c76a38f7ea497e4439f0e4e786a8.svg
Xuhui Jia
user
Jxh-cuit
656fda8234440bdf2ed80282
Yukun Zhu
admin_assigned
2023-12-06T13:46:44.840Z
646ec8a3ee36629c15bbf115
/avatars/169bf2a4a8b46715913be72eaa3ca2c4.svg
zhuyukun
user
kun888
656fda8234440bdf2ed80283
Ming-Hsuan Yang
Fine-grained Controllable Video Generation via Object Appearance and
  Context
Text-to-video generation has shown promising results. However, by taking only
natural languages as input, users often face difficulties in providing detailed
information to precisely control the model's output. In this work, we propose
fine-grained controllable video generation (FACTOR) to achieve detailed
control. Specifically, FACTOR aims to control objects' appearances and context,
including their location and category, in conjunction with the text prompt. To
achieve detailed control, we propose a unified framework to jointly inject
control signals into the existing text-to-video model. Our model consists of a
joint encoder and adaptive cross-attention layers. By optimizing the encoder
and the inserted layer, we adapt the model to generate videos that are aligned
with both text prompts and fine-grained control. Compared to existing methods
relying on dense control signals such as edge maps, we provide a more intuitive
and user-friendly interface to allow object-level fine-grained control. Our
method achieves controllability of object appearances without finetuning, which
reduces the per-subject optimization efforts for the users. Extensive
experiments on standard benchmark datasets and user-provided inputs validate
that our model obtains a 70% improvement in controllability metrics over
competitive baselines.
FACTOR, a unified framework with a joint encoder and adaptive cross-attention layers, enhances text-to-video generation by enabling detailed control over objects' appearances and context without fine-tuning.
fine-grained controllable video generation FACTOR unified framework joint encoder adaptive cross-attention layers controllability metrics dense control signals edge maps user-friendly interface object-level fine-grained control