title: Fine-grained Controllable Video Generation via Object Appearance and
    Context
summary: Text-to-video generation has shown promising results. However, by taking only
  natural languages as input, users often face difficulties in providing detailed
  information to precisely control the model's output. In this work, we propose
  fine-grained controllable video generation (FACTOR) to achieve detailed
  control. Specifically, FACTOR aims to control objects' appearances and context,
  including their location and category, in conjunction with the text prompt. To
  achieve detailed control, we propose a unified framework to jointly inject
  control signals into the existing text-to-video model. Our model consists of a
  joint encoder and adaptive cross-attention layers. By optimizing the encoder
  and the inserted layer, we adapt the model to generate videos that are aligned
  with both text prompts and fine-grained control. Compared to existing methods
  relying on dense control signals such as edge maps, we provide a more intuitive
  and user-friendly interface to allow object-level fine-grained control. Our
  method achieves controllability of object appearances without finetuning, which
  reduces the per-subject optimization efforts for the users. Extensive
  experiments on standard benchmark datasets and user-provided inputs validate
  that our model obtains a 70% improvement in controllability metrics over
  competitive baselines.
paper_authors._id: 656fda8234440bdf2ed8027d
paper_authors.name: Hsin-Ping Huang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-12-19T18:06:31.675Z
paper_authors._id: 667b2ee8e005e1dbcc76e2e2
paper_authors.avatarUrl: /avatars/8b2f5f997f0ed5ae9f4f274941933c40.svg
paper_authors.fullname: Hsin-Ping Huang
paper_authors.type: user
paper_authors.user: hsinh
paper_authors._id: 656fda8234440bdf2ed8027e
paper_authors.name: Yu-Chuan Su
paper_authors._id: 656fda8234440bdf2ed8027f
paper_authors.name: Deqing Sun
paper_authors._id: 656fda8234440bdf2ed80280
paper_authors.name: Lu Jiang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2023-12-23T11:18:13.047Z
paper_authors._id: 64a5cba3bea0116f8f7187a7
paper_authors.avatarUrl: /avatars/97b3bad82e359c57d1ab7d6e3aa7748b.svg
paper_authors.fullname: Lu Jiang
paper_authors.type: user
paper_authors.user: roadjiang
paper_authors._id: 656fda8234440bdf2ed80281
paper_authors.name: Xuhui Jia
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-12-06T13:46:17.668Z
paper_authors._id: 634ce7ce05f736dff37aae5f
paper_authors.avatarUrl: /avatars/1405c76a38f7ea497e4439f0e4e786a8.svg
paper_authors.fullname: Xuhui Jia
paper_authors.type: user
paper_authors.user: Jxh-cuit
paper_authors._id: 656fda8234440bdf2ed80282
paper_authors.name: Yukun Zhu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-12-06T13:46:44.840Z
paper_authors._id: 646ec8a3ee36629c15bbf115
paper_authors.avatarUrl: /avatars/169bf2a4a8b46715913be72eaa3ca2c4.svg
paper_authors.fullname: zhuyukun
paper_authors.type: user
paper_authors.user: kun888
paper_authors._id: 656fda8234440bdf2ed80283
paper_authors.name: Ming-Hsuan Yang
paper_title: Fine-grained Controllable Video Generation via Object Appearance and
    Context
paper_summary: Text-to-video generation has shown promising results. However, by taking only
  natural languages as input, users often face difficulties in providing detailed
  information to precisely control the model's output. In this work, we propose
  fine-grained controllable video generation (FACTOR) to achieve detailed
  control. Specifically, FACTOR aims to control objects' appearances and context,
  including their location and category, in conjunction with the text prompt. To
  achieve detailed control, we propose a unified framework to jointly inject
  control signals into the existing text-to-video model. Our model consists of a
  joint encoder and adaptive cross-attention layers. By optimizing the encoder
  and the inserted layer, we adapt the model to generate videos that are aligned
  with both text prompts and fine-grained control. Compared to existing methods
  relying on dense control signals such as edge maps, we provide a more intuitive
  and user-friendly interface to allow object-level fine-grained control. Our
  method achieves controllability of object appearances without finetuning, which
  reduces the per-subject optimization efforts for the users. Extensive
  experiments on standard benchmark datasets and user-provided inputs validate
  that our model obtains a 70% improvement in controllability metrics over
  competitive baselines.
paper_ai_summary: FACTOR, a unified framework with a joint encoder and adaptive cross-attention layers, enhances text-to-video generation by enabling detailed control over objects' appearances and context without fine-tuning.
paper_ai_keywords: fine-grained controllable video generation FACTOR unified framework joint encoder adaptive cross-attention layers controllability metrics dense control signals edge maps user-friendly interface object-level fine-grained control