title: Multi-subject Open-set Personalization in Video Generation
summary: Video personalization methods allow us to synthesize videos with specific
  concepts such as people, pets, and places. However, existing methods often
  focus on limited domains, require time-consuming optimization per subject, or
  support only a single subject. We present Video Alchemist - a video model
  with built-in multi-subject, open-set personalization capabilities for both
  foreground objects and background, eliminating the need for time-consuming
  test-time optimization. Our model is built on a new Diffusion Transformer
  module that fuses each conditional reference image and its corresponding
  subject-level text prompt with cross-attention layers. Developing such a large
  model presents two main challenges: dataset and evaluation. First, as paired
  datasets of reference images and videos are extremely hard to collect, we
  sample selected video frames as reference images and synthesize a clip of the
  target video. However, while models can easily denoise training videos given
  reference frames, they fail to generalize to new contexts. To mitigate this
  issue, we design a new automatic data construction pipeline with extensive
  image augmentations. Second, evaluating open-set video personalization is a
  challenge in itself. To address this, we introduce a personalization benchmark
  that focuses on accurate subject fidelity and supports diverse personalization
  scenarios. Finally, our extensive experiments show that our method
  significantly outperforms existing personalization methods in both quantitative
  and qualitative evaluations.
paper_authors._id: 6784c8f85ffcc11850583e40
paper_authors.name: Tsai-Shien Chen
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-01-13T09:39:01.856Z
paper_authors._id: 6330f570b68c7453d2ee8945
paper_authors.avatarUrl: /avatars/5b394a72f277aa6c886b21b477010ab0.svg
paper_authors.fullname: Tsai-Shien Chen
paper_authors.type: user
paper_authors.user: tschen
paper_authors._id: 6784c8f85ffcc11850583e41
paper_authors.name: Aliaksandr Siarohin
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-01-13T09:39:08.671Z
paper_authors._id: 64276311eb9a0ed86180715b
paper_authors.avatarUrl: /avatars/76f933cd549f10e5e2db379de235d304.svg
paper_authors.fullname: Aliaksandr Siarohin
paper_authors.type: user
paper_authors.user: aliaksandr-siarohin
paper_authors._id: 6784c8f85ffcc11850583e42
paper_authors.name: Willi Menapace
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-01-13T09:39:22.216Z
paper_authors._id: 6315358a362e3e95ea538081
paper_authors.avatarUrl: /avatars/3b089a25a87c2e83c6b23ccb5d2dc73e.svg
paper_authors.fullname: Willi Menapace
paper_authors.type: user
paper_authors.user: willi-menapace
paper_authors._id: 6784c8f85ffcc11850583e43
paper_authors.name: Yuwei Fang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-01-13T09:39:37.004Z
paper_authors._id: 6406d1dcc3ab325efa9db198
paper_authors.avatarUrl: /avatars/781f3cd455596f7cffa0906cc487f79b.svg
paper_authors.fullname: Yuwei Fang
paper_authors.type: user
paper_authors.user: studyfang
paper_authors._id: 6784c8f85ffcc11850583e44
paper_authors.name: Kwot Sin Lee
paper_authors._id: 6784c8f85ffcc11850583e45
paper_authors.name: Ivan Skorokhodov
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-01-13T09:39:50.742Z
paper_authors._id: 63610db13c7147fae7de88e3
paper_authors.avatarUrl: /avatars/d7e97a16cfee39e1e50d7a5b747876f1.svg
paper_authors.fullname: Ivan Skorokhodov
paper_authors.type: user
paper_authors.user: universome
paper_authors._id: 6784c8f85ffcc11850583e46
paper_authors.name: Kfir Aberman
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-01-13T09:39:55.987Z
paper_authors._id: 64db29097266618e853dd6ec
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/64db29097266618e853dd6ec/r0MaPQCfAxeKv3ycdKYLK.jpeg
paper_authors.fullname: Kfir Aberman
paper_authors.type: user
paper_authors.user: kaberman
paper_authors._id: 6784c8f85ffcc11850583e47
paper_authors.name: Jun-Yan Zhu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-01-13T09:40:02.174Z
paper_authors._id: 63a0acc32fabbbb899952a2b
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1671474335794-noauth.jpeg
paper_authors.fullname: Jun-Yan Zhu
paper_authors.type: user
paper_authors.user: junyanz
paper_authors._id: 6784c8f85ffcc11850583e48
paper_authors.name: Ming-Hsuan Yang
paper_authors._id: 6784c8f85ffcc11850583e49
paper_authors.name: Sergey Tulyakov
paper_title: Multi-subject Open-set Personalization in Video Generation
paper_summary: Video personalization methods allow us to synthesize videos with specific
  concepts such as people, pets, and places. However, existing methods often
  focus on limited domains, require time-consuming optimization per subject, or
  support only a single subject. We present Video Alchemist - a video model
  with built-in multi-subject, open-set personalization capabilities for both
  foreground objects and background, eliminating the need for time-consuming
  test-time optimization. Our model is built on a new Diffusion Transformer
  module that fuses each conditional reference image and its corresponding
  subject-level text prompt with cross-attention layers. Developing such a large
  model presents two main challenges: dataset and evaluation. First, as paired
  datasets of reference images and videos are extremely hard to collect, we
  sample selected video frames as reference images and synthesize a clip of the
  target video. However, while models can easily denoise training videos given
  reference frames, they fail to generalize to new contexts. To mitigate this
  issue, we design a new automatic data construction pipeline with extensive
  image augmentations. Second, evaluating open-set video personalization is a
  challenge in itself. To address this, we introduce a personalization benchmark
  that focuses on accurate subject fidelity and supports diverse personalization
  scenarios. Finally, our extensive experiments show that our method
  significantly outperforms existing personalization methods in both quantitative
  and qualitative evaluations.
paper_ai_summary: Video Alchemist uses a Diffusion Transformer with cross-attention to enable multi-subject personalization without test-time optimization, outperforming existing methods.
paper_ai_keywords: Diffusion Transformer cross-attention video personalization subject fidelity automatic data construction pipeline