title: Dynamic Concepts Personalization from Single Videos
summary: Personalizing generative text-to-image models has seen remarkable progress,
  but extending this personalization to text-to-video models presents unique
  challenges. Unlike static concepts, personalizing text-to-video models has the
  potential to capture dynamic concepts, i.e., entities defined not only by their
  appearance but also by their motion. In this paper, we introduce
  Set-and-Sequence, a novel framework for personalizing Diffusion Transformers
  (DiTs)-based generative video models with dynamic concepts. Our approach
  imposes a spatio-temporal weight space within an architecture that does not
  explicitly separate spatial and temporal features. This is achieved in two key
  stages. First, we fine-tune Low-Rank Adaptation (LoRA) layers using an
  unordered set of frames from the video to learn an identity LoRA basis that
  represents the appearance, free from temporal interference. In the second
  stage, with the identity LoRAs frozen, we augment their coefficients with
  Motion Residuals and fine-tune them on the full video sequence, capturing
  motion dynamics. Our Set-and-Sequence framework results in a spatio-temporal
  weight space that effectively embeds dynamic concepts into the video model's
  output domain, enabling unprecedented editability and compositionality while
  setting a new benchmark for personalizing dynamic concepts.
paper_authors._id: 67b7f5ee8b3dff28b749be78
paper_authors.name: Rameen Abdal
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-02-21T15:00:23.357Z
paper_authors._id: 630428fa7b50dd9d0a38cde0
paper_authors.avatarUrl: /avatars/d1baf7fd17daf4be16ba5bd6cd4f2277.svg
paper_authors.fullname: Rameen Abdal
paper_authors.type: user
paper_authors.user: RameenAbdal
paper_authors._id: 67b7f5ee8b3dff28b749be79
paper_authors.name: Or Patashnik
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-02-21T15:00:32.474Z
paper_authors._id: 62853516e483e0d37b354ce1
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/62853516e483e0d37b354ce1/t5Tyd3E07w26B9Z3XpZWI.jpeg
paper_authors.fullname: Or Patashnik
paper_authors.type: user
paper_authors.user: orpatashnik
paper_authors._id: 67b7f5ee8b3dff28b749be7a
paper_authors.name: Ivan Skorokhodov
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-02-21T15:00:39.038Z
paper_authors._id: 63610db13c7147fae7de88e3
paper_authors.avatarUrl: /avatars/d7e97a16cfee39e1e50d7a5b747876f1.svg
paper_authors.fullname: Ivan Skorokhodov
paper_authors.type: user
paper_authors.user: universome
paper_authors._id: 67b7f5ee8b3dff28b749be7b
paper_authors.name: Willi Menapace
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-02-21T15:00:45.032Z
paper_authors._id: 6315358a362e3e95ea538081
paper_authors.avatarUrl: /avatars/3b089a25a87c2e83c6b23ccb5d2dc73e.svg
paper_authors.fullname: Willi Menapace
paper_authors.type: user
paper_authors.user: willi-menapace
paper_authors._id: 67b7f5ee8b3dff28b749be7c
paper_authors.name: Aliaksandr Siarohin
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-02-21T15:00:51.221Z
paper_authors._id: 64276311eb9a0ed86180715b
paper_authors.avatarUrl: /avatars/76f933cd549f10e5e2db379de235d304.svg
paper_authors.fullname: Aliaksandr Siarohin
paper_authors.type: user
paper_authors.user: aliaksandr-siarohin
paper_authors._id: 67b7f5ee8b3dff28b749be7d
paper_authors.name: Sergey Tulyakov
paper_authors._id: 67b7f5ee8b3dff28b749be7e
paper_authors.name: Daniel Cohen-Or
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-02-21T15:01:00.737Z
paper_authors._id: 628507161949ebcae8e24ec3
paper_authors.avatarUrl: /avatars/008ecb3daa4c8187b5f339f1176b3c39.svg
paper_authors.fullname: Daniel Cohen-Or
paper_authors.type: user
paper_authors.user: cohenor
paper_authors._id: 67b7f5ee8b3dff28b749be7f
paper_authors.name: Kfir Aberman
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-02-21T15:01:06.434Z
paper_authors._id: 64db29097266618e853dd6ec
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/64db29097266618e853dd6ec/r0MaPQCfAxeKv3ycdKYLK.jpeg
paper_authors.fullname: Kfir Aberman
paper_authors.type: user
paper_authors.user: kaberman
paper_title: Dynamic Concepts Personalization from Single Videos
paper_summary: Personalizing generative text-to-image models has seen remarkable progress,
  but extending this personalization to text-to-video models presents unique
  challenges. Unlike static concepts, personalizing text-to-video models has the
  potential to capture dynamic concepts, i.e., entities defined not only by their
  appearance but also by their motion. In this paper, we introduce
  Set-and-Sequence, a novel framework for personalizing Diffusion Transformers
  (DiTs)-based generative video models with dynamic concepts. Our approach
  imposes a spatio-temporal weight space within an architecture that does not
  explicitly separate spatial and temporal features. This is achieved in two key
  stages. First, we fine-tune Low-Rank Adaptation (LoRA) layers using an
  unordered set of frames from the video to learn an identity LoRA basis that
  represents the appearance, free from temporal interference. In the second
  stage, with the identity LoRAs frozen, we augment their coefficients with
  Motion Residuals and fine-tune them on the full video sequence, capturing
  motion dynamics. Our Set-and-Sequence framework results in a spatio-temporal
  weight space that effectively embeds dynamic concepts into the video model's
  output domain, enabling unprecedented editability and compositionality while
  setting a new benchmark for personalizing dynamic concepts.
paper_ai_summary: The Set-and-Sequence framework personalizes Diffusion Transformers for generative video models by learning spatial identities and motion dynamics, significantly enhancing video editability and performance.
paper_ai_keywords: Diffusion Transformers Low-Rank Adaptation Motion Residuals spatio-temporal weight space identity LoRA spatial identities motion dynamics