title: TEDRA: Text-based Editing of Dynamic and Photoreal Actors
summary: Over the past years, significant progress has been made in creating
  photorealistic and drivable 3D avatars solely from videos of real humans.
  However, a core remaining challenge is the fine-grained and user-friendly
  editing of clothing styles by means of textual descriptions. To this end, we
  present TEDRA, the first method allowing text-based edits of an avatar, which
  maintains the avatar's high fidelity, space-time coherency, as well as
  dynamics, and enables skeletal pose and view control. We begin by training a
  model to create a controllable and high-fidelity digital replica of the real
  actor. Next, we personalize a pretrained generative diffusion model by
  fine-tuning it on various frames of the real character captured from different
  camera angles, ensuring the digital representation faithfully captures the
  dynamics and movements of the real person. This two-stage process lays the
  foundation for our approach to dynamic human avatar editing. Utilizing this
  personalized diffusion model, we modify the dynamic avatar based on a provided
  text prompt using our Personalized Normal Aligned Score Distillation Sampling
  (PNA-SDS) within a model-based guidance framework. Additionally, we propose a
  time step annealing strategy to ensure high-quality edits. Our results
  demonstrate a clear improvement over prior work in functionality and visual
  quality.
paper_authors._id: 66cfde5570b7b22757960c9e
paper_authors.name: Basavaraj Sunagad
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-08-29T07:39:38.468Z
paper_authors._id: 656df2b453703dd78fcd8072
paper_authors.avatarUrl: /avatars/1a7d76bb345804821146d064be1de203.svg
paper_authors.fullname: Basavaraj Sunagad
paper_authors.type: user
paper_authors.user: brsunagad
paper_authors._id: 66cfde5570b7b22757960c9f
paper_authors.name: Heming Zhu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-08-29T07:39:45.146Z
paper_authors._id: 64a41b3da3f7783f6c13bced
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/64a41b3da3f7783f6c13bced/M-w8PUAs64hG2IRguZuf7.jpeg
paper_authors.fullname: Heming  Zhu
paper_authors.type: user
paper_authors.user: kv2000
paper_authors._id: 66cfde5570b7b22757960ca0
paper_authors.name: Mohit Mendiratta
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-08-29T07:39:51.254Z
paper_authors._id: 6363ae61ff4b318d1b77a337
paper_authors.avatarUrl: /avatars/6a15518c49c66fef2e4f8b4eb2f56936.svg
paper_authors.fullname: Mohit Mendiratta
paper_authors.type: user
paper_authors.user: Mohit10051994
paper_authors._id: 66cfde5570b7b22757960ca1
paper_authors.name: Adam Kortylewski
paper_authors._id: 66cfde5570b7b22757960ca2
paper_authors.name: Christian Theobalt
paper_authors._id: 66cfde5570b7b22757960ca3
paper_authors.name: Marc Habermann
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-08-29T08:06:56.051Z
paper_authors._id: 66d02ac970b7b22757b2c631
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/-71YM7txHBu67bfS_Re8-.jpeg
paper_authors.fullname: Marc Habermann
paper_authors.type: user
paper_authors.user: MarcHabermann
paper_title: TEDRA: Text-based Editing of Dynamic and Photoreal Actors
paper_summary: Over the past years, significant progress has been made in creating
  photorealistic and drivable 3D avatars solely from videos of real humans.
  However, a core remaining challenge is the fine-grained and user-friendly
  editing of clothing styles by means of textual descriptions. To this end, we
  present TEDRA, the first method allowing text-based edits of an avatar, which
  maintains the avatar's high fidelity, space-time coherency, as well as
  dynamics, and enables skeletal pose and view control. We begin by training a
  model to create a controllable and high-fidelity digital replica of the real
  actor. Next, we personalize a pretrained generative diffusion model by
  fine-tuning it on various frames of the real character captured from different
  camera angles, ensuring the digital representation faithfully captures the
  dynamics and movements of the real person. This two-stage process lays the
  foundation for our approach to dynamic human avatar editing. Utilizing this
  personalized diffusion model, we modify the dynamic avatar based on a provided
  text prompt using our Personalized Normal Aligned Score Distillation Sampling
  (PNA-SDS) within a model-based guidance framework. Additionally, we propose a
  time step annealing strategy to ensure high-quality edits. Our results
  demonstrate a clear improvement over prior work in functionality and visual
  quality.
paper_ai_keywords: TEDRA digital replica generative diffusion model fine-tuning skeletal pose view control dynamic human avatar editing Personalized Normal Aligned Score Distillation Sampling (PNA-SDS) model-based guidance time step annealing high-quality edits