title: Interpreting the Weight Space of Customized Diffusion Models
summary: We investigate the space of weights spanned by a large collection of
  customized diffusion models. We populate this space by creating a dataset of
  over 60,000 models, each of which is a base model fine-tuned to insert a
  different person's visual identity. We model the underlying manifold of these
  weights as a subspace, which we term weights2weights. We demonstrate three
  immediate applications of this space -- sampling, editing, and inversion.
  First, as each point in the space corresponds to an identity, sampling a set of
  weights from it results in a model encoding a novel identity. Next, we find
  linear directions in this space corresponding to semantic edits of the identity
  (e.g., adding a beard). These edits persist in appearance across generated
  samples. Finally, we show that inverting a single image into this space
  reconstructs a realistic identity, even if the input image is out of
  distribution (e.g., a painting). Our results indicate that the weight space of
  fine-tuned diffusion models behaves as an interpretable latent space of
  identities.
paper_authors._id: 666bae6c3214e94388ea7606
paper_authors.name: Amil Dravid
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-06-14T07:22:40.931Z
paper_authors._id: 655e490d3beaa281e50aff5f
paper_authors.avatarUrl: /avatars/7e6d9a096a71d601f66279e317d11e1f.svg
paper_authors.fullname: Amil Dravid
paper_authors.type: user
paper_authors.user: amildravid4292
paper_authors._id: 666bae6c3214e94388ea7607
paper_authors.name: Yossi Gandelsman
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-06-14T09:57:15.404Z
paper_authors._id: 62f6942205ca68c0e0ff7b97
paper_authors.avatarUrl: /avatars/b0600de531b4c88f4aa7c30c5ceb06e1.svg
paper_authors.fullname: Yossi Gandelsman
paper_authors.type: user
paper_authors.user: yossig
paper_authors._id: 666bae6c3214e94388ea7608
paper_authors.name: Kuan-Chieh Wang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-06-14T09:57:20.946Z
paper_authors._id: 648ca58a39d2584ee47efef6
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/648ca58a39d2584ee47efef6/R7B72bnwc59mdK45rmzYS.png
paper_authors.fullname: Kuan-Chieh Wang
paper_authors.type: user
paper_authors.user: wangkua1
paper_authors._id: 666bae6c3214e94388ea7609
paper_authors.name: Rameen Abdal
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-06-14T09:57:26.935Z
paper_authors._id: 630428fa7b50dd9d0a38cde0
paper_authors.avatarUrl: /avatars/d1baf7fd17daf4be16ba5bd6cd4f2277.svg
paper_authors.fullname: Rameen Abdal
paper_authors.type: user
paper_authors.user: RameenAbdal
paper_authors._id: 666bae6c3214e94388ea760a
paper_authors.name: Gordon Wetzstein
paper_authors._id: 666bae6c3214e94388ea760b
paper_authors.name: Alexei A. Efros
paper_authors._id: 666bae6c3214e94388ea760c
paper_authors.name: Kfir Aberman
paper_title: Interpreting the Weight Space of Customized Diffusion Models
paper_summary: We investigate the space of weights spanned by a large collection of
  customized diffusion models. We populate this space by creating a dataset of
  over 60,000 models, each of which is a base model fine-tuned to insert a
  different person's visual identity. We model the underlying manifold of these
  weights as a subspace, which we term weights2weights. We demonstrate three
  immediate applications of this space -- sampling, editing, and inversion.
  First, as each point in the space corresponds to an identity, sampling a set of
  weights from it results in a model encoding a novel identity. Next, we find
  linear directions in this space corresponding to semantic edits of the identity
  (e.g., adding a beard). These edits persist in appearance across generated
  samples. Finally, we show that inverting a single image into this space
  reconstructs a realistic identity, even if the input image is out of
  distribution (e.g., a painting). Our results indicate that the weight space of
  fine-tuned diffusion models behaves as an interpretable latent space of
  identities.
paper_ai_summary: A large dataset of fine-tuned diffusion models for different visual identities reveals a manifold representing identity in the weight space, enabling sampling, semantic editing, and inversion tasks.
paper_ai_keywords: diffusion models weights2weights sampling editing inversion