title: DiffuseKronA: A Parameter Efficient Fine-tuning Method for Personalized
    Diffusion Model
summary: In the realm of subject-driven text-to-image (T2I) generative models, recent
  developments like DreamBooth and BLIP-Diffusion have led to impressive results
  yet encounter limitations due to their intensive fine-tuning demands and
  substantial parameter requirements. While the low-rank adaptation (LoRA) module
  within DreamBooth offers a reduction in trainable parameters, it introduces a
  pronounced sensitivity to hyperparameters, leading to a compromise between
  parameter efficiency and the quality of T2I personalized image synthesis.
  Addressing these constraints, we introduce \textit{DiffuseKronA}, a
  novel Kronecker product-based adaptation module that not only significantly
  reduces the parameter count by 35\% and 99.947\% compared to LoRA-DreamBooth
  and the original DreamBooth, respectively, but also enhances the quality of
  image synthesis. Crucially, DiffuseKronA mitigates the issue of
  hyperparameter sensitivity, delivering consistent high-quality generations
  across a wide range of hyperparameters, thereby diminishing the necessity for
  extensive fine-tuning. Furthermore, a more controllable decomposition makes
  DiffuseKronA more interpretable and even can achieve up to a 50\%
  reduction with results comparable to LoRA-Dreambooth. Evaluated against diverse
  and complex input images and text prompts, DiffuseKronA consistently
  outperforms existing models, producing diverse images of higher quality with
  improved fidelity and a more accurate color distribution of objects, all the
  while upholding exceptional parameter efficiency, thus presenting a substantial
  advancement in the field of T2I generative modeling. Our project page,
  consisting of links to the code, and pre-trained checkpoints, is available at
  https://diffusekrona.github.io/{https://diffusekrona.github.io/}.
paper_authors._id: 65ded73907334f484a6d53ff
paper_authors.name: Shyam Marjit
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-02-28T12:14:40.002Z
paper_authors._id: 6400fde28ba76abe4b6f8af6
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/6400fde28ba76abe4b6f8af6/UTIl4ygTLwpDBDloY3ldW.jpeg
paper_authors.fullname: Shyam Marjit
paper_authors.type: user
paper_authors.user: ShyamMarjit
paper_authors._id: 65ded73907334f484a6d5400
paper_authors.name: Harshit Singh
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-03-15T08:05:56.586Z
paper_authors._id: 646e3c668dc026869caf69f0
paper_authors.avatarUrl: /avatars/f87a452e7e4b048148b10361e6cdbf3f.svg
paper_authors.fullname: Harshit Singh
paper_authors.type: user
paper_authors.user: Harshan14
paper_authors._id: 65ded73907334f484a6d5401
paper_authors.name: Nityanand Mathur
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-02-28T12:57:52.398Z
paper_authors._id: 635c0de2cb0f36a40bb96fce
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/635c0de2cb0f36a40bb96fce/pqURAHT6eWbaHDWM-YJbM.jpeg
paper_authors.fullname: Nityanand Mathur
paper_authors.type: user
paper_authors.user: nityanandmathur
paper_authors._id: 65ded73907334f484a6d5402
paper_authors.name: Sayak Paul
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-02-28T12:15:39.061Z
paper_authors._id: 5f7fbd813e94f16a85448745
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1649681653581-5f7fbd813e94f16a85448745.jpeg
paper_authors.fullname: Sayak Paul
paper_authors.type: user
paper_authors.user: sayakpaul
paper_authors._id: 65ded73907334f484a6d5403
paper_authors.name: Chia-Mu Yu
paper_authors._id: 65ded73907334f484a6d5404
paper_authors.name: Pin-Yu Chen
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-02-28T12:15:54.986Z
paper_authors._id: 6495dd0b71f6708e0f990032
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/6495dd0b71f6708e0f990032/PBIjdKNnpkxvR_3djCGVm.png
paper_authors.fullname: Pin-Yu Chen
paper_authors.type: user
paper_authors.user: pinyuchen
paper_title: DiffuseKronA: A Parameter Efficient Fine-tuning Method for Personalized
    Diffusion Model
paper_summary: In the realm of subject-driven text-to-image (T2I) generative models, recent
  developments like DreamBooth and BLIP-Diffusion have led to impressive results
  yet encounter limitations due to their intensive fine-tuning demands and
  substantial parameter requirements. While the low-rank adaptation (LoRA) module
  within DreamBooth offers a reduction in trainable parameters, it introduces a
  pronounced sensitivity to hyperparameters, leading to a compromise between
  parameter efficiency and the quality of T2I personalized image synthesis.
  Addressing these constraints, we introduce \textit{DiffuseKronA}, a
  novel Kronecker product-based adaptation module that not only significantly
  reduces the parameter count by 35\% and 99.947\% compared to LoRA-DreamBooth
  and the original DreamBooth, respectively, but also enhances the quality of
  image synthesis. Crucially, DiffuseKronA mitigates the issue of
  hyperparameter sensitivity, delivering consistent high-quality generations
  across a wide range of hyperparameters, thereby diminishing the necessity for
  extensive fine-tuning. Furthermore, a more controllable decomposition makes
  DiffuseKronA more interpretable and even can achieve up to a 50\%
  reduction with results comparable to LoRA-Dreambooth. Evaluated against diverse
  and complex input images and text prompts, DiffuseKronA consistently
  outperforms existing models, producing diverse images of higher quality with
  improved fidelity and a more accurate color distribution of objects, all the
  while upholding exceptional parameter efficiency, thus presenting a substantial
  advancement in the field of T2I generative modeling. Our project page,
  consisting of links to the code, and pre-trained checkpoints, is available at
  https://diffusekrona.github.io/{https://diffusekrona.github.io/}.
paper_ai_summary: DiffuseKronA, a Kronecker product-based adaptation module, reduces parameter counts and enhances quality in text-to-image generative models compared to LoRA and DreamBooth.
paper_ai_keywords: subject-driven text-to-image DreamBooth BLIP-Diffusion low-rank adaptation Kronecker product DiffuseKronA hyperparameter sensitivity parameter efficiency image synthesis interpretability