LoRA.rar: Learning to Merge LoRAs via Hypernetworks for Subject-Style
  Conditioned Image Generation
Recent advancements in image generation models have enabled personalized
image creation with both user-defined subjects (content) and styles. Prior
works achieved personalization by merging corresponding low-rank adaptation
parameters (LoRAs) through optimization-based methods, which are
computationally demanding and unsuitable for real-time use on
resource-constrained devices like smartphones. To address this, we introduce
LoRA.rar, a method that not only improves image quality but also achieves a
remarkable speedup of over 4000times in the merging process. LoRA.rar
pre-trains a hypernetwork on a diverse set of content-style LoRA pairs,
learning an efficient merging strategy that generalizes to new, unseen
content-style pairs, enabling fast, high-quality personalization. Moreover, we
identify limitations in existing evaluation metrics for content-style quality
and propose a new protocol using multimodal large language models (MLLM) for
more accurate assessment. Our method significantly outperforms the current
state of the art in both content and style fidelity, as validated by MLLM
assessments and human evaluations.
67570c548e4eeb4b6f040bef
Donald Shenaj
claimed_verified
2024-12-11T09:20:52.312Z
651ed00e8ed730cc0066c2fb
/avatars/2790728f588075cf767a0db157115d19.svg
Donald Shenaj
user
donaldssh
67570c548e4eeb4b6f040bf0
Ondrej Bohdal
claimed_verified
2024-12-11T09:20:49.872Z
65b11831c9a5a7680faa8cc9
/avatars/e14a0f7ee2883926e7f6e146365c7ccf.svg
Ondrej Bohdal
user
obohdal
67570c548e4eeb4b6f040bf1
Mete Ozay
admin_assigned
2024-12-11T13:50:54.799Z
65bced24c63d6a8d7f400b8b
/avatars/91212d7f39eba91158d7a0d0c823bc45.svg
Mete Ozay
user
mosams
67570c548e4eeb4b6f040bf2
Pietro Zanuttigh
67570c548e4eeb4b6f040bf3
Umberto Michieli
admin_assigned
2024-12-11T13:51:08.508Z
646f1879c261dc4133830dd6
/avatars/f4134a8fb171f93c098a43f1c7a94eb6.svg
Umberto Michieli
user
u-michieli
LoRA.rar: Learning to Merge LoRAs via Hypernetworks for Subject-Style
  Conditioned Image Generation
Recent advancements in image generation models have enabled personalized
image creation with both user-defined subjects (content) and styles. Prior
works achieved personalization by merging corresponding low-rank adaptation
parameters (LoRAs) through optimization-based methods, which are
computationally demanding and unsuitable for real-time use on
resource-constrained devices like smartphones. To address this, we introduce
LoRA.rar, a method that not only improves image quality but also achieves a
remarkable speedup of over 4000times in the merging process. LoRA.rar
pre-trains a hypernetwork on a diverse set of content-style LoRA pairs,
learning an efficient merging strategy that generalizes to new, unseen
content-style pairs, enabling fast, high-quality personalization. Moreover, we
identify limitations in existing evaluation metrics for content-style quality
and propose a new protocol using multimodal large language models (MLLM) for
more accurate assessment. Our method significantly outperforms the current
state of the art in both content and style fidelity, as validated by MLLM
assessments and human evaluations.
LoRA.rar pre-trains a hypernetwork for efficient merging of content-style parameters, enabling fast and high-quality image personalization.
low-rank adaptation LoRAs hypernetwork content-style quality multimodal large language models MLLM