PaRa: Personalizing Text-to-Image Diffusion via Parameter Rank Reduction
Shangyu Chen Zizheng Pan Jianfei Cai Dinh Phung
Personalizing a large-scale pretrained Text-to-Image (T2I) diffusion model is chal-
lenging as it typically struggles to make an appropriate trade-off between its training
data distribution and the target distribution, i.e., learning a novel concept with only a
few target images to achieve personalization (aligning with the personalized target)
while preserving text editability (aligning with diverse text prompts). In this paper,
we propose PaRa, an effective and efficient Parameter Rank Reduction approach
for T2I model personalization by explicitly controlling the rank of the diffusion
model parameters to restrict its initial diverse generation space into a small and
well-balanced target space. Our design is motivated by the fact that taming a T2I
model toward a novel concept such as a specific art style implies a small generation
space. To this end, by reducing the rank of model parameters during finetuning, we
can effectively constrain the space of the denoising sampling trajectories towards
the target. With comprehensive experiments, we show that PaRa achieves great
advantages over existing finetuning approaches on single/multi-subject generation
as well as single-image editing. Notably, compared to the prevailing fine-tuning
technique LoRA, PaRa achieves better parameter efficiency (2× fewer learnable
parameters) and much better target image alignment.
Text-to-Image diffusion model, Diffusion model fine-tuning
Achieve lightweight and reliable personalized models through the subspace of the pre-trained Stable Diffusion model.