  Diffusion models have achieved great progress in face generation. However,
these models amplify the bias in the generation process, leading to an
imbalance in distribution of sensitive attributes such as age, gender and race.
This paper proposes a novel solution to this problem by balancing the facial
attributes of the generated images. We mitigate the bias by localizing the
means of the facial attributes in the latent space of the diffusion model using
Gaussian mixture models (GMM). Our motivation for choosing GMMs over other
clustering frameworks comes from the flexible latent structure of diffusion
model. Since each sampling step in diffusion models follows a Gaussian
distribution, we show that fitting a GMM model helps us to localize the
subspace responsible for generating a specific attribute. Furthermore, our
method does not require retraining, we instead localize the subspace on-the-fly
and mitigate the bias for generating a fair dataset. We evaluate our approach
on multiple face attribute datasets to demonstrate the effectiveness of our
approach. Our results demonstrate that our approach leads to a more fair data
generation in terms of representational fairness while preserving the quality
of generated samples.

Computer Vision and Pattern Recognition;Computers and Society