Iterative Object Count Optimization for Text-to-image Diffusion Models
We address a persistent challenge in text-to-image models: accurately
generating a specified number of objects. Current models, which learn from
image-text pairs, inherently struggle with counting, as training data cannot
depict every possible number of objects for any given object. To solve this, we
propose optimizing the generated image based on a counting loss derived from a
counting model that aggregates an object\'s potential. Employing an
out-of-the-box counting model is challenging for two reasons: first, the model
requires a scaling hyperparameter for the potential aggregation that varies
depending on the viewpoint of the objects, and second, classifier guidance
techniques require modified models that operate on noisy intermediate diffusion
steps. To address these challenges, we propose an iterated online training mode
that improves the accuracy of inferred images while altering the text
conditioning embedding and dynamically adjusting hyperparameters. Our method
offers three key advantages: (i) it can consider non-derivable counting
techniques based on detection models, (ii) it is a zero-shot plug-and-play
solution facilitating rapid changes to the counting techniques and image
generation methods, and (iii) the optimized counting token can be reused to
generate accurate images without additional optimization. We evaluate the
generation of various objects and show significant improvements in accuracy.
The project page is available at https://ozzafar.github.io/count_token.
66c767219e496a77e83e1954
Oz Zafar
66c767219e496a77e83e1955
Lior Wolf
66c767219e496a77e83e1956
Idan Schwartz
Iterative Object Count Optimization for Text-to-image Diffusion Models
We address a persistent challenge in text-to-image models: accurately
generating a specified number of objects. Current models, which learn from
image-text pairs, inherently struggle with counting, as training data cannot
depict every possible number of objects for any given object. To solve this, we
propose optimizing the generated image based on a counting loss derived from a
counting model that aggregates an object\'s potential. Employing an
out-of-the-box counting model is challenging for two reasons: first, the model
requires a scaling hyperparameter for the potential aggregation that varies
depending on the viewpoint of the objects, and second, classifier guidance
techniques require modified models that operate on noisy intermediate diffusion
steps. To address these challenges, we propose an iterated online training mode
that improves the accuracy of inferred images while altering the text
conditioning embedding and dynamically adjusting hyperparameters. Our method
offers three key advantages: (i) it can consider non-derivable counting
techniques based on detection models, (ii) it is a zero-shot plug-and-play
solution facilitating rapid changes to the counting techniques and image
generation methods, and (iii) the optimized counting token can be reused to
generate accurate images without additional optimization. We evaluate the
generation of various objects and show significant improvements in accuracy.
The project page is available at https://ozzafar.github.io/count_token.
A novel approach improves text-to-image generation accuracy by integrating counting techniques and dynamically adjusting hyperparameters during iterative online training.
text-to-image models counting model counting loss potential aggregation viewpoint classifier guidance diffusion steps text conditioning embedding zero-shot plug-and-play counting token