Make It Count: Text-to-Image Generation with an Accurate Number of
  Objects
Despite the unprecedented success of text-to-image diffusion models,
controlling the number of depicted objects using text is surprisingly hard.
This is important for various applications from technical documents, to
children's books to illustrating cooking recipes. Generating object-correct
counts is fundamentally challenging because the generative model needs to keep
a sense of separate identity for every instance of the object, even if several
objects look identical or overlap, and then carry out a global computation
implicitly during generation. It is still unknown if such representations
exist. To address count-correct generation, we first identify features within
the diffusion model that can carry the object identity information. We then use
them to separate and count instances of objects during the denoising process
and detect over-generation and under-generation. We fix the latter by training
a model that predicts both the shape and location of a missing object, based on
the layout of existing ones, and show how it can be used to guide denoising
with correct object count. Our approach, CountGen, does not depend on external
source to determine object layout, but rather uses the prior from the diffusion
model itself, creating prompt-dependent and seed-dependent layouts. Evaluated
on two benchmark datasets, we find that CountGen strongly outperforms the
count-accuracy of existing baselines.
666fe1d5b955b0e6551b19a8
Lital Binyamin
admin_assigned
2024-06-17T14:11:52.674Z
66701c543f16ff3d4540681e
/avatars/cfdfd398d4600dad3a17c20424033a6b.svg
Lital Binyamin
user
Litalby
666fe1d5b955b0e6551b19a9
Yoad Tewel
admin_assigned
2024-06-17T14:13:05.460Z
6201838f305742efd5a6a0e3
/avatars/4e47ab9ef1a4b9f59756a6d8c90a970f.svg
Yoad Tewel
user
YoadTew
666fe1d5b955b0e6551b19aa
Hilit Segev
admin_assigned
2024-06-17T14:13:11.770Z
6492bc477ddf9471f55de3b8
/avatars/ba42cf14b1ae558bd29e3ff51e59531a.svg
Hilit Segev
user
Hilit
666fe1d5b955b0e6551b19ab
Eran Hirsch
claimed_verified
2025-06-03T10:45:26.366Z
609eb1fc1172dedeac2200db
/avatars/a7384a8d3389610b38388c100a28c86d.svg
H
user
Eran
666fe1d5b955b0e6551b19ac
Royi Rassin
admin_assigned
2024-06-17T14:13:21.522Z
62a7581cf049be35252a2e7c
/avatars/91de4eb48f51bfd6e028c08ccfa98f8c.svg
Royi Rassin
user
Royir
666fe1d5b955b0e6551b19ad
Gal Chechik
admin_assigned
2024-06-17T14:13:27.138Z
6493393f357b252af72196c5
https://cdn-avatars.huggingface.co/v1/production/uploads/6493393f357b252af72196c5/EWSy18XRcMRa_4XMM3Fu-.jpeg
Gal Chechik
user
galchechik
Make It Count: Text-to-Image Generation with an Accurate Number of
  Objects
Despite the unprecedented success of text-to-image diffusion models,
controlling the number of depicted objects using text is surprisingly hard.
This is important for various applications from technical documents, to
children's books to illustrating cooking recipes. Generating object-correct
counts is fundamentally challenging because the generative model needs to keep
a sense of separate identity for every instance of the object, even if several
objects look identical or overlap, and then carry out a global computation
implicitly during generation. It is still unknown if such representations
exist. To address count-correct generation, we first identify features within
the diffusion model that can carry the object identity information. We then use
them to separate and count instances of objects during the denoising process
and detect over-generation and under-generation. We fix the latter by training
a model that predicts both the shape and location of a missing object, based on
the layout of existing ones, and show how it can be used to guide denoising
with correct object count. Our approach, CountGen, does not depend on external
source to determine object layout, but rather uses the prior from the diffusion
model itself, creating prompt-dependent and seed-dependent layouts. Evaluated
on two benchmark datasets, we find that CountGen strongly outperforms the
count-accuracy of existing baselines.
CountGen enhances text-to-image diffusion models by accurately controlling the number of depicted objects using features that carry object identity information, surpassing existing methods on benchmark datasets.
diffusion models denoising process object identity information count-correct generation CountGen