Greedy Growing Enables High-Resolution Pixel-Based Diffusion Models
We address the long-standing problem of how to learn effective pixel-based
image diffusion models at scale, introducing a remarkably simple greedy growing
method for stable training of large-scale, high-resolution models. without the
needs for cascaded super-resolution components. The key insight stems from
careful pre-training of core components, namely, those responsible for
text-to-image alignment {\it vs.} high-resolution rendering. We first
demonstrate the benefits of scaling a {\it Shallow UNet}, with no
down(up)-sampling enc(dec)oder. Scaling its deep core layers is shown to
improve alignment, object structure, and composition. Building on this core
model, we propose a greedy algorithm that grows the architecture into
high-resolution end-to-end models, while preserving the integrity of the
pre-trained representation, stabilizing training, and reducing the need for
large high-resolution datasets. This enables a single stage model capable of
generating high-resolution images without the need of a super-resolution
cascade. Our key results rely on public datasets and show that we are able to
train non-cascaded models up to 8B parameters with no further regularization
schemes. Vermeer, our full pipeline model trained with internal datasets to
produce 1024x1024 images, without cascades, is preferred by 44.0% vs. 21.4%
human evaluators over SDXL.
66554c9dac73f7759fa8ba1c
Cristina N. Vasconcelos
66554c9dac73f7759fa8ba1d
Abdullah Rashwan Austin Waters
66554c9dac73f7759fa8ba1e
Trevor Walker
66554c9dac73f7759fa8ba1f
Keyang Xu
66554c9dac73f7759fa8ba20
Jimmy Yan
66554c9dac73f7759fa8ba21
Rui Qian
66554c9dac73f7759fa8ba22
Shixin Luo
66554c9dac73f7759fa8ba23
Zarana Parekh
66554c9dac73f7759fa8ba24
Andrew Bunner
66554c9dac73f7759fa8ba25
Hongliang Fei
admin_assigned
2024-05-28T15:47:48.818Z
65aeb55e66ad44e2d567620c
/avatars/36b98e143c465b852a26f14307e571ca.svg
Hongliang Fei
user
feihongliang0
66554c9dac73f7759fa8ba26
Roopal Garg
admin_assigned
2024-05-28T15:47:55.198Z
6490ffd4f049c475bdb6a40f
https://cdn-avatars.huggingface.co/v1/production/uploads/6490ffd4f049c475bdb6a40f/EQQQamUGoBsoq0w_BFflM.jpeg
Roopal Garg
user
roopalgarg
66554c9dac73f7759fa8ba27
Mandy Guo
66554c9dac73f7759fa8ba28
Ivana Kajic
66554c9dac73f7759fa8ba29
Yeqing Li
admin_assigned
2024-05-28T15:48:09.284Z
65656057e5aac326bfb79a7c
/avatars/bb2b7956d47a022de3dab765459fd031.svg
Yeqing Li
user
yeqingli
66554c9dac73f7759fa8ba2a
Henna Nandwani
66554c9dac73f7759fa8ba2b
Jordi Pont-Tuset
admin_assigned
2024-05-28T15:48:19.659Z
66335b4b07f864a2f94b403d
/avatars/0a290942c4a282b48f870500cc6424b5.svg
Jordi Pont Tuset
user
jponttuset
66554c9dac73f7759fa8ba2c
Yasumasa Onoe
admin_assigned
2024-05-28T15:48:34.871Z
60591867ce97b2b478a3dc4d
/avatars/bc8fedbca5f60f99fe66b9ac37ab9a1a.svg
Yasumasa Onoe
user
yasumasaonoe
66554c9dac73f7759fa8ba2d
Sarah Rosston
admin_assigned
2024-05-28T15:48:41.982Z
64011dec0251f0f411684604
/avatars/aeb803b23ef4a7b2e24f9693c4f4c02a.svg
Sarah Rosston
user
srosston
66554c9dac73f7759fa8ba2e
Su Wang
66554c9dac73f7759fa8ba2f
Wenlei Zhou
66554c9dac73f7759fa8ba30
Kevin Swersky
admin_assigned
2024-05-28T15:49:06.032Z
630e6ef664f1f8d0c771b758
/avatars/1df2bfadb2b6fdf8307189936efc6ef0.svg
Kevin Swersky
user
kswersky
66554c9dac73f7759fa8ba31
David J. Fleet
66554c9dac73f7759fa8ba32
Jason M. Baldridge
admin_assigned
2024-05-28T15:49:16.516Z
630c9f648e3ff0c72327e8ee
/avatars/1a5262c6575daf92f62f5cc6d9f1f00f.svg
Jason Baldridge
user
jasonbaldridge
66554c9dac73f7759fa8ba33
Oliver Wang
admin_assigned
2024-05-28T15:49:32.673Z
62fa8bfdc95d426ff8f25fe9
/avatars/71fe9cff4ce79f6a208045984852e5ad.svg
Oliver Wang
user
oliverwang
Greedy Growing Enables High-Resolution Pixel-Based Diffusion Models
We address the long-standing problem of how to learn effective pixel-based
image diffusion models at scale, introducing a remarkably simple greedy growing
method for stable training of large-scale, high-resolution models. without the
needs for cascaded super-resolution components. The key insight stems from
careful pre-training of core components, namely, those responsible for
text-to-image alignment {\it vs.} high-resolution rendering. We first
demonstrate the benefits of scaling a {\it Shallow UNet}, with no
down(up)-sampling enc(dec)oder. Scaling its deep core layers is shown to
improve alignment, object structure, and composition. Building on this core
model, we propose a greedy algorithm that grows the architecture into
high-resolution end-to-end models, while preserving the integrity of the
pre-trained representation, stabilizing training, and reducing the need for
large high-resolution datasets. This enables a single stage model capable of
generating high-resolution images without the need of a super-resolution
cascade. Our key results rely on public datasets and show that we are able to
train non-cascaded models up to 8B parameters with no further regularization
schemes. Vermeer, our full pipeline model trained with internal datasets to
produce 1024x1024 images, without cascades, is preferred by 44.0% vs. 21.4%
human evaluators over SDXL.
A simple greedy growing method is introduced for stable training of large-scale, high-resolution image diffusion models without cascaded super-resolution components, achieving competitive results with a 1024x1024 image pipeline model.
image diffusion models greedy growing method Shallow UNet high-resolution models text-to-image alignment high-resolution rendering deep core layers parameter growth pre-trained representation training stabilization super-resolution cascade Vermeer SDXL