title: Greedy Growing Enables High-Resolution Pixel-Based Diffusion Models
summary: We address the long-standing problem of how to learn effective pixel-based
  image diffusion models at scale, introducing a remarkably simple greedy growing
  method for stable training of large-scale, high-resolution models. without the
  needs for cascaded super-resolution components. The key insight stems from
  careful pre-training of core components, namely, those responsible for
  text-to-image alignment {\it vs.} high-resolution rendering. We first
  demonstrate the benefits of scaling a {\it Shallow UNet}, with no
  down(up)-sampling enc(dec)oder. Scaling its deep core layers is shown to
  improve alignment, object structure, and composition. Building on this core
  model, we propose a greedy algorithm that grows the architecture into
  high-resolution end-to-end models, while preserving the integrity of the
  pre-trained representation, stabilizing training, and reducing the need for
  large high-resolution datasets. This enables a single stage model capable of
  generating high-resolution images without the need of a super-resolution
  cascade. Our key results rely on public datasets and show that we are able to
  train non-cascaded models up to 8B parameters with no further regularization
  schemes. Vermeer, our full pipeline model trained with internal datasets to
  produce 1024x1024 images, without cascades, is preferred by 44.0% vs. 21.4%
  human evaluators over SDXL.
paper_authors._id: 66554c9dac73f7759fa8ba1c
paper_authors.name: Cristina N. Vasconcelos
paper_authors._id: 66554c9dac73f7759fa8ba1d
paper_authors.name: Abdullah Rashwan Austin Waters
paper_authors._id: 66554c9dac73f7759fa8ba1e
paper_authors.name: Trevor Walker
paper_authors._id: 66554c9dac73f7759fa8ba1f
paper_authors.name: Keyang Xu
paper_authors._id: 66554c9dac73f7759fa8ba20
paper_authors.name: Jimmy Yan
paper_authors._id: 66554c9dac73f7759fa8ba21
paper_authors.name: Rui Qian
paper_authors._id: 66554c9dac73f7759fa8ba22
paper_authors.name: Shixin Luo
paper_authors._id: 66554c9dac73f7759fa8ba23
paper_authors.name: Zarana Parekh
paper_authors._id: 66554c9dac73f7759fa8ba24
paper_authors.name: Andrew Bunner
paper_authors._id: 66554c9dac73f7759fa8ba25
paper_authors.name: Hongliang Fei
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-05-28T15:47:48.818Z
paper_authors._id: 65aeb55e66ad44e2d567620c
paper_authors.avatarUrl: /avatars/36b98e143c465b852a26f14307e571ca.svg
paper_authors.fullname: Hongliang Fei
paper_authors.type: user
paper_authors.user: feihongliang0
paper_authors._id: 66554c9dac73f7759fa8ba26
paper_authors.name: Roopal Garg
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-05-28T15:47:55.198Z
paper_authors._id: 6490ffd4f049c475bdb6a40f
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/6490ffd4f049c475bdb6a40f/EQQQamUGoBsoq0w_BFflM.jpeg
paper_authors.fullname: Roopal Garg
paper_authors.type: user
paper_authors.user: roopalgarg
paper_authors._id: 66554c9dac73f7759fa8ba27
paper_authors.name: Mandy Guo
paper_authors._id: 66554c9dac73f7759fa8ba28
paper_authors.name: Ivana Kajic
paper_authors._id: 66554c9dac73f7759fa8ba29
paper_authors.name: Yeqing Li
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-05-28T15:48:09.284Z
paper_authors._id: 65656057e5aac326bfb79a7c
paper_authors.avatarUrl: /avatars/bb2b7956d47a022de3dab765459fd031.svg
paper_authors.fullname: Yeqing Li
paper_authors.type: user
paper_authors.user: yeqingli
paper_authors._id: 66554c9dac73f7759fa8ba2a
paper_authors.name: Henna Nandwani
paper_authors._id: 66554c9dac73f7759fa8ba2b
paper_authors.name: Jordi Pont-Tuset
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-05-28T15:48:19.659Z
paper_authors._id: 66335b4b07f864a2f94b403d
paper_authors.avatarUrl: /avatars/0a290942c4a282b48f870500cc6424b5.svg
paper_authors.fullname: Jordi Pont Tuset
paper_authors.type: user
paper_authors.user: jponttuset
paper_authors._id: 66554c9dac73f7759fa8ba2c
paper_authors.name: Yasumasa Onoe
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-05-28T15:48:34.871Z
paper_authors._id: 60591867ce97b2b478a3dc4d
paper_authors.avatarUrl: /avatars/bc8fedbca5f60f99fe66b9ac37ab9a1a.svg
paper_authors.fullname: Yasumasa Onoe
paper_authors.type: user
paper_authors.user: yasumasaonoe
paper_authors._id: 66554c9dac73f7759fa8ba2d
paper_authors.name: Sarah Rosston
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-05-28T15:48:41.982Z
paper_authors._id: 64011dec0251f0f411684604
paper_authors.avatarUrl: /avatars/aeb803b23ef4a7b2e24f9693c4f4c02a.svg
paper_authors.fullname: Sarah Rosston
paper_authors.type: user
paper_authors.user: srosston
paper_authors._id: 66554c9dac73f7759fa8ba2e
paper_authors.name: Su Wang
paper_authors._id: 66554c9dac73f7759fa8ba2f
paper_authors.name: Wenlei Zhou
paper_authors._id: 66554c9dac73f7759fa8ba30
paper_authors.name: Kevin Swersky
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-05-28T15:49:06.032Z
paper_authors._id: 630e6ef664f1f8d0c771b758
paper_authors.avatarUrl: /avatars/1df2bfadb2b6fdf8307189936efc6ef0.svg
paper_authors.fullname: Kevin Swersky
paper_authors.type: user
paper_authors.user: kswersky
paper_authors._id: 66554c9dac73f7759fa8ba31
paper_authors.name: David J. Fleet
paper_authors._id: 66554c9dac73f7759fa8ba32
paper_authors.name: Jason M. Baldridge
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-05-28T15:49:16.516Z
paper_authors._id: 630c9f648e3ff0c72327e8ee
paper_authors.avatarUrl: /avatars/1a5262c6575daf92f62f5cc6d9f1f00f.svg
paper_authors.fullname: Jason Baldridge
paper_authors.type: user
paper_authors.user: jasonbaldridge
paper_authors._id: 66554c9dac73f7759fa8ba33
paper_authors.name: Oliver Wang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-05-28T15:49:32.673Z
paper_authors._id: 62fa8bfdc95d426ff8f25fe9
paper_authors.avatarUrl: /avatars/71fe9cff4ce79f6a208045984852e5ad.svg
paper_authors.fullname: Oliver Wang
paper_authors.type: user
paper_authors.user: oliverwang
paper_title: Greedy Growing Enables High-Resolution Pixel-Based Diffusion Models
paper_summary: We address the long-standing problem of how to learn effective pixel-based
  image diffusion models at scale, introducing a remarkably simple greedy growing
  method for stable training of large-scale, high-resolution models. without the
  needs for cascaded super-resolution components. The key insight stems from
  careful pre-training of core components, namely, those responsible for
  text-to-image alignment {\it vs.} high-resolution rendering. We first
  demonstrate the benefits of scaling a {\it Shallow UNet}, with no
  down(up)-sampling enc(dec)oder. Scaling its deep core layers is shown to
  improve alignment, object structure, and composition. Building on this core
  model, we propose a greedy algorithm that grows the architecture into
  high-resolution end-to-end models, while preserving the integrity of the
  pre-trained representation, stabilizing training, and reducing the need for
  large high-resolution datasets. This enables a single stage model capable of
  generating high-resolution images without the need of a super-resolution
  cascade. Our key results rely on public datasets and show that we are able to
  train non-cascaded models up to 8B parameters with no further regularization
  schemes. Vermeer, our full pipeline model trained with internal datasets to
  produce 1024x1024 images, without cascades, is preferred by 44.0% vs. 21.4%
  human evaluators over SDXL.
paper_ai_keywords: image diffusion models greedy growing method stable training high-resolution models cascaded super-resolution components pre-training core components text-to-image alignment high-resolution rendering Shallow UNet deep core layers greedy algorithm end-to-end models pre-trained representation large high-resolution datasets single stage model high-resolution images cascades parameters regularization schemes