Scaling Rectified Flow Transformers for High-Resolution Image Synthesis
Diffusion models create data from noise by inverting the forward paths of
data towards noise and have emerged as a powerful generative modeling technique
for high-dimensional, perceptual data such as images and videos. Rectified flow
is a recent generative model formulation that connects data and noise in a
straight line. Despite its better theoretical properties and conceptual
simplicity, it is not yet decisively established as standard practice. In this
work, we improve existing noise sampling techniques for training rectified flow
models by biasing them towards perceptually relevant scales. Through a
large-scale study, we demonstrate the superior performance of this approach
compared to established diffusion formulations for high-resolution
text-to-image synthesis. Additionally, we present a novel transformer-based
architecture for text-to-image generation that uses separate weights for the
two modalities and enables a bidirectional flow of information between image
and text tokens, improving text comprehension, typography, and human preference
ratings. We demonstrate that this architecture follows predictable scaling
trends and correlates lower validation loss to improved text-to-image synthesis
as measured by various metrics and human evaluations. Our largest models
outperform state-of-the-art models, and we will make our experimental data,
code, and model weights publicly available.
65e7d25c9e860e781b24f11f
Patrick Esser
admin_assigned
2024-03-06T09:40:28.177Z
62f2576394fe062ca9adf3d5
https://cdn-avatars.huggingface.co/v1/production/uploads/1660049160797-noauth.png
Patrick Esser
user
pesser
65e7d25c9e860e781b24f120
Sumith Kulal
admin_assigned
2024-03-06T09:40:41.408Z
630fd26be52a259b856600d3
https://cdn-avatars.huggingface.co/v1/production/uploads/630fd26be52a259b856600d3/yAmMYc35ZDLDbE5Hpurs1.jpeg
Sumith Kulal
user
sumith1896
65e7d25c9e860e781b24f121
Andreas Blattmann
admin_assigned
2024-03-06T09:40:56.954Z
635fbc957c05eb9f5997b568
/avatars/0bf85960fa7ad445b0abb2d70c9d5721.svg
Andreas Blattmann
user
ablattmann
65e7d25c9e860e781b24f122
Rahim Entezari
admin_assigned
2024-03-06T09:41:04.031Z
62fca7e7b17dbe0ceae54f04
/avatars/90c55bd5e0c6842586cc0ec5db387f89.svg
Rahim Entezari
user
rahiment
65e7d25c9e860e781b24f123
Jonas Müller
admin_assigned
2024-03-06T09:41:29.350Z
64510d6304397681bcf9725b
/avatars/acb0f7305807539f892472ef4e847310.svg
Jonas Müller
user
jaymueller
65e7d25c9e860e781b24f124
Harry Saini
admin_assigned
2024-03-06T09:41:37.445Z
619252d5ded356549e20c5cd
/avatars/c3ea01b1054bbdfa891db94aa39e745c.svg
Harry 
user
HarrySaini
65e7d25c9e860e781b24f125
Yam Levi
admin_assigned
2024-03-06T09:41:43.643Z
64df2032842e7056589bc43e
/avatars/7cae8e7cf685b1368cc66ff981927522.svg
Yam Levi
user
yumlevi
65e7d25c9e860e781b24f126
Dominik Lorenz
admin_assigned
2024-03-06T09:41:56.184Z
64c0fd7c36db3901addfd28a
/avatars/8ff6247de6d5906e0e5431edbdf943ca.svg
Dominik Lorenz
user
dominiklorenz
65e7d25c9e860e781b24f127
Axel Sauer
admin_assigned
2024-03-06T09:42:06.543Z
61fc209cef99814f1705e934
https://cdn-avatars.huggingface.co/v1/production/uploads/61fc209cef99814f1705e934/PSgCbKtcCb5dg84513fo-.jpeg
Axel Sauer
user
xl-sr
65e7d25c9e860e781b24f128
Frederic Boesel
admin_assigned
2024-03-06T09:42:12.418Z
653beb48ccfb7ba1cbbc9218
/avatars/f14a52db958f56d3082c2545b88ea99a.svg
Frederic Boesel
user
fboeEl
65e7d25c9e860e781b24f129
Dustin Podell
admin_assigned
2024-03-06T09:42:25.480Z
63055631acc17ce4ad369a37
https://cdn-avatars.huggingface.co/v1/production/uploads/1676933320376-63055631acc17ce4ad369a37.png
Dustin Podell
user
TwoDukes
65e7d25c9e860e781b24f12a
Tim Dockhorn
admin_assigned
2024-03-06T09:42:32.066Z
633f7a8f4be90e06da248e0f
https://cdn-avatars.huggingface.co/v1/production/uploads/633f7a8f4be90e06da248e0f/Kf-vMGCSIar8UGezk-TK9.jpeg
Tim Dockhorn
user
timudk
65e7d25c9e860e781b24f12b
Zion English
65e7d25c9e860e781b24f12c
Kyle Lacey
admin_assigned
2024-03-06T09:42:42.635Z
6318c908acffc70bd4deebec
https://cdn-avatars.huggingface.co/v1/production/uploads/6318c908acffc70bd4deebec/2I88otvT1rDot_22yYPIi.png
TwoPerCent
user
TwoPerCent
65e7d25c9e860e781b24f12d
Alex Goodwin
admin_assigned
2024-03-06T09:42:49.028Z
64c12d35cac5f9ba52b68720
/avatars/d0c2709aa77c98b4401f0ae4bda68e04.svg
Alex Goodwin
user
AlexGoodwin
65e7d25c9e860e781b24f12e
Yannik Marek
65e7d25c9e860e781b24f12f
Robin Rombach
admin_assigned
2024-03-06T09:42:58.961Z
62cfefa74b3e8dc1e32e38bf
https://cdn-avatars.huggingface.co/v1/production/uploads/1657794439283-noauth.png
Robin Rombach
user
rromb
Scaling Rectified Flow Transformers for High-Resolution Image Synthesis
Diffusion models create data from noise by inverting the forward paths of
data towards noise and have emerged as a powerful generative modeling technique
for high-dimensional, perceptual data such as images and videos. Rectified flow
is a recent generative model formulation that connects data and noise in a
straight line. Despite its better theoretical properties and conceptual
simplicity, it is not yet decisively established as standard practice. In this
work, we improve existing noise sampling techniques for training rectified flow
models by biasing them towards perceptually relevant scales. Through a
large-scale study, we demonstrate the superior performance of this approach
compared to established diffusion formulations for high-resolution
text-to-image synthesis. Additionally, we present a novel transformer-based
architecture for text-to-image generation that uses separate weights for the
two modalities and enables a bidirectional flow of information between image
and text tokens, improving text comprehension, typography, and human preference
ratings. We demonstrate that this architecture follows predictable scaling
trends and correlates lower validation loss to improved text-to-image synthesis
as measured by various metrics and human evaluations. Our largest models
outperform state-of-the-art models, and we will make our experimental data,
code, and model weights publicly available.
Improved noise sampling techniques and a transformer-based architecture enhance rectified flow models for high-resolution text-to-image synthesis, outperforming existing methods.
diffusion models rectified flow generative modeling high-dimensional data images videos noise sampling perceptually relevant scales text-to-image synthesis transformer-based architecture bidirectional flow information text comprehension typography human preference validation loss