Emu: Enhancing Image Generation Models Using Photogenic Needles in a
  Haystack
Training text-to-image models with web scale image-text pairs enables the
generation of a wide range of visual concepts from text. However, these
pre-trained models often face challenges when it comes to generating highly
aesthetic images. This creates the need for aesthetic alignment post
pre-training. In this paper, we propose quality-tuning to effectively guide a
pre-trained model to exclusively generate highly visually appealing images,
while maintaining generality across visual concepts. Our key insight is that
supervised fine-tuning with a set of surprisingly small but extremely visually
appealing images can significantly improve the generation quality. We pre-train
a latent diffusion model on 1.1 billion image-text pairs and fine-tune it
with only a few thousand carefully selected high-quality images. The resulting
model, Emu, achieves a win rate of 82.9% compared with its pre-trained only
counterpart. Compared to the state-of-the-art SDXLv1.0, Emu is preferred
68.4% and 71.3% of the time on visual appeal on the standard PartiPrompts
and our Open User Input benchmark based on the real-world usage of
text-to-image models. In addition, we show that quality-tuning is a generic
approach that is also effective for other architectures, including pixel
diffusion and masked generative transformer models.
651503184b4f53b8268ac63b
Xiaoliang Dai
651503184b4f53b8268ac63c
Ji Hou
admin_assigned
2023-09-28T08:41:36.905Z
63084636f0dc38fb47c2dc6b
/avatars/04edff10a048b5d93981e6f5af3ecf40.svg
Ji Hou
user
jhou90
651503184b4f53b8268ac63d
Chih-Yao Ma
claimed_verified
2023-09-28T19:03:28.426Z
6515bf00c2aedaa7e692a1ec
https://cdn-avatars.huggingface.co/v1/production/uploads/6515bf00c2aedaa7e692a1ec/PYpS1QwTPdQYW5_BrvwzP.png
Chih-Yao Ma
user
chihyaoma
651503184b4f53b8268ac63e
Sam Tsai
claimed_verified
2024-10-18T15:31:28.279Z
67127b753a058517519fabe8
/avatars/6fabb1634efb8ad8a8a2ae96fd6cf1c6.svg
Sam Tsai
user
sstsai-adl
651503184b4f53b8268ac63f
Jialiang Wang
651503184b4f53b8268ac640
Rui Wang
claimed_verified
2023-09-28T19:03:31.458Z
6515bcb7f464644a73b3ed8e
/avatars/17872687e0117ffbdc2dcf0ddf273a60.svg
Rui Wang
user
theruiwang
651503184b4f53b8268ac641
Peizhao Zhang
admin_assigned
2023-09-28T08:43:07.975Z
6433a635a5aed21dd110381e
/avatars/b1fa4fe1c8bd7989875233820fa425af.svg
Peizhao Zhang
user
stzpz
651503184b4f53b8268ac642
Simon Vandenhende
admin_assigned
2023-09-28T08:43:15.471Z
62d9d1a4cf646c73e103524b
/avatars/f29da9044d0f98241b5114b583ce3453.svg
Simon Vandenhende
user
svandenh
651503184b4f53b8268ac643
Xiaofang Wang
admin_assigned
2023-09-28T08:43:23.770Z
6334c1b3762d2d0e9eb151ea
/avatars/45b3eb63f0515ab417aacd79aa9f8a41.svg
Xiaofang Wang
user
minione
651503184b4f53b8268ac644
Abhimanyu Dubey
admin_assigned
2023-09-28T08:43:30.965Z
6376f5a98272ecd8ddacacf8
/avatars/c57d5de9a90ec12b7f617327b1ac27fd.svg
Abhimanyu Dubey
user
dubeya
651503184b4f53b8268ac645
Matthew Yu
admin_assigned
2023-09-28T08:43:48.250Z
64043b47dbfbea2a0544e39d
/avatars/fc985078067c06a8ac3f70b1cade158a.svg
Matthew Yu
user
mattcyu
651503184b4f53b8268ac646
Abhishek Kadian
admin_assigned
2023-09-28T08:44:16.919Z
62c88a674fd4a24e1523f478
/avatars/5a21438eca713bd97f5a2c8c5cbd980a.svg
Abhishek Kadian
user
abhiskk
651503184b4f53b8268ac647
Filip Radenovic
claimed_verified
2023-09-29T07:04:28.704Z
6515dccbc979eb9ca72dcc8c
/avatars/b55101dc2b0052c36f0b2cbdea9473e7.svg
Filip Radenovic
user
filipradenovic
651503184b4f53b8268ac648
Dhruv Mahajan
admin_assigned
2023-09-28T08:44:39.690Z
64358a51f81a16e7436330f7
/avatars/71c350836e5d025b4c7b9f99f72c2e21.svg
Dhruv Mahajan
user
Dhruv2009
651503184b4f53b8268ac649
Kunpeng Li
admin_assigned
2023-09-28T08:44:47.175Z
6418098e927a8ce346e40633
/avatars/3b0e81e24e23db21b52c0a90c57faa8f.svg
Kunpeng Li
user
lkp
651503184b4f53b8268ac64a
Yue Zhao
claimed_verified
2023-09-28T19:03:37.634Z
641cc612f0b71a9743668604
/avatars/ab7d91df8de266fadb4e5632a1f0a37a.svg
Yue (R) Zhao
user
yzhao30
651503184b4f53b8268ac64b
Vladan Petrovic
claimed_verified
2023-09-28T19:03:22.005Z
6515caf71e7b9224c9d23851
/avatars/11b17ba6d4f8e7b97a1644cc551a9fe7.svg
Vladan Petrovic
user
vpetrovi
651503184b4f53b8268ac64c
Mitesh Kumar Singh
651503184b4f53b8268ac64d
Simran Motwani
claimed_verified
2023-09-28T19:03:34.603Z
6515bb7e99e8060f1e792047
/avatars/9c6400f83c3165de3154b0e6bee2e5a1.svg
Simran Motwani
user
smotwani
651503184b4f53b8268ac64e
Yi Wen
admin_assigned
2023-09-28T08:53:52.586Z
64519584938967fd069f7f45
/avatars/4c5ebff5df4d290873fd6901e00a67e4.svg
Yi Wen
user
w-yi
651503184b4f53b8268ac64f
Yiwen Song
claimed_verified
2023-10-02T07:00:16.343Z
6517326e335b4c728b3406a4
/avatars/cde4a672778fc69c172e6bdeee2d540a.svg
Yiwen Song
user
yiwen-song
651503184b4f53b8268ac650
Roshan Sumbaly
claimed_verified
2023-09-28T19:03:25.119Z
6515c595f464644a73b50064
/avatars/119102c41fb8a7986c72b3f55ec6200f.svg
Roshan sumbaly
user
rsumbaly
651503184b4f53b8268ac651
Vignesh Ramanathan
651503184b4f53b8268ac652
Zijian He
claimed_verified
2024-01-15T07:42:23.153Z
657cfa56504da7f6f393c7a1
/avatars/6add66f88a876147ef9a25c34ea6cf63.svg
zijian
user
Maxpooling
651503184b4f53b8268ac653
Peter Vajda
admin_assigned
2023-09-28T08:46:44.373Z
636f4b35af79dd193603342b
/avatars/577328195738d92ac41a3d0ff3108d7c.svg
Peter Vajda
user
vaphab
651503184b4f53b8268ac654
Devi Parikh
admin_assigned
2023-09-28T08:46:52.374Z
6340753995c20b9447379c67
/avatars/f793c00a98335cd8f7efce41d403da16.svg
Devi Parikh
user
deviparikh-genai
Emu: Enhancing Image Generation Models Using Photogenic Needles in a
  Haystack
Training text-to-image models with web scale image-text pairs enables the
generation of a wide range of visual concepts from text. However, these
pre-trained models often face challenges when it comes to generating highly
aesthetic images. This creates the need for aesthetic alignment post
pre-training. In this paper, we propose quality-tuning to effectively guide a
pre-trained model to exclusively generate highly visually appealing images,
while maintaining generality across visual concepts. Our key insight is that
supervised fine-tuning with a set of surprisingly small but extremely visually
appealing images can significantly improve the generation quality. We pre-train
a latent diffusion model on 1.1 billion image-text pairs and fine-tune it
with only a few thousand carefully selected high-quality images. The resulting
model, Emu, achieves a win rate of 82.9% compared with its pre-trained only
counterpart. Compared to the state-of-the-art SDXLv1.0, Emu is preferred
68.4% and 71.3% of the time on visual appeal on the standard PartiPrompts
and our Open User Input benchmark based on the real-world usage of
text-to-image models. In addition, we show that quality-tuning is a generic
approach that is also effective for other architectures, including pixel
diffusion and masked generative transformer models.
Quality-tuning enhances the aesthetic appeal of text-to-image models through supervised fine-tuning with a small set of high-quality images.
latent diffusion model quality-tuning text-to-image models visual appeal supervised fine-tuning Emu SDXLv1.0 PartiPrompts Open User Input benchmark pixel diffusion masked generative transformer models