Stylus: Automatic Adapter Selection for Diffusion Models
Beyond scaling base models with more data or parameters, fine-tuned adapters
provide an alternative way to generate high fidelity, custom images at reduced
costs. As such, adapters have been widely adopted by open-source communities,
accumulating a database of over 100K adapters-most of which are highly
customized with insufficient descriptions. This paper explores the problem of
matching the prompt to a set of relevant adapters, built on recent work that
highlight the performance gains of composing adapters. We introduce Stylus,
which efficiently selects and automatically composes task-specific adapters
based on a prompt's keywords. Stylus outlines a three-stage approach that first
summarizes adapters with improved descriptions and embeddings, retrieves
relevant adapters, and then further assembles adapters based on prompts'
keywords by checking how well they fit the prompt. To evaluate Stylus, we
developed StylusDocs, a curated dataset featuring 75K adapters with
pre-computed adapter embeddings. In our evaluation on popular Stable Diffusion
checkpoints, Stylus achieves greater CLIP-FID Pareto efficiency and is twice as
preferred, with humans and multimodal models as evaluators, over the base
model. See stylus-diffusion.github.io for more.
66307da7b6cca78994834530
Michael Luo
extracted_confirmed
2024-04-30T08:56:47.166Z
654037be97949fd2304aab7f
https://cdn-avatars.huggingface.co/v1/production/uploads/654037be97949fd2304aab7f/2cSME81gcwYa2OTeVlq5Q.jpeg
Michael Luo
user
michaelzhiluo
66307da7b6cca78994834531
Justin Wong
extracted_confirmed
2024-04-30T05:22:55.408Z
64b788fdff6d81ae297719d4
https://cdn-avatars.huggingface.co/v1/production/uploads/64b788fdff6d81ae297719d4/Uuzw94w-fZG9r55brsZ7I.jpeg
Justin Wong
user
JustinWong8314
66307da7b6cca78994834532
Brandon Trabucco
admin_assigned
2024-04-30T08:26:33.335Z
632d8b2e1d8a018adf4f98f1
https://cdn-avatars.huggingface.co/v1/production/uploads/632d8b2e1d8a018adf4f98f1/vGvpkxyGLNQSJmEONR2uX.jpeg
Brandon Trabucco
user
btrabucco
66307da7b6cca78994834533
Yanping Huang
66307da7b6cca78994834534
Joseph E. Gonzalez
extracted_pending
2024-04-30T05:12:11.836Z
645d2e8401f4eaab2a0878ce
/avatars/1273c5fb607b4b622a746a42692fa632.svg
Joseph E. Gonzalez
user
ProfJoeyG
66307da7b6cca78994834535
Zhifeng Chen
66307da7b6cca78994834536
Ruslan Salakhutdinov
admin_assigned
2024-04-30T08:27:13.889Z
6604dd83d35088f204cf658c
/avatars/0b624951cb7cc355854659a04ec3be9d.svg
Nobody
user
rsalakhu
66307da7b6cca78994834537
Ion Stoica
Stylus: Automatic Adapter Selection for Diffusion Models
Beyond scaling base models with more data or parameters, fine-tuned adapters
provide an alternative way to generate high fidelity, custom images at reduced
costs. As such, adapters have been widely adopted by open-source communities,
accumulating a database of over 100K adapters-most of which are highly
customized with insufficient descriptions. This paper explores the problem of
matching the prompt to a set of relevant adapters, built on recent work that
highlight the performance gains of composing adapters. We introduce Stylus,
which efficiently selects and automatically composes task-specific adapters
based on a prompt's keywords. Stylus outlines a three-stage approach that first
summarizes adapters with improved descriptions and embeddings, retrieves
relevant adapters, and then further assembles adapters based on prompts'
keywords by checking how well they fit the prompt. To evaluate Stylus, we
developed StylusDocs, a curated dataset featuring 75K adapters with
pre-computed adapter embeddings. In our evaluation on popular Stable Diffusion
checkpoints, Stylus achieves greater CLIP-FID Pareto efficiency and is twice as
preferred, with humans and multimodal models as evaluators, over the base
model. See stylus-diffusion.github.io for more.
Stylus automatically selects and composes task-specific adapters based on prompts, improving the efficiency and quality of custom image generation.
adapters fine-tuned high fidelity Stable Diffusion CLIP-FID Pareto efficiency multimodal models adapter embeddings keyword-based composition