title: Stylus: Automatic Adapter Selection for Diffusion Models
summary: Beyond scaling base models with more data or parameters, fine-tuned adapters
  provide an alternative way to generate high fidelity, custom images at reduced
  costs. As such, adapters have been widely adopted by open-source communities,
  accumulating a database of over 100K adapters-most of which are highly
  customized with insufficient descriptions. This paper explores the problem of
  matching the prompt to a set of relevant adapters, built on recent work that
  highlight the performance gains of composing adapters. We introduce Stylus,
  which efficiently selects and automatically composes task-specific adapters
  based on a prompt's keywords. Stylus outlines a three-stage approach that first
  summarizes adapters with improved descriptions and embeddings, retrieves
  relevant adapters, and then further assembles adapters based on prompts'
  keywords by checking how well they fit the prompt. To evaluate Stylus, we
  developed StylusDocs, a curated dataset featuring 75K adapters with
  pre-computed adapter embeddings. In our evaluation on popular Stable Diffusion
  checkpoints, Stylus achieves greater CLIP-FID Pareto efficiency and is twice as
  preferred, with humans and multimodal models as evaluators, over the base
  model. See stylus-diffusion.github.io for more.
paper_authors._id: 66307da7b6cca78994834530
paper_authors.name: Michael Luo
paper_authors.status: extracted_confirmed
paper_authors.statusLastChangedAt: 2024-04-30T08:56:47.166Z
paper_authors._id: 654037be97949fd2304aab7f
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/654037be97949fd2304aab7f/2cSME81gcwYa2OTeVlq5Q.jpeg
paper_authors.fullname: Michael Luo
paper_authors.type: user
paper_authors.user: michaelzhiluo
paper_authors._id: 66307da7b6cca78994834531
paper_authors.name: Justin Wong
paper_authors.status: extracted_confirmed
paper_authors.statusLastChangedAt: 2024-04-30T05:22:55.408Z
paper_authors._id: 64b788fdff6d81ae297719d4
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/64b788fdff6d81ae297719d4/Uuzw94w-fZG9r55brsZ7I.jpeg
paper_authors.fullname: Justin Wong
paper_authors.type: user
paper_authors.user: JustinWong8314
paper_authors._id: 66307da7b6cca78994834532
paper_authors.name: Brandon Trabucco
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-04-30T08:26:33.335Z
paper_authors._id: 632d8b2e1d8a018adf4f98f1
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/632d8b2e1d8a018adf4f98f1/vGvpkxyGLNQSJmEONR2uX.jpeg
paper_authors.fullname: Brandon Trabucco
paper_authors.type: user
paper_authors.user: btrabucco
paper_authors._id: 66307da7b6cca78994834533
paper_authors.name: Yanping Huang
paper_authors._id: 66307da7b6cca78994834534
paper_authors.name: Joseph E. Gonzalez
paper_authors.status: extracted_pending
paper_authors.statusLastChangedAt: 2024-04-30T05:12:11.836Z
paper_authors._id: 645d2e8401f4eaab2a0878ce
paper_authors.avatarUrl: /avatars/1273c5fb607b4b622a746a42692fa632.svg
paper_authors.fullname: Joseph E. Gonzalez
paper_authors.type: user
paper_authors.user: ProfJoeyG
paper_authors._id: 66307da7b6cca78994834535
paper_authors.name: Zhifeng Chen
paper_authors._id: 66307da7b6cca78994834536
paper_authors.name: Ruslan Salakhutdinov
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-04-30T08:27:13.889Z
paper_authors._id: 6604dd83d35088f204cf658c
paper_authors.avatarUrl: /avatars/0b624951cb7cc355854659a04ec3be9d.svg
paper_authors.fullname: Nobody
paper_authors.type: user
paper_authors.user: rsalakhu
paper_authors._id: 66307da7b6cca78994834537
paper_authors.name: Ion Stoica
paper_title: Stylus: Automatic Adapter Selection for Diffusion Models
paper_summary: Beyond scaling base models with more data or parameters, fine-tuned adapters
  provide an alternative way to generate high fidelity, custom images at reduced
  costs. As such, adapters have been widely adopted by open-source communities,
  accumulating a database of over 100K adapters-most of which are highly
  customized with insufficient descriptions. This paper explores the problem of
  matching the prompt to a set of relevant adapters, built on recent work that
  highlight the performance gains of composing adapters. We introduce Stylus,
  which efficiently selects and automatically composes task-specific adapters
  based on a prompt's keywords. Stylus outlines a three-stage approach that first
  summarizes adapters with improved descriptions and embeddings, retrieves
  relevant adapters, and then further assembles adapters based on prompts'
  keywords by checking how well they fit the prompt. To evaluate Stylus, we
  developed StylusDocs, a curated dataset featuring 75K adapters with
  pre-computed adapter embeddings. In our evaluation on popular Stable Diffusion
  checkpoints, Stylus achieves greater CLIP-FID Pareto efficiency and is twice as
  preferred, with humans and multimodal models as evaluators, over the base
  model. See stylus-diffusion.github.io for more.
paper_ai_keywords: fine-tuned adapters high fidelity custom images open-source communities Stylus prompt-based selection adapter composition improved descriptions embeddings prompt keywords StylusDocs adapter embeddings Stable Diffusion CLIP-FID Pareto efficiency human evaluators multimodal models