title: Text-to-Sticker: Style Tailoring Latent Diffusion Models for Human
    Expression
summary: We introduce Style Tailoring, a recipe to finetune Latent Diffusion Models
  (LDMs) in a distinct domain with high visual quality, prompt alignment and
  scene diversity. We choose sticker image generation as the target domain, as
  the images significantly differ from photorealistic samples typically generated
  by large-scale LDMs. We start with a competent text-to-image model, like Emu,
  and show that relying on prompt engineering with a photorealistic model to
  generate stickers leads to poor prompt alignment and scene diversity. To
  overcome these drawbacks, we first finetune Emu on millions of sticker-like
  images collected using weak supervision to elicit diversity. Next, we curate
  human-in-the-loop (HITL) Alignment and Style datasets from model generations,
  and finetune to improve prompt alignment and style alignment respectively.
  Sequential finetuning on these datasets poses a tradeoff between better style
  alignment and prompt alignment gains. To address this tradeoff, we propose a
  novel fine-tuning method called Style Tailoring, which jointly fits the content
  and style distribution and achieves best tradeoff. Evaluation results show our
  method improves visual quality by 14%, prompt alignment by 16.2% and scene
  diversity by 15.3%, compared to prompt engineering the base Emu model for
  stickers generation.
paper_authors._id: 655c2fecc4bb0926a684c1f5
paper_authors.name: Animesh Sinha
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2023-11-21T10:24:22.674Z
paper_authors._id: 63772d28e069bdcdbb8e4a29
paper_authors.avatarUrl: /avatars/be71b00e009f2093a3ff79f07815e68e.svg
paper_authors.fullname: Animesh Sinha
paper_authors.type: user
paper_authors.user: animeshsinha
paper_authors._id: 655c2fecc4bb0926a684c1f6
paper_authors.name: Bo Sun
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2023-11-22T09:48:36.015Z
paper_authors._id: 6552a36bd04b38b354592bd5
paper_authors.avatarUrl: /avatars/47c647e5d7297a60b6014a4f217c711c.svg
paper_authors.fullname: Bo Sun
paper_authors.type: user
paper_authors.user: bsun0802
paper_authors._id: 655c2fecc4bb0926a684c1f7
paper_authors.name: Anmol Kalia
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2023-12-11T08:16:48.968Z
paper_authors._id: 6573c2a8769f3ee9bdf98a74
paper_authors.avatarUrl: /avatars/df40b6153ec7368561049746a9c8adc2.svg
paper_authors.fullname: Anmol Kalia
paper_authors.type: user
paper_authors.user: anmolkalia
paper_authors._id: 655c2fecc4bb0926a684c1f8
paper_authors.name: Arantxa Casanova
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-11-21T13:21:43.154Z
paper_authors._id: 61798d543490eedabaa53911
paper_authors.avatarUrl: /avatars/8cd5eef76ce7ed0f1efd3debad8a3d08.svg
paper_authors.fullname: Arantxa Casanova
paper_authors.type: user
paper_authors.user: acasanova
paper_authors._id: 655c2fecc4bb0926a684c1f9
paper_authors.name: Elliot Blanchard
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2023-11-27T08:43:18.019Z
paper_authors._id: 6563cf4a718b9db5215d07b4
paper_authors.avatarUrl: /avatars/20e5be0bd3fcd888959a0af0f0b462e4.svg
paper_authors.fullname: Elliot Blanchard
paper_authors.type: user
paper_authors.user: elliotblanchard
paper_authors._id: 655c2fecc4bb0926a684c1fa
paper_authors.name: David Yan
paper_authors._id: 655c2fecc4bb0926a684c1fb
paper_authors.name: Winnie Zhang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-11-21T13:22:07.935Z
paper_authors._id: 64564b304a7ffb7d5a47f515
paper_authors.avatarUrl: /avatars/b848f02609d9ee8ab7df350ccb274c23.svg
paper_authors.fullname: Winnie Zhang
paper_authors.type: user
paper_authors.user: WinnieJackJack
paper_authors._id: 655c2fecc4bb0926a684c1fc
paper_authors.name: Tony Nelli
paper_authors._id: 655c2fecc4bb0926a684c1fd
paper_authors.name: Jiahui Chen
paper_authors._id: 655c2fecc4bb0926a684c1fe
paper_authors.name: Hardik Shah
paper_authors._id: 655c2fecc4bb0926a684c1ff
paper_authors.name: Licheng Yu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-11-21T13:23:49.720Z
paper_authors._id: 6372b1f9bd81fae2b3a712e8
paper_authors.avatarUrl: /avatars/bcdf7b540cc94882e0ae6035bb07a8f2.svg
paper_authors.fullname: Licheng Yu
paper_authors.type: user
paper_authors.user: lichengyu
paper_authors._id: 655c2fecc4bb0926a684c200
paper_authors.name: Mitesh Kumar Singh
paper_authors._id: 655c2fecc4bb0926a684c201
paper_authors.name: Ankit Ramchandani
paper_authors._id: 655c2fecc4bb0926a684c202
paper_authors.name: Maziar Sanjabi
paper_authors._id: 655c2fecc4bb0926a684c203
paper_authors.name: Sonal Gupta
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-11-21T13:24:40.879Z
paper_authors._id: 634ecb814677a5891c0c7026
paper_authors.avatarUrl: /avatars/a20cf325e4fe0a227ff80694ffea4a3d.svg
paper_authors.fullname: Sonal Gupta
paper_authors.type: user
paper_authors.user: thenextgenai
paper_authors._id: 655c2fecc4bb0926a684c204
paper_authors.name: Amy Bearman
paper_authors._id: 655c2fecc4bb0926a684c205
paper_authors.name: Dhruv Mahajan
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-11-21T13:24:50.780Z
paper_authors._id: 64358a51f81a16e7436330f7
paper_authors.avatarUrl: /avatars/71c350836e5d025b4c7b9f99f72c2e21.svg
paper_authors.fullname: Dhruv Mahajan
paper_authors.type: user
paper_authors.user: Dhruv2009
paper_title: Text-to-Sticker: Style Tailoring Latent Diffusion Models for Human
    Expression
paper_summary: We introduce Style Tailoring, a recipe to finetune Latent Diffusion Models
  (LDMs) in a distinct domain with high visual quality, prompt alignment and
  scene diversity. We choose sticker image generation as the target domain, as
  the images significantly differ from photorealistic samples typically generated
  by large-scale LDMs. We start with a competent text-to-image model, like Emu,
  and show that relying on prompt engineering with a photorealistic model to
  generate stickers leads to poor prompt alignment and scene diversity. To
  overcome these drawbacks, we first finetune Emu on millions of sticker-like
  images collected using weak supervision to elicit diversity. Next, we curate
  human-in-the-loop (HITL) Alignment and Style datasets from model generations,
  and finetune to improve prompt alignment and style alignment respectively.
  Sequential finetuning on these datasets poses a tradeoff between better style
  alignment and prompt alignment gains. To address this tradeoff, we propose a
  novel fine-tuning method called Style Tailoring, which jointly fits the content
  and style distribution and achieves best tradeoff. Evaluation results show our
  method improves visual quality by 14%, prompt alignment by 16.2% and scene
  diversity by 15.3%, compared to prompt engineering the base Emu model for
  stickers generation.
paper_ai_summary: Style Tailoring is a finetuning method for Latent Diffusion Models that enhances the generation of sticker images, improving visual quality, prompt alignment, and scene diversity.
paper_ai_keywords: Latent Diffusion Models Style Tailoring sticker image generation text-to-image model Emu weak supervision human-in-the-loop prompt alignment style alignment