title: YoChameleon: Personalized Vision and Language Generation
summary: Large Multimodal Models (e.g., GPT-4, Gemini, Chameleon) have evolved into
  powerful tools with millions of users. However, they remain generic models and
  lack personalized knowledge of specific user concepts. Previous work has
  explored personalization for text generation, yet it remains unclear how these
  methods can be adapted to new modalities, such as image generation. In this
  paper, we introduce Yo'Chameleon, the first attempt to study personalization
  for large multimodal models. Given 3-5 images of a particular concept,
  Yo'Chameleon leverages soft-prompt tuning to embed subject-specific information
  to (i) answer questions about the subject and (ii) recreate pixel-level details
  to produce images of the subject in new contexts. Yo'Chameleon is trained with
  (i) a self-prompting optimization mechanism to balance performance across
  multiple modalities, and (ii) a ``soft-positive" image generation approach to
  enhance image quality in a few-shot setting.
paper_authors._id: 6811899ba6198824c5589ed7
paper_authors.name: Thao Nguyen
paper_authors._id: 6811899ba6198824c5589ed8
paper_authors.name: Krishna Kumar Singh
paper_authors._id: 6811899ba6198824c5589ed9
paper_authors.name: Jing Shi
paper_authors._id: 6811899ba6198824c5589eda
paper_authors.name: Trung Bui
paper_authors._id: 6811899ba6198824c5589edb
paper_authors.name: Yong Jae Lee
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-04-30T12:19:30.281Z
paper_authors._id: 649f41ee70a478f8b36b2984
paper_authors.avatarUrl: /avatars/1c9a76717a450ac4aeb62a1e823d2e4a.svg
paper_authors.fullname: Yong Jae Lee
paper_authors.type: user
paper_authors.user: yjlee0222
paper_authors._id: 6811899ba6198824c5589edc
paper_authors.name: Yuheng Li
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-04-30T12:19:20.416Z
paper_authors._id: 63043c140547362a22a91137
paper_authors.avatarUrl: /avatars/d817f4e5f65c3fab7cef2fd6a3c72632.svg
paper_authors.fullname: Yuheng Li
paper_authors.type: user
paper_authors.user: Yuheng
paper_title: YoChameleon: Personalized Vision and Language Generation
paper_summary: Large Multimodal Models (e.g., GPT-4, Gemini, Chameleon) have evolved into
  powerful tools with millions of users. However, they remain generic models and
  lack personalized knowledge of specific user concepts. Previous work has
  explored personalization for text generation, yet it remains unclear how these
  methods can be adapted to new modalities, such as image generation. In this
  paper, we introduce Yo'Chameleon, the first attempt to study personalization
  for large multimodal models. Given 3-5 images of a particular concept,
  Yo'Chameleon leverages soft-prompt tuning to embed subject-specific information
  to (i) answer questions about the subject and (ii) recreate pixel-level details
  to produce images of the subject in new contexts. Yo'Chameleon is trained with
  (i) a self-prompting optimization mechanism to balance performance across
  multiple modalities, and (ii) a ``soft-positive" image generation approach to
  enhance image quality in a few-shot setting.
paper_ai_summary: Yo'Chameleon integrates soft-prompt tuning to personalize large multimodal models for answering questions and generating images of specific concepts from a few examples.
paper_ai_keywords: soft-prompt tuning self-prompting optimization soft-positive image generation