title: Personalized Visual Instruction Tuning
summary: Recent advancements in multimodal large language models (MLLMs) have
  demonstrated significant progress; however, these models exhibit a notable
  limitation, which we refer to as "face blindness". Specifically, they can
  engage in general conversations but fail to conduct personalized dialogues
  targeting at specific individuals. This deficiency hinders the application of
  MLLMs in personalized settings, such as tailored visual assistants on mobile
  devices, or domestic robots that need to recognize members of the family. In
  this paper, we introduce Personalized Visual Instruction Tuning (PVIT), a novel
  data curation and training framework designed to enable MLLMs to identify
  target individuals within an image and engage in personalized and coherent
  dialogues. Our approach involves the development of a sophisticated pipeline
  that autonomously generates training data containing personalized
  conversations. This pipeline leverages the capabilities of various visual
  experts, image generation models, and (multi-modal) large language models. To
  evaluate the personalized potential of MLLMs, we present a benchmark called
  P-Bench, which encompasses various question types with different levels of
  difficulty. The experiments demonstrate a substantial personalized performance
  enhancement after fine-tuning with our curated dataset.
paper_authors._id: 67075c2716abdf6976cd06c6
paper_authors.name: Renjie Pi
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-10-10T08:13:26.509Z
paper_authors._id: 63f45b8d520c14618930d175
paper_authors.avatarUrl: /avatars/a20994594579b52a8be8bd2c4acbb913.svg
paper_authors.fullname: renjie
paper_authors.type: user
paper_authors.user: renjiepi
paper_authors._id: 67075c2716abdf6976cd06c7
paper_authors.name: Jianshu Zhang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-10-10T08:13:33.559Z
paper_authors._id: 65d8b0f0661492b25c6623de
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/65d8b0f0661492b25c6623de/c6LPDse8NIV_3BHIu8dYe.png
paper_authors.fullname: Jianshu Zhang
paper_authors.type: user
paper_authors.user: Sterzhang
paper_authors._id: 67075c2716abdf6976cd06c8
paper_authors.name: Tianyang Han
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-10-11T07:44:07.595Z
paper_authors._id: 63425f57a7582111c3ff0939
paper_authors.avatarUrl: /avatars/4693fb6c6ad145eb438ff2fac39dbd90.svg
paper_authors.fullname: Masaiah
paper_authors.type: user
paper_authors.user: ScottHan
paper_authors._id: 67075c2716abdf6976cd06c9
paper_authors.name: Jipeng Zhang
paper_authors._id: 67075c2716abdf6976cd06ca
paper_authors.name: Rui Pan
paper_authors.status: extracted_confirmed
paper_authors.statusLastChangedAt: 2024-10-10T16:19:32.485Z
paper_authors._id: 64cb1ad1667f4f80852f6050
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/64cb1ad1667f4f80852f6050/iOn5q_RyyBS99tObrO5Tc.png
paper_authors.fullname: Rui Pan
paper_authors.type: user
paper_authors.user: research4pan
paper_authors._id: 67075c2716abdf6976cd06cb
paper_authors.name: Tong Zhang
paper_title: Personalized Visual Instruction Tuning
paper_summary: Recent advancements in multimodal large language models (MLLMs) have
  demonstrated significant progress; however, these models exhibit a notable
  limitation, which we refer to as "face blindness". Specifically, they can
  engage in general conversations but fail to conduct personalized dialogues
  targeting at specific individuals. This deficiency hinders the application of
  MLLMs in personalized settings, such as tailored visual assistants on mobile
  devices, or domestic robots that need to recognize members of the family. In
  this paper, we introduce Personalized Visual Instruction Tuning (PVIT), a novel
  data curation and training framework designed to enable MLLMs to identify
  target individuals within an image and engage in personalized and coherent
  dialogues. Our approach involves the development of a sophisticated pipeline
  that autonomously generates training data containing personalized
  conversations. This pipeline leverages the capabilities of various visual
  experts, image generation models, and (multi-modal) large language models. To
  evaluate the personalized potential of MLLMs, we present a benchmark called
  P-Bench, which encompasses various question types with different levels of
  difficulty. The experiments demonstrate a substantial personalized performance
  enhancement after fine-tuning with our curated dataset.
paper_ai_keywords: multimodal large language models (MLLMs) face blindness personalized dialogues self-supervised supervision Personalized Visual Instruction Tuning (PVIT) visual assistants domestic robots recognition multi-modal training data generation image generation models autonomous pipeline benchmark (P-Bench) fine-tuning