title: Remember, Retrieve and Generate: Understanding Infinite Visual Concepts
    as Your Personalized Assistant
summary: The development of large language models (LLMs) has significantly enhanced
  the capabilities of multimodal LLMs (MLLMs) as general assistants. However,
  lack of user-specific knowledge still restricts their application in human's
  daily life. In this paper, we introduce the Retrieval Augmented Personalization
  (RAP) framework for MLLMs' personalization. Starting from a general MLLM, we
  turn it into a personalized assistant in three steps. (a) Remember: We design a
  key-value database to store user-related information, e.g., user's name, avatar
  and other attributes. (b) Retrieve: When the user initiates a conversation, RAP
  will retrieve relevant information from the database using a multimodal
  retriever. (c) Generate: The input query and retrieved concepts' information
  are fed into MLLMs to generate personalized, knowledge-augmented responses.
  Unlike previous methods, RAP allows real-time concept editing via updating the
  external database. To further improve generation quality and alignment with
  user-specific information, we design a pipeline for data collection and create
  a specialized dataset for personalized training of MLLMs. Based on the dataset,
  we train a series of MLLMs as personalized multimodal assistants. By
  pretraining on large-scale dataset, RAP-MLLMs can generalize to infinite visual
  concepts without additional finetuning. Our models demonstrate outstanding
  flexibility and generation quality across a variety of tasks, such as
  personalized image captioning, question answering and visual recognition. The
  code, data and models are available at https://github.com/Hoar012/RAP-MLLM.
paper_authors._id: 6712176ef8feb84b4a03f70a
paper_authors.name: Haoran Hao
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-10-18T08:12:22.907Z
paper_authors._id: 669e221425f2081c6e3d8b61
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/669e221425f2081c6e3d8b61/OearXzKrvoIZtDKTH4kje.jpeg
paper_authors.fullname: Haoran Hao
paper_authors.type: user
paper_authors.user: Hoar012
paper_authors._id: 6712176ef8feb84b4a03f70b
paper_authors.name: Jiaming Han
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-10-21T07:48:10.684Z
paper_authors._id: 62318c0386753f5f41d0e261
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/62318c0386753f5f41d0e261/xO_5PvOf7lXhQPnQLcmnq.jpeg
paper_authors.fullname: Jiaming Han
paper_authors.type: user
paper_authors.user: csuhan
paper_authors._id: 6712176ef8feb84b4a03f70c
paper_authors.name: Changsheng Li
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-10-18T16:03:04.912Z
paper_authors._id: 653a54c87e372a9e13642952
paper_authors.avatarUrl: /avatars/0f34962610b4b22b8509c4b724821b96.svg
paper_authors.fullname: CHANGSHENG LI
paper_authors.type: user
paper_authors.user: Praised414
paper_authors._id: 6712176ef8feb84b4a03f70d
paper_authors.name: Yu-Feng Li
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-10-18T16:02:58.587Z
paper_authors._id: 64e76d5f31f7cea634cbeadc
paper_authors.avatarUrl: /avatars/f9708afafe1cb935bae80c1a1fc4359c.svg
paper_authors.fullname: Li, Yufeng
paper_authors.type: user
paper_authors.user: yufengli
paper_authors._id: 6712176ef8feb84b4a03f70e
paper_authors.name: Xiangyu Yue
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-10-18T16:02:47.827Z
paper_authors._id: 666a8f24e2990b0cb16b7bf9
paper_authors.avatarUrl: /avatars/fcbaf8f1e3e53a2a4a819b7cb2c53aa4.svg
paper_authors.fullname: Xiangyu Yue
paper_authors.type: user
paper_authors.user: xyyue
paper_title: Remember, Retrieve and Generate: Understanding Infinite Visual Concepts
    as Your Personalized Assistant
paper_summary: The development of large language models (LLMs) has significantly enhanced
  the capabilities of multimodal LLMs (MLLMs) as general assistants. However,
  lack of user-specific knowledge still restricts their application in human's
  daily life. In this paper, we introduce the Retrieval Augmented Personalization
  (RAP) framework for MLLMs' personalization. Starting from a general MLLM, we
  turn it into a personalized assistant in three steps. (a) Remember: We design a
  key-value database to store user-related information, e.g., user's name, avatar
  and other attributes. (b) Retrieve: When the user initiates a conversation, RAP
  will retrieve relevant information from the database using a multimodal
  retriever. (c) Generate: The input query and retrieved concepts' information
  are fed into MLLMs to generate personalized, knowledge-augmented responses.
  Unlike previous methods, RAP allows real-time concept editing via updating the
  external database. To further improve generation quality and alignment with
  user-specific information, we design a pipeline for data collection and create
  a specialized dataset for personalized training of MLLMs. Based on the dataset,
  we train a series of MLLMs as personalized multimodal assistants. By
  pretraining on large-scale dataset, RAP-MLLMs can generalize to infinite visual
  concepts without additional finetuning. Our models demonstrate outstanding
  flexibility and generation quality across a variety of tasks, such as
  personalized image captioning, question answering and visual recognition. The
  code, data and models are available at https://github.com/Hoar012/RAP-MLLM.
paper_ai_keywords: large language models (LLMs) multimodal LLMs (MLLMs) Retrieval Augmented Personalization (RAP) framework key-value database multimodal retriever personalized, knowledge-augmented responses data collection pipeline specialized dataset pretraining visual concepts personalized image captioning question answering visual recognition