Remember, Retrieve and Generate: Understanding Infinite Visual Concepts
  as Your Personalized Assistant
The development of large language models (LLMs) has significantly enhanced
the capabilities of multimodal LLMs (MLLMs) as general assistants. However,
lack of user-specific knowledge still restricts their application in human's
daily life. In this paper, we introduce the Retrieval Augmented Personalization
(RAP) framework for MLLMs' personalization. Starting from a general MLLM, we
turn it into a personalized assistant in three steps. (a) Remember: We design a
key-value database to store user-related information, e.g., user's name, avatar
and other attributes. (b) Retrieve: When the user initiates a conversation, RAP
will retrieve relevant information from the database using a multimodal
retriever. (c) Generate: The input query and retrieved concepts' information
are fed into MLLMs to generate personalized, knowledge-augmented responses.
Unlike previous methods, RAP allows real-time concept editing via updating the
external database. To further improve generation quality and alignment with
user-specific information, we design a pipeline for data collection and create
a specialized dataset for personalized training of MLLMs. Based on the dataset,
we train a series of MLLMs as personalized multimodal assistants. By
pretraining on large-scale dataset, RAP-MLLMs can generalize to infinite visual
concepts without additional finetuning. Our models demonstrate outstanding
flexibility and generation quality across a variety of tasks, such as
personalized image captioning, question answering and visual recognition. The
code, data and models are available at https://github.com/Hoar012/RAP-MLLM.
6712176ef8feb84b4a03f70a
Haoran Hao
claimed_verified
2024-10-18T08:12:22.907Z
669e221425f2081c6e3d8b61
https://cdn-avatars.huggingface.co/v1/production/uploads/669e221425f2081c6e3d8b61/OearXzKrvoIZtDKTH4kje.jpeg
Haoran Hao
user
Hoar012
6712176ef8feb84b4a03f70b
Jiaming Han
claimed_verified
2024-10-21T07:48:10.684Z
62318c0386753f5f41d0e261
https://cdn-avatars.huggingface.co/v1/production/uploads/62318c0386753f5f41d0e261/xO_5PvOf7lXhQPnQLcmnq.jpeg
Jiaming Han
user
csuhan
6712176ef8feb84b4a03f70c
Changsheng Li
admin_assigned
2024-10-18T16:03:04.912Z
653a54c87e372a9e13642952
/avatars/0f34962610b4b22b8509c4b724821b96.svg
CHANGSHENG LI
user
Praised414
6712176ef8feb84b4a03f70d
Yu-Feng Li
admin_assigned
2024-10-18T16:02:58.587Z
64e76d5f31f7cea634cbeadc
/avatars/f9708afafe1cb935bae80c1a1fc4359c.svg
Li, Yufeng
user
yufengli
6712176ef8feb84b4a03f70e
Xiangyu Yue
admin_assigned
2024-10-18T16:02:47.827Z
666a8f24e2990b0cb16b7bf9
/avatars/fcbaf8f1e3e53a2a4a819b7cb2c53aa4.svg
Xiangyu Yue
user
xyyue
Remember, Retrieve and Generate: Understanding Infinite Visual Concepts
  as Your Personalized Assistant
The development of large language models (LLMs) has significantly enhanced
the capabilities of multimodal LLMs (MLLMs) as general assistants. However,
lack of user-specific knowledge still restricts their application in human's
daily life. In this paper, we introduce the Retrieval Augmented Personalization
(RAP) framework for MLLMs' personalization. Starting from a general MLLM, we
turn it into a personalized assistant in three steps. (a) Remember: We design a
key-value database to store user-related information, e.g., user's name, avatar
and other attributes. (b) Retrieve: When the user initiates a conversation, RAP
will retrieve relevant information from the database using a multimodal
retriever. (c) Generate: The input query and retrieved concepts' information
are fed into MLLMs to generate personalized, knowledge-augmented responses.
Unlike previous methods, RAP allows real-time concept editing via updating the
external database. To further improve generation quality and alignment with
user-specific information, we design a pipeline for data collection and create
a specialized dataset for personalized training of MLLMs. Based on the dataset,
we train a series of MLLMs as personalized multimodal assistants. By
pretraining on large-scale dataset, RAP-MLLMs can generalize to infinite visual
concepts without additional finetuning. Our models demonstrate outstanding
flexibility and generation quality across a variety of tasks, such as
personalized image captioning, question answering and visual recognition. The
code, data and models are available at https://github.com/Hoar012/RAP-MLLM.
The Retrieval Augmented Personalization (RAP) framework enhances multimodal language models by integrating user-specific knowledge in real-time for improved, personalized responses.
large language models multimodal LLMs personalized assistants key-value database multimodal retriever real-time concept editing specialized dataset pretraining generalized visual concepts personalized image captioning question answering visual recognition