Aligning VLM Assistants with Personalized Situated Cognition
Vision-language models (VLMs) aligned with general human objectives, such as
being harmless and hallucination-free, have become valuable assistants of
humans in managing visual tasks. However, people with diversified backgrounds
have different cognition even in the same situation. Consequently, they may
have personalized expectations for VLM assistants. This highlights the urgent
need to align VLM assistants with personalized situated cognition for
real-world assistance. To study this problem, we first simplify it by
characterizing individuals based on the sociological concept of Role-Set. Then,
we propose to evaluate the individuals' actions to examine whether the
personalized alignment is achieved. Further, we construct a benchmark named
PCogAlignBench, which includes 18k instances and 20 individuals with different
Role-Sets. Finally, we present a framework called PCogAlign, which constructs a
cognition-aware and action-based reward model for personalized alignment.
Experimental results and human evaluations demonstrate the reliability of the
PCogAlignBench and the effectiveness of our proposed PCogAlign. We will
open-source the constructed benchmark and code at
https://github.com/NLPGM/PCogAlign.
683e6f7a118b68a69239ef94
Yongqi Li
claimed_verified
2025-06-03T10:45:17.727Z
64b73c51be30d6567a706974
https://cdn-avatars.huggingface.co/v1/production/uploads/64b73c51be30d6567a706974/VrpQoHC0ZltYT8lNmEHqK.png
Yongqi Li
user
YongqiLi
683e6f7a118b68a69239ef95
Shen Zhou
683e6f7a118b68a69239ef96
Xiaohu Li
683e6f7a118b68a69239ef97
Xin Miao
683e6f7a118b68a69239ef98
Jintao Wen
683e6f7a118b68a69239ef99
Mayi Xu
683e6f7a118b68a69239ef9a
Jianhao Chen
683e6f7a118b68a69239ef9b
Birong Pan
683e6f7a118b68a69239ef9c
Hankun Kang
683e6f7a118b68a69239ef9d
Yuanyuan Zhu
683e6f7a118b68a69239ef9e
Ming Zhong
683e6f7a118b68a69239ef9f
Tieyun Qian
Aligning VLM Assistants with Personalized Situated Cognition
Vision-language models (VLMs) aligned with general human objectives, such as
being harmless and hallucination-free, have become valuable assistants of
humans in managing visual tasks. However, people with diversified backgrounds
have different cognition even in the same situation. Consequently, they may
have personalized expectations for VLM assistants. This highlights the urgent
need to align VLM assistants with personalized situated cognition for
real-world assistance. To study this problem, we first simplify it by
characterizing individuals based on the sociological concept of Role-Set. Then,
we propose to evaluate the individuals' actions to examine whether the
personalized alignment is achieved. Further, we construct a benchmark named
PCogAlignBench, which includes 18k instances and 20 individuals with different
Role-Sets. Finally, we present a framework called PCogAlign, which constructs a
cognition-aware and action-based reward model for personalized alignment.
Experimental results and human evaluations demonstrate the reliability of the
PCogAlignBench and the effectiveness of our proposed PCogAlign. We will
open-source the constructed benchmark and code at
https://github.com/NLPGM/PCogAlign.
A framework called PCogAlign constructs a reward model for aligning vision-language models with personalized situated cognition, using a benchmark with varied Role-Sets.
Vision-language models Role-Set cognition-aware action-based reward model personalized alignment