title: Aligning VLM Assistants with Personalized Situated Cognition
summary: Vision-language models (VLMs) aligned with general human objectives, such as
  being harmless and hallucination-free, have become valuable assistants of
  humans in managing visual tasks. However, people with diversified backgrounds
  have different cognition even in the same situation. Consequently, they may
  have personalized expectations for VLM assistants. This highlights the urgent
  need to align VLM assistants with personalized situated cognition for
  real-world assistance. To study this problem, we first simplify it by
  characterizing individuals based on the sociological concept of Role-Set. Then,
  we propose to evaluate the individuals' actions to examine whether the
  personalized alignment is achieved. Further, we construct a benchmark named
  PCogAlignBench, which includes 18k instances and 20 individuals with different
  Role-Sets. Finally, we present a framework called PCogAlign, which constructs a
  cognition-aware and action-based reward model for personalized alignment.
  Experimental results and human evaluations demonstrate the reliability of the
  PCogAlignBench and the effectiveness of our proposed PCogAlign. We will
  open-source the constructed benchmark and code at
  https://github.com/NLPGM/PCogAlign.
paper_authors._id: 683e6f7a118b68a69239ef94
paper_authors.name: Yongqi Li
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-06-03T10:45:17.727Z
paper_authors._id: 64b73c51be30d6567a706974
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/64b73c51be30d6567a706974/VrpQoHC0ZltYT8lNmEHqK.png
paper_authors.fullname: Yongqi Li
paper_authors.type: user
paper_authors.user: YongqiLi
paper_authors._id: 683e6f7a118b68a69239ef95
paper_authors.name: Shen Zhou
paper_authors._id: 683e6f7a118b68a69239ef96
paper_authors.name: Xiaohu Li
paper_authors._id: 683e6f7a118b68a69239ef97
paper_authors.name: Xin Miao
paper_authors._id: 683e6f7a118b68a69239ef98
paper_authors.name: Jintao Wen
paper_authors._id: 683e6f7a118b68a69239ef99
paper_authors.name: Mayi Xu
paper_authors._id: 683e6f7a118b68a69239ef9a
paper_authors.name: Jianhao Chen
paper_authors._id: 683e6f7a118b68a69239ef9b
paper_authors.name: Birong Pan
paper_authors._id: 683e6f7a118b68a69239ef9c
paper_authors.name: Hankun Kang
paper_authors._id: 683e6f7a118b68a69239ef9d
paper_authors.name: Yuanyuan Zhu
paper_authors._id: 683e6f7a118b68a69239ef9e
paper_authors.name: Ming Zhong
paper_authors._id: 683e6f7a118b68a69239ef9f
paper_authors.name: Tieyun Qian
paper_title: Aligning VLM Assistants with Personalized Situated Cognition
paper_summary: Vision-language models (VLMs) aligned with general human objectives, such as
  being harmless and hallucination-free, have become valuable assistants of
  humans in managing visual tasks. However, people with diversified backgrounds
  have different cognition even in the same situation. Consequently, they may
  have personalized expectations for VLM assistants. This highlights the urgent
  need to align VLM assistants with personalized situated cognition for
  real-world assistance. To study this problem, we first simplify it by
  characterizing individuals based on the sociological concept of Role-Set. Then,
  we propose to evaluate the individuals' actions to examine whether the
  personalized alignment is achieved. Further, we construct a benchmark named
  PCogAlignBench, which includes 18k instances and 20 individuals with different
  Role-Sets. Finally, we present a framework called PCogAlign, which constructs a
  cognition-aware and action-based reward model for personalized alignment.
  Experimental results and human evaluations demonstrate the reliability of the
  PCogAlignBench and the effectiveness of our proposed PCogAlign. We will
  open-source the constructed benchmark and code at
  https://github.com/NLPGM/PCogAlign.
paper_ai_summary: A framework called PCogAlign constructs a reward model for aligning vision-language models with personalized situated cognition, using a benchmark with varied Role-Sets.
paper_ai_keywords: Vision-language models Role-Set cognition-aware action-based reward model personalized alignment