title: SocialGPT: Prompting LLMs for Social Relation Reasoning via Greedy
    Segment Optimization
summary: Social relation reasoning aims to identify relation categories such as
  friends, spouses, and colleagues from images. While current methods adopt the
  paradigm of training a dedicated network end-to-end using labeled image data,
  they are limited in terms of generalizability and interpretability. To address
  these issues, we first present a simple yet well-crafted framework named
  {\name}, which combines the perception capability of Vision Foundation Models
  (VFMs) and the reasoning capability of Large Language Models (LLMs) within a
  modular framework, providing a strong baseline for social relation recognition.
  Specifically, we instruct VFMs to translate image content into a textual social
  story, and then utilize LLMs for text-based reasoning. {\name} introduces
  systematic design principles to adapt VFMs and LLMs separately and bridge their
  gaps. Without additional model training, it achieves competitive zero-shot
  results on two databases while offering interpretable answers, as LLMs can
  generate language-based explanations for the decisions. The manual prompt
  design process for LLMs at the reasoning phase is tedious and an automated
  prompt optimization method is desired. As we essentially convert a visual
  classification task into a generative task of LLMs, automatic prompt
  optimization encounters a unique long prompt optimization issue. To address
  this issue, we further propose the Greedy Segment Prompt Optimization (GSPO),
  which performs a greedy search by utilizing gradient information at the segment
  level. Experimental results show that GSPO significantly improves performance,
  and our method also generalizes to different image styles. The code is
  available at https://github.com/Mengzibin/SocialGPT.
paper_authors._id: 6721931db02ab0e490b45411
paper_authors.name: Wanhua Li
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-10-30T09:52:46.803Z
paper_authors._id: 658bb7e47459b6e471b9d2e6
paper_authors.avatarUrl: /avatars/efd8051b468b4dbcb5d149479de67c58.svg
paper_authors.fullname: Wanhua Li
paper_authors.type: user
paper_authors.user: EthanTaylor
paper_authors._id: 6721931db02ab0e490b45412
paper_authors.name: Zibin Meng
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-10-30T09:52:48.983Z
paper_authors._id: 64d3a3a29d84115700581720
paper_authors.avatarUrl: /avatars/db23c6585975e08c6629393538ef8980.svg
paper_authors.fullname: Zibin Meng
paper_authors.type: user
paper_authors.user: zmeng0116
paper_authors._id: 6721931db02ab0e490b45413
paper_authors.name: Jiawei Zhou
paper_authors._id: 6721931db02ab0e490b45414
paper_authors.name: Donglai Wei
paper_authors._id: 6721931db02ab0e490b45415
paper_authors.name: Chuang Gan
paper_authors._id: 6721931db02ab0e490b45416
paper_authors.name: Hanspeter Pfister
paper_title: SocialGPT: Prompting LLMs for Social Relation Reasoning via Greedy
    Segment Optimization
paper_summary: Social relation reasoning aims to identify relation categories such as
  friends, spouses, and colleagues from images. While current methods adopt the
  paradigm of training a dedicated network end-to-end using labeled image data,
  they are limited in terms of generalizability and interpretability. To address
  these issues, we first present a simple yet well-crafted framework named
  {\name}, which combines the perception capability of Vision Foundation Models
  (VFMs) and the reasoning capability of Large Language Models (LLMs) within a
  modular framework, providing a strong baseline for social relation recognition.
  Specifically, we instruct VFMs to translate image content into a textual social
  story, and then utilize LLMs for text-based reasoning. {\name} introduces
  systematic design principles to adapt VFMs and LLMs separately and bridge their
  gaps. Without additional model training, it achieves competitive zero-shot
  results on two databases while offering interpretable answers, as LLMs can
  generate language-based explanations for the decisions. The manual prompt
  design process for LLMs at the reasoning phase is tedious and an automated
  prompt optimization method is desired. As we essentially convert a visual
  classification task into a generative task of LLMs, automatic prompt
  optimization encounters a unique long prompt optimization issue. To address
  this issue, we further propose the Greedy Segment Prompt Optimization (GSPO),
  which performs a greedy search by utilizing gradient information at the segment
  level. Experimental results show that GSPO significantly improves performance,
  and our method also generalizes to different image styles. The code is
  available at https://github.com/Mengzibin/SocialGPT.
paper_ai_summary: A modular framework that leverages Vision Foundation Models and Large Language Models for social relation reasoning achieves competitive zero-shot accuracy and interpretability through text-based explanations, with automatic prompt optimization to enhance performance.
paper_ai_keywords: Vision Foundation Models Large Language Models textual social story text-based reasoning systematic design principles zero-shot results automatic prompt optimization Greedy Segment Prompt Optimization gradient information segment level long prompt optimization