title: InterFeedback: Unveiling Interactive Intelligence of Large Multimodal
    Models via Human Feedback
summary: Existing benchmarks do not test Large Multimodal Models (LMMs) on their
  interactive intelligence with human users which is vital for developing
  general-purpose AI assistants. We design InterFeedback, an interactive
  framework, which can be applied to any LMM and dataset to assess this ability
  autonomously. On top of this, we introduce InterFeedback-Bench which evaluates
  interactive intelligence using two representative datasets, MMMU-Pro and
  MathVerse, to test 10 different open-source LMMs. Additionally, we present
  InterFeedback-Human, a newly collected dataset of 120 cases designed for
  manually testing interactive performance in leading models such as OpenAI-o1
  and Claude-3.5-Sonnet. Our evaluation results show that even state-of-the-art
  LMM (like OpenAI-o1) can correct their results through human feedback less than
  50%. Our findings point to the need for methods that can enhance the LMMs'
  capability to interpret and benefit from feedback.
paper_authors._id: 67bbdcec79fcd85f09ddd869
paper_authors.name: Henry Hengyuan Zhao
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-02-24T13:32:25.051Z
paper_authors._id: 647d7eb9770c299e56f5b39b
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/647d7eb9770c299e56f5b39b/CC5JJgkyLkXOxw-BeT4G5.jpeg
paper_authors.fullname: Henry Hengyuan Zhao
paper_authors.type: user
paper_authors.user: hhenryz
paper_authors._id: 67bbdcec79fcd85f09ddd86a
paper_authors.name: Wenqi Pei
paper_authors._id: 67bbdcec79fcd85f09ddd86b
paper_authors.name: Yifei Tao
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-02-24T15:52:28.951Z
paper_authors._id: 6684c542cc72cbbde88ccf55
paper_authors.avatarUrl: /avatars/dafa8b5b44dbb8be859fbae94a6cd953.svg
paper_authors.fullname: yifeitao
paper_authors.type: user
paper_authors.user: yifeitao
paper_authors._id: 67bbdcec79fcd85f09ddd86c
paper_authors.name: Haiyang Mei
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-02-24T15:52:15.485Z
paper_authors._id: 66fcfa6e05638227c44233a9
paper_authors.avatarUrl: /avatars/4a88765c7f5c5ca77da6d21eb01f73e0.svg
paper_authors.fullname: Haiyang Mei
paper_authors.type: user
paper_authors.user: meihaiyang
paper_authors._id: 67bbdcec79fcd85f09ddd86d
paper_authors.name: Mike Zheng Shou
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-02-24T15:52:21.365Z
paper_authors._id: 661ab3da2b14565c7acccf5c
paper_authors.avatarUrl: /avatars/fa4fc03664803e02aede4d4c3d50b393.svg
paper_authors.fullname: Mike Zheng Shou
paper_authors.type: user
paper_authors.user: AnalMom
paper_title: InterFeedback: Unveiling Interactive Intelligence of Large Multimodal
    Models via Human Feedback
paper_summary: Existing benchmarks do not test Large Multimodal Models (LMMs) on their
  interactive intelligence with human users which is vital for developing
  general-purpose AI assistants. We design InterFeedback, an interactive
  framework, which can be applied to any LMM and dataset to assess this ability
  autonomously. On top of this, we introduce InterFeedback-Bench which evaluates
  interactive intelligence using two representative datasets, MMMU-Pro and
  MathVerse, to test 10 different open-source LMMs. Additionally, we present
  InterFeedback-Human, a newly collected dataset of 120 cases designed for
  manually testing interactive performance in leading models such as OpenAI-o1
  and Claude-3.5-Sonnet. Our evaluation results show that even state-of-the-art
  LMM (like OpenAI-o1) can correct their results through human feedback less than
  50%. Our findings point to the need for methods that can enhance the LMMs'
  capability to interpret and benefit from feedback.
paper_ai_keywords: Large Multimodal Models (LMMs) interactive intelligence continuous framework InterFeedback InterFeedback-Bench MMMU-Pro MathVerse open-source LMMs InterFeedback-Human OpenAI-o1 Claude-3.5-Sonnet human feedback state-of-the-art LMMs interpret and benefit from feedback