title: FIRE: A Dataset for Feedback Integration and Refinement Evaluation of
    Multimodal Models
summary: Vision language models (VLMs) have achieved impressive progress in diverse
  applications, becoming a prevalent research direction. In this paper, we build
  FIRE, a feedback-refinement dataset, consisting of 1.1M multi-turn
  conversations that are derived from 27 source datasets, empowering VLMs to
  spontaneously refine their responses based on user feedback across diverse
  tasks. To scale up the data collection, FIRE is collected in two components:
  FIRE-100K and FIRE-1M, where FIRE-100K is generated by GPT-4V, and FIRE-1M is
  freely generated via models trained on FIRE-100K. Then, we build FIRE-Bench, a
  benchmark to comprehensively evaluate the feedback-refining capability of VLMs,
  which contains 11K feedback-refinement conversations as the test data, two
  evaluation settings, and a model to provide feedback for VLMs. We develop the
  FIRE-LLaVA model by fine-tuning LLaVA on FIRE-100K and FIRE-1M, which shows
  remarkable feedback-refining capability on FIRE-Bench and outperforms untrained
  VLMs by 50%, making more efficient user-agent interactions and underscoring the
  significance of the FIRE dataset.
paper_authors._id: 6697a6d09ad36faed17bee68
paper_authors.name: Pengxiang Li
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-08-09T07:48:38.848Z
paper_authors._id: 65d6bd8ebb862d1b367580fe
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/65d6bd8ebb862d1b367580fe/Pf5o-Oxi2IZxJPa-ZgEtV.jpeg
paper_authors.fullname: Pengxiang Li
paper_authors.type: user
paper_authors.user: PengxiangLi
paper_authors._id: 6697a6d09ad36faed17bee69
paper_authors.name: Zhi Gao
paper_authors._id: 6697a6d09ad36faed17bee6a
paper_authors.name: Bofei Zhang
paper_authors._id: 6697a6d09ad36faed17bee6b
paper_authors.name: Tao Yuan
paper_authors._id: 6697a6d09ad36faed17bee6c
paper_authors.name: Yuwei Wu
paper_authors._id: 6697a6d09ad36faed17bee6d
paper_authors.name: Mehrtash Harandi
paper_authors._id: 6697a6d09ad36faed17bee6e
paper_authors.name: Yunde Jia
paper_authors._id: 6697a6d09ad36faed17bee6f
paper_authors.name: Song-Chun Zhu
paper_authors._id: 6697a6d09ad36faed17bee70
paper_authors.name: Qing Li
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-08-03T08:26:08.641Z
paper_authors._id: 6455f8c2c8f569b995d603c9
paper_authors.avatarUrl: /avatars/3cb2cad3ab123887c270234bb6a4ca43.svg
paper_authors.fullname: Qing Li
paper_authors.type: user
paper_authors.user: li-qing
paper_title: FIRE: A Dataset for Feedback Integration and Refinement Evaluation of
    Multimodal Models
paper_summary: Vision language models (VLMs) have achieved impressive progress in diverse
  applications, becoming a prevalent research direction. In this paper, we build
  FIRE, a feedback-refinement dataset, consisting of 1.1M multi-turn
  conversations that are derived from 27 source datasets, empowering VLMs to
  spontaneously refine their responses based on user feedback across diverse
  tasks. To scale up the data collection, FIRE is collected in two components:
  FIRE-100K and FIRE-1M, where FIRE-100K is generated by GPT-4V, and FIRE-1M is
  freely generated via models trained on FIRE-100K. Then, we build FIRE-Bench, a
  benchmark to comprehensively evaluate the feedback-refining capability of VLMs,
  which contains 11K feedback-refinement conversations as the test data, two
  evaluation settings, and a model to provide feedback for VLMs. We develop the
  FIRE-LLaVA model by fine-tuning LLaVA on FIRE-100K and FIRE-1M, which shows
  remarkable feedback-refining capability on FIRE-Bench and outperforms untrained
  VLMs by 50%, making more efficient user-agent interactions and underscoring the
  significance of the FIRE dataset.
paper_ai_keywords: neural language models (VLMs) feedback-refinement dataset multi-turn conversations source datasets depth source datasets response refinement user feedback FIRE-100K FIRE-1M GPT-4V FIRE-Bench feedback-refinement conversations evaluation settings FIRE-LLaVA model fine-tuning LLaVA feedback-refining capability