title: ReFocus: Visual Editing as a Chain of Thought for Structured Image
    Understanding
summary: Structured image understanding, such as interpreting tables and charts,
  requires strategically refocusing across various structures and texts within an
  image, forming a reasoning sequence to arrive at the final answer. However,
  current multimodal large language models (LLMs) lack this multihop selective
  attention capability. In this work, we introduce ReFocus, a simple yet
  effective framework that equips multimodal LLMs with the ability to generate
  "visual thoughts" by performing visual editing on the input image through code,
  shifting and refining their visual focuses. Specifically, ReFocus enables
  multimodal LLMs to generate Python codes to call tools and modify the input
  image, sequentially drawing boxes, highlighting sections, and masking out
  areas, thereby enhancing the visual reasoning process. We experiment upon a
  wide range of structured image understanding tasks involving tables and charts.
  ReFocus largely improves performance on all tasks over GPT-4o without visual
  editing, yielding an average gain of 11.0% on table tasks and 6.8% on chart
  tasks. We present an in-depth analysis of the effects of different visual
  edits, and reasons why ReFocus can improve the performance without introducing
  additional information. Further, we collect a 14k training set using ReFocus,
  and prove that such visual chain-of-thought with intermediate information
  offers a better supervision than standard VQA data, reaching a 8.0% average
  gain over the same model trained with QA pairs and 2.6% over CoT.
paper_authors._id: 6784b75e90f970b02598154e
paper_authors.name: Xingyu Fu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-01-13T09:33:05.255Z
paper_authors._id: 6336091b2db86a181ccd6054
paper_authors.avatarUrl: /avatars/829f69436225d05d2c2136bc90f640d7.svg
paper_authors.fullname: Xingyu Fu
paper_authors.type: user
paper_authors.user: Fiaa
paper_authors._id: 6784b75e90f970b02598154f
paper_authors.name: Minqian Liu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-01-13T09:33:12.267Z
paper_authors._id: 64c32a75d15a8812b71afc48
paper_authors.avatarUrl: /avatars/4186c592b00aea73fb8c5bb719935ce7.svg
paper_authors.fullname: Minqian Liu
paper_authors.type: user
paper_authors.user: mqliu
paper_authors._id: 6784b75e90f970b025981550
paper_authors.name: Zhengyuan Yang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-01-13T09:33:18.418Z
paper_authors._id: 630713411801ecc7d2592a7c
paper_authors.avatarUrl: /avatars/fb36f69f03421c3a2a7f72ba0858fa60.svg
paper_authors.fullname: Zhengyuan Yang
paper_authors.type: user
paper_authors.user: zyang39
paper_authors._id: 6784b75e90f970b025981551
paper_authors.name: John Corring
paper_authors._id: 6784b75e90f970b025981552
paper_authors.name: Yijuan Lu
paper_authors._id: 6784b75e90f970b025981553
paper_authors.name: Jianwei Yang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-01-13T09:33:56.577Z
paper_authors._id: 6125df7f25027fb1ea9c7a41
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1629871954341-noauth.jpeg
paper_authors.fullname: Jianwei Yang
paper_authors.type: user
paper_authors.user: jw2yang
paper_authors._id: 6784b75e90f970b025981554
paper_authors.name: Dan Roth
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-01-13T09:34:09.978Z
paper_authors._id: 63191aa1f18d0b5d999a47fc
paper_authors.avatarUrl: /avatars/d7eadb22aa3d8a067189b924767864e9.svg
paper_authors.fullname: Dan Roth
paper_authors.type: user
paper_authors.user: danielpaulroth
paper_authors._id: 6784b75e90f970b025981555
paper_authors.name: Dinei Florencio
paper_authors._id: 6784b75e90f970b025981556
paper_authors.name: Cha Zhang
paper_title: ReFocus: Visual Editing as a Chain of Thought for Structured Image
    Understanding
paper_summary: Structured image understanding, such as interpreting tables and charts,
  requires strategically refocusing across various structures and texts within an
  image, forming a reasoning sequence to arrive at the final answer. However,
  current multimodal large language models (LLMs) lack this multihop selective
  attention capability. In this work, we introduce ReFocus, a simple yet
  effective framework that equips multimodal LLMs with the ability to generate
  "visual thoughts" by performing visual editing on the input image through code,
  shifting and refining their visual focuses. Specifically, ReFocus enables
  multimodal LLMs to generate Python codes to call tools and modify the input
  image, sequentially drawing boxes, highlighting sections, and masking out
  areas, thereby enhancing the visual reasoning process. We experiment upon a
  wide range of structured image understanding tasks involving tables and charts.
  ReFocus largely improves performance on all tasks over GPT-4o without visual
  editing, yielding an average gain of 11.0% on table tasks and 6.8% on chart
  tasks. We present an in-depth analysis of the effects of different visual
  edits, and reasons why ReFocus can improve the performance without introducing
  additional information. Further, we collect a 14k training set using ReFocus,
  and prove that such visual chain-of-thought with intermediate information
  offers a better supervision than standard VQA data, reaching a 8.0% average
  gain over the same model trained with QA pairs and 2.6% over CoT.
paper_ai_summary: ReFocus enhances multimodal LLMs for structured image understanding tasks by enabling visual edits and reasoning improvements through generated Python code, outperforming GPT-4 and other models.
paper_ai_keywords: multimodal LLMs ReFocus visual thoughts visual editing visual reasoning table tasks chart tasks intermediate information chain-of-thought