ReFocus: Visual Editing as a Chain of Thought for Structured Image
  Understanding
Structured image understanding, such as interpreting tables and charts,
requires strategically refocusing across various structures and texts within an
image, forming a reasoning sequence to arrive at the final answer. However,
current multimodal large language models (LLMs) lack this multihop selective
attention capability. In this work, we introduce ReFocus, a simple yet
effective framework that equips multimodal LLMs with the ability to generate
"visual thoughts" by performing visual editing on the input image through code,
shifting and refining their visual focuses. Specifically, ReFocus enables
multimodal LLMs to generate Python codes to call tools and modify the input
image, sequentially drawing boxes, highlighting sections, and masking out
areas, thereby enhancing the visual reasoning process. We experiment upon a
wide range of structured image understanding tasks involving tables and charts.
ReFocus largely improves performance on all tasks over GPT-4o without visual
editing, yielding an average gain of 11.0% on table tasks and 6.8% on chart
tasks. We present an in-depth analysis of the effects of different visual
edits, and reasons why ReFocus can improve the performance without introducing
additional information. Further, we collect a 14k training set using ReFocus,
and prove that such visual chain-of-thought with intermediate information
offers a better supervision than standard VQA data, reaching a 8.0% average
gain over the same model trained with QA pairs and 2.6% over CoT.
6784b75e90f970b02598154e
Xingyu Fu
admin_assigned
2025-01-13T09:33:05.255Z
6336091b2db86a181ccd6054
/avatars/829f69436225d05d2c2136bc90f640d7.svg
Xingyu Fu
user
Fiaa
6784b75e90f970b02598154f
Minqian Liu
admin_assigned
2025-01-13T09:33:12.267Z
64c32a75d15a8812b71afc48
/avatars/4186c592b00aea73fb8c5bb719935ce7.svg
Minqian Liu
user
mqliu
6784b75e90f970b025981550
Zhengyuan Yang
admin_assigned
2025-01-13T09:33:18.418Z
630713411801ecc7d2592a7c
/avatars/fb36f69f03421c3a2a7f72ba0858fa60.svg
Zhengyuan Yang
user
zyang39
6784b75e90f970b025981551
John Corring
6784b75e90f970b025981552
Yijuan Lu
6784b75e90f970b025981553
Jianwei Yang
admin_assigned
2025-01-13T09:33:56.577Z
6125df7f25027fb1ea9c7a41
https://cdn-avatars.huggingface.co/v1/production/uploads/1629871954341-noauth.jpeg
Jianwei Yang
user
jw2yang
6784b75e90f970b025981554
Dan Roth
admin_assigned
2025-01-13T09:34:09.978Z
63191aa1f18d0b5d999a47fc
/avatars/d7eadb22aa3d8a067189b924767864e9.svg
Dan Roth
user
danielpaulroth
6784b75e90f970b025981555
Dinei Florencio
6784b75e90f970b025981556
Cha Zhang
ReFocus: Visual Editing as a Chain of Thought for Structured Image
  Understanding
Structured image understanding, such as interpreting tables and charts,
requires strategically refocusing across various structures and texts within an
image, forming a reasoning sequence to arrive at the final answer. However,
current multimodal large language models (LLMs) lack this multihop selective
attention capability. In this work, we introduce ReFocus, a simple yet
effective framework that equips multimodal LLMs with the ability to generate
"visual thoughts" by performing visual editing on the input image through code,
shifting and refining their visual focuses. Specifically, ReFocus enables
multimodal LLMs to generate Python codes to call tools and modify the input
image, sequentially drawing boxes, highlighting sections, and masking out
areas, thereby enhancing the visual reasoning process. We experiment upon a
wide range of structured image understanding tasks involving tables and charts.
ReFocus largely improves performance on all tasks over GPT-4o without visual
editing, yielding an average gain of 11.0% on table tasks and 6.8% on chart
tasks. We present an in-depth analysis of the effects of different visual
edits, and reasons why ReFocus can improve the performance without introducing
additional information. Further, we collect a 14k training set using ReFocus,
and prove that such visual chain-of-thought with intermediate information
offers a better supervision than standard VQA data, reaching a 8.0% average
gain over the same model trained with QA pairs and 2.6% over CoT.
ReFocus enhances multimodal LLMs for structured image understanding tasks by enabling visual edits and reasoning improvements through generated Python code, outperforming GPT-4 and other models.
multimodal LLMs ReFocus visual thoughts visual editing visual reasoning table tasks chart tasks intermediate information chain-of-thought