title: Mitigating Visual Forgetting via Take-along Visual Conditioning for
    Multi-modal Long CoT Reasoning
summary: Recent advancements in Large Language Models (LLMs) have demonstrated
  enhanced reasoning capabilities, evolving from Chain-of-Thought (CoT) prompting
  to advanced, product-oriented solutions like OpenAI o1. During our
  re-implementation of this model, we noticed that in multimodal tasks requiring
  visual input (e.g., geometry problems), Multimodal LLMs (MLLMs) struggle to
  maintain focus on the visual information, in other words, MLLMs suffer from a
  gradual decline in attention to visual information as reasoning progresses,
  causing text-over-relied outputs. To investigate this, we ablate image inputs
  during long-chain reasoning. Concretely, we truncate the reasoning process
  midway, then re-complete the reasoning process with the input image removed. We
  observe only a ~2% accuracy drop on MathVista's test-hard subset, revealing the
  model's textual outputs dominate the following reasoning process. Motivated by
  this, we propose Take-along Visual Conditioning (TVC), a strategy that shifts
  image input to critical reasoning stages and compresses redundant visual tokens
  via dynamic pruning. This methodology helps the model retain attention to the
  visual components throughout the reasoning. Our approach achieves
  state-of-the-art performance on average across five mathematical reasoning
  benchmarks (+3.4% vs previous sota), demonstrating the effectiveness of TVC in
  enhancing multimodal reasoning systems.
paper_authors._id: 67d8e21dea26d6d743f2adde
paper_authors.name: Hai-Long Sun
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-03-20T10:46:16.064Z
paper_authors._id: 6623975c728f756224d4b768
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/6623975c728f756224d4b768/US9n0k45Y1TF6WqOhgBhZ.jpeg
paper_authors.fullname: Allen Sun
paper_authors.type: user
paper_authors.user: Allen8
paper_authors._id: 67d8e21dea26d6d743f2addf
paper_authors.name: Zhun Sun
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-03-20T13:28:46.228Z
paper_authors._id: 64b75c9c479b934973de0e98
paper_authors.avatarUrl: /avatars/99a02865ddc6af27a77fcf7701f9f666.svg
paper_authors.fullname: Zhun Sun
paper_authors.type: user
paper_authors.user: zhunsun
paper_authors._id: 67d8e21dea26d6d743f2ade0
paper_authors.name: Houwen Peng
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-03-20T13:28:40.023Z
paper_authors._id: 631952b6f18d0b5d999ca397
paper_authors.avatarUrl: /avatars/eb87dfc142a7602f8fb888f7b7b60d38.svg
paper_authors.fullname: Peng
paper_authors.type: user
paper_authors.user: Houwen
paper_authors._id: 67d8e21dea26d6d743f2ade1
paper_authors.name: Han-Jia Ye
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-03-20T13:28:54.161Z
paper_authors._id: 67ab41b0ef4586b3ea65e7ed
paper_authors.avatarUrl: /avatars/4fa69250c6657624276a9af2aea6cd89.svg
paper_authors.fullname: Han-Jia Ye
paper_authors.type: user
paper_authors.user: han-jia
paper_title: Mitigating Visual Forgetting via Take-along Visual Conditioning for
    Multi-modal Long CoT Reasoning
paper_summary: Recent advancements in Large Language Models (LLMs) have demonstrated
  enhanced reasoning capabilities, evolving from Chain-of-Thought (CoT) prompting
  to advanced, product-oriented solutions like OpenAI o1. During our
  re-implementation of this model, we noticed that in multimodal tasks requiring
  visual input (e.g., geometry problems), Multimodal LLMs (MLLMs) struggle to
  maintain focus on the visual information, in other words, MLLMs suffer from a
  gradual decline in attention to visual information as reasoning progresses,
  causing text-over-relied outputs. To investigate this, we ablate image inputs
  during long-chain reasoning. Concretely, we truncate the reasoning process
  midway, then re-complete the reasoning process with the input image removed. We
  observe only a ~2% accuracy drop on MathVista's test-hard subset, revealing the
  model's textual outputs dominate the following reasoning process. Motivated by
  this, we propose Take-along Visual Conditioning (TVC), a strategy that shifts
  image input to critical reasoning stages and compresses redundant visual tokens
  via dynamic pruning. This methodology helps the model retain attention to the
  visual components throughout the reasoning. Our approach achieves
  state-of-the-art performance on average across five mathematical reasoning
  benchmarks (+3.4% vs previous sota), demonstrating the effectiveness of TVC in
  enhancing multimodal reasoning systems.
paper_ai_summary: Take-along Visual Conditioning (TVC) strategy enhances multimodal reasoning by focusing visual input at critical stages and compressing redundant visual tokens, achieving better performance in mathematical reasoning benchmarks.
paper_ai_keywords: Large Language Models (LLMs) Chain-of-Thought (CoT) OpenAI o1 Multimodal LLMs (MLLMs) attention (MathVista's test-hard subset Take-along Visual Conditioning (TVC) dynamic pruning multimodal reasoning