title: Diving into Self-Evolving Training for Multimodal Reasoning
summary: Reasoning ability is essential for Large Multimodal Models (LMMs). In the
  absence of multimodal chain-of-thought annotated data, self-evolving training,
  where the model learns from its own outputs, has emerged as an effective and
  scalable approach for enhancing reasoning abilities. Despite its growing usage,
  a comprehensive understanding of self-evolving training, particularly in the
  context of multimodal reasoning, remains limited. In this paper, we delve into
  the intricacies of self-evolving training for multimodal reasoning, pinpointing
  three key factors: Training Method, Reward Model, and Prompt Variation. We
  systematically examine each factor and explore how various configurations
  affect the training's effectiveness. Our analysis leads to a set of best
  practices for each factor, aimed at optimizing multimodal reasoning.
  Furthermore, we explore the Self-Evolution Dynamics during training and the
  impact of automatic balancing mechanisms in boosting performance. After all the
  investigations, we present a final recipe for self-evolving training in
  multimodal reasoning, encapsulating these design choices into a framework we
  call MSTaR (Multimodal Self-evolving Training for Reasoning), which is
  universally effective for models with different sizes on various benchmarks,
  e.g., surpassing the pre-evolved model significantly on 5 multimodal reasoning
  benchmarks without using additional human annotations, as demonstrated on
  MiniCPM-V-2.5 (8B), Phi-3.5-Vision (4B) and InternVL2 (2B). We believe this
  study fills a significant gap in the understanding of self-evolving training
  for multimodal reasoning and offers a robust framework for future research. Our
  policy and reward models, as well as the collected data, is released to
  facilitate further investigation in multimodal reasoning.
paper_authors._id: 676a25e38ffab02f2c91a99e
paper_authors.name: Wei Liu
paper_authors._id: 676a25e38ffab02f2c91a99f
paper_authors.name: Junlong Li
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-12-30T19:35:34.958Z
paper_authors._id: 621e40ac944c7e36aaec2369
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/621e40ac944c7e36aaec2369/Yj-FJRWps3rvsS_B2bnKo.jpeg
paper_authors.fullname: Junlong Li
paper_authors.type: user
paper_authors.user: lockon
paper_authors._id: 676a25e38ffab02f2c91a9a0
paper_authors.name: Xiwen Zhang
paper_authors._id: 676a25e38ffab02f2c91a9a1
paper_authors.name: Fan Zhou
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-12-30T19:35:32.979Z
paper_authors._id: 628f6e5ab90dde28ef57d293
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/628f6e5ab90dde28ef57d293/AxNzR2nvrND6Rf3RPkYMk.jpeg
paper_authors.fullname: Fan Zhou
paper_authors.type: user
paper_authors.user: koalazf99
paper_authors._id: 676a25e38ffab02f2c91a9a2
paper_authors.name: Yu Cheng
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-01-23T15:05:17.014Z
paper_authors._id: 67017abfe4d49b157ac534d9
paper_authors.avatarUrl: /avatars/997e1b9f54b27a7728a9d4abfee4ba91.svg
paper_authors.fullname: Yu Cheng
paper_authors.type: user
paper_authors.user: ych133
paper_authors._id: 676a25e38ffab02f2c91a9a3
paper_authors.name: Junxian He
paper_title: Diving into Self-Evolving Training for Multimodal Reasoning
paper_summary: Reasoning ability is essential for Large Multimodal Models (LMMs). In the
  absence of multimodal chain-of-thought annotated data, self-evolving training,
  where the model learns from its own outputs, has emerged as an effective and
  scalable approach for enhancing reasoning abilities. Despite its growing usage,
  a comprehensive understanding of self-evolving training, particularly in the
  context of multimodal reasoning, remains limited. In this paper, we delve into
  the intricacies of self-evolving training for multimodal reasoning, pinpointing
  three key factors: Training Method, Reward Model, and Prompt Variation. We
  systematically examine each factor and explore how various configurations
  affect the training's effectiveness. Our analysis leads to a set of best
  practices for each factor, aimed at optimizing multimodal reasoning.
  Furthermore, we explore the Self-Evolution Dynamics during training and the
  impact of automatic balancing mechanisms in boosting performance. After all the
  investigations, we present a final recipe for self-evolving training in
  multimodal reasoning, encapsulating these design choices into a framework we
  call MSTaR (Multimodal Self-evolving Training for Reasoning), which is
  universally effective for models with different sizes on various benchmarks,
  e.g., surpassing the pre-evolved model significantly on 5 multimodal reasoning
  benchmarks without using additional human annotations, as demonstrated on
  MiniCPM-V-2.5 (8B), Phi-3.5-Vision (4B) and InternVL2 (2B). We believe this
  study fills a significant gap in the understanding of self-evolving training
  for multimodal reasoning and offers a robust framework for future research. Our
  policy and reward models, as well as the collected data, is released to
  facilitate further investigation in multimodal reasoning.
paper_ai_keywords: Large Multimodal Models (LMMs) self-evolving training multimodal reasoning Training Method Reward Model Prompt Variation Self-Evolution Dynamics automatic balancing mechanisms MSTaR (Multimodal Self-evolving Training for Reasoning) MiniCPM-V-2.5 (8B) Phi-3.5-Vision (4B) InternVL2 (2B)