title: MoDoMoDo: Multi-Domain Data Mixtures for Multimodal LLM Reinforcement
    Learning
summary: Reinforcement Learning with Verifiable Rewards (RLVR) has recently emerged as
  a powerful paradigm for post-training large language models (LLMs), achieving
  state-of-the-art performance on tasks with structured, verifiable answers.
  Applying RLVR to Multimodal LLMs (MLLMs) presents significant opportunities but
  is complicated by the broader, heterogeneous nature of vision-language tasks
  that demand nuanced visual, logical, and spatial capabilities. As such,
  training MLLMs using RLVR on multiple datasets could be beneficial but creates
  challenges with conflicting objectives from interaction among diverse datasets,
  highlighting the need for optimal dataset mixture strategies to improve
  generalization and reasoning. We introduce a systematic post-training framework
  for Multimodal LLM RLVR, featuring a rigorous data mixture problem formulation
  and benchmark implementation. Specifically, (1) We developed a multimodal RLVR
  framework for multi-dataset post-training by curating a dataset that contains
  different verifiable vision-language problems and enabling multi-domain online
  RL learning with different verifiable rewards; (2) We proposed a data mixture
  strategy that learns to predict the RL fine-tuning outcome from the data
  mixture distribution, and consequently optimizes the best mixture.
  Comprehensive experiments showcase that multi-domain RLVR training, when
  combined with mixture prediction strategies, can significantly boost MLLM
  general reasoning capacities. Our best mixture improves the post-trained
  model's accuracy on out-of-distribution benchmarks by an average of 5.24%
  compared to the same model post-trained with uniform data mixture, and by a
  total of 20.74% compared to the pre-finetuning baseline.
paper_authors._id: 683dcb779187578a8ae24286
paper_authors.name: Yiqing Liang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-06-03T08:48:57.342Z
paper_authors._id: 6581d57b283fdb0c8dcb9851
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/elZhAKdRizAiAKtiAWBr3.png
paper_authors.fullname: Yiqing Liang
paper_authors.type: user
paper_authors.user: yiqingliang
paper_authors._id: 683dcb779187578a8ae24287
paper_authors.name: Jielin Qiu
paper_authors._id: 683dcb779187578a8ae24288
paper_authors.name: Wenhao Ding
paper_authors._id: 683dcb779187578a8ae24289
paper_authors.name: Zuxin Liu
paper_authors._id: 683dcb779187578a8ae2428a
paper_authors.name: James Tompkin
paper_authors._id: 683dcb779187578a8ae2428b
paper_authors.name: Mengdi Xu
paper_authors._id: 683dcb779187578a8ae2428c
paper_authors.name: Mengzhou Xia
paper_authors._id: 683dcb779187578a8ae2428d
paper_authors.name: Zhengzhong Tu
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-06-03T08:48:54.428Z
paper_authors._id: 62548d5fef3debb2ddf91217
paper_authors.avatarUrl: /avatars/14975b45568f9c399c92c3986b6ce83e.svg
paper_authors.fullname: Zhengzhong Tu
paper_authors.type: user
paper_authors.user: vztu
paper_authors._id: 683dcb779187578a8ae2428e
paper_authors.name: Laixi Shi
paper_authors._id: 683dcb779187578a8ae2428f
paper_authors.name: Jiacheng Zhu
paper_title: MoDoMoDo: Multi-Domain Data Mixtures for Multimodal LLM Reinforcement
    Learning
paper_summary: Reinforcement Learning with Verifiable Rewards (RLVR) has recently emerged as
  a powerful paradigm for post-training large language models (LLMs), achieving
  state-of-the-art performance on tasks with structured, verifiable answers.
  Applying RLVR to Multimodal LLMs (MLLMs) presents significant opportunities but
  is complicated by the broader, heterogeneous nature of vision-language tasks
  that demand nuanced visual, logical, and spatial capabilities. As such,
  training MLLMs using RLVR on multiple datasets could be beneficial but creates
  challenges with conflicting objectives from interaction among diverse datasets,
  highlighting the need for optimal dataset mixture strategies to improve
  generalization and reasoning. We introduce a systematic post-training framework
  for Multimodal LLM RLVR, featuring a rigorous data mixture problem formulation
  and benchmark implementation. Specifically, (1) We developed a multimodal RLVR
  framework for multi-dataset post-training by curating a dataset that contains
  different verifiable vision-language problems and enabling multi-domain online
  RL learning with different verifiable rewards; (2) We proposed a data mixture
  strategy that learns to predict the RL fine-tuning outcome from the data
  mixture distribution, and consequently optimizes the best mixture.
  Comprehensive experiments showcase that multi-domain RLVR training, when
  combined with mixture prediction strategies, can significantly boost MLLM
  general reasoning capacities. Our best mixture improves the post-trained
  model's accuracy on out-of-distribution benchmarks by an average of 5.24%
  compared to the same model post-trained with uniform data mixture, and by a
  total of 20.74% compared to the pre-finetuning baseline.
paper_ai_summary: A framework for post-training multimodal large language models using reinforcement learning with verifiable rewards introduces a data mixture strategy to enhance general reasoning abilities and benchmark performance.
paper_ai_keywords: reinforcement learning with verifiable rewards multimodal large language models multi-dataset post-training data mixture problem online RL learning RL fine-tuning outcome mixture prediction strategies