title: Unsupervised Post-Training for Multi-Modal LLM Reasoning via GRPO
summary: Improving Multi-modal Large Language Models (MLLMs) in the post-training
  stage typically relies on supervised fine-tuning (SFT) or reinforcement
  learning (RL). However, these supervised methods require expensive and manually
  annotated multi-modal data--an ultimately unsustainable resource. While recent
  efforts have explored unsupervised post-training, their methods are complex and
  difficult to iterate. In this work, we are the first to investigate the use of
  GRPO, a stable and scalable online RL algorithm, for enabling continual
  self-improvement without any external supervision. We propose MM-UPT, a simple
  yet effective framework for unsupervised post-training of MLLMs. MM-UPT builds
  upon GRPO, replacing traditional reward signals with a self-rewarding mechanism
  based on majority voting over multiple sampled responses. Our experiments
  demonstrate that MM-UPT significantly improves the reasoning ability of
  Qwen2.5-VL-7B (e.g., 66.3 %rightarrow72.9 % on MathVista, 62.9
  %rightarrow68.7 % on We-Math), using standard dataset without ground truth
  labels. MM-UPT also outperforms prior unsupervised baselines and even
  approaches the results of supervised GRPO. Furthermore, we show that
  incorporating synthetic questions, generated solely by MLLM itself, can boost
  performance as well, highlighting a promising approach for scalable
  self-improvement. Overall, MM-UPT offers a new paradigm for continual,
  autonomous enhancement of MLLMs in the absence of external supervision. Our
  code is available at https://github.com/waltonfuture/MM-UPT.
paper_authors._id: 6837c318a4e378954486e45d
paper_authors.name: Lai Wei
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-05-29T07:42:35.145Z
paper_authors._id: 64a16b1aeacb4b50ba1c889d
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/64a16b1aeacb4b50ba1c889d/EhWoBGL6LlFGIVTUsp2F4.jpeg
paper_authors.fullname: Lai Wei
paper_authors.type: user
paper_authors.user: WaltonFuture
paper_authors._id: 6837c318a4e378954486e45e
paper_authors.name: Yuting Li
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-06-05T12:42:14.354Z
paper_authors._id: 66e3e801ec0343c253da37c3
paper_authors.avatarUrl: /avatars/72fc67fa2b25473fd3acba1fbd132f62.svg
paper_authors.fullname: Yuting Li
paper_authors.type: user
paper_authors.user: Yuting6
paper_authors._id: 6837c318a4e378954486e45f
paper_authors.name: Chen Wang
paper_authors._id: 6837c318a4e378954486e460
paper_authors.name: Yue Wang
paper_authors.status: extracted_confirmed
paper_authors.statusLastChangedAt: 2025-06-01T07:52:59.306Z
paper_authors._id: 65e095da35ad8b2fe8c80e71
paper_authors.avatarUrl: /avatars/225d4be6411dcec2460047ea1a88a5c3.svg
paper_authors.fullname: Weiran Huang
paper_authors.type: user
paper_authors.user: weiranhuang
paper_authors._id: 6837c318a4e378954486e461
paper_authors.name: Linghe Kong
paper_authors._id: 6837c318a4e378954486e462
paper_authors.name: Weiran Huang
paper_authors.status: extracted_pending
paper_authors.statusLastChangedAt: 2025-05-29T02:14:48.838Z
paper_authors._id: 65e095da35ad8b2fe8c80e71
paper_authors.avatarUrl: /avatars/225d4be6411dcec2460047ea1a88a5c3.svg
paper_authors.fullname: Weiran Huang
paper_authors.type: user
paper_authors.user: weiranhuang
paper_authors._id: 6837c318a4e378954486e463
paper_authors.name: Lichao Sun
paper_title: Unsupervised Post-Training for Multi-Modal LLM Reasoning via GRPO
paper_summary: Improving Multi-modal Large Language Models (MLLMs) in the post-training
  stage typically relies on supervised fine-tuning (SFT) or reinforcement
  learning (RL). However, these supervised methods require expensive and manually
  annotated multi-modal data--an ultimately unsustainable resource. While recent
  efforts have explored unsupervised post-training, their methods are complex and
  difficult to iterate. In this work, we are the first to investigate the use of
  GRPO, a stable and scalable online RL algorithm, for enabling continual
  self-improvement without any external supervision. We propose MM-UPT, a simple
  yet effective framework for unsupervised post-training of MLLMs. MM-UPT builds
  upon GRPO, replacing traditional reward signals with a self-rewarding mechanism
  based on majority voting over multiple sampled responses. Our experiments
  demonstrate that MM-UPT significantly improves the reasoning ability of
  Qwen2.5-VL-7B (e.g., 66.3 %rightarrow72.9 % on MathVista, 62.9
  %rightarrow68.7 % on We-Math), using standard dataset without ground truth
  labels. MM-UPT also outperforms prior unsupervised baselines and even
  approaches the results of supervised GRPO. Furthermore, we show that
  incorporating synthetic questions, generated solely by MLLM itself, can boost
  performance as well, highlighting a promising approach for scalable
  self-improvement. Overall, MM-UPT offers a new paradigm for continual,
  autonomous enhancement of MLLMs in the absence of external supervision. Our
  code is available at https://github.com/waltonfuture/MM-UPT.
paper_ai_summary: MM-UPT, a framework employing GRPO and self-rewarding, enhances multi-modal LLMs through unsupervised continual learning, showing performance improvements without manual annotations.
paper_ai_keywords: GRPO MM-UPT reinforcement learning unsupervised post-training multi-modal large language models self-rewarding mechanism majority voting synthetic questions MathVista We-Math