title: NoisyRollout: Reinforcing Visual Reasoning with Data Augmentation
summary: Recent advances in reinforcement learning (RL) have strengthened the
  reasoning capabilities of vision-language models (VLMs). However, enhancing
  policy exploration to more effectively scale test-time compute remains
  underexplored in VLMs. In addition, VLMs continue to struggle with imperfect
  visual perception, which in turn affects the subsequent reasoning process. To
  this end, we propose NoisyRollout, a simple yet effective RL approach that
  mixes trajectories from both clean and moderately distorted images to introduce
  targeted diversity in visual perception and the resulting reasoning patterns.
  Without additional training cost, NoisyRollout enhances the exploration
  capabilities of VLMs by incorporating a vision-oriented inductive bias.
  Furthermore, NoisyRollout employs a noise annealing schedule that gradually
  reduces distortion strength over training, ensuring benefit from noisy signals
  early while maintaining training stability and scalability in later stages.
  With just 2.1K training samples, NoisyRollout achieves state-of-the-art
  performance among open-source RL-tuned models on 5 out-of-domain benchmarks
  spanning both reasoning and perception tasks, while preserving comparable or
  even better in-domain performance.
paper_authors._id: 6801eb7e3881da18a86691c3
paper_authors.name: Xiangyan Liu
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-05-20T07:23:09.128Z
paper_authors._id: 6486b09e8315b19342f0bf5e
paper_authors.avatarUrl: /avatars/bc5f22f231c884146d373fe1042d81bd.svg
paper_authors.fullname: Xiangyan Liu
paper_authors.type: user
paper_authors.user: xyliu6
paper_authors._id: 6801eb7e3881da18a86691c4
paper_authors.name: Jinjie Ni
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-04-28T12:54:19.516Z
paper_authors._id: 62a7362fd1e7a011fd4e31a7
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/62a7362fd1e7a011fd4e31a7/ZY_mwH-sI0o05SHpLqwc7.png
paper_authors.fullname: Jinjie Ni
paper_authors.type: user
paper_authors.user: jinjieni
paper_authors._id: 6801eb7e3881da18a86691c5
paper_authors.name: Zijian Wu
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-06-04T09:04:54.594Z
paper_authors._id: 626d268d5f7327906f05cad1
paper_authors.avatarUrl: /avatars/18bda74612a3ee63a17f991bcc695106.svg
paper_authors.fullname: Zijian Wu
paper_authors.type: user
paper_authors.user: Jakumetsu
paper_authors._id: 6801eb7e3881da18a86691c6
paper_authors.name: Chao Du
paper_authors._id: 6801eb7e3881da18a86691c7
paper_authors.name: Longxu Dou
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-04-18T09:38:28.304Z
paper_authors._id: 6214e4ee1e35c843d42d1f88
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/6214e4ee1e35c843d42d1f88/fj-9wuIdPhvogh3BrcXTB.jpeg
paper_authors.fullname: Longxu Dou
paper_authors.type: user
paper_authors.user: dreamerdeo
paper_authors._id: 6801eb7e3881da18a86691c8
paper_authors.name: Haonan Wang
paper_authors._id: 6801eb7e3881da18a86691c9
paper_authors.name: Tianyu Pang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-04-20T15:03:24.060Z
paper_authors._id: 63d91b6d255ef6add20e1b38
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1675921369867-63d91b6d255ef6add20e1b38.jpeg
paper_authors.fullname: Tianyu Pang
paper_authors.type: user
paper_authors.user: P2333
paper_authors._id: 6801eb7e3881da18a86691ca
paper_authors.name: Michael Qizhe Shieh
paper_title: NoisyRollout: Reinforcing Visual Reasoning with Data Augmentation
paper_summary: Recent advances in reinforcement learning (RL) have strengthened the
  reasoning capabilities of vision-language models (VLMs). However, enhancing
  policy exploration to more effectively scale test-time compute remains
  underexplored in VLMs. In addition, VLMs continue to struggle with imperfect
  visual perception, which in turn affects the subsequent reasoning process. To
  this end, we propose NoisyRollout, a simple yet effective RL approach that
  mixes trajectories from both clean and moderately distorted images to introduce
  targeted diversity in visual perception and the resulting reasoning patterns.
  Without additional training cost, NoisyRollout enhances the exploration
  capabilities of VLMs by incorporating a vision-oriented inductive bias.
  Furthermore, NoisyRollout employs a noise annealing schedule that gradually
  reduces distortion strength over training, ensuring benefit from noisy signals
  early while maintaining training stability and scalability in later stages.
  With just 2.1K training samples, NoisyRollout achieves state-of-the-art
  performance among open-source RL-tuned models on 5 out-of-domain benchmarks
  spanning both reasoning and perception tasks, while preserving comparable or
  even better in-domain performance.
paper_ai_summary: NoisyRollout, an RL approach that introduces targeted diversity through noise in image trajectories, enhances VLM policy exploration without additional training cost, achieving state-of-the-art performance on out-of-domain benchmarks.
paper_ai_keywords: reinforcement learning vision-language models policy exploration noisy signals noise annealing vision-oriented inductive bias