NoisyRollout: Reinforcing Visual Reasoning with Data Augmentation
Recent advances in reinforcement learning (RL) have strengthened the
reasoning capabilities of vision-language models (VLMs). However, enhancing
policy exploration to more effectively scale test-time compute remains
underexplored in VLMs. In addition, VLMs continue to struggle with imperfect
visual perception, which in turn affects the subsequent reasoning process. To
this end, we propose NoisyRollout, a simple yet effective RL approach that
mixes trajectories from both clean and moderately distorted images to introduce
targeted diversity in visual perception and the resulting reasoning patterns.
Without additional training cost, NoisyRollout enhances the exploration
capabilities of VLMs by incorporating a vision-oriented inductive bias.
Furthermore, NoisyRollout employs a noise annealing schedule that gradually
reduces distortion strength over training, ensuring benefit from noisy signals
early while maintaining training stability and scalability in later stages.
With just 2.1K training samples, NoisyRollout achieves state-of-the-art
performance among open-source RL-tuned models on 5 out-of-domain benchmarks
spanning both reasoning and perception tasks, while preserving comparable or
even better in-domain performance.
6801eb7e3881da18a86691c3
Xiangyan Liu
claimed_verified
2025-05-20T07:23:09.128Z
6486b09e8315b19342f0bf5e
/avatars/bc5f22f231c884146d373fe1042d81bd.svg
Xiangyan Liu
user
xyliu6
6801eb7e3881da18a86691c4
Jinjie Ni
claimed_verified
2025-04-28T12:54:19.516Z
62a7362fd1e7a011fd4e31a7
https://cdn-avatars.huggingface.co/v1/production/uploads/62a7362fd1e7a011fd4e31a7/ZY_mwH-sI0o05SHpLqwc7.png
Jinjie Ni
user
jinjieni
6801eb7e3881da18a86691c5
Zijian Wu
claimed_verified
2025-06-04T09:04:54.594Z
626d268d5f7327906f05cad1
/avatars/18bda74612a3ee63a17f991bcc695106.svg
Zijian Wu
user
Jakumetsu
6801eb7e3881da18a86691c6
Chao Du
6801eb7e3881da18a86691c7
Longxu Dou
claimed_verified
2025-04-18T09:38:28.304Z
6214e4ee1e35c843d42d1f88
https://cdn-avatars.huggingface.co/v1/production/uploads/6214e4ee1e35c843d42d1f88/fj-9wuIdPhvogh3BrcXTB.jpeg
Longxu Dou
user
dreamerdeo
6801eb7e3881da18a86691c8
Haonan Wang
6801eb7e3881da18a86691c9
Tianyu Pang
claimed_verified
2025-04-20T15:03:24.060Z
63d91b6d255ef6add20e1b38
https://cdn-avatars.huggingface.co/v1/production/uploads/1675921369867-63d91b6d255ef6add20e1b38.jpeg
Tianyu Pang
user
P2333
6801eb7e3881da18a86691ca
Michael Qizhe Shieh
NoisyRollout: Reinforcing Visual Reasoning with Data Augmentation
Recent advances in reinforcement learning (RL) have strengthened the
reasoning capabilities of vision-language models (VLMs). However, enhancing
policy exploration to more effectively scale test-time compute remains
underexplored in VLMs. In addition, VLMs continue to struggle with imperfect
visual perception, which in turn affects the subsequent reasoning process. To
this end, we propose NoisyRollout, a simple yet effective RL approach that
mixes trajectories from both clean and moderately distorted images to introduce
targeted diversity in visual perception and the resulting reasoning patterns.
Without additional training cost, NoisyRollout enhances the exploration
capabilities of VLMs by incorporating a vision-oriented inductive bias.
Furthermore, NoisyRollout employs a noise annealing schedule that gradually
reduces distortion strength over training, ensuring benefit from noisy signals
early while maintaining training stability and scalability in later stages.
With just 2.1K training samples, NoisyRollout achieves state-of-the-art
performance among open-source RL-tuned models on 5 out-of-domain benchmarks
spanning both reasoning and perception tasks, while preserving comparable or
even better in-domain performance.
NoisyRollout, an RL approach that introduces targeted diversity through noise in image trajectories, enhances VLM policy exploration without additional training cost, achieving state-of-the-art performance on out-of-domain benchmarks.
reinforcement learning vision-language models policy exploration noisy signals noise annealing vision-oriented inductive bias