Omni-R1: Reinforcement Learning for Omnimodal Reasoning via Two-System
  Collaboration
Long-horizon video-audio reasoning and fine-grained pixel understanding
impose conflicting requirements on omnimodal models: dense temporal coverage
demands many low-resolution frames, whereas precise grounding calls for
high-resolution inputs. We tackle this trade-off with a two-system
architecture: a Global Reasoning System selects informative keyframes and
rewrites the task at low spatial cost, while a Detail Understanding System
performs pixel-level grounding on the selected high-resolution snippets.
Because ``optimal'' keyframe selection and reformulation are ambiguous and hard
to supervise, we formulate them as a reinforcement learning (RL) problem and
present Omni-R1, an end-to-end RL framework built on Group Relative Policy
Optimization. Omni-R1 trains the Global Reasoning System through hierarchical
rewards obtained via online collaboration with the Detail Understanding System,
requiring only one epoch of RL on small task splits.
  Experiments on two challenging benchmarks, namely Referring Audio-Visual
Segmentation (RefAVS) and Reasoning Video Object Segmentation (REVOS), show
that Omni-R1 not only surpasses strong supervised baselines but also
outperforms specialized state-of-the-art models, while substantially improving
out-of-domain generalization and mitigating multimodal hallucination. Our
results demonstrate the first successful application of RL to large-scale
omnimodal reasoning and highlight a scalable path toward universally foundation
models.
68352e44c829f2ea1e0484b5
Hao Zhong
68352e44c829f2ea1e0484b6
Muzhi Zhu
claimed_verified
2025-05-28T09:00:37.809Z
632179745fc60c44fd91fc33
/avatars/37d4fefbcc19f091dccffefec9706de2.svg
zhumuzhi
user
Z-MU-Z
68352e44c829f2ea1e0484b7
Zongze Du
68352e44c829f2ea1e0484b8
Zheng Huang
68352e44c829f2ea1e0484b9
Canyu Zhao
claimed_verified
2025-05-27T07:50:50.579Z
646efd223dd912a539e0bd46
https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/EOFAv5xvOgJOzuDgh4nSb.png
Canyu Zhao
user
Canyu
68352e44c829f2ea1e0484ba
Mingyu Liu
claimed_verified
2025-05-28T09:00:35.196Z
652e25d2e647b0ee0a024f26
/avatars/b5c65cf6c8d0ddc9b8ef0226e0295d56.svg
Mingyu Liu
user
MingyuLiu
68352e44c829f2ea1e0484bb
Wen Wang
68352e44c829f2ea1e0484bc
Hao Chen
68352e44c829f2ea1e0484bd
Chunhua Shen
Omni-R1: Reinforcement Learning for Omnimodal Reasoning via Two-System
  Collaboration
Long-horizon video-audio reasoning and fine-grained pixel understanding
impose conflicting requirements on omnimodal models: dense temporal coverage
demands many low-resolution frames, whereas precise grounding calls for
high-resolution inputs. We tackle this trade-off with a two-system
architecture: a Global Reasoning System selects informative keyframes and
rewrites the task at low spatial cost, while a Detail Understanding System
performs pixel-level grounding on the selected high-resolution snippets.
Because ``optimal'' keyframe selection and reformulation are ambiguous and hard
to supervise, we formulate them as a reinforcement learning (RL) problem and
present Omni-R1, an end-to-end RL framework built on Group Relative Policy
Optimization. Omni-R1 trains the Global Reasoning System through hierarchical
rewards obtained via online collaboration with the Detail Understanding System,
requiring only one epoch of RL on small task splits.
  Experiments on two challenging benchmarks, namely Referring Audio-Visual
Segmentation (RefAVS) and Reasoning Video Object Segmentation (REVOS), show
that Omni-R1 not only surpasses strong supervised baselines but also
outperforms specialized state-of-the-art models, while substantially improving
out-of-domain generalization and mitigating multimodal hallucination. Our
results demonstrate the first successful application of RL to large-scale
omnimodal reasoning and highlight a scalable path toward universally foundation
models.
An end-to-end reinforcement learning framework, Omni-R1, achieves superior performance in long-horizon video-audio reasoning and fine-grained pixel understanding tasks by combining global reasoning and detail understanding systems.
reinforcement learning Group Relative Policy Optimization hierarchical rewards Referring Audio-Visual Segmentation Reasoning Video Object Segmentation out-of-domain generalization multimodal hallucination universally foundation models