HumanOmniV2: From Understanding to Omni-Modal Reasoning with Context
With the rapid evolution of multimodal large language models, the capacity to
deeply understand and interpret human intentions has emerged as a critical
capability, which demands detailed and thoughtful reasoning. In recent studies,
Reinforcement Learning (RL) has demonstrated potential in enhancing the
reasoning capabilities of Large Language Models (LLMs). Nonetheless, the
challenges associated with adapting RL to multimodal data and formats remain
largely unaddressed. In this paper, we identify two issues in existing
multimodal reasoning models: insufficient global context understanding and
shortcut problems. Insufficient context understanding can happen when a model
misinterprets multimodal context, resulting in incorrect answers. The shortcut
problem occurs when the model overlooks crucial clues in multimodal inputs,
directly addressing the query without considering the multimodal information.
To tackle these issues, we emphasize the necessity for the model to reason with
a clear understanding of the global context within multimodal inputs. This
global context understanding can effectively prevent the model from overlooking
key multimodal cues and ensure a thorough reasoning process. To ensure the
accurate interpretation of multimodal context information, we implement a
context reward judged by a large language model, alongside format and accuracy
rewards. Additionally, to improve complex reasoning capability, we employ the
LLM to assess the logical reward, determining whether the reasoning process
successfully integrates multimodal information with logical methods. We also
introduce a reasoning omni-modal benchmark, IntentBench, aimed at evaluating
models in understanding complex human intentions and emotions. Our proposed
method demonstrates advanced performance across multiple omni-modal benchmarks
compared to other open-source omni-modal models.
68634a2c588cea0da970c8ae
Qize Yang
claimed_verified
2025-07-02T07:40:53.951Z
66a097801a26a2350395edc7
/avatars/1e7e127cb7222df7d56e5bfda6bab519.svg
Qize Yang
user
PhilipC
68634a2c588cea0da970c8af
Shimin Yao
68634a2c588cea0da970c8b0
Weixuan Chen
68634a2c588cea0da970c8b1
Shenghao Fu
claimed_verified
2025-07-02T07:40:56.838Z
67067633351e0c16a5c27497
/avatars/356aa3431198c8931b820a714bcfb19d.svg
Shenghao Fu
user
fushh7
68634a2c588cea0da970c8b2
Detao Bai
68634a2c588cea0da970c8b3
Jiaxing Zhao
68634a2c588cea0da970c8b4
Boyuan Sun
claimed_verified
2025-07-01T10:13:33.669Z
66ef2611fcc1c455f8dce832
/avatars/c73ef2dfcd1e6ec8414a31226ad38e3b.svg
Boyuan Sun
user
BBBBCHAN
68634a2c588cea0da970c8b5
Bowen Yin
68634a2c588cea0da970c8b6
Xihan Wei
68634a2c588cea0da970c8b7
Jingren Zhou
HumanOmniV2: From Understanding to Omni-Modal Reasoning with Context
With the rapid evolution of multimodal large language models, the capacity to
deeply understand and interpret human intentions has emerged as a critical
capability, which demands detailed and thoughtful reasoning. In recent studies,
Reinforcement Learning (RL) has demonstrated potential in enhancing the
reasoning capabilities of Large Language Models (LLMs). Nonetheless, the
challenges associated with adapting RL to multimodal data and formats remain
largely unaddressed. In this paper, we identify two issues in existing
multimodal reasoning models: insufficient global context understanding and
shortcut problems. Insufficient context understanding can happen when a model
misinterprets multimodal context, resulting in incorrect answers. The shortcut
problem occurs when the model overlooks crucial clues in multimodal inputs,
directly addressing the query without considering the multimodal information.
To tackle these issues, we emphasize the necessity for the model to reason with
a clear understanding of the global context within multimodal inputs. This
global context understanding can effectively prevent the model from overlooking
key multimodal cues and ensure a thorough reasoning process. To ensure the
accurate interpretation of multimodal context information, we implement a
context reward judged by a large language model, alongside format and accuracy
rewards. Additionally, to improve complex reasoning capability, we employ the
LLM to assess the logical reward, determining whether the reasoning process
successfully integrates multimodal information with logical methods. We also
introduce a reasoning omni-modal benchmark, IntentBench, aimed at evaluating
models in understanding complex human intentions and emotions. Our proposed
method demonstrates advanced performance across multiple omni-modal benchmarks
compared to other open-source omni-modal models.
The paper addresses challenges in multimodal reasoning by implementing context and logical rewards to enhance global context understanding and prevent shortcut problems, using a reasoning omni-modal benchmark called IntentBench.
Reinforcement Learning Large Language Models multimodal reasoning global context understanding shortcut problems context reward logical reward IntentBench