EmbRACE-3K: Embodied Reasoning and Action in Complex Environments
Recent advanced vision-language models(VLMs) have demonstrated strong
performance on passive, offline image and video understanding tasks. However,
their effectiveness in embodied settings, which require online interaction and
active scene understanding remains limited. In such scenarios, an agent
perceives the environment from a first-person perspective, with each action
dynamically shaping subsequent observations. Even state-of-the-art models such
as GPT-4o, Claude 3.5 Sonnet, and Gemini 2.5 Pro struggle in open-environment
interactions, exhibiting clear limitations in spatial reasoning and
long-horizon planning. To address this gap, we introduce EmRACE-3K, a dataset
of over 3,000 language-guided tasks situated in diverse, photorealistic
environments constructed using Unreal Engine and the UnrealCV-Zoo framework.
The tasks encompass a wide range of embodied challenges, including navigation,
object manipulation, and multi-stage goal execution. Each task unfolds as a
multi-step trajectory, pairing first-person visual observations with high-level
instructions, grounded actions, and natural language rationales that express
the agent's intent at every step. Using EmRACE-3K, we establish a benchmark to
evaluate the embodied reasoning capabilities of VLMs across three key
dimensions: Exploration, Dynamic Spatial-Semantic Reasoning, and Multi-stage
Goal Execution. In zero-shot settings, all models achieve success rates below
20%, underscoring the challenge posed by our benchmark and the current
limitations of VLMs in interactive environments. To demonstrate the utility of
EmRACE-3K, we further fine-tune Qwen2.5-VL-7B using supervised learning
followed by reinforcement learning. This approach yields substantial
improvements across all three challenge categories, highlighting the dataset's
effectiveness in enabling the development of embodied reasoning capabilities.
6875d6e7257d4f04353705b5
Mingxian Lin
claimed_verified
2025-07-15T19:09:31.521Z
6548662e08568852409762f6
/avatars/d12ff2564375d018669248caaeed1e1a.svg
Mingxian Lin
user
mxllc
6875d6e7257d4f04353705b6
Wei Huang
6875d6e7257d4f04353705b7
Yitang Li
6875d6e7257d4f04353705b8
Chengjie Jiang
6875d6e7257d4f04353705b9
Kui Wu
6875d6e7257d4f04353705ba
Fangwei Zhong
6875d6e7257d4f04353705bb
Shengju Qian
6875d6e7257d4f04353705bc
Xin Wang
6875d6e7257d4f04353705bd
Xiaojuan Qi
EmbRACE-3K: Embodied Reasoning and Action in Complex Environments
Recent advanced vision-language models(VLMs) have demonstrated strong
performance on passive, offline image and video understanding tasks. However,
their effectiveness in embodied settings, which require online interaction and
active scene understanding remains limited. In such scenarios, an agent
perceives the environment from a first-person perspective, with each action
dynamically shaping subsequent observations. Even state-of-the-art models such
as GPT-4o, Claude 3.5 Sonnet, and Gemini 2.5 Pro struggle in open-environment
interactions, exhibiting clear limitations in spatial reasoning and
long-horizon planning. To address this gap, we introduce EmRACE-3K, a dataset
of over 3,000 language-guided tasks situated in diverse, photorealistic
environments constructed using Unreal Engine and the UnrealCV-Zoo framework.
The tasks encompass a wide range of embodied challenges, including navigation,
object manipulation, and multi-stage goal execution. Each task unfolds as a
multi-step trajectory, pairing first-person visual observations with high-level
instructions, grounded actions, and natural language rationales that express
the agent's intent at every step. Using EmRACE-3K, we establish a benchmark to
evaluate the embodied reasoning capabilities of VLMs across three key
dimensions: Exploration, Dynamic Spatial-Semantic Reasoning, and Multi-stage
Goal Execution. In zero-shot settings, all models achieve success rates below
20%, underscoring the challenge posed by our benchmark and the current
limitations of VLMs in interactive environments. To demonstrate the utility of
EmRACE-3K, we further fine-tune Qwen2.5-VL-7B using supervised learning
followed by reinforcement learning. This approach yields substantial
improvements across all three challenge categories, highlighting the dataset's
effectiveness in enabling the development of embodied reasoning capabilities.
EmRACE-3K, a dataset of language-guided tasks in diverse environments, evaluates and improves embodied reasoning capabilities of vision-language models through zero-shot and fine-tuned performance.
vision-language models embodied settings first-person perspective spatial reasoning long-horizon planning Unreal Engine UnrealCV-Zoo navigation object manipulation multi-stage goal execution multi-step trajectory high-level instructions grounded actions natural language rationales zero-shot settings supervised learning reinforcement learning embodied reasoning capabilities