title: Visual Embodied Brain: Let Multimodal Large Language Models See, Think,
    and Control in Spaces
summary: The remarkable progress of Multimodal Large Language Models (MLLMs) has
  attracted increasing attention to extend them to physical entities like legged
  robot. This typically requires MLLMs to not only grasp multimodal understanding
  abilities, but also integrate visual-spatial reasoning and physical interaction
  capabilities. Nevertheless,existing methods struggle to unify these
  capabilities due to their fundamental differences.In this paper, we present the
  Visual Embodied Brain (VeBrain), a unified framework for perception, reasoning,
  and control in real world. VeBrain reformulates robotic control into common
  text-based MLLM tasks in the 2D visual space, thus unifying the objectives and
  mapping spaces of different tasks. Then, a novel robotic adapter is proposed to
  convert textual control signals from MLLMs to motion policies of real robots.
  From the data perspective, we further introduce VeBrain-600k, a high-quality
  instruction dataset encompassing various capabilities of VeBrain. In
  VeBrain-600k, we take hundreds of hours to collect, curate and annotate the
  data, and adopt multimodal chain-of-thought(CoT) to mix the different
  capabilities into a single conversation. Extensive experiments on 13 multimodal
  benchmarks and 5 spatial intelligence benchmarks demonstrate the superior
  performance of VeBrain to existing MLLMs like Qwen2.5-VL. When deployed to
  legged robots and robotic arms, VeBrain shows strong adaptability, flexibility,
  and compositional capabilities compared to existing methods. For example,
  compared to Qwen2.5-VL, VeBrain not only achieves substantial gains on MMVet by
  +5.6%, but also excels in legged robot tasks with +50% average gains.
paper_authors._id: 683e709a3a4c2c3b2750fc32
paper_authors.name: Gen Luo
paper_authors._id: 683e709a3a4c2c3b2750fc33
paper_authors.name: Ganlin Yang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-06-03T08:45:03.857Z
paper_authors._id: 6565d7149afd51867e55520b
paper_authors.avatarUrl: /avatars/027b17651e61df598af53f69b92e7771.svg
paper_authors.fullname: Ganlin Yang
paper_authors.type: user
paper_authors.user: ganlinyang
paper_authors._id: 683e709a3a4c2c3b2750fc34
paper_authors.name: Ziyang Gong
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-06-03T08:45:00.364Z
paper_authors._id: 660691330be1fbe3b9e4c33d
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/660691330be1fbe3b9e4c33d/TxrDFH_cRu3AlpMC3xmhv.jpeg
paper_authors.fullname: ZiYang Gong
paper_authors.type: user
paper_authors.user: Cusyoung
paper_authors._id: 683e709a3a4c2c3b2750fc35
paper_authors.name: Guanzhou Chen
paper_authors._id: 683e709a3a4c2c3b2750fc36
paper_authors.name: Haonan Duan
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-06-04T09:03:03.236Z
paper_authors._id: 66ab30dfd456f0408b93f27b
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/66ab30dfd456f0408b93f27b/nps4Kni_eOExO5Z92RiiF.jpeg
paper_authors.fullname: Haonan Duan
paper_authors.type: user
paper_authors.user: robot-haonan
paper_authors._id: 683e709a3a4c2c3b2750fc37
paper_authors.name: Erfei Cui
paper_authors._id: 683e709a3a4c2c3b2750fc38
paper_authors.name: Ronglei Tong
paper_authors._id: 683e709a3a4c2c3b2750fc39
paper_authors.name: Zhi Hou
paper_authors._id: 683e709a3a4c2c3b2750fc3a
paper_authors.name: Tianyi Zhang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-06-07T05:49:46.792Z
paper_authors._id: 64c9e86a6a26cddbecd9bae2
paper_authors.avatarUrl: /avatars/61a84989dbbc1898ebcba3236dbed039.svg
paper_authors.fullname: Tianyi Zhang
paper_authors.type: user
paper_authors.user: TianyiZhang0213
paper_authors._id: 683e709a3a4c2c3b2750fc3b
paper_authors.name: Zhe Chen
paper_authors._id: 683e709a3a4c2c3b2750fc3c
paper_authors.name: Shenglong Ye
paper_authors._id: 683e709a3a4c2c3b2750fc3d
paper_authors.name: Lewei Lu
paper_authors._id: 683e709a3a4c2c3b2750fc3e
paper_authors.name: Jingbo Wang
paper_authors._id: 683e709a3a4c2c3b2750fc3f
paper_authors.name: Wenhai Wang
paper_authors._id: 683e709a3a4c2c3b2750fc40
paper_authors.name: Jifeng Dai
paper_authors._id: 683e709a3a4c2c3b2750fc41
paper_authors.name: Yu Qiao
paper_authors._id: 683e709a3a4c2c3b2750fc42
paper_authors.name: Rongrong Ji
paper_authors._id: 683e709a3a4c2c3b2750fc43
paper_authors.name: Xizhou Zhu
paper_title: Visual Embodied Brain: Let Multimodal Large Language Models See, Think,
    and Control in Spaces
paper_summary: The remarkable progress of Multimodal Large Language Models (MLLMs) has
  attracted increasing attention to extend them to physical entities like legged
  robot. This typically requires MLLMs to not only grasp multimodal understanding
  abilities, but also integrate visual-spatial reasoning and physical interaction
  capabilities. Nevertheless,existing methods struggle to unify these
  capabilities due to their fundamental differences.In this paper, we present the
  Visual Embodied Brain (VeBrain), a unified framework for perception, reasoning,
  and control in real world. VeBrain reformulates robotic control into common
  text-based MLLM tasks in the 2D visual space, thus unifying the objectives and
  mapping spaces of different tasks. Then, a novel robotic adapter is proposed to
  convert textual control signals from MLLMs to motion policies of real robots.
  From the data perspective, we further introduce VeBrain-600k, a high-quality
  instruction dataset encompassing various capabilities of VeBrain. In
  VeBrain-600k, we take hundreds of hours to collect, curate and annotate the
  data, and adopt multimodal chain-of-thought(CoT) to mix the different
  capabilities into a single conversation. Extensive experiments on 13 multimodal
  benchmarks and 5 spatial intelligence benchmarks demonstrate the superior
  performance of VeBrain to existing MLLMs like Qwen2.5-VL. When deployed to
  legged robots and robotic arms, VeBrain shows strong adaptability, flexibility,
  and compositional capabilities compared to existing methods. For example,
  compared to Qwen2.5-VL, VeBrain not only achieves substantial gains on MMVet by
  +5.6%, but also excels in legged robot tasks with +50% average gains.
paper_ai_summary: VeBrain is a unified framework that integrates multimodal understanding, visual-spatial reasoning, and physical interaction for legged robots, demonstrating superior performance compared to existing methods across various benchmarks.
paper_ai_keywords: Multimodal Large Language Models VeBrain Visual Embodied Brain text-based MLLM tasks robotic adapter VeBrain-600k multimodal chain-of-thought MMVet compositional capabilities