title: Cosmos-Reason1: From Physical Common Sense To Embodied Reasoning
summary: Physical AI systems need to perceive, understand, and perform complex actions
  in the physical world. In this paper, we present the Cosmos-Reason1 models that
  can understand the physical world and generate appropriate embodied decisions
  (e.g., next step action) in natural language through long chain-of-thought
  reasoning processes. We begin by defining key capabilities for Physical AI
  reasoning, with a focus on physical common sense and embodied reasoning. To
  represent physical common sense, we use a hierarchical ontology that captures
  fundamental knowledge about space, time, and physics. For embodied reasoning,
  we rely on a two-dimensional ontology that generalizes across different
  physical embodiments. Building on these capabilities, we develop two multimodal
  large language models, Cosmos-Reason1-8B and Cosmos-Reason1-56B. We curate data
  and train our models in four stages: vision pre-training, general supervised
  fine-tuning (SFT), Physical AI SFT, and Physical AI reinforcement learning (RL)
  as the post-training. To evaluate our models, we build comprehensive benchmarks
  for physical common sense and embodied reasoning according to our ontologies.
  Evaluation results show that Physical AI SFT and reinforcement learning bring
  significant improvements. To facilitate the development of Physical AI, we will
  make our code and pre-trained models available under the NVIDIA Open Model
  License at https://github.com/nvidia-cosmos/cosmos-reason1.
paper_authors._id: 67dcadafb2cd7d4f3a266037
paper_authors.name: NVIDIA
paper_authors._id: 67dcadafb2cd7d4f3a266039
paper_authors.name: Alisson Azzolini
paper_authors._id: 67dcadafb2cd7d4f3a26603a
paper_authors.name: Hannah Brandon
paper_authors._id: 67dcadafb2cd7d4f3a26603b
paper_authors.name: Prithvijit Chattopadhyay
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-03-21T12:55:45.738Z
paper_authors._id: 628d451386d23ad1560882c4
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/628d451386d23ad1560882c4/UMxez0DEvX5qdP5ddqi-8.png
paper_authors.fullname: Prithvijit Chattopadhyay
paper_authors.type: user
paper_authors.user: prithv1
paper_authors._id: 67dcadafb2cd7d4f3a26603c
paper_authors.name: Huayu Chen
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-03-21T12:56:00.301Z
paper_authors._id: 6630f87ee53fcb71c3887df0
paper_authors.avatarUrl: /avatars/50191a3d45bebf90cf08df09477e95db.svg
paper_authors.fullname: HuayuChen
paper_authors.type: user
paper_authors.user: HuayuChen
paper_authors._id: 67dcadafb2cd7d4f3a26603d
paper_authors.name: Jinju Chu
paper_authors._id: 67dcadafb2cd7d4f3a26603e
paper_authors.name: Yin Cui
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-03-24T08:54:25.189Z
paper_authors._id: 649f05367b57fab3a5b27c8b
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/649f05367b57fab3a5b27c8b/UDJB4yqF2NmaRwCyTOfcl.jpeg
paper_authors.fullname: Yin Cui
paper_authors.type: user
paper_authors.user: richardaecn
paper_authors._id: 67dcadafb2cd7d4f3a26603f
paper_authors.name: Jenna Diamond
paper_authors._id: 67dcadafb2cd7d4f3a266040
paper_authors.name: Yifan Ding
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-03-21T12:56:21.932Z
paper_authors._id: 5e08d1e0fcf41d740b69964f
paper_authors.avatarUrl: /avatars/926fef14351d247a0287f6f9dc9e3cb6.svg
paper_authors.fullname: Yifan Ding
paper_authors.type: user
paper_authors.user: YifanDing
paper_authors._id: 67dcadafb2cd7d4f3a266041
paper_authors.name: Francesco Ferroni
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-03-21T11:41:11.413Z
paper_authors._id: 6353d3e95eac2d2efa7501f9
paper_authors.avatarUrl: /avatars/4b063f54000bed4bfb1bfcc3cde1a09e.svg
paper_authors.fullname: Francesco Ferroni
paper_authors.type: user
paper_authors.user: fferroni
paper_authors._id: 67dcadafb2cd7d4f3a266042
paper_authors.name: Rama Govindaraju
paper_authors._id: 67dcadafb2cd7d4f3a266043
paper_authors.name: Jinwei Gu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-03-21T12:56:32.134Z
paper_authors._id: 647e8118770c299e56fc2bc8
paper_authors.avatarUrl: /avatars/adf80f3473dda42450148789ae5c208f.svg
paper_authors.fullname: Jinwei Gu
paper_authors.type: user
paper_authors.user: jwgu
paper_authors._id: 67dcadafb2cd7d4f3a266044
paper_authors.name: Siddharth Gururani
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-03-21T12:56:41.456Z
paper_authors._id: 679c387ac4fd876ea127b49f
paper_authors.avatarUrl: /avatars/605f04dce6c2bf0f428642650d6642f5.svg
paper_authors.fullname: Siddharth Gururani
paper_authors.type: user
paper_authors.user: siddfisher
paper_authors._id: 67dcadafb2cd7d4f3a266045
paper_authors.name: Imad El Hanafi
paper_authors._id: 67dcadafb2cd7d4f3a266046
paper_authors.name: Zekun Hao
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-03-21T12:56:51.384Z
paper_authors._id: 6721a3afef642d85f1f6eed5
paper_authors.avatarUrl: /avatars/1689e9ef452a7b71189b5d81fadeff01.svg
paper_authors.fullname: Zekun Hao
paper_authors.type: user
paper_authors.user: zekunhao
paper_authors._id: 67dcadafb2cd7d4f3a266047
paper_authors.name: Jacob Huffman
paper_authors._id: 67dcadafb2cd7d4f3a266048
paper_authors.name: Jingyi Jin
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-03-21T12:57:01.419Z
paper_authors._id: 678ec9e693649b2688c1d2ad
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/no-auth/tLzqFzkmPu8o-A8PtViGO.png
paper_authors.fullname: Jingyi Jin
paper_authors.type: user
paper_authors.user: jingyijin03
paper_authors._id: 67dcadafb2cd7d4f3a266049
paper_authors.name: Brendan Johnson
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-03-21T12:57:09.652Z
paper_authors._id: 646f8b3d850a938d6c537775
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/646f8b3d850a938d6c537775/-bO4-nu6KocS9Wa1gcX4H.png
paper_authors.fullname: Brendan Johnson
paper_authors.type: user
paper_authors.user: BrendanBJohnson
paper_authors._id: 67dcadafb2cd7d4f3a26604a
paper_authors.name: Rizwan Khan
paper_authors._id: 67dcadafb2cd7d4f3a26604b
paper_authors.name: George Kurian
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-03-21T12:57:31.615Z
paper_authors._id: 66d05ee16447c6c45f54d65d
paper_authors.avatarUrl: /avatars/21df19be6a2e3426c858ad373e44f39b.svg
paper_authors.fullname: George Kurian
paper_authors.type: user
paper_authors.user: Duke09
paper_authors._id: 67dcadafb2cd7d4f3a26604c
paper_authors.name: Elena Lantz
paper_authors._id: 67dcadafb2cd7d4f3a26604d
paper_authors.name: Nayeon Lee
paper_authors._id: 67dcadafb2cd7d4f3a26604e
paper_authors.name: Zhaoshuo Li
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-03-21T12:57:42.436Z
paper_authors._id: 647f13d85e1bc4753743f79e
paper_authors.avatarUrl: /avatars/13a2a7fbd55d2e089fcf9ff0f9a41635.svg
paper_authors.fullname: Zhaoshuo Li
paper_authors.type: user
paper_authors.user: mli0603
paper_authors._id: 67dcadafb2cd7d4f3a26604f
paper_authors.name: Xuan Li
paper_authors._id: 67dcadafb2cd7d4f3a266050
paper_authors.name: Tsung-Yi Lin
paper_authors._id: 67dcadafb2cd7d4f3a266051
paper_authors.name: Yen-Chen Lin
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-03-21T12:58:29.625Z
paper_authors._id: 654c025dfda290a640090714
paper_authors.avatarUrl: /avatars/83b730d2ed91f824e3f35e1999f5377f.svg
paper_authors.fullname: Yen-Chen Lin
paper_authors.type: user
paper_authors.user: yenchenl-nvidia
paper_authors._id: 67dcadafb2cd7d4f3a266052
paper_authors.name: Ming-Yu Liu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-03-21T12:58:21.711Z
paper_authors._id: 62f049afdf4b93aad5c7f2d6
paper_authors.avatarUrl: /avatars/e272e58ad996733d7098e50248e5b57e.svg
paper_authors.fullname: Ming-Yu Liu
paper_authors.type: user
paper_authors.user: mingyuliutw
paper_authors._id: 67dcadafb2cd7d4f3a266053
paper_authors.name: Andrew Mathau
paper_authors._id: 67dcadafb2cd7d4f3a266054
paper_authors.name: Yun Ni
paper_authors._id: 67dcadafb2cd7d4f3a266055
paper_authors.name: Lindsey Pavao
paper_authors._id: 67dcadafb2cd7d4f3a266056
paper_authors.name: Wei Ping
paper_authors._id: 67dcadafb2cd7d4f3a266057
paper_authors.name: David W. Romero
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-03-24T08:54:37.389Z
paper_authors._id: 6734efd27d73f338440d2420
paper_authors.avatarUrl: /avatars/390187e6670dea73987840ba2de8706d.svg
paper_authors.fullname: David W Romero
paper_authors.type: user
paper_authors.user: dwromero-nv
paper_authors._id: 67dcadafb2cd7d4f3a266058
paper_authors.name: Misha Smelyanskiy
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-03-21T12:58:13.481Z
paper_authors._id: 6606f3def4ab651901fe8602
paper_authors.avatarUrl: /avatars/bacd64657a0154d2a319c9b411ef9fe4.svg
paper_authors.fullname: Misha Smelyanskiy
paper_authors.type: user
paper_authors.user: Msmelyan
paper_authors._id: 67dcadafb2cd7d4f3a266059
paper_authors.name: Shuran Song
paper_authors._id: 67dcadafb2cd7d4f3a26605a
paper_authors.name: Lyne Tchapmi
paper_authors._id: 67dcadafb2cd7d4f3a26605b
paper_authors.name: Andrew Z. Wang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-06-02T07:52:20.071Z
paper_authors._id: 6690456b54b460a2a6ad0817
paper_authors.avatarUrl: /avatars/c0ca52463ed2157be6390daed2a484b5.svg
paper_authors.fullname: Andrew Z. Wang
paper_authors.type: user
paper_authors.user: andrewzwang
paper_authors._id: 67dcadafb2cd7d4f3a26605c
paper_authors.name: Boxin Wang
paper_authors._id: 67dcadafb2cd7d4f3a26605d
paper_authors.name: Haoxiang Wang
paper_authors._id: 67dcadafb2cd7d4f3a26605e
paper_authors.name: Fangyin Wei
paper_authors._id: 67dcadafb2cd7d4f3a26605f
paper_authors.name: Jiashu Xu
paper_authors._id: 67dcadafb2cd7d4f3a266060
paper_authors.name: Yao Xu
paper_authors._id: 67dcadafb2cd7d4f3a266061
paper_authors.name: Xiaodong Yang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-03-21T12:58:02.852Z
paper_authors._id: 64ab727051756fb15c275c5d
paper_authors.avatarUrl: /avatars/26f77a1f81f0f4703eb87e542cd81fa9.svg
paper_authors.fullname: Yangxiaodong
paper_authors.type: user
paper_authors.user: Xiaodongyang
paper_authors._id: 67dcadafb2cd7d4f3a266062
paper_authors.name: Zhuolin Yang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-03-21T11:41:13.584Z
paper_authors._id: 67d75b0117c2acac528f47b6
paper_authors.avatarUrl: /avatars/619aacd1a619aab64de3499ac3ee2229.svg
paper_authors.fullname: Zhuolin Yang
paper_authors.type: user
paper_authors.user: zhuoliny
paper_authors._id: 67dcadafb2cd7d4f3a266063
paper_authors.name: Xiaohui Zeng
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-03-21T12:57:54.702Z
paper_authors._id: 64d9cf178767727dff1da0c7
paper_authors.avatarUrl: /avatars/c2ec61b511754a8d312e2db3229215ec.svg
paper_authors.fullname: Xiaohui  ZENG
paper_authors.type: user
paper_authors.user: eccony
paper_authors._id: 67dcadafb2cd7d4f3a266064
paper_authors.name: Zhe Zhang
paper_title: Cosmos-Reason1: From Physical Common Sense To Embodied Reasoning
paper_summary: Physical AI systems need to perceive, understand, and perform complex actions
  in the physical world. In this paper, we present the Cosmos-Reason1 models that
  can understand the physical world and generate appropriate embodied decisions
  (e.g., next step action) in natural language through long chain-of-thought
  reasoning processes. We begin by defining key capabilities for Physical AI
  reasoning, with a focus on physical common sense and embodied reasoning. To
  represent physical common sense, we use a hierarchical ontology that captures
  fundamental knowledge about space, time, and physics. For embodied reasoning,
  we rely on a two-dimensional ontology that generalizes across different
  physical embodiments. Building on these capabilities, we develop two multimodal
  large language models, Cosmos-Reason1-8B and Cosmos-Reason1-56B. We curate data
  and train our models in four stages: vision pre-training, general supervised
  fine-tuning (SFT), Physical AI SFT, and Physical AI reinforcement learning (RL)
  as the post-training. To evaluate our models, we build comprehensive benchmarks
  for physical common sense and embodied reasoning according to our ontologies.
  Evaluation results show that Physical AI SFT and reinforcement learning bring
  significant improvements. To facilitate the development of Physical AI, we will
  make our code and pre-trained models available under the NVIDIA Open Model
  License at https://github.com/nvidia-cosmos/cosmos-reason1.
paper_ai_summary: Cosmos-Reason1 models, using hierarchical and two-dimensional ontologies for physical common sense and embodied reasoning, generate embodied decisions through multimodal large language models trained in vision and Physical AI stages.
paper_ai_keywords: Physical AI reasoning chain-of-thought hierarchical ontology two-dimensional ontology physical common sense embodied reasoning multimodal large language models vision pre-training supervised fine-tuning Physical AI SFT Physical AI reinforcement learning benchmarks