title: Magma: A Foundation Model for Multimodal AI Agents
summary: We present Magma, a foundation model that serves multimodal AI agentic tasks
  in both the digital and physical worlds. Magma is a significant extension of
  vision-language (VL) models in that it not only retains the VL understanding
  ability (verbal intelligence) of the latter, but is also equipped with the
  ability to plan and act in the visual-spatial world (spatial-temporal
  intelligence) and complete agentic tasks ranging from UI navigation to robot
  manipulation. To endow the agentic capabilities, Magma is pretrained on large
  amounts of heterogeneous datasets spanning from images, videos to robotics
  data, where the actionable visual objects (e.g., clickable buttons in GUI) in
  images are labeled by Set-of-Mark (SoM) for action grounding, and the object
  movements (e.g., the trace of human hands or robotic arms) in videos are
  labeled by Trace-of-Mark (ToM) for action planning. Extensive experiments show
  that SoM and ToM reach great synergy and facilitate the acquisition of
  spatial-temporal intelligence for our Magma model, which is fundamental to a
  wide range of tasks as shown in Fig.1. In particular, Magma creates new
  state-of-the-art results on UI navigation and robotic manipulation tasks,
  outperforming previous models that are specifically tailored to these tasks. On
  image and video-related multimodal tasks, Magma also compares favorably to
  popular large multimodal models that are trained on much larger datasets. We
  make our model and code public for reproducibility at
  https://microsoft.github.io/Magma.
paper_authors._id: 67b5625fb27eb6046b2ceec5
paper_authors.name: Jianwei Yang
paper_authors._id: 67b5625fb27eb6046b2ceec6
paper_authors.name: Reuben Tan
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-02-19T10:19:01.753Z
paper_authors._id: 674774e8eb8fb5ea40877838
paper_authors.avatarUrl: /avatars/7ee8599cb1f7bb4402bc8512faf6ca12.svg
paper_authors.fullname: Reuben Tan
paper_authors.type: user
paper_authors.user: tanreuben
paper_authors._id: 67b5625fb27eb6046b2ceec7
paper_authors.name: Qianhui Wu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-02-19T10:18:38.510Z
paper_authors._id: 63ef330b1e695b35aa484e11
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/63ef330b1e695b35aa484e11/bXwpGy0dl8JXeJwJ--ilr.jpeg
paper_authors.fullname: Qianhui WU
paper_authors.type: user
paper_authors.user: qianhuiwu
paper_authors._id: 67b5625fb27eb6046b2ceec8
paper_authors.name: Ruijie Zheng
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-02-19T10:18:32.761Z
paper_authors._id: 653b24ef8f8d60f204872f0a
paper_authors.avatarUrl: /avatars/45a55219e8a78be53fd32e96ba460282.svg
paper_authors.fullname: Ruijie Zheng
paper_authors.type: user
paper_authors.user: rzheng12
paper_authors._id: 67b5625fb27eb6046b2ceec9
paper_authors.name: Baolin Peng
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-02-19T10:18:25.610Z
paper_authors._id: 61942296d5c2ba6daa290357
paper_authors.avatarUrl: /avatars/594021cc183c4922d48b46f43772a062.svg
paper_authors.fullname: Baolin Peng
paper_authors.type: user
paper_authors.user: Baolin
paper_authors._id: 67b5625fb27eb6046b2ceeca
paper_authors.name: Yongyuan Liang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-02-19T10:18:00.072Z
paper_authors._id: 6646d5819bb34d2b6b7455d3
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/JFH3ZTPvlaVSg4RJJBb6L.jpeg
paper_authors.fullname: Yongyuan Liang
paper_authors.type: user
paper_authors.user: cheryyunl
paper_authors._id: 67b5625fb27eb6046b2ceecb
paper_authors.name: Yu Gu
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-02-20T09:37:05.283Z
paper_authors._id: 645a02972abf6165a3ba5df8
paper_authors.avatarUrl: /avatars/6166e4f213b9020ba99f853cb04f8db0.svg
paper_authors.fullname: Yu Gu
paper_authors.type: user
paper_authors.user: aidenygu
paper_authors._id: 67b5625fb27eb6046b2ceecc
paper_authors.name: Mu Cai
paper_authors._id: 67b5625fb27eb6046b2ceecd
paper_authors.name: Seonghyeon Ye
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-02-19T10:17:47.576Z
paper_authors._id: 62551f7767f0b85962624047
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1664552038624-62551f7767f0b85962624047.png
paper_authors.fullname: Seonghyeon Ye
paper_authors.type: user
paper_authors.user: seonghyeonye
paper_authors._id: 67b5625fb27eb6046b2ceece
paper_authors.name: Joel Jang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-02-19T10:17:40.305Z
paper_authors._id: 613e1a9267835521a6816b04
paper_authors.avatarUrl: /avatars/49edaa425bbce04dff92bbfb12a6b41c.svg
paper_authors.fullname: Joel Jang
paper_authors.type: user
paper_authors.user: wkddydpf
paper_authors._id: 67b5625fb27eb6046b2ceecf
paper_authors.name: Yuquan Deng
paper_authors._id: 67b5625fb27eb6046b2ceed0
paper_authors.name: Lars Liden
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-02-19T10:17:09.378Z
paper_authors._id: 65bc13c7719492167d777718
paper_authors.avatarUrl: /avatars/d1a3dc52c130b84a47c8a4ddd2e74be8.svg
paper_authors.fullname: Lars Liden
paper_authors.type: user
paper_authors.user: larsliden
paper_authors._id: 67b5625fb27eb6046b2ceed1
paper_authors.name: Jianfeng Gao
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-02-19T10:17:03.043Z
paper_authors._id: 641904caf9d6f1d772ec7af7
paper_authors.avatarUrl: /avatars/4a63eac71eb30f70b1a0e9d4708f26c1.svg
paper_authors.fullname: Jianfeng Gao
paper_authors.type: user
paper_authors.user: wyngjf
paper_title: Magma: A Foundation Model for Multimodal AI Agents
paper_summary: We present Magma, a foundation model that serves multimodal AI agentic tasks
  in both the digital and physical worlds. Magma is a significant extension of
  vision-language (VL) models in that it not only retains the VL understanding
  ability (verbal intelligence) of the latter, but is also equipped with the
  ability to plan and act in the visual-spatial world (spatial-temporal
  intelligence) and complete agentic tasks ranging from UI navigation to robot
  manipulation. To endow the agentic capabilities, Magma is pretrained on large
  amounts of heterogeneous datasets spanning from images, videos to robotics
  data, where the actionable visual objects (e.g., clickable buttons in GUI) in
  images are labeled by Set-of-Mark (SoM) for action grounding, and the object
  movements (e.g., the trace of human hands or robotic arms) in videos are
  labeled by Trace-of-Mark (ToM) for action planning. Extensive experiments show
  that SoM and ToM reach great synergy and facilitate the acquisition of
  spatial-temporal intelligence for our Magma model, which is fundamental to a
  wide range of tasks as shown in Fig.1. In particular, Magma creates new
  state-of-the-art results on UI navigation and robotic manipulation tasks,
  outperforming previous models that are specifically tailored to these tasks. On
  image and video-related multimodal tasks, Magma also compares favorably to
  popular large multimodal models that are trained on much larger datasets. We
  make our model and code public for reproducibility at
  https://microsoft.github.io/Magma.
paper_ai_summary: Magma is a multimodal foundation model with both verbal intelligence and spatial-temporal intelligence, trained on diverse datasets to perform agentic tasks like UI navigation and robotic manipulation, outperforming specialized models.
paper_ai_keywords: vision-language models spatial-temporal intelligence actionable visual objects Set-of-Mark Trace-of-Mark action grounding action planning spatial-temporal intelligence UI navigation robotic manipulation