Magma: A Foundation Model for Multimodal AI Agents
We present Magma, a foundation model that serves multimodal AI agentic tasks
in both the digital and physical worlds. Magma is a significant extension of
vision-language (VL) models in that it not only retains the VL understanding
ability (verbal intelligence) of the latter, but is also equipped with the
ability to plan and act in the visual-spatial world (spatial-temporal
intelligence) and complete agentic tasks ranging from UI navigation to robot
manipulation. To endow the agentic capabilities, Magma is pretrained on large
amounts of heterogeneous datasets spanning from images, videos to robotics
data, where the actionable visual objects (e.g., clickable buttons in GUI) in
images are labeled by Set-of-Mark (SoM) for action grounding, and the object
movements (e.g., the trace of human hands or robotic arms) in videos are
labeled by Trace-of-Mark (ToM) for action planning. Extensive experiments show
that SoM and ToM reach great synergy and facilitate the acquisition of
spatial-temporal intelligence for our Magma model, which is fundamental to a
wide range of tasks as shown in Fig.1. In particular, Magma creates new
state-of-the-art results on UI navigation and robotic manipulation tasks,
outperforming previous models that are specifically tailored to these tasks. On
image and video-related multimodal tasks, Magma also compares favorably to
popular large multimodal models that are trained on much larger datasets. We
make our model and code public for reproducibility at
https://microsoft.github.io/Magma.
67b5625fb27eb6046b2ceec5
Jianwei Yang
67b5625fb27eb6046b2ceec6
Reuben Tan
admin_assigned
2025-02-19T10:19:01.753Z
674774e8eb8fb5ea40877838
/avatars/7ee8599cb1f7bb4402bc8512faf6ca12.svg
Reuben Tan
user
tanreuben
67b5625fb27eb6046b2ceec7
Qianhui Wu
admin_assigned
2025-02-19T10:18:38.510Z
63ef330b1e695b35aa484e11
https://cdn-avatars.huggingface.co/v1/production/uploads/63ef330b1e695b35aa484e11/bXwpGy0dl8JXeJwJ--ilr.jpeg
Qianhui WU
user
qianhuiwu
67b5625fb27eb6046b2ceec8
Ruijie Zheng
admin_assigned
2025-02-19T10:18:32.761Z
653b24ef8f8d60f204872f0a
/avatars/45a55219e8a78be53fd32e96ba460282.svg
Ruijie Zheng
user
rzheng12
67b5625fb27eb6046b2ceec9
Baolin Peng
admin_assigned
2025-02-19T10:18:25.610Z
61942296d5c2ba6daa290357
/avatars/594021cc183c4922d48b46f43772a062.svg
Baolin Peng
user
Baolin
67b5625fb27eb6046b2ceeca
Yongyuan Liang
admin_assigned
2025-02-19T10:18:00.072Z
6646d5819bb34d2b6b7455d3
https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/JFH3ZTPvlaVSg4RJJBb6L.jpeg
Yongyuan Liang
user
cheryyunl
67b5625fb27eb6046b2ceecb
Yu Gu
claimed_verified
2025-02-20T09:37:05.283Z
645a02972abf6165a3ba5df8
/avatars/6166e4f213b9020ba99f853cb04f8db0.svg
Yu Gu
user
aidenygu
67b5625fb27eb6046b2ceecc
Mu Cai
67b5625fb27eb6046b2ceecd
Seonghyeon Ye
admin_assigned
2025-02-19T10:17:47.576Z
62551f7767f0b85962624047
https://cdn-avatars.huggingface.co/v1/production/uploads/1664552038624-62551f7767f0b85962624047.png
Seonghyeon Ye
user
seonghyeonye
67b5625fb27eb6046b2ceece
Joel Jang
admin_assigned
2025-02-19T10:17:40.305Z
613e1a9267835521a6816b04
/avatars/49edaa425bbce04dff92bbfb12a6b41c.svg
Joel Jang
user
wkddydpf
67b5625fb27eb6046b2ceecf
Yuquan Deng
67b5625fb27eb6046b2ceed0
Lars Liden
admin_assigned
2025-02-19T10:17:09.378Z
65bc13c7719492167d777718
/avatars/d1a3dc52c130b84a47c8a4ddd2e74be8.svg
Lars Liden
user
larsliden
67b5625fb27eb6046b2ceed1
Jianfeng Gao
admin_assigned
2025-02-19T10:17:03.043Z
641904caf9d6f1d772ec7af7
/avatars/4a63eac71eb30f70b1a0e9d4708f26c1.svg
Jianfeng Gao
user
wyngjf
Magma: A Foundation Model for Multimodal AI Agents
We present Magma, a foundation model that serves multimodal AI agentic tasks
in both the digital and physical worlds. Magma is a significant extension of
vision-language (VL) models in that it not only retains the VL understanding
ability (verbal intelligence) of the latter, but is also equipped with the
ability to plan and act in the visual-spatial world (spatial-temporal
intelligence) and complete agentic tasks ranging from UI navigation to robot
manipulation. To endow the agentic capabilities, Magma is pretrained on large
amounts of heterogeneous datasets spanning from images, videos to robotics
data, where the actionable visual objects (e.g., clickable buttons in GUI) in
images are labeled by Set-of-Mark (SoM) for action grounding, and the object
movements (e.g., the trace of human hands or robotic arms) in videos are
labeled by Trace-of-Mark (ToM) for action planning. Extensive experiments show
that SoM and ToM reach great synergy and facilitate the acquisition of
spatial-temporal intelligence for our Magma model, which is fundamental to a
wide range of tasks as shown in Fig.1. In particular, Magma creates new
state-of-the-art results on UI navigation and robotic manipulation tasks,
outperforming previous models that are specifically tailored to these tasks. On
image and video-related multimodal tasks, Magma also compares favorably to
popular large multimodal models that are trained on much larger datasets. We
make our model and code public for reproducibility at
https://microsoft.github.io/Magma.
Magma is a multimodal foundation model with both verbal intelligence and spatial-temporal intelligence, trained on diverse datasets to perform agentic tasks like UI navigation and robotic manipulation, outperforming specialized models.
vision-language models spatial-temporal intelligence actionable visual objects Set-of-Mark Trace-of-Mark action grounding action planning spatial-temporal intelligence UI navigation robotic manipulation