title: An Interactive Agent Foundation Model
summary: The development of artificial intelligence systems is transitioning from
  creating static, task-specific models to dynamic, agent-based systems capable
  of performing well in a wide range of applications. We propose an Interactive
  Agent Foundation Model that uses a novel multi-task agent training paradigm for
  training AI agents across a wide range of domains, datasets, and tasks. Our
  training paradigm unifies diverse pre-training strategies, including visual
  masked auto-encoders, language modeling, and next-action prediction, enabling a
  versatile and adaptable AI framework. We demonstrate the performance of our
  framework across three separate domains -- Robotics, Gaming AI, and Healthcare.
  Our model demonstrates its ability to generate meaningful and contextually
  relevant outputs in each area. The strength of our approach lies in its
  generality, leveraging a variety of data sources such as robotics sequences,
  gameplay data, large-scale video datasets, and textual information for
  effective multimodal and multi-task learning. Our approach provides a promising
  avenue for developing generalist, action-taking, multimodal systems.
paper_authors._id: 65c59916e286dbda4e966b55
paper_authors.name: Zane Durante
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-02-09T10:10:22.306Z
paper_authors._id: 650259fda33ce3ae28620bf3
paper_authors.avatarUrl: /avatars/5fb455b2e3a9523f07230d81d1947e15.svg
paper_authors.fullname: Zane Durante
paper_authors.type: user
paper_authors.user: ZaneDurante
paper_authors._id: 65c59916e286dbda4e966b56
paper_authors.name: Bidipta Sarkar
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-02-09T08:05:04.783Z
paper_authors._id: 63abbf74ad514ca8d14a0548
paper_authors.avatarUrl: /avatars/b1357b73b8f9a8ff9908710ad64154ef.svg
paper_authors.fullname: Bidipta Sarkar
paper_authors.type: user
paper_authors.user: bidiptas
paper_authors._id: 65c59916e286dbda4e966b57
paper_authors.name: Ran Gong
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-02-09T10:10:28.674Z
paper_authors._id: 64acdaf75c6fc404f0d95e5b
paper_authors.avatarUrl: /avatars/a4235fb6984e5945eb0e11c3ce581b59.svg
paper_authors.fullname: Ran Gong
paper_authors.type: user
paper_authors.user: nikepupu
paper_authors._id: 65c59916e286dbda4e966b58
paper_authors.name: Rohan Taori
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-02-09T10:10:35.170Z
paper_authors._id: 622fca0d8d46dde5d605f83e
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1647489114181-622fca0d8d46dde5d605f83e.jpeg
paper_authors.fullname: Rohan Taori
paper_authors.type: user
paper_authors.user: rtaori
paper_authors._id: 65c59916e286dbda4e966b59
paper_authors.name: Yusuke Noda
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-02-09T10:10:49.522Z
paper_authors._id: 6430f78767688e5e12a408e4
paper_authors.avatarUrl: /avatars/a7a10d30123495a3eb87d944ae9f2b55.svg
paper_authors.fullname: Yusuke Noda
paper_authors.type: user
paper_authors.user: yusukenoda
paper_authors._id: 65c59916e286dbda4e966b5a
paper_authors.name: Paul Tang
paper_authors._id: 65c59916e286dbda4e966b5b
paper_authors.name: Ehsan Adeli
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-02-12T08:39:55.429Z
paper_authors._id: 65c673e773a47fa85dfc7ca3
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/4e1JO03Gh2l5s-8DW8MV2.jpeg
paper_authors.fullname: Ehsan
paper_authors.type: user
paper_authors.user: eadeli
paper_authors._id: 65c59916e286dbda4e966b5c
paper_authors.name: Shrinidhi Kowshika Lakshmikanth
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-02-12T08:39:53.443Z
paper_authors._id: 6456a0f04cfac49227918439
paper_authors.avatarUrl: /avatars/9fa5191d1e9897f68ea2d21978b19cca.svg
paper_authors.fullname: Shrinidhi KL
paper_authors.type: user
paper_authors.user: klshrinidhi
paper_authors._id: 65c59916e286dbda4e966b5d
paper_authors.name: Kevin Schulman
paper_authors._id: 65c59916e286dbda4e966b5e
paper_authors.name: Arnold Milstein
paper_authors._id: 65c59916e286dbda4e966b5f
paper_authors.name: Demetri Terzopoulos
paper_authors._id: 65c59916e286dbda4e966b60
paper_authors.name: Ade Famoti
paper_authors._id: 65c59916e286dbda4e966b61
paper_authors.name: Noboru Kuno
paper_authors._id: 65c59916e286dbda4e966b62
paper_authors.name: Ashley Llorens
paper_authors._id: 65c59916e286dbda4e966b63
paper_authors.name: Hoi Vo
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-02-09T10:16:06.984Z
paper_authors._id: 631aae9bfac58c9c816636e1
paper_authors.avatarUrl: /avatars/92d030f9eb552a334aa129e02d3f2ca6.svg
paper_authors.fullname: Hoi Vo
paper_authors.type: user
paper_authors.user: Hoivo63
paper_authors._id: 65c59916e286dbda4e966b64
paper_authors.name: Katsu Ikeuchi
paper_authors._id: 65c59916e286dbda4e966b65
paper_authors.name: Li Fei-Fei
paper_authors._id: 65c59916e286dbda4e966b66
paper_authors.name: Jianfeng Gao
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-02-09T10:16:26.696Z
paper_authors._id: 641904caf9d6f1d772ec7af7
paper_authors.avatarUrl: /avatars/4a63eac71eb30f70b1a0e9d4708f26c1.svg
paper_authors.fullname: Jianfeng Gao
paper_authors.type: user
paper_authors.user: wyngjf
paper_authors._id: 65c59916e286dbda4e966b67
paper_authors.name: Naoki Wake
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-02-09T10:16:33.608Z
paper_authors._id: 650bd16424da30e773ec74ca
paper_authors.avatarUrl: /avatars/1ff8a1c9d4c6bde1893e4f80c5fd9fdb.svg
paper_authors.fullname: Wake
paper_authors.type: user
paper_authors.user: NaokiWake
paper_authors._id: 65c59916e286dbda4e966b68
paper_authors.name: Qiuyuan Huang
paper_title: An Interactive Agent Foundation Model
paper_summary: The development of artificial intelligence systems is transitioning from
  creating static, task-specific models to dynamic, agent-based systems capable
  of performing well in a wide range of applications. We propose an Interactive
  Agent Foundation Model that uses a novel multi-task agent training paradigm for
  training AI agents across a wide range of domains, datasets, and tasks. Our
  training paradigm unifies diverse pre-training strategies, including visual
  masked auto-encoders, language modeling, and next-action prediction, enabling a
  versatile and adaptable AI framework. We demonstrate the performance of our
  framework across three separate domains -- Robotics, Gaming AI, and Healthcare.
  Our model demonstrates its ability to generate meaningful and contextually
  relevant outputs in each area. The strength of our approach lies in its
  generality, leveraging a variety of data sources such as robotics sequences,
  gameplay data, large-scale video datasets, and textual information for
  effective multimodal and multi-task learning. Our approach provides a promising
  avenue for developing generalist, action-taking, multimodal systems.
paper_ai_keywords: Interactive Agent Foundation Model multi-task agents training paradigm visual masked auto-encoders language modeling next-action prediction multimodal multi-task learning generality multimodal systems