title: Alexa, play with robot: Introducing the First Alexa Prize SimBot
    Challenge on Embodied AI
summary: The Alexa Prize program has empowered numerous university students to
  explore, experiment, and showcase their talents in building conversational
  agents through challenges like the SocialBot Grand Challenge and the TaskBot
  Challenge. As conversational agents increasingly appear in multimodal and
  embodied contexts, it is important to explore the affordances of conversational
  interaction augmented with computer vision and physical embodiment. This paper
  describes the SimBot Challenge, a new challenge in which university teams
  compete to build robot assistants that complete tasks in a simulated physical
  environment. This paper provides an overview of the SimBot Challenge, which
  included both online and offline challenge phases. We describe the
  infrastructure and support provided to the teams including Alexa Arena, the
  simulated environment, and the ML toolkit provided to teams to accelerate their
  building of vision and language models. We summarize the approaches the
  participating teams took to overcome research challenges and extract key
  lessons learned. Finally, we provide analysis of the performance of the
  competing SimBots during the competition.
paper_authors._id: 64d596d0d9d00dcb114e7472
paper_authors.name: Hangjie Shi
paper_authors._id: 64d596d0d9d00dcb114e7473
paper_authors.name: Leslie Ball
paper_authors._id: 64d596d0d9d00dcb114e7474
paper_authors.name: Govind Thattai
paper_authors._id: 64d596d0d9d00dcb114e7475
paper_authors.name: Desheng Zhang
paper_authors._id: 64d596d0d9d00dcb114e7476
paper_authors.name: Lucy Hu
paper_authors._id: 64d596d0d9d00dcb114e7477
paper_authors.name: Qiaozi Gao
paper_authors._id: 64d596d0d9d00dcb114e7478
paper_authors.name: Suhaila Shakiah
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-08-11T13:31:22.507Z
paper_authors._id: 645d7db5ce72244df7b3d939
paper_authors.avatarUrl: /avatars/9fd48ebd3233bdf336664600deff3ea5.svg
paper_authors.fullname: Suhaila Shakiah
paper_authors.type: user
paper_authors.user: suhailashakiah
paper_authors._id: 64d596d0d9d00dcb114e7479
paper_authors.name: Xiaofeng Gao
paper_authors._id: 64d596d0d9d00dcb114e747a
paper_authors.name: Aishwarya Padmakumar
paper_authors._id: 64d596d0d9d00dcb114e747b
paper_authors.name: Bofei Yang
paper_authors._id: 64d596d0d9d00dcb114e747c
paper_authors.name: Cadence Chung
paper_authors._id: 64d596d0d9d00dcb114e747d
paper_authors.name: Dinakar Guthy
paper_authors._id: 64d596d0d9d00dcb114e747e
paper_authors.name: Gaurav Sukhatme
paper_authors._id: 64d596d0d9d00dcb114e747f
paper_authors.name: Karthika Arumugam
paper_authors._id: 64d596d0d9d00dcb114e7480
paper_authors.name: Matthew Wen
paper_authors._id: 64d596d0d9d00dcb114e7481
paper_authors.name: Osman Ipek
paper_authors._id: 64d596d0d9d00dcb114e7482
paper_authors.name: Patrick Lange
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-08-11T13:34:08.473Z
paper_authors._id: 640f53dd208821a59b740bae
paper_authors.avatarUrl: /avatars/460de2ab74dddfbc1502ff87acae314b.svg
paper_authors.fullname: Patrick Lange
paper_authors.type: user
paper_authors.user: langep
paper_authors._id: 64d596d0d9d00dcb114e7483
paper_authors.name: Rohan Khanna
paper_authors._id: 64d596d0d9d00dcb114e7484
paper_authors.name: Shreyas Pansare
paper_authors._id: 64d596d0d9d00dcb114e7485
paper_authors.name: Vasu Sharma
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-01-10T08:22:24.768Z
paper_authors._id: 65492730a3d1682f79f3ab7a
paper_authors.avatarUrl: /avatars/fc62605bdf10947dab393305c51ace96.svg
paper_authors.fullname: Vasu Sharma
paper_authors.type: user
paper_authors.user: vasusharma55
paper_authors._id: 64d596d0d9d00dcb114e7486
paper_authors.name: Chao Zhang
paper_authors._id: 64d596d0d9d00dcb114e7487
paper_authors.name: Cris Flagg
paper_authors._id: 64d596d0d9d00dcb114e7488
paper_authors.name: Daniel Pressel
paper_authors._id: 64d596d0d9d00dcb114e7489
paper_authors.name: Lavina Vaz
paper_authors._id: 64d596d0d9d00dcb114e748a
paper_authors.name: Luke Dai
paper_authors._id: 64d596d0d9d00dcb114e748b
paper_authors.name: Prasoon Goyal
paper_authors._id: 64d596d0d9d00dcb114e748c
paper_authors.name: Sattvik Sahai
paper_authors._id: 64d596d0d9d00dcb114e748d
paper_authors.name: Shaohua Liu
paper_authors._id: 64d596d0d9d00dcb114e748e
paper_authors.name: Yao Lu
paper_authors._id: 64d596d0d9d00dcb114e748f
paper_authors.name: Anna Gottardi
paper_authors._id: 64d596d0d9d00dcb114e7490
paper_authors.name: Shui Hu
paper_authors._id: 64d596d0d9d00dcb114e7491
paper_authors.name: Yang Liu
paper_authors._id: 64d596d0d9d00dcb114e7492
paper_authors.name: Dilek Hakkani-Tur
paper_authors._id: 64d596d0d9d00dcb114e7493
paper_authors.name: Kate Bland
paper_authors._id: 64d596d0d9d00dcb114e7494
paper_authors.name: Heather Rocker
paper_authors._id: 64d596d0d9d00dcb114e7495
paper_authors.name: James Jeun
paper_authors._id: 64d596d0d9d00dcb114e7496
paper_authors.name: Yadunandana Rao
paper_authors._id: 64d596d0d9d00dcb114e7497
paper_authors.name: Michael Johnston
paper_authors._id: 64d596d0d9d00dcb114e7498
paper_authors.name: Akshaya Iyengar
paper_authors._id: 64d596d0d9d00dcb114e7499
paper_authors.name: Arindam Mandal
paper_authors._id: 64d596d0d9d00dcb114e749a
paper_authors.name: Prem Natarajan
paper_authors._id: 64d596d0d9d00dcb114e749b
paper_authors.name: Reza Ghanadan
paper_title: Alexa, play with robot: Introducing the First Alexa Prize SimBot
    Challenge on Embodied AI
paper_summary: The Alexa Prize program has empowered numerous university students to
  explore, experiment, and showcase their talents in building conversational
  agents through challenges like the SocialBot Grand Challenge and the TaskBot
  Challenge. As conversational agents increasingly appear in multimodal and
  embodied contexts, it is important to explore the affordances of conversational
  interaction augmented with computer vision and physical embodiment. This paper
  describes the SimBot Challenge, a new challenge in which university teams
  compete to build robot assistants that complete tasks in a simulated physical
  environment. This paper provides an overview of the SimBot Challenge, which
  included both online and offline challenge phases. We describe the
  infrastructure and support provided to the teams including Alexa Arena, the
  simulated environment, and the ML toolkit provided to teams to accelerate their
  building of vision and language models. We summarize the approaches the
  participating teams took to overcome research challenges and extract key
  lessons learned. Finally, we provide analysis of the performance of the
  competing SimBots during the competition.
paper_ai_keywords: conversational agents computer vision physical embodiment SimBot Challenge Alexa Arena simulated environment ML toolkit vision and language models SimBots