title: MOSAIC: A Modular System for Assistive and Interactive Cooking
summary: We present MOSAIC, a modular architecture for home robots to perform complex
  collaborative tasks, such as cooking with everyday users. MOSAIC tightly
  collaborates with humans, interacts with users using natural language,
  coordinates multiple robots, and manages an open vocabulary of everyday
  objects. At its core, MOSAIC employs modularity: it leverages multiple
  large-scale pre-trained models for general tasks like language and image
  recognition, while using streamlined modules designed for task-specific
  control. We extensively evaluate MOSAIC on 60 end-to-end trials where two
  robots collaborate with a human user to cook a combination of 6 recipes. We
  also extensively test individual modules with 180 episodes of visuomotor
  picking, 60 episodes of human motion forecasting, and 46 online user
  evaluations of the task planner. We show that MOSAIC is able to efficiently
  collaborate with humans by running the overall system end-to-end with a real
  human user, completing 68.3% (41/60) collaborative cooking trials of 6
  different recipes with a subtask completion rate of 91.6%. Finally, we discuss
  the limitations of the current system and exciting open challenges in this
  domain. The project's website is at https://portal-cornell.github.io/MOSAIC/
paper_authors._id: 65e16f65257de6e24f14f956
paper_authors.name: Huaxiaoyue Wang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-03-02T19:13:31.973Z
paper_authors._id: 64fc78067021123688e2c2f2
paper_authors.avatarUrl: /avatars/e7007c6136a60d7073e7ccdf7fbb9faa.svg
paper_authors.fullname: Yuki Wang
paper_authors.type: user
paper_authors.user: LunaY0Yuki
paper_authors._id: 65e16f65257de6e24f14f957
paper_authors.name: Kushal Kedia
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-03-02T19:13:40.773Z
paper_authors._id: 65e1ff68ec41b21de6201cbd
paper_authors.avatarUrl: /avatars/498f8d0b5d83c18e9aefdf730f5be170.svg
paper_authors.fullname: Kushal Kedia
paper_authors.type: user
paper_authors.user: Kushal20
paper_authors._id: 65e16f65257de6e24f14f958
paper_authors.name: Juntao Ren
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-03-02T19:13:42.777Z
paper_authors._id: 6490f24f985cd48c3e21ca77
paper_authors.avatarUrl: /avatars/1586c514dc9b4653f9b9d6e84496c75f.svg
paper_authors.fullname: Juntao Ren
paper_authors.type: user
paper_authors.user: jren123
paper_authors._id: 65e16f65257de6e24f14f959
paper_authors.name: Rahma Abdullah
paper_authors._id: 65e16f65257de6e24f14f95a
paper_authors.name: Atiksh Bhardwaj
paper_authors._id: 65e16f65257de6e24f14f95b
paper_authors.name: Angela Chao
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-03-04T09:00:14.690Z
paper_authors._id: 65e3d60d00680b19bc015b0a
paper_authors.avatarUrl: /avatars/f0d8e56a8181f0a59439ebe961ef5b06.svg
paper_authors.fullname: Angela Chao
paper_authors.type: user
paper_authors.user: ac2323
paper_authors._id: 65e16f65257de6e24f14f95c
paper_authors.name: Kelly Y Chen
paper_authors._id: 65e16f65257de6e24f14f95d
paper_authors.name: Nathaniel Chin
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-03-01T17:33:00.244Z
paper_authors._id: 65e20cd74dbf9514fbe52560
paper_authors.avatarUrl: /avatars/f6bb14123be6cda2136f2575f4576a3c.svg
paper_authors.fullname: Nathaniel Chin
paper_authors.type: user
paper_authors.user: nlauchin
paper_authors._id: 65e16f65257de6e24f14f95e
paper_authors.name: Prithwish Dan
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-03-07T09:16:00.389Z
paper_authors._id: 65e8c6cae214f37d85e341c0
paper_authors.avatarUrl: /avatars/1c1663ed4725dddeeac50f7ad955a61d.svg
paper_authors.fullname: Prithwish Dan
paper_authors.type: user
paper_authors.user: prithwishdan
paper_authors._id: 65e16f65257de6e24f14f95f
paper_authors.name: Xinyi Fan
paper_authors._id: 65e16f65257de6e24f14f960
paper_authors.name: Gonzalo Gonzalez-Pumariega
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-03-02T19:13:24.926Z
paper_authors._id: 6421d2972143035270db37b9
paper_authors.avatarUrl: /avatars/4fadeafc273d32cf72fe2f12d444c5e8.svg
paper_authors.fullname: Gonzalo Gonzalez
paper_authors.type: user
paper_authors.user: chalo2000
paper_authors._id: 65e16f65257de6e24f14f961
paper_authors.name: Aditya Kompella
paper_authors._id: 65e16f65257de6e24f14f962
paper_authors.name: Maximus Adrian Pace
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-03-02T19:13:38.683Z
paper_authors._id: 65e20780ec41b21de623e65c
paper_authors.avatarUrl: /avatars/5aaa4540eca3cbddc0a956d8dea2b27c.svg
paper_authors.fullname: Max Pace
paper_authors.type: user
paper_authors.user: map438
paper_authors._id: 65e16f65257de6e24f14f963
paper_authors.name: Yash Sharma
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-03-02T19:13:36.672Z
paper_authors._id: 63882d3429851e77efd07536
paper_authors.avatarUrl: /avatars/1534f4d828ba953c16c36c72d45d844f.svg
paper_authors.fullname: Yash Sharma
paper_authors.type: user
paper_authors.user: yashs20
paper_authors._id: 65e16f65257de6e24f14f964
paper_authors.name: Xiangwan Sun
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-03-02T19:13:27.612Z
paper_authors._id: 65e2112e3e48a43bd73b4e9f
paper_authors.avatarUrl: /avatars/df2aab5ee33ea65ec8f80f4da3566050.svg
paper_authors.fullname: Xiangwan Sun
paper_authors.type: user
paper_authors.user: XiangwanSun
paper_authors._id: 65e16f65257de6e24f14f965
paper_authors.name: Neha Sunkara
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-03-02T19:13:34.701Z
paper_authors._id: 65e20cf4edc2f7306eb31909
paper_authors.avatarUrl: /avatars/67b9abd0441baa83429662f64624e0dc.svg
paper_authors.fullname: Neha Sunkara
paper_authors.type: user
paper_authors.user: nehasunkara
paper_authors._id: 65e16f65257de6e24f14f966
paper_authors.name: Sanjiban Choudhury
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-03-01T12:50:13.281Z
paper_authors._id: 652b04fbf1205983227accfe
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/652b04fbf1205983227accfe/RrISowuG-5ELLolqkye0i.jpeg
paper_authors.fullname: Sanjiban Choudhury
paper_authors.type: user
paper_authors.user: sc2582
paper_title: MOSAIC: A Modular System for Assistive and Interactive Cooking
paper_summary: We present MOSAIC, a modular architecture for home robots to perform complex
  collaborative tasks, such as cooking with everyday users. MOSAIC tightly
  collaborates with humans, interacts with users using natural language,
  coordinates multiple robots, and manages an open vocabulary of everyday
  objects. At its core, MOSAIC employs modularity: it leverages multiple
  large-scale pre-trained models for general tasks like language and image
  recognition, while using streamlined modules designed for task-specific
  control. We extensively evaluate MOSAIC on 60 end-to-end trials where two
  robots collaborate with a human user to cook a combination of 6 recipes. We
  also extensively test individual modules with 180 episodes of visuomotor
  picking, 60 episodes of human motion forecasting, and 46 online user
  evaluations of the task planner. We show that MOSAIC is able to efficiently
  collaborate with humans by running the overall system end-to-end with a real
  human user, completing 68.3% (41/60) collaborative cooking trials of 6
  different recipes with a subtask completion rate of 91.6%. Finally, we discuss
  the limitations of the current system and exciting open challenges in this
  domain. The project's website is at https://portal-cornell.github.io/MOSAIC/
paper_ai_keywords: natural language modularity large-scale pre-trained models language and image recognition task-specific control visuomotor picking human motion forecasting task planner collaborative cooking trials subtask completion rate