MOSAIC: A Modular System for Assistive and Interactive Cooking
We present MOSAIC, a modular architecture for home robots to perform complex
collaborative tasks, such as cooking with everyday users. MOSAIC tightly
collaborates with humans, interacts with users using natural language,
coordinates multiple robots, and manages an open vocabulary of everyday
objects. At its core, MOSAIC employs modularity: it leverages multiple
large-scale pre-trained models for general tasks like language and image
recognition, while using streamlined modules designed for task-specific
control. We extensively evaluate MOSAIC on 60 end-to-end trials where two
robots collaborate with a human user to cook a combination of 6 recipes. We
also extensively test individual modules with 180 episodes of visuomotor
picking, 60 episodes of human motion forecasting, and 46 online user
evaluations of the task planner. We show that MOSAIC is able to efficiently
collaborate with humans by running the overall system end-to-end with a real
human user, completing 68.3% (41/60) collaborative cooking trials of 6
different recipes with a subtask completion rate of 91.6%. Finally, we discuss
the limitations of the current system and exciting open challenges in this
domain. The project's website is at https://portal-cornell.github.io/MOSAIC/
65e16f65257de6e24f14f956
Huaxiaoyue Wang
claimed_verified
2024-03-02T19:13:31.973Z
64fc78067021123688e2c2f2
/avatars/e7007c6136a60d7073e7ccdf7fbb9faa.svg
Yuki Wang
user
LunaY0Yuki
65e16f65257de6e24f14f957
Kushal Kedia
claimed_verified
2024-03-02T19:13:40.773Z
65e1ff68ec41b21de6201cbd
/avatars/498f8d0b5d83c18e9aefdf730f5be170.svg
Kushal Kedia
user
Kushal20
65e16f65257de6e24f14f958
Juntao Ren
claimed_verified
2024-03-02T19:13:42.777Z
6490f24f985cd48c3e21ca77
/avatars/1586c514dc9b4653f9b9d6e84496c75f.svg
Juntao Ren
user
jren123
65e16f65257de6e24f14f959
Rahma Abdullah
65e16f65257de6e24f14f95a
Atiksh Bhardwaj
65e16f65257de6e24f14f95b
Angela Chao
claimed_verified
2024-03-04T09:00:14.690Z
65e3d60d00680b19bc015b0a
/avatars/f0d8e56a8181f0a59439ebe961ef5b06.svg
Angela Chao
user
ac2323
65e16f65257de6e24f14f95c
Kelly Y Chen
65e16f65257de6e24f14f95d
Nathaniel Chin
admin_assigned
2024-03-01T17:33:00.244Z
65e20cd74dbf9514fbe52560
/avatars/f6bb14123be6cda2136f2575f4576a3c.svg
Nathaniel Chin
user
nlauchin
65e16f65257de6e24f14f95e
Prithwish Dan
claimed_verified
2024-03-07T09:16:00.389Z
65e8c6cae214f37d85e341c0
https://cdn-avatars.huggingface.co/v1/production/uploads/65e8c6cae214f37d85e341c0/CtpYeBEqap9TwomncZI8V.png
Prithwish Dan
user
prithwishdan
65e16f65257de6e24f14f95f
Xinyi Fan
65e16f65257de6e24f14f960
Gonzalo Gonzalez-Pumariega
claimed_verified
2024-03-02T19:13:24.926Z
6421d2972143035270db37b9
/avatars/4fadeafc273d32cf72fe2f12d444c5e8.svg
Gonzalo Gonzalez
user
chalo2000
65e16f65257de6e24f14f961
Aditya Kompella
65e16f65257de6e24f14f962
Maximus Adrian Pace
claimed_verified
2024-03-02T19:13:38.683Z
65e20780ec41b21de623e65c
/avatars/5aaa4540eca3cbddc0a956d8dea2b27c.svg
Max Pace
user
map438
65e16f65257de6e24f14f963
Yash Sharma
claimed_verified
2024-03-02T19:13:36.672Z
63882d3429851e77efd07536
/avatars/1534f4d828ba953c16c36c72d45d844f.svg
Yash Sharma
user
yashs20
65e16f65257de6e24f14f964
Xiangwan Sun
claimed_verified
2024-03-02T19:13:27.612Z
65e2112e3e48a43bd73b4e9f
/avatars/df2aab5ee33ea65ec8f80f4da3566050.svg
Xiangwan Sun
user
XiangwanSun
65e16f65257de6e24f14f965
Neha Sunkara
claimed_verified
2024-03-02T19:13:34.701Z
65e20cf4edc2f7306eb31909
/avatars/67b9abd0441baa83429662f64624e0dc.svg
Neha Sunkara
user
nehasunkara
65e16f65257de6e24f14f966
Sanjiban Choudhury
admin_assigned
2024-03-01T12:50:13.281Z
652b04fbf1205983227accfe
https://cdn-avatars.huggingface.co/v1/production/uploads/652b04fbf1205983227accfe/RrISowuG-5ELLolqkye0i.jpeg
Sanjiban Choudhury
user
sc2582
MOSAIC: A Modular System for Assistive and Interactive Cooking
We present MOSAIC, a modular architecture for home robots to perform complex
collaborative tasks, such as cooking with everyday users. MOSAIC tightly
collaborates with humans, interacts with users using natural language,
coordinates multiple robots, and manages an open vocabulary of everyday
objects. At its core, MOSAIC employs modularity: it leverages multiple
large-scale pre-trained models for general tasks like language and image
recognition, while using streamlined modules designed for task-specific
control. We extensively evaluate MOSAIC on 60 end-to-end trials where two
robots collaborate with a human user to cook a combination of 6 recipes. We
also extensively test individual modules with 180 episodes of visuomotor
picking, 60 episodes of human motion forecasting, and 46 online user
evaluations of the task planner. We show that MOSAIC is able to efficiently
collaborate with humans by running the overall system end-to-end with a real
human user, completing 68.3% (41/60) collaborative cooking trials of 6
different recipes with a subtask completion rate of 91.6%. Finally, we discuss
the limitations of the current system and exciting open challenges in this
domain. The project's website is at https://portal-cornell.github.io/MOSAIC/
MOSAIC, a modular home robot architecture, collaborates with humans to perform complex cooking tasks using pre-trained models and specific modules, achieving 68.3% success in 60 collaborative trials with various recipes.
modular architecture home robots complex collaborative tasks natural language language recognition image recognition task-specific control end-to-end trials visuomotor picking human motion forecasting task planner subtask completion rate