Theia: Distilling Diverse Vision Foundation Models for Robot Learning
Vision-based robot policy learning, which maps visual inputs to actions,
necessitates a holistic understanding of diverse visual tasks beyond
single-task needs like classification or segmentation. Inspired by this, we
introduce Theia, a vision foundation model for robot learning that distills
multiple off-the-shelf vision foundation models trained on varied vision tasks.
Theia's rich visual representations encode diverse visual knowledge, enhancing
downstream robot learning. Extensive experiments demonstrate that Theia
outperforms its teacher models and prior robot learning models using less
training data and smaller model sizes. Additionally, we quantify the quality of
pre-trained visual representations and hypothesize that higher entropy in
feature norm distributions leads to improved robot learning performance. Code
and models are available at https://github.com/bdaiinstitute/theia.
66a8542b26f6962f854fd356
Jinghuan Shang
admin_assigned
2024-07-30T08:15:44.374Z
63f686c49cbd6730302cf0f2
/avatars/b553823be640a07eeb4c7edc2c176d5d.svg
Jinghuan Shang
user
Jinghuan
66a8542b26f6962f854fd357
Karl Schmeckpeper
admin_assigned
2024-07-30T08:15:49.919Z
662c012cf5f38dec89bf8ec1
/avatars/672c96fbf25225be51cd023e8be8e1f9.svg
Karl Schmeckpeper
user
kschmeckpeper
66a8542b26f6962f854fd358
Brandon B. May
admin_assigned
2024-07-30T08:16:27.428Z
666c76ad3ae28e1bcfe02c16
https://cdn-avatars.huggingface.co/v1/production/uploads/666c76ad3ae28e1bcfe02c16/GG9ImF8QF8BC__7ofbaxO.jpeg
Brandon B. May
user
bmay
66a8542b26f6962f854fd359
Maria Vittoria Minniti
admin_assigned
2024-07-30T08:16:33.590Z
665a13ce042dcc43ad3cefec
/avatars/28a1c0948431eac2e166dcc88468ffd8.svg
Maria Vittoria Minniti
user
mminniti-bdai
66a8542b26f6962f854fd35a
Tarik Kelestemur
admin_assigned
2024-07-30T08:16:38.905Z
63bb24c9df5897db7f05047f
/avatars/80cfd3dc400f9e2dccd275dc54d3e92b.svg
Tarik Kelestemur
user
kelestemur
66a8542b26f6962f854fd35b
David Watkins
claimed_verified
2024-07-30T11:38:47.253Z
66a8cc342c7c3ebdd77652f7
https://cdn-avatars.huggingface.co/v1/production/uploads/66a8cc342c7c3ebdd77652f7/K9gKXX2ciBcSZOm0IPXiq.jpeg
David Watkins
user
davidwatkins
66a8542b26f6962f854fd35c
Laura Herlant
admin_assigned
2024-07-30T08:17:08.638Z
66a10bba8645e7fa2629bde6
https://cdn-avatars.huggingface.co/v1/production/uploads/66a10bba8645e7fa2629bde6/m5ZXO3u1IPABiGCqhABVp.jpeg
Laura Herlant
user
lherlant
Theia: Distilling Diverse Vision Foundation Models for Robot Learning
Vision-based robot policy learning, which maps visual inputs to actions,
necessitates a holistic understanding of diverse visual tasks beyond
single-task needs like classification or segmentation. Inspired by this, we
introduce Theia, a vision foundation model for robot learning that distills
multiple off-the-shelf vision foundation models trained on varied vision tasks.
Theia's rich visual representations encode diverse visual knowledge, enhancing
downstream robot learning. Extensive experiments demonstrate that Theia
outperforms its teacher models and prior robot learning models using less
training data and smaller model sizes. Additionally, we quantify the quality of
pre-trained visual representations and hypothesize that higher entropy in
feature norm distributions leads to improved robot learning performance. Code
and models are available at https://github.com/bdaiinstitute/theia.
Theia, a vision foundation model for robot learning, outperforms existing models with less data and smaller sizes by distilling knowledge from multiple vision foundation models.
vision foundation model robot policy learning visual representations off-the-shelf models feature norm distributions entropy