IGOR: Image-GOal Representations are the Atomic Control Units for
  Foundation Models in Embodied AI
We introduce Image-GOal Representations (IGOR), aiming to learn a unified,
semantically consistent action space across human and various robots. Through
this unified latent action space, IGOR enables knowledge transfer among
large-scale robot and human activity data. We achieve this by compressing
visual changes between an initial image and its goal state into latent actions.
IGOR allows us to generate latent action labels for internet-scale video data.
This unified latent action space enables the training of foundation policy and
world models across a wide variety of tasks performed by both robots and
humans. We demonstrate that: (1) IGOR learns a semantically consistent action
space for both human and robots, characterizing various possible motions of
objects representing the physical interaction knowledge; (2) IGOR can "migrate"
the movements of the object in the one video to other videos, even across human
and robots, by jointly using the latent action model and world model; (3) IGOR
can learn to align latent actions with natural language through the foundation
policy model, and integrate latent actions with a low-level policy model to
achieve effective robot control. We believe IGOR opens new possibilities for
human-to-robot knowledge transfer and control.
6729a6429c7db29615bae05a
Xiaoyu Chen
6729a6429c7db29615bae05b
Junliang Guo
6729a6429c7db29615bae05c
Tianyu He
claimed_verified
2024-12-09T14:48:11.819Z
619b7b1cab4c7b7f16a7d59e
https://cdn-avatars.huggingface.co/v1/production/uploads/619b7b1cab4c7b7f16a7d59e/6TvXaAqBghAMYO1-j5l4v.jpeg
Tianyu He
user
deeptimhe
6729a6429c7db29615bae05d
Chuheng Zhang
claimed_verified
2024-11-15T09:38:14.572Z
64c32727c370f29a10334d35
/avatars/a3284bdd2e51f437433b89a96a904448.svg
ZhangChuheng
user
zhangchuheng123
6729a6429c7db29615bae05e
Pushi Zhang
claimed_verified
2024-11-05T07:57:21.751Z
62a4ab8106b784ece3860b88
/avatars/042be18e0586984e73fa7c9c0a124ed9.svg
Pushi
user
zpschang
6729a6429c7db29615bae05f
Derek Cathera Yang
6729a6429c7db29615bae060
Li Zhao
extracted_pending
2025-08-01T03:31:14.613Z
673ad4944271de2b747bdb28
/avatars/eb5d420fd11990632547cdc686cf0970.svg
Li Zhao
user
lizo1113
6729a6429c7db29615bae061
Jiang Bian
claimed_verified
2025-06-19T09:11:51.553Z
683843c8a9bdf98f92657042
/avatars/4983876f615fa830cfc8f2cb5a00fa08.svg
Jiang Bian
user
jiangbian22
IGOR: Image-GOal Representations are the Atomic Control Units for
  Foundation Models in Embodied AI
We introduce Image-GOal Representations (IGOR), aiming to learn a unified,
semantically consistent action space across human and various robots. Through
this unified latent action space, IGOR enables knowledge transfer among
large-scale robot and human activity data. We achieve this by compressing
visual changes between an initial image and its goal state into latent actions.
IGOR allows us to generate latent action labels for internet-scale video data.
This unified latent action space enables the training of foundation policy and
world models across a wide variety of tasks performed by both robots and
humans. We demonstrate that: (1) IGOR learns a semantically consistent action
space for both human and robots, characterizing various possible motions of
objects representing the physical interaction knowledge; (2) IGOR can "migrate"
the movements of the object in the one video to other videos, even across human
and robots, by jointly using the latent action model and world model; (3) IGOR
can learn to align latent actions with natural language through the foundation
policy model, and integrate latent actions with a low-level policy model to
achieve effective robot control. We believe IGOR opens new possibilities for
human-to-robot knowledge transfer and control.
Image-GOal Representations (IGOR) unify human and robotic action spaces through latent action labels, enabling cross-domain knowledge transfer and control.
Image-GOal Representations IGOR latent action space action labels knowledge transfer world models foundation policy semantically consistent action space latent action model low-level policy model human-to-robot knowledge transfer