title: IGOR: Image-GOal Representations are the Atomic Control Units for
    Foundation Models in Embodied AI
summary: We introduce Image-GOal Representations (IGOR), aiming to learn a unified,
  semantically consistent action space across human and various robots. Through
  this unified latent action space, IGOR enables knowledge transfer among
  large-scale robot and human activity data. We achieve this by compressing
  visual changes between an initial image and its goal state into latent actions.
  IGOR allows us to generate latent action labels for internet-scale video data.
  This unified latent action space enables the training of foundation policy and
  world models across a wide variety of tasks performed by both robots and
  humans. We demonstrate that: (1) IGOR learns a semantically consistent action
  space for both human and robots, characterizing various possible motions of
  objects representing the physical interaction knowledge; (2) IGOR can "migrate"
  the movements of the object in the one video to other videos, even across human
  and robots, by jointly using the latent action model and world model; (3) IGOR
  can learn to align latent actions with natural language through the foundation
  policy model, and integrate latent actions with a low-level policy model to
  achieve effective robot control. We believe IGOR opens new possibilities for
  human-to-robot knowledge transfer and control.
paper_authors._id: 6729a6429c7db29615bae05a
paper_authors.name: Xiaoyu Chen
paper_authors._id: 6729a6429c7db29615bae05b
paper_authors.name: Junliang Guo
paper_authors._id: 6729a6429c7db29615bae05c
paper_authors.name: Tianyu He
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-12-09T14:48:11.819Z
paper_authors._id: 619b7b1cab4c7b7f16a7d59e
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/619b7b1cab4c7b7f16a7d59e/6TvXaAqBghAMYO1-j5l4v.jpeg
paper_authors.fullname: Tianyu He
paper_authors.type: user
paper_authors.user: deeptimhe
paper_authors._id: 6729a6429c7db29615bae05d
paper_authors.name: Chuheng Zhang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-11-15T09:38:14.572Z
paper_authors._id: 64c32727c370f29a10334d35
paper_authors.avatarUrl: /avatars/a3284bdd2e51f437433b89a96a904448.svg
paper_authors.fullname: ZhangChuheng
paper_authors.type: user
paper_authors.user: zhangchuheng123
paper_authors._id: 6729a6429c7db29615bae05e
paper_authors.name: Pushi Zhang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-11-05T07:57:21.751Z
paper_authors._id: 62a4ab8106b784ece3860b88
paper_authors.avatarUrl: /avatars/042be18e0586984e73fa7c9c0a124ed9.svg
paper_authors.fullname: Pushi
paper_authors.type: user
paper_authors.user: zpschang
paper_authors._id: 6729a6429c7db29615bae05f
paper_authors.name: Derek Cathera Yang
paper_authors._id: 6729a6429c7db29615bae060
paper_authors.name: Li Zhao
paper_authors._id: 6729a6429c7db29615bae061
paper_authors.name: Jiang Bian
paper_title: IGOR: Image-GOal Representations are the Atomic Control Units for
    Foundation Models in Embodied AI
paper_summary: We introduce Image-GOal Representations (IGOR), aiming to learn a unified,
  semantically consistent action space across human and various robots. Through
  this unified latent action space, IGOR enables knowledge transfer among
  large-scale robot and human activity data. We achieve this by compressing
  visual changes between an initial image and its goal state into latent actions.
  IGOR allows us to generate latent action labels for internet-scale video data.
  This unified latent action space enables the training of foundation policy and
  world models across a wide variety of tasks performed by both robots and
  humans. We demonstrate that: (1) IGOR learns a semantically consistent action
  space for both human and robots, characterizing various possible motions of
  objects representing the physical interaction knowledge; (2) IGOR can "migrate"
  the movements of the object in the one video to other videos, even across human
  and robots, by jointly using the latent action model and world model; (3) IGOR
  can learn to align latent actions with natural language through the foundation
  policy model, and integrate latent actions with a low-level policy model to
  achieve effective robot control. We believe IGOR opens new possibilities for
  human-to-robot knowledge transfer and control.
paper_ai_keywords: Image-GOal Representations (IGOR) Unified latent action space Knowledge transfer Latent actions Internet-scale video data Foundation policy World models Semantic consistency Physical interaction knowledge Motion characterization Object migration Natural language alignment Low-level policy model Robot control Human-to-robot knowledge transfer