title: In-Context Imitation Learning via Next-Token Prediction
summary: We explore how to enhance next-token prediction models to perform in-context
  imitation learning on a real robot, where the robot executes new tasks by
  interpreting contextual information provided during the input phase, without
  updating its underlying policy parameters. We propose In-Context Robot
  Transformer (ICRT), a causal transformer that performs autoregressive
  prediction on sensorimotor trajectories without relying on any linguistic data
  or reward function. This formulation enables flexible and training-free
  execution of new tasks at test time, achieved by prompting the model with
  sensorimotor trajectories of the new task composing of image observations,
  actions and states tuples, collected through human teleoperation. Experiments
  with a Franka Emika robot demonstrate that the ICRT can adapt to new tasks
  specified by prompts, even in environment configurations that differ from both
  the prompt and the training data. In a multitask environment setup, ICRT
  significantly outperforms current state-of-the-art next-token prediction models
  in robotics on generalizing to unseen tasks. Code, checkpoints and data are
  available on https://icrt.dev/
paper_authors._id: 66cfe2810c8862832e552b36
paper_authors.name: Letian Fu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-08-29T07:53:10.487Z
paper_authors._id: 64908d6b0c18343a09361b93
paper_authors.avatarUrl: /avatars/2fa44344fe760f68f19de775b76f45b8.svg
paper_authors.fullname: Max (Letian) Fu
paper_authors.type: user
paper_authors.user: mlfu7
paper_authors._id: 66cfe2810c8862832e552b37
paper_authors.name: Huang Huang
paper_authors._id: 66cfe2810c8862832e552b38
paper_authors.name: Gaurav Datta
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-08-29T07:54:05.965Z
paper_authors._id: 65caf5f998a46a5626426bd1
paper_authors.avatarUrl: /avatars/636301ed1f3577b85787959651b93279.svg
paper_authors.fullname: Gaurav Datta
paper_authors.type: user
paper_authors.user: gdatta
paper_authors._id: 66cfe2810c8862832e552b39
paper_authors.name: Lawrence Yunliang Chen
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-08-29T07:54:46.288Z
paper_authors._id: 6683696648edefb453aa39f1
paper_authors.avatarUrl: /avatars/b056d8499420966049b40bb6a03a8782.svg
paper_authors.fullname: Lawrence Chen
paper_authors.type: user
paper_authors.user: yunliangchen
paper_authors._id: 66cfe2810c8862832e552b3a
paper_authors.name: William Chung-Ho Panitch
paper_authors._id: 66cfe2810c8862832e552b3b
paper_authors.name: Fangchen Liu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-08-29T07:55:11.431Z
paper_authors._id: 63609ef63605bd411c1c7509
paper_authors.avatarUrl: /avatars/3fadc3c4117cdf17d25f3efc545388cd.svg
paper_authors.fullname: Fangchen Liu
paper_authors.type: user
paper_authors.user: fangchenliu
paper_authors._id: 66cfe2810c8862832e552b3c
paper_authors.name: Hui Li
paper_authors._id: 66cfe2810c8862832e552b3d
paper_authors.name: Ken Goldberg
paper_title: In-Context Imitation Learning via Next-Token Prediction
paper_summary: We explore how to enhance next-token prediction models to perform in-context
  imitation learning on a real robot, where the robot executes new tasks by
  interpreting contextual information provided during the input phase, without
  updating its underlying policy parameters. We propose In-Context Robot
  Transformer (ICRT), a causal transformer that performs autoregressive
  prediction on sensorimotor trajectories without relying on any linguistic data
  or reward function. This formulation enables flexible and training-free
  execution of new tasks at test time, achieved by prompting the model with
  sensorimotor trajectories of the new task composing of image observations,
  actions and states tuples, collected through human teleoperation. Experiments
  with a Franka Emika robot demonstrate that the ICRT can adapt to new tasks
  specified by prompts, even in environment configurations that differ from both
  the prompt and the training data. In a multitask environment setup, ICRT
  significantly outperforms current state-of-the-art next-token prediction models
  in robotics on generalizing to unseen tasks. Code, checkpoints and data are
  available on https://icrt.dev/
paper_ai_keywords: In-Context Robot Transformer (ICRT) causal transformer autoregressive prediction sensorimotor trajectories image observations human teleoperation Franka Emika robot