title: MineWorld: a Real-Time and Open-Source Interactive World Model on
    Minecraft
summary: World modeling is a crucial task for enabling intelligent agents to
  effectively interact with humans and operate in dynamic environments. In this
  work, we propose MineWorld, a real-time interactive world model on Minecraft,
  an open-ended sandbox game which has been utilized as a common testbed for
  world modeling. MineWorld is driven by a visual-action autoregressive
  Transformer, which takes paired game scenes and corresponding actions as input,
  and generates consequent new scenes following the actions. Specifically, by
  transforming visual game scenes and actions into discrete token ids with an
  image tokenizer and an action tokenizer correspondingly, we consist the model
  input with the concatenation of the two kinds of ids interleaved. The model is
  then trained with next token prediction to learn rich representations of game
  states as well as the conditions between states and actions simultaneously. In
  inference, we develop a novel parallel decoding algorithm that predicts the
  spatial redundant tokens in each frame at the same time, letting models in
  different scales generate 4 to 7 frames per second and enabling real-time
  interactions with game players. In evaluation, we propose new metrics to assess
  not only visual quality but also the action following capacity when generating
  new scenes, which is crucial for a world model. Our comprehensive evaluation
  shows the efficacy of MineWorld, outperforming SoTA open-sourced diffusion
  based world models significantly. The code and model have been released.
paper_authors._id: 67fc7367df5f5d1e87c14c6a
paper_authors.name: Junliang Guo
paper_authors._id: 67fc7367df5f5d1e87c14c6b
paper_authors.name: Yang Ye
paper_authors._id: 67fc7367df5f5d1e87c14c6c
paper_authors.name: Tianyu He
paper_authors._id: 67fc7367df5f5d1e87c14c6d
paper_authors.name: Haoyu Wu
paper_authors._id: 67fc7367df5f5d1e87c14c6e
paper_authors.name: Yushu Jiang
paper_authors._id: 67fc7367df5f5d1e87c14c6f
paper_authors.name: Tim Pearce
paper_authors._id: 67fc7367df5f5d1e87c14c70
paper_authors.name: Jiang Bian
paper_title: MineWorld: a Real-Time and Open-Source Interactive World Model on
    Minecraft
paper_summary: World modeling is a crucial task for enabling intelligent agents to
  effectively interact with humans and operate in dynamic environments. In this
  work, we propose MineWorld, a real-time interactive world model on Minecraft,
  an open-ended sandbox game which has been utilized as a common testbed for
  world modeling. MineWorld is driven by a visual-action autoregressive
  Transformer, which takes paired game scenes and corresponding actions as input,
  and generates consequent new scenes following the actions. Specifically, by
  transforming visual game scenes and actions into discrete token ids with an
  image tokenizer and an action tokenizer correspondingly, we consist the model
  input with the concatenation of the two kinds of ids interleaved. The model is
  then trained with next token prediction to learn rich representations of game
  states as well as the conditions between states and actions simultaneously. In
  inference, we develop a novel parallel decoding algorithm that predicts the
  spatial redundant tokens in each frame at the same time, letting models in
  different scales generate 4 to 7 frames per second and enabling real-time
  interactions with game players. In evaluation, we propose new metrics to assess
  not only visual quality but also the action following capacity when generating
  new scenes, which is crucial for a world model. Our comprehensive evaluation
  shows the efficacy of MineWorld, outperforming SoTA open-sourced diffusion
  based world models significantly. The code and model have been released.
paper_ai_summary: MineWorld, a real-time interactive world model built on Minecraft, uses a visual-action autoregressive Transformer to generate consequent game scenes based on inputs of game scenes and actions, outperforming state-of-the-art diffusion-based models.
paper_ai_keywords: visual-action autoregressive Transformer image tokenizer action tokenizer next token prediction parallel decoding algorithm visual quality action following capacity world modeling