title: AlphaStar Unplugged: Large-Scale Offline Reinforcement Learning
summary: StarCraft II is one of the most challenging simulated reinforcement learning
  environments; it is partially observable, stochastic, multi-agent, and
  mastering StarCraft II requires strategic planning over long time horizons with
  real-time low-level execution. It also has an active professional competitive
  scene. StarCraft II is uniquely suited for advancing offline RL algorithms,
  both because of its challenging nature and because Blizzard has released a
  massive dataset of millions of StarCraft II games played by human players. This
  paper leverages that and establishes a benchmark, called AlphaStar Unplugged,
  introducing unprecedented challenges for offline reinforcement learning. We
  define a dataset (a subset of Blizzard's release), tools standardizing an API
  for machine learning methods, and an evaluation protocol. We also present
  baseline agents, including behavior cloning, offline variants of actor-critic
  and MuZero. We improve the state of the art of agents using only offline data,
  and we achieve 90% win rate against previously published AlphaStar behavior
  cloning agent.
paper_authors._id: 64d1a9f79617774ce42efbbf
paper_authors.name: Michaël Mathieu
paper_authors._id: 64d1a9f79617774ce42efbc0
paper_authors.name: Sherjil Ozair
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-08-08T13:58:49.891Z
paper_authors._id: 6311bda23a773344e068edc7
paper_authors.avatarUrl: /avatars/4a7cdbdac92ad6bcb91d2f7d52f03613.svg
paper_authors.fullname: Sherjil
paper_authors.type: user
paper_authors.user: sherjilozair
paper_authors._id: 64d1a9f79617774ce42efbc1
paper_authors.name: Srivatsan Srinivasan
paper_authors._id: 64d1a9f79617774ce42efbc2
paper_authors.name: Caglar Gulcehre
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2023-08-09T00:43:02.684Z
paper_authors._id: 64d28085db0b63b977a6c3f8
paper_authors.avatarUrl: /avatars/994cc3ff7c18650fe2535e473d56b240.svg
paper_authors.fullname: Caglar Gulcehre
paper_authors.type: user
paper_authors.user: caglarg
paper_authors._id: 64d1a9f79617774ce42efbc3
paper_authors.name: Shangtong Zhang
paper_authors._id: 64d1a9f79617774ce42efbc4
paper_authors.name: Ray Jiang
paper_authors._id: 64d1a9f79617774ce42efbc5
paper_authors.name: Tom Le Paine
paper_authors._id: 64d1a9f79617774ce42efbc6
paper_authors.name: Richard Powell
paper_authors._id: 64d1a9f79617774ce42efbc7
paper_authors.name: Konrad Żołna
paper_authors._id: 64d1a9f79617774ce42efbc8
paper_authors.name: Julian Schrittwieser
paper_authors._id: 64d1a9f79617774ce42efbc9
paper_authors.name: David Choi
paper_authors._id: 64d1a9f79617774ce42efbca
paper_authors.name: Petko Georgiev
paper_authors._id: 64d1a9f79617774ce42efbcb
paper_authors.name: Daniel Toyama
paper_authors._id: 64d1a9f79617774ce42efbcc
paper_authors.name: Aja Huang
paper_authors._id: 64d1a9f79617774ce42efbcd
paper_authors.name: Roman Ring
paper_authors._id: 64d1a9f79617774ce42efbce
paper_authors.name: Igor Babuschkin
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-08-08T14:19:44.331Z
paper_authors._id: 62d99bf239a1c86eec71bc41
paper_authors.avatarUrl: /avatars/d2f3693eb76b95c7c7097cc6b5411fc4.svg
paper_authors.fullname: Igor Babuschkin
paper_authors.type: user
paper_authors.user: ibab
paper_authors._id: 64d1a9f79617774ce42efbcf
paper_authors.name: Timo Ewalds
paper_authors._id: 64d1a9f79617774ce42efbd0
paper_authors.name: Mahyar Bordbar
paper_authors._id: 64d1a9f79617774ce42efbd1
paper_authors.name: Sarah Henderson
paper_authors._id: 64d1a9f79617774ce42efbd2
paper_authors.name: Sergio Gómez Colmenarejo
paper_authors._id: 64d1a9f79617774ce42efbd3
paper_authors.name: Aäron van den Oord
paper_authors._id: 64d1a9f79617774ce42efbd4
paper_authors.name: Wojciech Marian Czarnecki
paper_authors._id: 64d1a9f79617774ce42efbd5
paper_authors.name: Nando de Freitas
paper_authors._id: 64d1a9f79617774ce42efbd6
paper_authors.name: Oriol Vinyals
paper_title: AlphaStar Unplugged: Large-Scale Offline Reinforcement Learning
paper_summary: StarCraft II is one of the most challenging simulated reinforcement learning
  environments; it is partially observable, stochastic, multi-agent, and
  mastering StarCraft II requires strategic planning over long time horizons with
  real-time low-level execution. It also has an active professional competitive
  scene. StarCraft II is uniquely suited for advancing offline RL algorithms,
  both because of its challenging nature and because Blizzard has released a
  massive dataset of millions of StarCraft II games played by human players. This
  paper leverages that and establishes a benchmark, called AlphaStar Unplugged,
  introducing unprecedented challenges for offline reinforcement learning. We
  define a dataset (a subset of Blizzard's release), tools standardizing an API
  for machine learning methods, and an evaluation protocol. We also present
  baseline agents, including behavior cloning, offline variants of actor-critic
  and MuZero. We improve the state of the art of agents using only offline data,
  and we achieve 90% win rate against previously published AlphaStar behavior
  cloning agent.
paper_ai_keywords: reinforcement learning partially observable stochastic multi-agent long time horizons real-time low-level execution offline RL algorithms AlphaStar Unplugged dataset API behavior cloning offline variants actor-critic MuZero win rate