title: MaxInfoRL: Boosting exploration in reinforcement learning through
    information gain maximization
summary: Reinforcement learning (RL) algorithms aim to balance exploiting the current
  best strategy with exploring new options that could lead to higher rewards.
  Most common RL algorithms use undirected exploration, i.e., select random
  sequences of actions. Exploration can also be directed using intrinsic rewards,
  such as curiosity or model epistemic uncertainty. However, effectively
  balancing task and intrinsic rewards is challenging and often task-dependent.
  In this work, we introduce a framework, MaxInfoRL, for balancing intrinsic and
  extrinsic exploration. MaxInfoRL steers exploration towards informative
  transitions, by maximizing intrinsic rewards such as the information gain about
  the underlying task. When combined with Boltzmann exploration, this approach
  naturally trades off maximization of the value function with that of the
  entropy over states, rewards, and actions. We show that our approach achieves
  sublinear regret in the simplified setting of multi-armed bandits. We then
  apply this general formulation to a variety of off-policy model-free RL methods
  for continuous state-action spaces, yielding novel algorithms that achieve
  superior performance across hard exploration problems and complex scenarios
  such as visual control tasks.
paper_authors._id: 6761b55db5d954758bb63957
paper_authors.name: Bhavya Sukhija
paper_authors._id: 6761b55db5d954758bb63958
paper_authors.name: Stelian Coros
paper_authors._id: 6761b55db5d954758bb63959
paper_authors.name: Andreas Krause
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-12-18T13:37:28.986Z
paper_authors._id: 67192065fecffc9f731c89d3
paper_authors.avatarUrl: /avatars/3fdc97c13a825f2eb785abe1d74a08c1.svg
paper_authors.fullname: Andreas Krause
paper_authors.type: user
paper_authors.user: andreas85123
paper_authors._id: 6761b55db5d954758bb6395a
paper_authors.name: Pieter Abbeel
paper_authors._id: 6761b55db5d954758bb6395b
paper_authors.name: Carmelo Sferrazza
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-12-18T13:37:16.009Z
paper_authors._id: 65ce68aec1912289cad81816
paper_authors.avatarUrl: /avatars/eb9289243a533a6660e111185b012190.svg
paper_authors.fullname: Carmelo Sferrazza
paper_authors.type: user
paper_authors.user: csferrazza
paper_title: MaxInfoRL: Boosting exploration in reinforcement learning through
    information gain maximization
paper_summary: Reinforcement learning (RL) algorithms aim to balance exploiting the current
  best strategy with exploring new options that could lead to higher rewards.
  Most common RL algorithms use undirected exploration, i.e., select random
  sequences of actions. Exploration can also be directed using intrinsic rewards,
  such as curiosity or model epistemic uncertainty. However, effectively
  balancing task and intrinsic rewards is challenging and often task-dependent.
  In this work, we introduce a framework, MaxInfoRL, for balancing intrinsic and
  extrinsic exploration. MaxInfoRL steers exploration towards informative
  transitions, by maximizing intrinsic rewards such as the information gain about
  the underlying task. When combined with Boltzmann exploration, this approach
  naturally trades off maximization of the value function with that of the
  entropy over states, rewards, and actions. We show that our approach achieves
  sublinear regret in the simplified setting of multi-armed bandits. We then
  apply this general formulation to a variety of off-policy model-free RL methods
  for continuous state-action spaces, yielding novel algorithms that achieve
  superior performance across hard exploration problems and complex scenarios
  such as visual control tasks.
paper_ai_summary: MaxInfoRL balances intrinsic and extrinsic rewards by maximizing information gain, reducing regret in bandits and improving performance in continuous RL tasks.
paper_ai_keywords: reinforcement learning RL algorithms undirected exploration intrinsic rewards curiosity model epistemic uncertainty MaxInfoRL informative transitions Boltzmann exploration value function entropy off-policy model-free RL continuous state-action spaces superlinear regret multi-armed bandits visual control tasks