MaxInfoRL: Boosting exploration in reinforcement learning through
  information gain maximization
Reinforcement learning (RL) algorithms aim to balance exploiting the current
best strategy with exploring new options that could lead to higher rewards.
Most common RL algorithms use undirected exploration, i.e., select random
sequences of actions. Exploration can also be directed using intrinsic rewards,
such as curiosity or model epistemic uncertainty. However, effectively
balancing task and intrinsic rewards is challenging and often task-dependent.
In this work, we introduce a framework, MaxInfoRL, for balancing intrinsic and
extrinsic exploration. MaxInfoRL steers exploration towards informative
transitions, by maximizing intrinsic rewards such as the information gain about
the underlying task. When combined with Boltzmann exploration, this approach
naturally trades off maximization of the value function with that of the
entropy over states, rewards, and actions. We show that our approach achieves
sublinear regret in the simplified setting of multi-armed bandits. We then
apply this general formulation to a variety of off-policy model-free RL methods
for continuous state-action spaces, yielding novel algorithms that achieve
superior performance across hard exploration problems and complex scenarios
such as visual control tasks.
6761b55db5d954758bb63957
Bhavya Sukhija
6761b55db5d954758bb63958
Stelian Coros
6761b55db5d954758bb63959
Andreas Krause
admin_assigned
2024-12-18T13:37:28.986Z
67192065fecffc9f731c89d3
/avatars/3fdc97c13a825f2eb785abe1d74a08c1.svg
Andreas Krause
user
andreas85123
6761b55db5d954758bb6395a
Pieter Abbeel
6761b55db5d954758bb6395b
Carmelo Sferrazza
admin_assigned
2024-12-18T13:37:16.009Z
65ce68aec1912289cad81816
/avatars/eb9289243a533a6660e111185b012190.svg
Carmelo Sferrazza
user
csferrazza
MaxInfoRL: Boosting exploration in reinforcement learning through
  information gain maximization
Reinforcement learning (RL) algorithms aim to balance exploiting the current
best strategy with exploring new options that could lead to higher rewards.
Most common RL algorithms use undirected exploration, i.e., select random
sequences of actions. Exploration can also be directed using intrinsic rewards,
such as curiosity or model epistemic uncertainty. However, effectively
balancing task and intrinsic rewards is challenging and often task-dependent.
In this work, we introduce a framework, MaxInfoRL, for balancing intrinsic and
extrinsic exploration. MaxInfoRL steers exploration towards informative
transitions, by maximizing intrinsic rewards such as the information gain about
the underlying task. When combined with Boltzmann exploration, this approach
naturally trades off maximization of the value function with that of the
entropy over states, rewards, and actions. We show that our approach achieves
sublinear regret in the simplified setting of multi-armed bandits. We then
apply this general formulation to a variety of off-policy model-free RL methods
for continuous state-action spaces, yielding novel algorithms that achieve
superior performance across hard exploration problems and complex scenarios
such as visual control tasks.
MaxInfoRL balances intrinsic and extrinsic rewards by maximizing information gain, reducing regret in bandits and improving performance in continuous RL tasks.
reinforcement learning RL algorithms undirected exploration intrinsic rewards curiosity model epistemic uncertainty MaxInfoRL informative transitions Boltzmann exploration value function entropy off-policy model-free RL continuous state-action spaces superlinear regret multi-armed bandits visual control tasks