Answers.Body: <p>For 2D (labyrinth-like) tasks the measure is usually percentage of the space that was covered. Another option is distance moved (in any direction).</p>
  <p>Once you have an objective measure for exploration, there will also be some algorithm that uses it as a target. Measuring the entropy of actions leads you to epsilon-greedy exploration. Coverage of (state, action) pairs leads to count-based exploration, I guess. And so on.</p>
  <p>MAP-Elites and Novelty Search: Task-specific behaviour criteria (BCs) are designed (in addition to the numerical reward/return). Then either create a grid with fixed bins and measure grid coverage, or maintain an archive of known solutions and measure the distance of new solutions to the k nearest neighbours.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-05-05T05:00:57.413Z
Answers.LastActivityDate: 2023-05-05T05:00:57.413Z
Answers.UserDisplayName: maxy
Body: <p>I am looking for metrics to compare the exploration under different RL Algos/reward functions. I want to somehow quantify how big of a region of the policy space is explored. What are common measures to do so?</p>
  <p>Thanks in advance for any help!</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2023-05-02T13:04:28.127Z
Comments.Text: Welcome to AI stack exchange! Please put your question in the title of your post. Try to make it as clear as possible from the title what kind of answer you would like to receive.
Comments.UserDisplayName: Robin van Hoorn
Tags: <reinforcement-learning><exploration-exploitation-tradeoff><exploration-strategies>
Title: Metrics to compare the exploraation of RL Algorithms
UserDisplayName: L208
ThreadText: # Metrics to compare the exploraation of RL Algorithms
  
  (Asked by: username_0 on 2023-05-02)
  
  I am looking for metrics to compare the exploration under different RL Algos/reward functions. I want to somehow quantify how big of a region of the policy space is explored. What are common measures to do so?
  
  Thanks in advance for any help!
  
  **Tags:** reinforcement-learning, exploration-exploitation-tradeoff, exploration-strategies
  
  > **Comments:**
  > **username_1**: Welcome to AI stack exchange! Please put your question in the title of your post. Try to make it as clear as possible from the title what kind of answer you would like to receive.  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_2 on 2023-05-05. Score: 1
  
  For 2D (labyrinth-like) tasks the measure is usually percentage of the space that was covered. Another option is distance moved (in any direction).
  
  Once you have an objective measure for exploration, there will also be some algorithm that uses it as a target. Measuring the entropy of actions leads you to epsilon-greedy exploration. Coverage of (state, action) pairs leads to count-based exploration, I guess. And so on.
  
  MAP-Elites and Novelty Search: Task-specific behaviour criteria (BCs) are designed (in addition to the numerical reward/return). Then either create a grid with fixed bins and measure grid coverage, or maintain an archive of known solutions and measure the distance of new solutions to the k nearest neighbours.
  
  
  ---
  
  