Answers.Body: <blockquote>
  <p>What are the pros and cons of sparse and dense rewards in reinforcement learning?</p>
  </blockquote>
  <p>It is unusual to refer to this difference as &quot;pros and cons&quot; because that term is often used to make comparisons between difference <em>choices</em>. Assuming you have a specific problem to solve, then whether or not the rewards are <em>naturally</em> sparse or dense is not a choice. You cannot say &quot;I want to solve MountainCar, I will use a dense reward setting&quot;, because MountainCar has (relatively, for a starting problem) sparse rewards. You can only say &quot;I won't attempt MountainCar, it is too difficult&quot;.</p>
  <p>In short however, your assessment is correct:</p>
  <blockquote>
  <p>if the rewards are sparse the agent will have to explore more to get rewards and learn the optimal policy, whereas if the rewards are dense in time, the agent is quickly guided towards its learning goal</p>
  </blockquote>
  <p>There is not really any other difference at the top level. Essentially, sparser rewards make for a harder problem to solve. All RL algorithms can cope with sparse rewards to some degree, the whole concept of returns and value backup is designed to deal with sparseness at a theoretical level. In practical terms however, it may take some algorithms an unreasonable amount of time to determine a good policy beyond certain levels of sparseness.</p>
  <blockquote>
  <p>On a side-note, I feel that the inability to specify rewards that are dense in time is what makes imitation learning useful.</p>
  </blockquote>
  <p>Imitation learning is one of many techniques available to work around or deal with problems that have sparse reward structure. Others include:</p>
  <ul>
  <li><p>Reward shaping, which attempts to convert a sparse reward scheme to a dense one using domain knowledge of the researcher.</p>
  </li>
  <li><p>Eligibility traces, which back up individual TD errors across multiple time steps.</p>
  </li>
  <li><p>Prioritised sweeping, which focuses updates on &quot;surprising&quot; reward data.</p>
  </li>
  <li><p>Action selection planning algorithms that look ahead from the current state.</p>
  </li>
  <li><p>&quot;Curiousity&quot; driven reinforcement learning that guides exploration to new state spaces independently of any reward signal.</p>
  </li>
  </ul>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-08-13T07:49:18.347Z
Answers.Text: Could you point me in the direction of interesting papers related to the techniques you've mentioned, besides imitation learning? That'd be helpful!
Answers.UserDisplayName: stoic-santiago
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-08-13T07:53:06.810Z
Answers.Text: @cogito_ai: Not really, most are broad topics and you are better off using a web search to find tutorials. For eligibility traces and planning, Sutton & Barto has a good chapter on each. It also has a section on prioritised sweeping in planning context (DQN may also use it to select replay memory). Here's something on cuirosity: https://towardsdatascience.com/curiosity-driven-learning-made-easy-part-i-d3e5a2263359 - there were a few papers on that in 2018
Answers.UserDisplayName: Neil Slater
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-08-13T07:34:27.970Z
Answers.LastActivityDate: 2020-08-13T07:34:27.970Z
Answers.UserDisplayName: Neil Slater
Body: <p>From what I understand, if the rewards are <strong>sparse</strong> the agent will have to <strong>explore more</strong> to get rewards and learn the optimal policy, whereas if the rewards are <strong>dense</strong> in time, the agent is <strong>quickly</strong> guided towards its learning goal.</p>
  <p>Are the above thoughts correct, and are there any other pros and cons of the two contrasting settings? On a side-note, I feel that the inability to specify rewards that are dense in time is what makes imitation learning useful.</p>
  
Tags: <reinforcement-learning><comparison><reward-functions><sparse-rewards><dense-rewards>
Title: What are the pros and cons of sparse and dense rewards in reinforcement learning?
UserDisplayName: stoic-santiago
ThreadText: # What are the pros and cons of sparse and dense rewards in reinforcement learning?
  
  (Asked by: username_0 on 2020-08-13)
  
  From what I understand, if the rewards are **sparse** the agent will have to **explore more** to get rewards and learn the optimal policy, whereas if the rewards are **dense** in time, the agent is **quickly** guided towards its learning goal.
  
  Are the above thoughts correct, and are there any other pros and cons of the two contrasting settings? On a side-note, I feel that the inability to specify rewards that are dense in time is what makes imitation learning useful.
  
  **Tags:** reinforcement-learning, comparison, reward-functions, sparse-rewards, dense-rewards
  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2020-08-13. Score: 4 | Accepted answer
  
  > What are the pros and cons of sparse and dense rewards in reinforcement learning?
  
  It is unusual to refer to this difference as "pros and cons" because that term is often used to make comparisons between difference *choices*. Assuming you have a specific problem to solve, then whether or not the rewards are *naturally* sparse or dense is not a choice. You cannot say "I want to solve MountainCar, I will use a dense reward setting", because MountainCar has (relatively, for a starting problem) sparse rewards. You can only say "I won't attempt MountainCar, it is too difficult".
  
  In short however, your assessment is correct:
  
  > if the rewards are sparse the agent will have to explore more to get rewards and learn the optimal policy, whereas if the rewards are dense in time, the agent is quickly guided towards its learning goal
  
  There is not really any other difference at the top level. Essentially, sparser rewards make for a harder problem to solve. All RL algorithms can cope with sparse rewards to some degree, the whole concept of returns and value backup is designed to deal with sparseness at a theoretical level. In practical terms however, it may take some algorithms an unreasonable amount of time to determine a good policy beyond certain levels of sparseness.
  
  > On a side-note, I feel that the inability to specify rewards that are dense in time is what makes imitation learning useful.
  
  Imitation learning is one of many techniques available to work around or deal with problems that have sparse reward structure. Others include:
  
  * Reward shaping, which attempts to convert a sparse reward scheme to a dense one using domain knowledge of the researcher.
  * Eligibility traces, which back up individual TD errors across multiple time steps.
  * Prioritised sweeping, which focuses updates on "surprising" reward data.
  * Action selection planning algorithms that look ahead from the current state.
  * "Curiousity" driven reinforcement learning that guides exploration to new state spaces independently of any reward signal.
  
  > **Comments:**
  > **username_0**: Could you point me in the direction of interesting papers related to the techniques you've mentioned, besides imitation learning? That'd be helpful!  
  > **username_1**: @cogito_ai: Not really, most are broad topics and you are better off using a web search to find tutorials. For eligibility traces and planning, Sutton & Barto has a good chapter on each. It also has a section on prioritised sweeping in planning context (DQN may also use it to select replay memory). Here's something on cuirosity: https://towardsdatascience.com/curiosity-driven-learning-made-easy-part-i-d3e5a2263359 - there were a few papers on that in 2018  
  
  ---
  
  