Answers.Body: <blockquote>
  <p>The reward is given only at the end of the episode (or when there is timeout there is no reward)</p>
  </blockquote>
  <p>This is a common case. E.g. winning a board game, or reaching a goal state.</p>
  <blockquote>
  <p>How could we learn the value function?</p>
  </blockquote>
  <p><strong>All</strong> RL algorithms are designed to cope with this scenario. Actor-Critic is not an exception. Value-based algorithms (including the critic in Actor-Critic) learn through time step backup updates. The simplest backup is to copy data about experience at time <span class="math-container">$t+1$</span> into an update for the state or state/action experienced at time <span class="math-container">$t$</span>. That is what single-step temporal difference algorithms do. Other value-based algorithms can be more sophisticated and more efficient with assigning the update signal back in time.</p>
  <p>A <em>very</em> sparse reward can be difficult for an agent to find or learn from. So some methods may work better than others. Without knowing the environment, and what the specific difficulties might be, it is not possible to recommend an approach, or to even suggest whether your algorithm needs any help.</p>
  <blockquote>
  <p>Do you recommend computing intermediate rewards?</p>
  </blockquote>
  <p>In general, no. However, these can help when the &quot;natural&quot; rewards in a problem are both sparse and hard for the agent to discover through exploration. Constructing extra reward signals to guide a learning agent is called reward shaping.</p>
  <p>Reward shaping needs to be done with care because it can inadvertently change what the optimal solution is. But if done well, it can make a problem much easier to solve for an agent.</p>
  <p>A starting rule of thumb for whether to add some kind of reward shaping is based on how often a random agent might accidentally obtain a difference in end rewards. Once the agent has experienced a difference between rewards, it can start to refine its predictions and begin to prefer the higher reward, which then usually leads to exploration nearer more valuable states. It may do this even if the higher reward only occurs one time in a thousand initially, say. However, if initial random actions gain no useful signal even for millions of trial-and-error episodes, then you will need to do something to assist the agent.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-10-05T09:07:31.210Z
Answers.LastActivityDate: 2021-10-05T16:31:39.387Z
Answers.LastEditDate: 2021-10-05T16:31:39.387Z
Answers.UserDisplayName: Neil Slater
Body: <p>Consider the actor-critic reinforcement learning setting (actor and critic parameterized by a neural network). The reward is given only at the end of the episode (or when there is a timeout there is no reward).</p>
  <p>How could we learn the value function? Do you recommend computing intermediate rewards?</p>
  
Tags: <reinforcement-learning><actor-critic-methods><reward-functions><sparse-rewards>
Title: How do I compute the value function when the reward is only at the end in the context of actor-critic algorithms?
UserDisplayName: cerebrou
ThreadText: # How do I compute the value function when the reward is only at the end in the context of actor-critic algorithms?
  
  (Asked by: username_0 on 2021-10-05)
  
  Consider the actor-critic reinforcement learning setting (actor and critic parameterized by a neural network). The reward is given only at the end of the episode (or when there is a timeout there is no reward).
  
  How could we learn the value function? Do you recommend computing intermediate rewards?
  
  **Tags:** reinforcement-learning, actor-critic-methods, reward-functions, sparse-rewards
  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2021-10-05. Score: 2
  
  > The reward is given only at the end of the episode (or when there is timeout there is no reward)
  
  This is a common case. E.g. winning a board game, or reaching a goal state.
  
  > How could we learn the value function?
  
  **All** RL algorithms are designed to cope with this scenario. Actor-Critic is not an exception. Value-based algorithms (including the critic in Actor-Critic) learn through time step backup updates. The simplest backup is to copy data about experience at time $t+1$ into an update for the state or state/action experienced at time $t$. That is what single-step temporal difference algorithms do. Other value-based algorithms can be more sophisticated and more efficient with assigning the update signal back in time.
  
  A *very* sparse reward can be difficult for an agent to find or learn from. So some methods may work better than others. Without knowing the environment, and what the specific difficulties might be, it is not possible to recommend an approach, or to even suggest whether your algorithm needs any help.
  
  > Do you recommend computing intermediate rewards?
  
  In general, no. However, these can help when the "natural" rewards in a problem are both sparse and hard for the agent to discover through exploration. Constructing extra reward signals to guide a learning agent is called reward shaping.
  
  Reward shaping needs to be done with care because it can inadvertently change what the optimal solution is. But if done well, it can make a problem much easier to solve for an agent.
  
  A starting rule of thumb for whether to add some kind of reward shaping is based on how often a random agent might accidentally obtain a difference in end rewards. Once the agent has experienced a difference between rewards, it can start to refine its predictions and begin to prefer the higher reward, which then usually leads to exploration nearer more valuable states. It may do this even if the higher reward only occurs one time in a thousand initially, say. However, if initial random actions gain no useful signal even for millions of trial-and-error episodes, then you will need to do something to assist the agent.
  
  
  ---
  
  