Body: <p>In the <a href="https://arxiv.org/abs/1707.06347" rel="nofollow noreferrer">original paper</a>, the objective of PPO is as follows:<a href="https://i.sstatic.net/L6JnO.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/L6JnO.png" alt="enter image description here" /></a>. My question is, how does this objective behave in a sparse reward setting (i.e., reward is only given after a sequence of actions were taken)? In this case we don't have <span class="math-container">$\hat{A}_{t}$</span> defined for every <span class="math-container">$t$</span>.</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2023-03-06T01:40:40.667Z
Comments.Text: Why won't we A_{hat}_{t} for every t? There is usually a tuple of (state, reward, next_state) associated with every time step t. For sparse reward settings, the reward will be 0 for non-reward states.
Comments.UserDisplayName: desert_ranger
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2023-03-06T10:58:31.170Z
Comments.Text: @desert_ranger yes you can think of reward as 0 for those states, but in some situations it might be undefined
Comments.UserDisplayName: Sam
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2023-03-06T23:28:08.853Z
Comments.Text: It is the user who designs the reward for each step. Therefore, as long as the environment is formulated correctly, this shouldn't happen.
Comments.UserDisplayName: desert_ranger
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2023-03-07T02:45:11.967Z
Comments.Text: @desert_ranger think of Go. By default, not every move has a reward assigned. Are you suggesting going down the reward-shaping route to introduce artificial rewards?
Comments.UserDisplayName: Sam
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2023-04-05T10:23:57.637Z
Comments.Text: A reward _must_ exist for every $t$. As we are operating in an MDP, each $t$ corresponds to a transition from a state $s$ to another state $s'$ given an action $a$. By the definition of the MDP, a reward _must_ be associated with this transition, otherwise you are not working in a proper MDP. This is true for the game of Go. I believe the work by Deepmind just assigned a score of +1 for winning, -1 for losing, and every intermediate step was assigned a reward of 0.
Comments.UserDisplayName: David
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2023-04-06T14:06:25.243Z
Comments.Text: @DavidIreland yeah so to fit into MDP framework, can assign dummy reward 0 to the intermediate states.
Comments.UserDisplayName: Sam
Tags: <proximal-policy-optimization><sparse-rewards>
Title: How does Proximal Policy Optimization deal with sparse reward
UserDisplayName: Sam
ThreadText: # How does Proximal Policy Optimization deal with sparse reward
  
  (Asked by: username_0 on 2023-03-04)
  
  In the [original paper](https://arxiv.org/abs/1707.06347), the objective of PPO is as follows:[![enter image description here](https://i.sstatic.net/L6JnO.png)](https://i.sstatic.net/L6JnO.png). My question is, how does this objective behave in a sparse reward setting (i.e., reward is only given after a sequence of actions were taken)? In this case we don't have $\hat{A}\_{t}$ defined for every $t$.
  
  **Tags:** proximal-policy-optimization, sparse-rewards
  
  > **Comments:**
  > **username_1**: Why won't we A_{hat}_{t} for every t? There is usually a tuple of (state, reward, next_state) associated with every time step t. For sparse reward settings, the reward will be 0 for non-reward states.  
  > **username_0**: @username_1 yes you can think of reward as 0 for those states, but in some situations it might be undefined  
  > **username_1**: It is the user who designs the reward for each step. Therefore, as long as the environment is formulated correctly, this shouldn't happen.  
  > **username_0**: @username_1 think of Go. By default, not every move has a reward assigned. Are you suggesting going down the reward-shaping route to introduce artificial rewards?  
  > **username_2**: A reward _must_ exist for every $t$. As we are operating in an MDP, each $t$ corresponds to a transition from a state $s$ to another state $s'$ given an action $a$. By the definition of the MDP, a reward _must_ be associated with this transition, otherwise you are not working in a proper MDP. This is true for the game of Go. I believe the work by Deepmind just assigned a score of +1 for winning, -1 for losing, and every intermediate step was assigned a reward of 0.  
  > **username_0**: @DavidIreland yeah so to fit into MDP framework, can assign dummy reward 0 to the intermediate states.  
  
  ---
  
  ## 0 answers
  
  