Answers.Body: <p>I convinced myself that the problem has to do with the fact that rollouts terminate when a terminal state is reached. In principle, a reward shift should not affect the optimal policy. However, it does matter in this case because we do not see the terminal rewards (except the first one). As such, all subsequent states effectively have <code>0</code> reward. If <code>0</code> reward is bigger than your typical reward, it can entice the actor to seek out a terminal state.</p>
  <p>Note that due to discounting, 1 state’s worth of terminal reward is being compared to <code>gamma / (1 - gamma) * ave_reward</code>, which means if the first is <code>-100</code> and gamma is <code>0.99</code> for cart-pole, <code>ave_reward</code> has to be above <code>-1.0101</code> to be more enticing. In my case that wasn’t true.</p>
  <p>Potential mitigations include:</p>
  <ol>
  <li>Making your non-terminal rewards positive</li>
  <li>Having a very large negative first terminal reward</li>
  <li>Not terminating your rollout when a terminal state is encountered</li>
  </ol>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-03-23T16:58:32.683Z
Answers.LastActivityDate: 2023-04-02T04:03:31.937Z
Answers.LastEditDate: 2023-04-02T04:03:31.937Z
Answers.UserDisplayName: Mageek
Body: <p>I’ve been toying around with cart pole swingup, and have been flummoxed by a problem related to reward shifts. Traditionally, cart pole gives you positive reward if you’re stable and zero reward if you’re out of the acceptable set.
  In my version of cart pole swingup, when I have 0 terminal reward and strictly positive normal reward, the system learns to balance.</p>
  <pre><code>if is_terminal(s)
    return 0
  end
  
  return 100 - x^2 - signed_dist(theta, 0)^2
  </code></pre>
  <p>However, if I shift this reward by 100, it flounders and fails.</p>
  <pre><code>if is_terminal(s)
    return -100
  end
  
  return - x^2 - signed_dist(theta, 0)^2
  </code></pre>
  <p>My understanding of the theory is that shifting the reward shouldn’t affect anything. Any idea what I am missing? Thank you!</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2023-03-21T16:46:30.590Z
Comments.Text: Does your program use a learning rate, if so, have you tried to tune it? Have you shown your whole reward? Do you have a reference to your entire codebase?
Comments.UserDisplayName: cinch
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2023-03-21T19:29:07.983Z
Comments.Text: @Mageek, What is the maximum value of the positive reward you can receive in the first case? I.e., the reward is in the range [0, y] for what value of y?
Comments.UserDisplayName: DeepQZero
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2023-03-21T19:54:15.870Z
Comments.Text: Yes I have tried tuning the learning rate (both for Adam and basic descent). I think it has more to do with episodes ending when they terminate rather than the entity actually being exposed to the negative rewards.
Comments.UserDisplayName: Mageek
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2023-03-21T19:56:31.167Z
Comments.Text: The max reward in the first case is 100, since dx and dtheta can be zero.
Comments.UserDisplayName: Mageek
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2023-03-21T19:57:34.950Z
Comments.Text: I employed the trick where a rollout does not end the first step where the state is terminal, but only on the second. That hasn’t fixed the problem
Comments.UserDisplayName: Mageek
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2023-03-22T15:35:58.223Z
Comments.Text: @Mageek, what does your neural network architecture look like? Is there any chance that it's only outputting positive values?
Comments.UserDisplayName: DeepQZero
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2023-03-23T03:07:45.167Z
Comments.Text: My actor neural net does in fact only output non-negative values, as it has a softmax output that produces a probability distribution over the actions. My critic neural net does not have an activation, so it is free to output positive or negative values. They are both otherwise simple sequential affine layers with ReLUs in between.
Comments.UserDisplayName: Mageek
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2023-03-23T15:47:09.960Z
Comments.Text: @Mageek At this point, I would probably need to see more code to have an idea about possible solutions.
Comments.UserDisplayName: DeepQZero
Tags: <actor-critic-methods>
Title: Should Cart Pole be immune to Reward Shifting?
UserDisplayName: Mageek
ThreadText: # Should Cart Pole be immune to Reward Shifting?
  
  (Asked by: username_0 on 2023-03-21)
  
  I’ve been toying around with cart pole swingup, and have been flummoxed by a problem related to reward shifts. Traditionally, cart pole gives you positive reward if you’re stable and zero reward if you’re out of the acceptable set.
  In my version of cart pole swingup, when I have 0 terminal reward and strictly positive normal reward, the system learns to balance.
  
  ```
  if is_terminal(s)
    return 0
  end
  
  return 100 - x^2 - signed_dist(theta, 0)^2
  
  ```
  
  However, if I shift this reward by 100, it flounders and fails.
  
  ```
  if is_terminal(s)
    return -100
  end
  
  return - x^2 - signed_dist(theta, 0)^2
  
  ```
  
  My understanding of the theory is that shifting the reward shouldn’t affect anything. Any idea what I am missing? Thank you!
  
  **Tags:** actor-critic-methods
  
  > **Comments:**
  > **username_1**: Does your program use a learning rate, if so, have you tried to tune it? Have you shown your whole reward? Do you have a reference to your entire codebase?  
  > **username_2**: @username_0, What is the maximum value of the positive reward you can receive in the first case? I.e., the reward is in the range [0, y] for what value of y?  
  > **username_0**: Yes I have tried tuning the learning rate (both for Adam and basic descent). I think it has more to do with episodes ending when they terminate rather than the entity actually being exposed to the negative rewards.  
  > **username_0**: The max reward in the first case is 100, since dx and dtheta can be zero.  
  > **username_0**: I employed the trick where a rollout does not end the first step where the state is terminal, but only on the second. That hasn’t fixed the problem  
  > **username_2**: @username_0, what does your neural network architecture look like? Is there any chance that it's only outputting positive values?  
  > **username_0**: My actor neural net does in fact only output non-negative values, as it has a softmax output that produces a probability distribution over the actions. My critic neural net does not have an activation, so it is free to output positive or negative values. They are both otherwise simple sequential affine layers with ReLUs in between.  
  > **username_2**: @username_0 At this point, I would probably need to see more code to have an idea about possible solutions.  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_0 on 2023-03-23. Score: 1 | Accepted answer
  
  I convinced myself that the problem has to do with the fact that rollouts terminate when a terminal state is reached. In principle, a reward shift should not affect the optimal policy. However, it does matter in this case because we do not see the terminal rewards (except the first one). As such, all subsequent states effectively have `0` reward. If `0` reward is bigger than your typical reward, it can entice the actor to seek out a terminal state.
  
  Note that due to discounting, 1 state’s worth of terminal reward is being compared to `gamma / (1 - gamma) * ave_reward`, which means if the first is `-100` and gamma is `0.99` for cart-pole, `ave_reward` has to be above `-1.0101` to be more enticing. In my case that wasn’t true.
  
  Potential mitigations include:
  
  1. Making your non-terminal rewards positive
  2. Having a very large negative first terminal reward
  3. Not terminating your rollout when a terminal state is encountered
  
  
  ---
  
  