Answers.Body: <p>Your second equation is the BOE (Bellman optimality equation) of the altered transition probabilities for the new environment where any <span class="math-container">$\epsilon$</span>-soft policy is moved inside as explained in the same page:</p>
  <blockquote>
  <p>The new environment has the same action and state set as the original and behaves as follows. If in state <span class="math-container">$s$</span> and taking action <span class="math-container">$a$</span>, then with probability <span class="math-container">$1-\epsilon$</span> the new environment behaves exactly like the old environment. With probability <span class="math-container">$\epsilon$</span> it repicks the action at random, with equal probabilities, and then behaves like the old environment with the new, random action... Let <span class="math-container">$\overset{\sim}{v}_*$</span> and <span class="math-container">$\overset{\sim}{q}_*$</span> denote the optimal value functions for the new environment. Then a policy <span class="math-container">$\pi$</span> is optimal among <span class="math-container">$\epsilon$</span>-soft policies if and only if <span class="math-container">$v_\pi=\overset{\sim}{v}_*$</span></p>
  </blockquote>
  <p>Therefore the max operator needs to range over the action <span class="math-container">$a$</span> in the first part of your second equation for learning the general (non-<span class="math-container">$\epsilon$</span>-soft) optimal policy of the new environment. The uniformly repicked random actions <span class="math-container">$a'$</span> are summed out in the second part of your second equation, thus they're not really variables but placeholders.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-04-16T20:19:11.087Z
Answers.Text: but why is it "a" in the first part  which is the same variable as the variable over which max is taken?
Answers.UserDisplayName: DSPinfinity
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-04-16T21:08:39.293Z
Answers.Text: As stated in my answer here the first part action $a$ is the real single non-dummy variable to be applied by max operator of BOE, it's the action of a general non-ϵ
  -soft/greedy optimal policy of the new environment, as in any generic model-based value iteration dynamic programming method. The possible confusing part is the transition probability $p$ here belongs to the old environment, you have to use the altered form to express the new environment's transition probability based on the said given $p$.
Answers.UserDisplayName: cinch
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-04-16T06:01:29.017Z
Answers.LastActivityDate: 2024-04-16T06:01:29.017Z
Answers.UserDisplayName: cinch
Body: <p>Sutton-Barto, page 102 (second edition).</p>
  <p>How does the Belman optimality with altered transition probabilities in the second equation follow? The point which confuses me is the first part inside the summation in second equation where the action variable is <span class="math-container">$a$</span> (the one over which max is taken) but the variable in the second part in the summation is <span class="math-container">$a^\prime$</span> which is independent of the variable over which max is taken. This second part is understandable because the actions
  are random. But why is still the variable in the first part of the summation in the second equation <span class="math-container">$a$</span>  which is the variable over which max is taken?</p>
  <p><a href="https://i.sstatic.net/pe1FyGfg.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/pe1FyGfg.png" alt="enter image description here" /></a></p>
  
Tags: <reinforcement-learning><sutton-barto>
Title: How does the Belman optimality equation with altered transition probabilities in the second equation follow?
UserDisplayName: DSPinfinity
ThreadText: # How does the Belman optimality equation with altered transition probabilities in the second equation follow?
  
  (Asked by: username_0 on 2024-04-16)
  
  Sutton-Barto, page 102 (second edition).
  
  How does the Belman optimality with altered transition probabilities in the second equation follow? The point which confuses me is the first part inside the summation in second equation where the action variable is $a$ (the one over which max is taken) but the variable in the second part in the summation is $a^\prime$ which is independent of the variable over which max is taken. This second part is understandable because the actions
  are random. But why is still the variable in the first part of the summation in the second equation $a$ which is the variable over which max is taken?
  
  [![enter image description here](https://i.sstatic.net/pe1FyGfg.png)](https://i.sstatic.net/pe1FyGfg.png)
  
  **Tags:** reinforcement-learning, sutton-barto
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2024-04-16. Score: 2 | Accepted answer
  
  Your second equation is the BOE (Bellman optimality equation) of the altered transition probabilities for the new environment where any $\epsilon$-soft policy is moved inside as explained in the same page:
  
  > The new environment has the same action and state set as the original and behaves as follows. If in state $s$ and taking action $a$, then with probability $1-\epsilon$ the new environment behaves exactly like the old environment. With probability $\epsilon$ it repicks the action at random, with equal probabilities, and then behaves like the old environment with the new, random action... Let $\overset{\sim}{v}\_\*$ and $\overset{\sim}{q}\_\*$ denote the optimal value functions for the new environment. Then a policy $\pi$ is optimal among $\epsilon$-soft policies if and only if $v\_\pi=\overset{\sim}{v}\_\*$
  
  Therefore the max operator needs to range over the action $a$ in the first part of your second equation for learning the general (non-$\epsilon$-soft) optimal policy of the new environment. The uniformly repicked random actions $a'$ are summed out in the second part of your second equation, thus they're not really variables but placeholders.
  
  > **Comments:**
  > **username_0**: but why is it "a" in the first part  which is the same variable as the variable over which max is taken?  
  > **username_1**: As stated in my answer here the first part action $a$ is the real single non-dummy variable to be applied by max operator of BOE, it's the action of a general non-ϵ -soft/greedy optimal policy of the new environment, as in any generic model-based value iteration dynamic programming method. The possible confusing part is the transition probability $p$ here belongs to the old environment, you have to use the altered form to express the new environment's transition probability based on the said given $p$.  
  
  ---
  
  