Body: <p>This is question 6.13 in Sutton-Barto,page 136.</p>
  <blockquote>
  <p>What are the update equations for Double Expected Sarsa with an <span class="math-container">$\epsilon$</span>-greedy target policy?</p>
  </blockquote>
  <p>The answer is given as follows:</p>
  <blockquote>
  <p>Let <span class="math-container">$Q_1$</span> and <span class="math-container">$Q_2$</span> be the two action-value functions and let <span class="math-container">$\pi_1$</span>
  and <span class="math-container">$\pi_2$</span> be their <span class="math-container">$\epsilon$</span>-greedy policies, respectively. Then, the updates would be:</p>
  </blockquote>
  <blockquote>
  <p><span class="math-container">$Q_1(S_t, A_t) \leftarrow Q_1(S_t, A_t)+\alpha\Big[R_{t+1}+\gamma \displaystyle \sum_{a}\pi_2(a|S_{t+1})Q_1(S_{t+1},a)-Q_1(S_t, A_t)\Big] $</span></p>
  </blockquote>
  <blockquote>
  <p><span class="math-container">$Q_2(S_t, A_t) \leftarrow Q_2(S_t, A_t)+\alpha\Big[R_{t+1}+\gamma \displaystyle \sum_{a}\pi_1(a|S_{t+1})Q_2(S_{t+1},a)-Q_2(S_t, A_t)\Big] $</span></p>
  </blockquote>
  <p>Could someone explain how they were obtained similar to a reasoning corresponding to double Q-learning?</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2024-05-18T22:30:22.213Z
Comments.Text: I thought you had the official answers already? You do know you can get them from the author?
Comments.UserDisplayName: Neil Slater
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2024-05-18T23:44:06.637Z
Comments.Text: Yes, I have but they just put the equations without any explanation.
Comments.UserDisplayName: DSPinfinity
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2024-05-18T23:57:54.577Z
Comments.Text: Now, I also put the equations  here. What I need is a good explanation on how they were obtained similar to a reasoning for double Q-learning.
Comments.UserDisplayName: DSPinfinity
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2024-05-19T11:10:33.547Z
Comments.Text: That's not how they were obtained though. The two component ideas - double-learning, and expected SARSA - make entirely different changes to the formulation of the update rules, and are compatible. So you simply make both changes: Sum over expected results from policy, and alternate action choice table vs bootstrap estimate table. There's no need to derive these from scratch, and no benefit to doing so at all. In fact I think that defeats the purpose of understanding what is going on - having a feel for these changes as "operators" that modify the equations is more useful and intuitive
Comments.UserDisplayName: Neil Slater
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2024-05-19T22:42:05.450Z
Comments.Text: What are "action choice table" and "bootstrap estimation table"? Where is the "alternate action choice table vs bootstrap estimate table" in the given updates?
Comments.UserDisplayName: DSPinfinity
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2024-05-20T06:57:06.533Z
Comments.Text: This is a tabular method, so Q(s, a) is a table
Comments.UserDisplayName: Neil Slater
Tags: <sutton-barto><double-q-learning><expected-sarsa>
Title: What are the update equations for Double Expected Sarsa with an $\epsilon$-greedy target policy?
UserDisplayName: DSPinfinity
ThreadText: # What are the update equations for Double Expected Sarsa with an $\epsilon$-greedy target policy?
  
  (Asked by: username_0 on 2024-05-18)
  
  This is question 6.13 in Sutton-Barto,page 136.
  
  > What are the update equations for Double Expected Sarsa with an $\epsilon$-greedy target policy?
  
  The answer is given as follows:
  
  > Let $Q\_1$ and $Q\_2$ be the two action-value functions and let $\pi\_1$
  > and $\pi\_2$ be their $\epsilon$-greedy policies, respectively. Then, the updates would be:
  
  > $Q\_1(S\_t, A\_t) \leftarrow Q\_1(S\_t, A\_t)+\alpha\Big[R\_{t+1}+\gamma \displaystyle \sum\_{a}\pi\_2(a|S\_{t+1})Q\_1(S\_{t+1},a)-Q\_1(S\_t, A\_t)\Big] $
  
  > $Q\_2(S\_t, A\_t) \leftarrow Q\_2(S\_t, A\_t)+\alpha\Big[R\_{t+1}+\gamma \displaystyle \sum\_{a}\pi\_1(a|S\_{t+1})Q\_2(S\_{t+1},a)-Q\_2(S\_t, A\_t)\Big] $
  
  Could someone explain how they were obtained similar to a reasoning corresponding to double Q-learning?
  
  **Tags:** sutton-barto, double-q-learning, expected-sarsa
  > **Comments:**
  > **username_1**: I thought you had the official answers already? You do know you can get them from the author?  
  > **username_0**: Yes, I have but they just put the equations without any explanation.  
  > **username_0**: Now, I also put the equations  here. What I need is a good explanation on how they were obtained similar to a reasoning for double Q-learning.  
  > **username_1**: That's not how they were obtained though. The two component ideas - double-learning, and expected SARSA - make entirely different changes to the formulation of the update rules, and are compatible. So you simply make both changes: Sum over expected results from policy, and alternate action choice table vs bootstrap estimate table. There's no need to derive these from scratch, and no benefit to doing so at all. In fact I think that defeats the purpose of understanding what is going on - having a feel for these changes as "operators" that modify the equations is more useful and intuitive  
  > **username_0**: What are "action choice table" and "bootstrap estimation table"? Where is the "alternate action choice table vs bootstrap estimate table" in the given updates?  
  > **username_1**: This is a tabular method, so Q(s, a) is a table  
  
  ---
  
  ## 0 answers
  
  