Answers.Body: <p>I don't think the section was written in haste. I think they just didn't have space to include the whole proof. It's a bit involved, so they just gave concepts.  </p>
  
  <p><a href="https://armahmood.github.io/files/SMW-emphasis-2016.pdf" rel="nofollow noreferrer">An Emphatic Approach to the Problem
  of Off-policy Temporal-Difference Learning</a> gives a proof of stability. 
  At least parts of it should seem familiar if you've read Sutton and Barto's proof of the convergence of linear TD(0) on page 206 of their 2nd edition RL book. </p>
  
  <p><a href="http://proceedings.mlr.press/v40/Yu15.pdf" rel="nofollow noreferrer">On Convergence of Emphatic Temporal-Difference Learning</a> gives a proof of convergence.</p>
  
  <p>I confess that I don't understand these papers well enough to give a summary. If you eventually do, I would greatly appreciate an update.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-06-07T01:06:00.047Z
Answers.Text: Hi, after skimming through all cites of these two papers. I found it seems that there is no paper on how to combine Deep Networks with Emphatic TD. Is it possible to use it in DQN, TRPO or PPO?
Answers.UserDisplayName: GoingMyWay
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-03-13T06:48:28.987Z
Answers.LastActivityDate: 2019-03-13T07:29:41.120Z
Answers.LastEditDate: 2019-03-13T07:29:41.120Z
Answers.UserDisplayName: Philip Raeisghasem
Body: <p>The book by Sutton and Barto discusses in <a href="http://incompleteideas.net/book/RLbook2020.pdf#page=303" rel="nofollow noreferrer">section 11.8</a> that the convergence of off-policy TD function approximation can be improved by correcting for the distribution of states encountered. The section seems to be written in haste and doesn't do a good job in explaining why will <span class="math-container">$M_t$</span>, the <em>emphasis</em>, help in getting a state distribution closer to the target policy.</p>
  <p>My understanding of <em>on-policy distribution</em> is not clear at the moment. I think it is the distribution of states encountered under the target policy (the policy for which we want to state-action/state values).</p>
  <p>The importance sampling ratio corrects for update distribution (by multiplying the correction term with the ratio), but how is <span class="math-container">$M_t$</span> helping in correcting for the state distribution?</p>
  
Tags: <reinforcement-learning><temporal-difference-methods><off-policy-methods><sutton-barto>
Title: Why is $M_t$ (the emphasis) helping in correcting for the state distribution in the Emphatic TD algorithm?
UserDisplayName: pg2455
ThreadText: # Why is $M_t$ (the emphasis) helping in correcting for the state distribution in the Emphatic TD algorithm?
  
  (Asked by: username_0 on 2019-03-12)
  
  The book by Sutton and Barto discusses in [section 11.8](http://incompleteideas.net/book/RLbook2020.pdf#page=303) that the convergence of off-policy TD function approximation can be improved by correcting for the distribution of states encountered. The section seems to be written in haste and doesn't do a good job in explaining why will $M\_t$, the *emphasis*, help in getting a state distribution closer to the target policy.
  
  My understanding of *on-policy distribution* is not clear at the moment. I think it is the distribution of states encountered under the target policy (the policy for which we want to state-action/state values).
  
  The importance sampling ratio corrects for update distribution (by multiplying the correction term with the ratio), but how is $M\_t$ helping in correcting for the state distribution?
  
  **Tags:** reinforcement-learning, temporal-difference-methods, off-policy-methods, sutton-barto
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2019-03-13. Score: 2
  
  I don't think the section was written in haste. I think they just didn't have space to include the whole proof. It's a bit involved, so they just gave concepts.
  
  [An Emphatic Approach to the Problem
  of Off-policy Temporal-Difference Learning](https://armahmood.github.io/files/SMW-emphasis-2016.pdf) gives a proof of stability.
  At least parts of it should seem familiar if you've read Sutton and Barto's proof of the convergence of linear TD(0) on page 206 of their 2nd edition RL book.
  
  [On Convergence of Emphatic Temporal-Difference Learning](http://proceedings.mlr.press/v40/Yu15.pdf) gives a proof of convergence.
  
  I confess that I don't understand these papers well enough to give a summary. If you eventually do, I would greatly appreciate an update.
  
  > **Comments:**
  > **username_2**: Hi, after skimming through all cites of these two papers. I found it seems that there is no paper on how to combine Deep Networks with Emphatic TD. Is it possible to use it in DQN, TRPO or PPO?  
  
  ---
  
  