Answers.Body: <p>It is a mathematical problem. Usually, we use contraction mapping theorem for the proof of convergence. You should apply the Banach fixed point theorem for Bellman's functions.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-09-22T17:20:20.627Z
Answers.LastActivityDate: 2019-09-22T17:20:20.627Z
Answers.UserDisplayName: David Nguyen
Body: <p>Prediction's goal is to get an estimate of a performance of a policy given a specific state. </p>
  
  <p>Control's goal is to improve the policy wrt. the prediction. </p>
  
  <p>The alternation between the two is the basis of reinforcement learning algorithms. </p>
  
  <p>In the paper <a href="https://arxiv.org/abs/1606.02647" rel="nofollow noreferrer">“Safe and Efficient Off-Policy Reinforcement Learning.” (Munos, 2016)</a>, the section 3.1) "Policy evaluation" assumes that the target policy is fixed, while the section 3.2) "Control" extends to where the target policy is a sequence of policies improved by a sequence of increasingly greedy operations. </p>
  
  <p>This suggests that even a proof of convergence is established with a fixed target policy, one cannot immediately imply that of the case where the target policy is a sequence of improving policies.</p>
  
  <p>I wonder why it is the case. If an algorithm converges under a fixed target policy assumption, any policy during the chain of improvement should have no problem with this algorithm as well. With the merit of policy improvement, each policy in sequence is increasingly better hence converging to an optimal policy.</p>
  
  <p>This should be obvious from the policy improvement perspective and should require no further proof at all?</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2019-09-19T13:20:05.193Z
Comments.Text: I don't understand what you are asking. What do you mean by "This suggests that even a proof of convergence is established with a fixed target policy". Where does the suggestion come from? Also, you should rephrase this "one cannot immediately imply that of the case where the target policy is a sequence of improving policies.", which is not very clear.
Comments.UserDisplayName: nbro
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2019-09-19T13:42:39.507Z
Comments.Text: @nbro Actually, my simpler question is what is the need for section 3.2) "control" at all?
Comments.UserDisplayName: Phizaz
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2019-09-19T13:43:46.747Z
Comments.Text: @nbro About the suggestion, I imply from the existence of section 3.2. If the proof provided in the section 3.1 (fixed policy case) is suffice, there should be no section 3.2.
Comments.UserDisplayName: Phizaz
Tags: <reinforcement-learning>
Title: Why doesn't stability in prediction imply stability in control in off-policy reinforcement learning?
UserDisplayName: Phizaz
ThreadText: # Why doesn't stability in prediction imply stability in control in off-policy reinforcement learning?
  
  (Asked by: username_0 on 2019-09-19)
  
  Prediction's goal is to get an estimate of a performance of a policy given a specific state.
  
  Control's goal is to improve the policy wrt. the prediction.
  
  The alternation between the two is the basis of reinforcement learning algorithms.
  
  In the paper [“Safe and Efficient Off-Policy Reinforcement Learning.” (Munos, 2016)](https://arxiv.org/abs/1606.02647), the section 3.1) "Policy evaluation" assumes that the target policy is fixed, while the section 3.2) "Control" extends to where the target policy is a sequence of policies improved by a sequence of increasingly greedy operations.
  
  This suggests that even a proof of convergence is established with a fixed target policy, one cannot immediately imply that of the case where the target policy is a sequence of improving policies.
  
  I wonder why it is the case. If an algorithm converges under a fixed target policy assumption, any policy during the chain of improvement should have no problem with this algorithm as well. With the merit of policy improvement, each policy in sequence is increasingly better hence converging to an optimal policy.
  
  This should be obvious from the policy improvement perspective and should require no further proof at all?
  
  **Tags:** reinforcement-learning
  > **Comments:**
  > **username_1**: I don't understand what you are asking. What do you mean by "This suggests that even a proof of convergence is established with a fixed target policy". Where does the suggestion come from? Also, you should rephrase this "one cannot immediately imply that of the case where the target policy is a sequence of improving policies.", which is not very clear.  
  > **username_0**: @nbro Actually, my simpler question is what is the need for section 3.2) "control" at all?  
  > **username_0**: @nbro About the suggestion, I imply from the existence of section 3.2. If the proof provided in the section 3.1 (fixed policy case) is suffice, there should be no section 3.2.  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_2 on 2019-09-22. Score: 1
  
  It is a mathematical problem. Usually, we use contraction mapping theorem for the proof of convergence. You should apply the Banach fixed point theorem for Bellman's functions.
  
  
  ---
  
  