Answers.Body: <blockquote>
    <p>In step 2 I need to decide for an initial estimate <span class="math-container">$\tilde{Q}_n$</span>. Is it a decent option to use <span class="math-container">$\tilde{Q}_n=Q_{n-1}$</span>?</p>
  </blockquote>
  
  <p>Yes, this is a common choice. It's actually common to update the table for <span class="math-container">$\tilde{Q}$</span> in place, without any separate initialisation per step. The separate phases of estimation and policy improvement are easier to analyse for theoretical correctness, but in practice updates made in place can be faster because new information is used as soon as it is available.</p>
  
  <p>Depending on how the policy was changed, and how accurate the previous estimate was, this could place the estimates closer convergence for the next step. Often the previous estimates will be closer to the new targets than any fixed or random initialisation scheme you could set up.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-11-10T11:15:40.677Z
Answers.Text: Thank you!  However I am not sure whether I understand the term "in place" does this mean that I improve my policy after every Monte-Carlo Update?
Answers.UserDisplayName: Sebastian
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-11-10T14:56:41.777Z
Answers.Text: I mean that you only have one Q table, and overwrite values as you calculate them. So no need to initialise a new table on each time step
Answers.UserDisplayName: Neil Slater
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-11-10T11:08:31.653Z
Answers.LastActivityDate: 2019-11-10T11:08:31.653Z
Answers.UserDisplayName: Neil Slater
Body: <p>I am trying to implement a tabular-based GLIE Monte-Carlo learning algorithm. 
  So I repeat n times: </p>
  
  <ol>
  <li>create observations using my previous policy <span class="math-container">$\pi_{n-1}(s)$</span></li>
  <li>update my state-action values using the observations generated in 1 with the monte-carlo update rule: <span class="math-container">$Q_n(s_t,a_t)= Q_n(s_t,a_t)+1/N(s_t,a_t)\times(G_t-Q_n(S_t,a_t))$</span></li>
  <li>update my policy to <span class="math-container">$\pi_{n}$</span> using epsilon-geedy improvement with <span class="math-container">$\epsilon=1/(n+1)$</span>. </li>
  </ol>
  
  <p>In step 2 I need to decide for an initial estimate <span class="math-container">$\tilde{Q}_n$</span>. Is it a decent option to use <span class="math-container">$\tilde{Q}_n=Q_{n-1}$</span>?</p>
  
Tags: <reinforcement-learning><monte-carlo-methods>
Title: Can I use my previous estimate of the state-action values as initialisation in GLIE-Monte Carlo Control?
UserDisplayName: Sebastian
ThreadText: # Can I use my previous estimate of the state-action values as initialisation in GLIE-Monte Carlo Control?
  
  (Asked by: username_0 on 2019-11-10)
  
  I am trying to implement a tabular-based GLIE Monte-Carlo learning algorithm.
  So I repeat n times:
  
  1. create observations using my previous policy $\pi\_{n-1}(s)$
  2. update my state-action values using the observations generated in 1 with the monte-carlo update rule: $Q\_n(s\_t,a\_t)= Q\_n(s\_t,a\_t)+1/N(s\_t,a\_t)\times(G\_t-Q\_n(S\_t,a\_t))$
  3. update my policy to $\pi\_{n}$ using epsilon-geedy improvement with $\epsilon=1/(n+1)$.
  
  In step 2 I need to decide for an initial estimate $\tilde{Q}\_n$. Is it a decent option to use $\tilde{Q}\_n=Q\_{n-1}$?
  
  **Tags:** reinforcement-learning, monte-carlo-methods
  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2019-11-10. Score: 1 | Accepted answer
  
  > In step 2 I need to decide for an initial estimate $\tilde{Q}\_n$. Is it a decent option to use $\tilde{Q}\_n=Q\_{n-1}$?
  
  Yes, this is a common choice. It's actually common to update the table for $\tilde{Q}$ in place, without any separate initialisation per step. The separate phases of estimation and policy improvement are easier to analyse for theoretical correctness, but in practice updates made in place can be faster because new information is used as soon as it is available.
  
  Depending on how the policy was changed, and how accurate the previous estimate was, this could place the estimates closer convergence for the next step. Often the previous estimates will be closer to the new targets than any fixed or random initialisation scheme you could set up.
  
  > **Comments:**
  > **username_0**: Thank you!  However I am not sure whether I understand the term "in place" does this mean that I improve my policy after every Monte-Carlo Update?  
  > **username_1**: I mean that you only have one Q table, and overwrite values as you calculate them. So no need to initialise a new table on each time step  
  
  ---
  
  