Answers.Body: <p>In S&amp;B RL book page 332 pseudocode for one-step actor-critic method with 2 nested loops, both policy and critic parameters could be initialized arbitrarily, and for example they're both initialized to <span class="math-container">$0$</span> vectors in the pseudocode. And <span class="math-container">$\alpha, \beta$</span> above are learning rate step size for policy and critic parameters, respectively.</p>
  <p>However, your reference notations are not based on one-step but based on accumulated Monte Carlo simulated return of the m-th trajectory under the current (stochastic) policy, therefore, following your reference’s notations, policy and critic parameters can still both be initialized as <span class="math-container">$0$</span> vectors, but there’s actually no need to initialize <span class="math-container">$\Delta \phi$</span> as it’s computed only based on the advantage functions <span class="math-container">$A^i_t$</span> after your quoted two inner loops. The reason why <span class="math-container">$\Delta \theta$</span> is initialized to <span class="math-container">$0$</span> after the outermost episodic loop is that it needs to be reset to <span class="math-container">$0$</span> at the start of each episode after accumulation in the two inner loops while <span class="math-container">$\Delta \phi$</span> is not accumulating there thus no need to reset or initialize.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-09-02T08:13:16.463Z
Answers.LastActivityDate: 2024-09-02T08:18:32.390Z
Answers.LastEditDate: 2024-09-02T08:18:32.390Z
Answers.UserDisplayName: cinch
Body: <p>See slide 92 here: <a href="https://cs231n.stanford.edu/slides/2017/cs231n_2017_lecture14.pdf" rel="nofollow noreferrer">https://cs231n.stanford.edu/slides/2017/cs231n_2017_lecture14.pdf</a></p>
  <p><a href="https://i.sstatic.net/XWaH4sgc.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/XWaH4sgc.png" alt="enter image description here" /></a></p>
  <p>I did not understand the expression for critic in update of <span class="math-container">$\Delta \phi$</span> and also why is it updated at the end of inner for-loop? Where is the initialization for <span class="math-container">$\Delta \phi$</span>?</p>
  
Tags: <reinforcement-learning><actor-critic-methods>
Title: Is this actor-critic algorithm correct?
UserDisplayName: DSPinfinity
ThreadText: # Is this actor-critic algorithm correct?
  
  (Asked by: username_0 on 2024-09-01)
  
  See slide 92 here: <https://cs231n.stanford.edu/slides/2017/cs231n_2017_lecture14.pdf>
  
  [![enter image description here](https://i.sstatic.net/XWaH4sgc.png)](https://i.sstatic.net/XWaH4sgc.png)
  
  I did not understand the expression for critic in update of $\Delta \phi$ and also why is it updated at the end of inner for-loop? Where is the initialization for $\Delta \phi$?
  
  **Tags:** reinforcement-learning, actor-critic-methods
  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2024-09-02. Score: 1 | Accepted answer
  
  In S&B RL book page 332 pseudocode for one-step actor-critic method with 2 nested loops, both policy and critic parameters could be initialized arbitrarily, and for example they're both initialized to $0$ vectors in the pseudocode. And $\alpha, \beta$ above are learning rate step size for policy and critic parameters, respectively.
  
  However, your reference notations are not based on one-step but based on accumulated Monte Carlo simulated return of the m-th trajectory under the current (stochastic) policy, therefore, following your reference’s notations, policy and critic parameters can still both be initialized as $0$ vectors, but there’s actually no need to initialize $\Delta \phi$ as it’s computed only based on the advantage functions $A^i\_t$ after your quoted two inner loops. The reason why $\Delta \theta$ is initialized to $0$ after the outermost episodic loop is that it needs to be reset to $0$ at the start of each episode after accumulation in the two inner loops while $\Delta \phi$ is not accumulating there thus no need to reset or initialize.
  
  
  ---
  
  