Answers.Body: <p>After some research (thanks, <a href="https://chat.openai.com/chat" rel="nofollow noreferrer">ChatGPT</a>!) it seems that the implementation provided by PyTorch is known as <em>Monte Carlo actor-critic</em>, where the value function is updated using the Monte Carlo returns (i.e., the sum of the rewards received in an episode) rather than the TD error.</p>
  <p>One advantage of Monte Carlo actor-critic is that it can be easier to implement and may be more stable in some environments. However, it is less sample efficient than the online version, since it only learns from completed episodes rather than making updates at each step.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-12-16T07:41:50.883Z
Answers.LastActivityDate: 2022-12-16T07:41:50.883Z
Answers.UserDisplayName: Hadar Sharvit
Body: <p>In the <a href="https://github.com/pytorch/examples/blob/main/reinforcement_learning/actor_critic.py" rel="nofollow noreferrer">Actor-Critic</a> example, provided by PyTorch, it seems that the update rule only occurs when the episode ends (like in a Monte-Carlo process). Specifically, in their <code>main</code> function they sample the environment in a loop, saving the log probabilities and state values <code>model.saved_actions.append(SavedAction(m.log_prob(action), state_value))</code> right after the agent takes an action. Later, when <code>done</code> was reached, the loss term is aggregated and back-propagated.</p>
  <p>The thing is - in every literature book I've come across they distinguish that one of the critical points in AC method is that the update rule can be performed after every step in the environment - something along the lines of:</p>
  <ul>
  <li>take action <span class="math-container">$a\sim\pi_\theta(\cdot|s)$</span> and observe <span class="math-container">$s',r$</span></li>
  <li>calculate TD error <span class="math-container">$\delta\gets r+\gamma \hat{V}(s',\theta_V)-\hat{V}(s,\theta_V)$</span></li>
  <li>update <span class="math-container">$\hat{V}$</span>'s weights <span class="math-container">$\theta_V\gets\theta_V+\alpha_V  \delta \nabla_{\theta_V}\hat{V}(s,\theta_v)$</span></li>
  <li>update <span class="math-container">$\pi_\theta$</span>'s weights <span class="math-container">$\theta\gets\theta+\alpha_\theta I \delta \nabla_{\theta} \ln{\pi_\theta(a|s)}$</span></li>
  </ul>
  <p>what am I missing here?</p>
  
Tags: <reinforcement-learning><pytorch><actor-critic-methods>
Title: Pytorch's Actor-critic implementation seems to be implemented in a Monte-Carlo fashion - why?
UserDisplayName: Hadar Sharvit
ThreadText: # Pytorch's Actor-critic implementation seems to be implemented in a Monte-Carlo fashion - why?
  
  (Asked by: username_0 on 2022-12-15)
  
  In the [Actor-Critic](https://github.com/pytorch/examples/blob/main/reinforcement_learning/actor_critic.py) example, provided by PyTorch, it seems that the update rule only occurs when the episode ends (like in a Monte-Carlo process). Specifically, in their `main` function they sample the environment in a loop, saving the log probabilities and state values `model.saved_actions.append(SavedAction(m.log_prob(action), state_value))` right after the agent takes an action. Later, when `done` was reached, the loss term is aggregated and back-propagated.
  
  The thing is - in every literature book I've come across they distinguish that one of the critical points in AC method is that the update rule can be performed after every step in the environment - something along the lines of:
  
  * take action $a\sim\pi\_\theta(\cdot|s)$ and observe $s',r$
  * calculate TD error $\delta\gets r+\gamma \hat{V}(s',\theta\_V)-\hat{V}(s,\theta\_V)$
  * update $\hat{V}$'s weights $\theta\_V\gets\theta\_V+\alpha\_V \delta \nabla\_{\theta\_V}\hat{V}(s,\theta\_v)$
  * update $\pi\_\theta$'s weights $\theta\gets\theta+\alpha\_\theta I \delta \nabla\_{\theta} \ln{\pi\_\theta(a|s)}$
  
  what am I missing here?
  
  **Tags:** reinforcement-learning, pytorch, actor-critic-methods
  
  ---
  
  ## 1 answers
  
  ### Answer by username_0 on 2022-12-16. Score: 1
  
  After some research (thanks, [ChatGPT](https://chat.openai.com/chat)!) it seems that the implementation provided by PyTorch is known as *Monte Carlo actor-critic*, where the value function is updated using the Monte Carlo returns (i.e., the sum of the rewards received in an episode) rather than the TD error.
  
  One advantage of Monte Carlo actor-critic is that it can be easier to implement and may be more stable in some environments. However, it is less sample efficient than the online version, since it only learns from completed episodes rather than making updates at each step.
  
  
  ---
  
  