Answers.Body: <p>In <a href="https://marcinbogdanski.github.io/rl-sketchpad/RL_An_Introduction_2018/1305a_One_Step_Actor_Critic.html" rel="nofollow noreferrer">One-Step Actor-Critic</a>, the update rule is performed after every step using the TD error. might want to check that out</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-12-27T07:35:17.873Z
Answers.LastActivityDate: 2022-12-27T07:35:17.873Z
Answers.UserDisplayName: Hadar Sharvit
Body: <p>Waiting an entire episode before doing a backprop can build up a very large computational graph, which is a burden on memory. Could one still learn a good policy by doing a backprop every fixed number of steps <em>within</em> an episode? E.g. backprop every 500 steps. Is there any literature about this that I can check out?</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2022-12-26T05:50:55.870Z
Comments.Text: in One-step actor-critic, the update rule is performed after every step using the td error. might want to check that our
Comments.UserDisplayName: Hadar Sharvit
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2022-12-27T01:00:08.143Z
Comments.Text: @HadarSharvit you might as well write this an answer
Comments.UserDisplayName: David
Tags: <reinforcement-learning><reference-request><policy-gradients>
Title: Could one still learn a good policy by doing a backprop every fixed number of steps within an episode?
UserDisplayName: Archie Gertsman
ThreadText: # Could one still learn a good policy by doing a backprop every fixed number of steps within an episode?
  
  (Asked by: username_0 on 2022-12-25)
  
  Waiting an entire episode before doing a backprop can build up a very large computational graph, which is a burden on memory. Could one still learn a good policy by doing a backprop every fixed number of steps *within* an episode? E.g. backprop every 500 steps. Is there any literature about this that I can check out?
  
  **Tags:** reinforcement-learning, reference-request, policy-gradients
  > **Comments:**
  > **username_1**: in One-step actor-critic, the update rule is performed after every step using the td error. might want to check that our  
  > **username_2**: @HadarSharvit you might as well write this an answer  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2022-12-27. Score: 1
  
  In [One-Step Actor-Critic](https://marcinbogdanski.github.io/rl-sketchpad/RL_An_Introduction_2018/1305a_One_Step_Actor_Critic.html), the update rule is performed after every step using the TD error. might want to check that out
  
  
  ---
  
  