<p>In <a href="https://marcinbogdanski.github.io/rl-sketchpad/RL_An_Introduction_2018/1305a_One_Step_Actor_Critic.html" rel="nofollow noreferrer">One-Step Actor-Critic</a>, the update rule is performed after every step using the TD error. might want to check that out</p>

CC BY-SA 4.0
2022-12-27T07:35:17.873Z
2022-12-27T07:35:17.873Z
Hadar Sharvit
<p>Waiting an entire episode before doing a backprop can build up a very large computational graph, which is a burden on memory. Could one still learn a good policy by doing a backprop every fixed number of steps <em>within</em> an episode? E.g. backprop every 500 steps. Is there any literature about this that I can check out?</p>

CC BY-SA 4.0
2022-12-26T05:50:55.870Z
in One-step actor-critic, the update rule is performed after every step using the td error. might want to check that our
Hadar Sharvit
CC BY-SA 4.0
2022-12-27T01:00:08.143Z
@HadarSharvit you might as well write this an answer
David
<reinforcement-learning><reference-request><policy-gradients>
Could one still learn a good policy by doing a backprop every fixed number of steps within an episode?
Archie Gertsman
# Could one still learn a good policy by doing a backprop every fixed number of steps within an episode?

(Asked by: username_0 on 2022-12-25)

Waiting an entire episode before doing a backprop can build up a very large computational graph, which is a burden on memory. Could one still learn a good policy by doing a backprop every fixed number of steps *within* an episode? E.g. backprop every 500 steps. Is there any literature about this that I can check out?

**Tags:** reinforcement-learning, reference-request, policy-gradients

> **Comments:**
> **username_1**: in One-step actor-critic, the update rule is performed after every step using the td error. might want to check that our  
> **username_2**: @HadarSharvit you might as well write this an answer  

---

## 1 answers

### Answer by username_1 on 2022-12-27. Score: 1

In [One-Step Actor-Critic](https://marcinbogdanski.github.io/rl-sketchpad/RL_An_Introduction_2018/1305a_One_Step_Actor_Critic.html), the update rule is performed after every step using the TD error. might want to check that out


---

