Answers.Body: <p>The loss function is estimated in every batch training cycle. Gradients of the loss are computed and propagation back to the network happens in every cycle. This means that you use a small batch (e.g. 100 instances) from the replay memory, and by having the states you can input them to the respective network and have the <span class="math-container">$Q(s)$</span> for every state in your batch. Then you estimate the loss and run backpropagation and the networks' weights are getting updated. You continue gathering experience by interacting with the environment and after a threshold that you specify you repeat the cycle by re-sampling a new batch from the memory.</p>
  
  <p>Just a suggestion: Start moving towards Asynchronous/synchronous methods for RL and use one network with different "heads" for Actor and Critic. Then you use one loss function plus the experience that now is collected from multiple instances of your agent-environment interaction (in contrast to your description in which one agent collects and stores experience from one instance).</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-02-01T19:13:11.330Z
Answers.LastActivityDate: 2019-02-01T19:13:11.330Z
Answers.UserDisplayName: Constantinos
Body: <p><a href="https://datascience.stackexchange.com/questions/20535/what-is-experience-replay-and-what-are-its-benefits">Experience replay</a> is buffer (or a "memory") of transactions <span class="math-container">$e_t = (s_t, a_t, r_t, s_{t+1})$</span>.</p>
  
  <p><a href="https://medium.freecodecamp.org/an-intro-to-advantage-actor-critic-methods-lets-play-sonic-the-hedgehog-86d6240171d" rel="nofollow noreferrer">The equations for calculating the loss in actor critic</a> are an </p>
  
  <p><em>actor</em> loss (parameterized by <span class="math-container">$\theta$</span>) <span class="math-container">$$log[\pi_\theta(s_t,a_t)]Q_w(s_t,a_t)$$</span> and a <em>critic</em> loss (parameterized by <span class="math-container">$w$</span>) <span class="math-container">$$r(s_t,a_t) + \gamma Q_w(s_{t+1},a_{t+1}) - Q_w(s_{t},a_{t}).$$</span></p>
  
  <p>As I see it, there are two more elements that need to be saved for later use:</p>
  
  <ol>
  <li><p>The expected Q value at the time <span class="math-container">$t$</span>: <span class="math-container">$Q_w(s_{t},a_{t})$</span></p></li>
  <li><p>The log probability for action <span class="math-container">$a_t$</span>: <span class="math-container">$log[\pi_\theta(s_t,a_t)]$</span></p></li>
  </ol>
  
  <p>If they are not saved, how will we be able to later calculate the loss for learning? I didn't see anywhere stating to save those, and I must be missing something.</p>
  
  <p>Do these elements need to be saved or not?</p>
  
Tags: <machine-learning><reinforcement-learning><policy-gradients><experience-replay>
Title: What information should be cached in experience replay for actor-critic?
UserDisplayName: Gulzar
ThreadText: # What information should be cached in experience replay for actor-critic?
  
  (Asked by: username_0 on 2019-01-31)
  
  [Experience replay](https://datascience.stackexchange.com/questions/20535/what-is-experience-replay-and-what-are-its-benefits) is buffer (or a "memory") of transactions $e\_t = (s\_t, a\_t, r\_t, s\_{t+1})$.
  
  [The equations for calculating the loss in actor critic](https://medium.freecodecamp.org/an-intro-to-advantage-actor-critic-methods-lets-play-sonic-the-hedgehog-86d6240171d) are an
  
  *actor* loss (parameterized by $\theta$) $$log[\pi\_\theta(s\_t,a\_t)]Q\_w(s\_t,a\_t)$$ and a *critic* loss (parameterized by $w$) $$r(s\_t,a\_t) + \gamma Q\_w(s\_{t+1},a\_{t+1}) - Q\_w(s\_{t},a\_{t}).$$
  
  As I see it, there are two more elements that need to be saved for later use:
  
  1. The expected Q value at the time $t$: $Q\_w(s\_{t},a\_{t})$
  2. The log probability for action $a\_t$: $log[\pi\_\theta(s\_t,a\_t)]$
  
  If they are not saved, how will we be able to later calculate the loss for learning? I didn't see anywhere stating to save those, and I must be missing something.
  
  Do these elements need to be saved or not?
  
  **Tags:** machine-learning, reinforcement-learning, policy-gradients, experience-replay
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2019-02-01. Score: 4 | Accepted answer
  
  The loss function is estimated in every batch training cycle. Gradients of the loss are computed and propagation back to the network happens in every cycle. This means that you use a small batch (e.g. 100 instances) from the replay memory, and by having the states you can input them to the respective network and have the $Q(s)$ for every state in your batch. Then you estimate the loss and run backpropagation and the networks' weights are getting updated. You continue gathering experience by interacting with the environment and after a threshold that you specify you repeat the cycle by re-sampling a new batch from the memory.
  
  Just a suggestion: Start moving towards Asynchronous/synchronous methods for RL and use one network with different "heads" for Actor and Critic. Then you use one loss function plus the experience that now is collected from multiple instances of your agent-environment interaction (in contrast to your description in which one agent collects and stores experience from one instance).
  
  
  ---
  
  