Body: <p>I have skimmed through a bunch of deep learning books, but I have not yet understood whether we must use the experience replay buffer with the A3C algorithm.</p>
  
  <p>The approached I used is the following:</p>
  
  <ul>
  <li>I have some threaded agents that play their own copy of an enviroment; they all use a shared NN to predict 'best' move given a current state;</li>
  <li>At the end of each episode, they push the episode (in the form of a list of steps) in a shared memory</li>
  <li>A separated thread reads from the shared memory and executes the train step on the shared NN, training it episode after episode.</li>
  </ul>
  
  <p>Is this an appropriate approach? More specifically, do I need to sample data to train the NN from the shared memory? Or should I push in the shared memory each step, just when it's done by a single agent? In this last case, I wonder how I could calculate discounted rewards.</p>
  
  <p>I'm afraid that with my current approach I'm doing nothing more that presenting n episodes to the NN, with the hope that each agent explores the enviroment differently from other agents (so that NN is presented a richer variety of states).</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2019-05-02T11:09:04.270Z
Comments.Text: A3C is an actor critic reinforcement learning algorithm with asynchronous updates, hence it is not really relevant for a DL book to cover it. You'll want to check the original paper for details (https://arxiv.org/abs/1602.01783). There are many implementations which cover the details of the algo, random eg: https://medium.com/emergent-future/simple-reinforcement-learning-with-tensorflow-part-8-asynchronous-actor-critic-agents-a3c-c88f72a5e9f2.
Comments.UserDisplayName: vega
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2019-05-02T11:09:58.467Z
Comments.Text: Being on-policy AC, it's not using a replay buffer like off-policy methods such as DQN. However, It does collect data in order to compute discounted rewards. After you update your network, you should not reuse old experiences (that would be off-policy).
Comments.UserDisplayName: vega
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2019-05-02T11:37:35.363Z
Comments.Text: Thanks vega for your quick answer. I'll try to follow the tutorial you suggested as well as the original paper.
Comments.UserDisplayName: Scorpio76
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2019-05-02T13:26:00.187Z
Comments.Text: It's good to try to implement the algo, good learning experience. But be warned, it is notoriously difficult. Best to use a known stable version, like [ikostrikov](https://github.com/ikostrikov/pytorch-a3c), or try a better algo like [ppo](https://github.com/ikostrikov/pytorch-a2c-ppo-acktr-gail). Good luck!
Comments.UserDisplayName: vega
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2019-05-02T15:01:58.450Z
Comments.Text: Well, from the few experiments done so far, I noticed that RL is really difficult. Let me ask another question: if I understood it right, the example you linked proposes a scenario where each agent has its own copy of the model. In examples I've read until now, there is only a single NN copy, and each agent is, more or less, an "example producer" (i.e provides a list of steps). What am I missing?
Comments.UserDisplayName: Scorpio76
Tags: <reinforcement-learning><ai-design><deep-rl><actor-critic-methods><experience-replay>
Title: Do we need to use the experience replay buffer with the A3C algorithm?
UserDisplayName: Scorpio76
ThreadText: # Do we need to use the experience replay buffer with the A3C algorithm?
  
  (Asked by: username_0 on 2019-05-02)
  
  I have skimmed through a bunch of deep learning books, but I have not yet understood whether we must use the experience replay buffer with the A3C algorithm.
  
  The approached I used is the following:
  
  * I have some threaded agents that play their own copy of an enviroment; they all use a shared NN to predict 'best' move given a current state;
  * At the end of each episode, they push the episode (in the form of a list of steps) in a shared memory
  * A separated thread reads from the shared memory and executes the train step on the shared NN, training it episode after episode.
  
  Is this an appropriate approach? More specifically, do I need to sample data to train the NN from the shared memory? Or should I push in the shared memory each step, just when it's done by a single agent? In this last case, I wonder how I could calculate discounted rewards.
  
  I'm afraid that with my current approach I'm doing nothing more that presenting n episodes to the NN, with the hope that each agent explores the enviroment differently from other agents (so that NN is presented a richer variety of states).
  
  **Tags:** reinforcement-learning, ai-design, deep-rl, actor-critic-methods, experience-replay
  
  > **Comments:**
  > **username_1**: A3C is an actor critic reinforcement learning algorithm with asynchronous updates, hence it is not really relevant for a DL book to cover it. You'll want to check the original paper for details (https://arxiv.org/abs/1602.01783). There are many implementations which cover the details of the algo, random eg: https://medium.com/emergent-future/simple-reinforcement-learning-with-tensorflow-part-8-asynchronous-actor-critic-agents-a3c-c88f72a5e9f2.  
  > **username_1**: Being on-policy AC, it's not using a replay buffer like off-policy methods such as DQN. However, It does collect data in order to compute discounted rewards. After you update your network, you should not reuse old experiences (that would be off-policy).  
  > **username_0**: Thanks username_1 for your quick answer. I'll try to follow the tutorial you suggested as well as the original paper.  
  > **username_1**: It's good to try to implement the algo, good learning experience. But be warned, it is notoriously difficult. Best to use a known stable version, like [ikostrikov](https://github.com/ikostrikov/pytorch-a3c), or try a better algo like [ppo](https://github.com/ikostrikov/pytorch-a2c-ppo-acktr-gail). Good luck!  
  > **username_0**: Well, from the few experiments done so far, I noticed that RL is really difficult. Let me ask another question: if I understood it right, the example you linked proposes a scenario where each agent has its own copy of the model. In examples I've read until now, there is only a single NN copy, and each agent is, more or less, an "example producer" (i.e provides a list of steps). What am I missing?  
  
  ---
  
  ## 0 answers
  
  