Answers.Body: <p>You might want to check out this paper relating to Phizaz's comment: <a href="https://arxiv.org/pdf/1602.01783.pdf" rel="nofollow noreferrer">Asynchronous Methods for Deep Reinforcement Learning</a> (specifically search for Hogwild).</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-03-25T15:01:51.373Z
Answers.Text: Thank you. I've since discovered that A2C can still have multiple environments/agents to garner experience more quickly (as you would in A3C), but the difference being that A2C waits for all agents to complete their experiences and apply gradient changes across all agents' batches, whereas A3C performs the gradient updates as soon as it has collected the specified size of experience data for that one agent. The benefit of multi-agent A2C is the agent is always working with the most up-to-date policy, and possible speed improvements if you use a GPU to combine the batches into one large batch.
Answers.UserDisplayName: BigBadMe
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-03-23T23:50:00.453Z
Answers.LastActivityDate: 2019-03-24T13:38:52.290Z
Answers.LastEditDate: 2019-03-24T13:38:52.290Z
Answers.UserDisplayName: apuffin
Body: <p>To make A2C into A3C you make it asynchronous. From what I understand the 'correct' way to do that is to thread off workers with a copy of the policy and critic, and then return the state/action/reward tuples to the main thread, which then performs the gradients updates on the main policy and critic, and then repeat the process.</p>
  
  <p>I understand why the copying would be necessary in a distributed environment, but if I were to always run it locally then could I just perform the updates on a global variable of the policy and critic, i.e. avoid the need for copying? Provided the concurrency of the updates was handled correctly, would that be fine?</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2019-03-03T06:19:20.093Z
Comments.Text: I don't think it is possible. If the global net updates every N iterations but local net (each agent) updates every M iterations and each agent runs asynchronously, having one copy of the weights is not possible since each agent might need a different copy of the weights.
Comments.UserDisplayName: Phizaz
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2019-03-03T07:14:39.340Z
Comments.Text: The global net would be updated every M iterations also. To be clear; the worker would need to 'freeze' a copy of the policy/critic in order to do the updates, it would just get an immediate/online copy through the global variables. The only difference, as I see it, is that each worker has direct access to global variables for the policy/critic to perform the updates, rather than needing to return out with the *S, a, r* tuples for the main handler thread to do the updates. As I say, I understand the need for that in distributed, but I don't see why it'd be needed on a purely local environment
Comments.UserDisplayName: BigBadMe
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2019-03-03T11:45:47.810Z
Comments.Text: If you allow each worker to share the global net and update on it live, it is possible to do with only one copy of the network. This is called "Hogwild" fashion. I have seen this in Pytorch [tutorial](https://github.com/pytorch/examples/tree/master/mnist_hogwild) itself.
Comments.UserDisplayName: Phizaz
Tags: <reinforcement-learning><actor-critic-methods>
Title: Can A3C update the policy / critic on a local machine without needing to copy?
UserDisplayName: BigBadMe
ThreadText: # Can A3C update the policy / critic on a local machine without needing to copy?
  
  (Asked by: username_0 on 2019-03-02)
  
  To make A2C into A3C you make it asynchronous. From what I understand the 'correct' way to do that is to thread off workers with a copy of the policy and critic, and then return the state/action/reward tuples to the main thread, which then performs the gradients updates on the main policy and critic, and then repeat the process.
  
  I understand why the copying would be necessary in a distributed environment, but if I were to always run it locally then could I just perform the updates on a global variable of the policy and critic, i.e. avoid the need for copying? Provided the concurrency of the updates was handled correctly, would that be fine?
  
  **Tags:** reinforcement-learning, actor-critic-methods
  > **Comments:**
  > **username_1**: I don't think it is possible. If the global net updates every N iterations but local net (each agent) updates every M iterations and each agent runs asynchronously, having one copy of the weights is not possible since each agent might need a different copy of the weights.  
  > **username_0**: The global net would be updated every M iterations also. To be clear; the worker would need to 'freeze' a copy of the policy/critic in order to do the updates, it would just get an immediate/online copy through the global variables. The only difference, as I see it, is that each worker has direct access to global variables for the policy/critic to perform the updates, rather than needing to return out with the *S, a, r* tuples for the main handler thread to do the updates. As I say, I understand the need for that in distributed, but I don't see why it'd be needed on a purely local environment  
  > **username_1**: If you allow each worker to share the global net and update on it live, it is possible to do with only one copy of the network. This is called "Hogwild" fashion. I have seen this in Pytorch [tutorial](https://github.com/pytorch/examples/tree/master/mnist_hogwild) itself.  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_2 on 2019-03-23. Score: 0
  
  You might want to check out this paper relating to Phizaz's comment: [Asynchronous Methods for Deep Reinforcement Learning](https://arxiv.org/pdf/1602.01783.pdf) (specifically search for Hogwild).
  
  > **Comments:**
  > **username_0**: Thank you. I've since discovered that A2C can still have multiple environments/agents to garner experience more quickly (as you would in A3C), but the difference being that A2C waits for all agents to complete their experiences and apply gradient changes across all agents' batches, whereas A3C performs the gradient updates as soon as it has collected the specified size of experience data for that one agent. The benefit of multi-agent A2C is the agent is always working with the most up-to-date policy, and possible speed improvements if you use a GPU to combine the batches into one large batch.  
  
  ---
  
  