Answers.Body: <p>This sounds like distributed RL, and most of the work goes in building the distributed system; the actual RL part is just a DQN (+some tricks from Rainbow DQN).
  NB: multi-agent RL arises when the agents interact with each other in the same environment (like Hanabi the card game), while in this case we have multiple agents that collect experiences in parallel.
  Here's a possible design from the <a href="https://openreview.net/pdf?id=H1Dy---0Z" rel="nofollow noreferrer">Ape-X paper</a>:
  <a href="https://i.sstatic.net/a6zb7.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/a6zb7.png" alt="enter image description here" /></a></p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-11-26T20:25:23.470Z
Answers.Text: Thanks @kaiwenw for your response and for reference! From my continued reading I was also leaning to towards this definition of dsitributed RL as well. In this case, does distributed RL assume parallel episodes "start", with different learners, at the same time, or is that just an aspect of the design, and not related? For example, where one episode starts with a learner, then say a minute later another episode starts with another new learner (with the original still running), and they are both updating a common policy.
Answers.UserDisplayName: RL_NOOB
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-11-26T22:04:53.687Z
Answers.Text: Excuse me, I meant actor, not learner in the above comment.
Answers.UserDisplayName: RL_NOOB
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-11-26T23:44:40.050Z
Answers.Text: @RL_NOOB everything can be asynchronous, and should be to minimize wait times; you can have some "master", perhaps the learner, decide when to share weights with the actors, or you can have a separate weight server that actor threads will periodically fetch from (in that case you have no direct interaction between learner and actor threads)
Answers.UserDisplayName: kaiwenw
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-11-26T07:23:43.607Z
Answers.LastActivityDate: 2020-11-26T07:23:43.607Z
Answers.UserDisplayName: kaiwenw
Body: <p>I would be grateful for some guidance on a RL problem I am trying to solve where multiple RL agents use a common/global policy at the initial state of an episode in the RL Environment, and then update this common/shared policy once the episode is completed.</p>
  <p>Below is an example of the problem scenario:</p>
  <ul>
  <li>An alert triggers a RL agent to execute a &quot;episode&quot; in the Environment</li>
  <li>Multiple alerts (e.g., episodes) can occur at the same time, or, one alert may still be being processed (e.g., the episode has not finished) before another alert is triggered (e.g., another episode begins).</li>
  </ul>
  <p>Below are the conditions of the Environment and desired behaviour of the RL Agent:</p>
  <ul>
  <li>Multiple episodes can run at once (e.g., another episode starts before another finishes).</li>
  <li>For each episode a &quot;instance&quot; of the RL agent uses the latest version of a common policy.</li>
  <li>After each episode the RL agent updates the common policy.</li>
  <li>Common policy updates are &quot;queued&quot; using versioning in code to prevent race conditions.</li>
  </ul>
  <p>Q: How can multiple RL agents in this case use a common policy at the beginning of an episode and then update a common policy after completing it? - All I have found are discussions related to Q-Learning, where agents can update a shared Q-table, or later update a &quot;global&quot; Q-table without any examples of how this can be achieved and whether there are also methods for other approaches such as TD rather than only Q-Learning, for <a href="https://www.researchgate.net/post/How_can_I_solve_the_problem_of_multi-agent_learning_while_agents_interact_just_through_the_environment_by_using_a_single-agent_Q-Learning_algorithm" rel="nofollow noreferrer">example</a></p>
  <p>Q: Does this sound like a traditional multi-agent scenario, at least conceptually? If so, how might one go about implementing this, any examples would be really helpful.</p>
  <p>Any help on this is greatly appreciated!</p>
  <p>EDIT:</p>
  <p>Since doing more investigation I have found this reference on Mathworks:
  <a href="https://www.mathworks.com/help/reinforcement-learning/ug/train-dqn-agent-for-lane-keeping-assist-using-parallel-computing.html" rel="nofollow noreferrer">Link</a>, which is similar to the above problem, but not exact.</p>
  
Tags: <reinforcement-learning><environment><policies>
Title: How to use and update a shared/global policy between Reinforcement Learning Agents
UserDisplayName: RL_NOOB
ThreadText: # How to use and update a shared/global policy between Reinforcement Learning Agents
  
  (Asked by: username_0 on 2020-11-25)
  
  I would be grateful for some guidance on a RL problem I am trying to solve where multiple RL agents use a common/global policy at the initial state of an episode in the RL Environment, and then update this common/shared policy once the episode is completed.
  
  Below is an example of the problem scenario:
  
  * An alert triggers a RL agent to execute a "episode" in the Environment
  * Multiple alerts (e.g., episodes) can occur at the same time, or, one alert may still be being processed (e.g., the episode has not finished) before another alert is triggered (e.g., another episode begins).
  
  Below are the conditions of the Environment and desired behaviour of the RL Agent:
  
  * Multiple episodes can run at once (e.g., another episode starts before another finishes).
  * For each episode a "instance" of the RL agent uses the latest version of a common policy.
  * After each episode the RL agent updates the common policy.
  * Common policy updates are "queued" using versioning in code to prevent race conditions.
  
  Q: How can multiple RL agents in this case use a common policy at the beginning of an episode and then update a common policy after completing it? - All I have found are discussions related to Q-Learning, where agents can update a shared Q-table, or later update a "global" Q-table without any examples of how this can be achieved and whether there are also methods for other approaches such as TD rather than only Q-Learning, for [example](https://www.researchgate.net/post/How_can_I_solve_the_problem_of_multi-agent_learning_while_agents_interact_just_through_the_environment_by_using_a_single-agent_Q-Learning_algorithm)
  
  Q: Does this sound like a traditional multi-agent scenario, at least conceptually? If so, how might one go about implementing this, any examples would be really helpful.
  
  Any help on this is greatly appreciated!
  
  EDIT:
  
  Since doing more investigation I have found this reference on Mathworks:
  [Link](https://www.mathworks.com/help/reinforcement-learning/ug/train-dqn-agent-for-lane-keeping-assist-using-parallel-computing.html), which is similar to the above problem, but not exact.
  
  **Tags:** reinforcement-learning, environment, policies
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2020-11-26. Score: 0 | Accepted answer
  
  This sounds like distributed RL, and most of the work goes in building the distributed system; the actual RL part is just a DQN (+some tricks from Rainbow DQN).
  NB: multi-agent RL arises when the agents interact with each other in the same environment (like Hanabi the card game), while in this case we have multiple agents that collect experiences in parallel.
  Here's a possible design from the [Ape-X paper](https://openreview.net/pdf?id=H1Dy---0Z):
  [![enter image description here](https://i.sstatic.net/a6zb7.png)](https://i.sstatic.net/a6zb7.png)
  
  > **Comments:**
  > **username_0**: Thanks @kaiwenw for your response and for reference! From my continued reading I was also leaning to towards this definition of dsitributed RL as well. In this case, does distributed RL assume parallel episodes "start", with different learners, at the same time, or is that just an aspect of the design, and not related? For example, where one episode starts with a learner, then say a minute later another episode starts with another new learner (with the original still running), and they are both updating a common policy.  
  > **username_0**: Excuse me, I meant actor, not learner in the above comment.  
  > **username_1**: @RL_NOOB everything can be asynchronous, and should be to minimize wait times; you can have some "master", perhaps the learner, decide when to share weights with the actors, or you can have a separate weight server that actor threads will periodically fetch from (in that case you have no direct interaction between learner and actor threads)  
  
  ---
  
  