Answers.Body: <p>I depends on your overall model architecture (and problem specification). As I understand it, you take the observations of all agents together and feed it into one model, a central controller, which then predicts the action per available agent.</p>
  <p>I believe that this varying number of applicable observations (depending on the number of currently present agents) is what you mean by <em>changing state spaces</em>?</p>
  <p>One option that works in these kinds of settings is to use padding. So, a fixed number of individual observations gets predetermined before training, being set to the average number of agents expected to be present in the environment at any timestep.</p>
  <p>Then, during training, local, individual observations of all agents (for the current time step) get stacked together, resulting in an overall observation being fed into central controller.</p>
  <p>Then, the controller predicts (concurrently/in a single pass) one action (or multiple, depends on what you want it to predict per agent...) per agent.</p>
  <p>In the case that the number of present agents currently (=in a given time step) is smaller than the predetermined number of expected agents, then padding is used to fill the empty observation slots by zero-pseudo-observations. The quality of the predicted actions is consequently only assessed for the present agents, neglecting predictions for absent agents, indicated by their empty observation slots.
  If the number of controlled agents happens to exceed the expected maximal number of agents present in the simulation, excess agents are selected at random and randomly controlled for the next time step. Then, this (randomized control of a random selection of some agents) continues until no excess agents are present in the simulation anymore. Alternatively, if available, other reasonable controllers might be used to temporarily steer excess agents.</p>
  <p>I have seen researchers doing this in the context of traffic control, where a variable number of cars was controlled by a central controller.
  See <a href="http://proceedings.mlr.press/v87/vinitsky18a/vinitsky18a.pdf" rel="nofollow noreferrer">this paper</a> (Section 3.2) for an example of the aforementioned technique.</p>
  <p>Alternatively, there exist approaches where each agent gets a periodically updated local copy of some central controller and then contributes to jointly updating the central controller. <a href="http://proceedings.mlr.press/v80/liang18b/liang18b.pdf" rel="nofollow noreferrer">This paper</a> shortly summarizes a lot of different options and possibilities to choose from in terms of different RL algorithms, where different options might apply depending on the concrete task at hand. Also, the paper discusses different options with respect to algorithms implemented in the python Reinforcement Learning (RL) library RLlib, which might already provide a suitable implementation for a class of algorithm you want to work with.</p>
  <p>Unfortunately, a lot of papers don't reveal explicitly whether their (partly) distributed RL training procedures are more suitable, i.e. converge stably to a good solution, for single-agent environments or equally suitable for multi-agent environments as well.</p>
  <p>When it comes to introducing new agents to the simulation, this depends entirely on your concrete problem at hand. In the context of vehicle control, you could position a vehicle at the start of the simulated road, steer it (safely) a few time steps straight ahead (to initialize its observation space) and then hand over control to the RL controller. For other types of agents (e.g. soccer playing robots etc.) that might look entirely different. In that case, you might have them sitting or standing next to the field and then have them enter the play ground or so. So, that really depends on your task at hand.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-08-28T16:19:48.670Z
Answers.Text: Thanks for your answer. Actually, every agent has a different state. Also,  feeding the state with random information may affect the learning results. Also, The only changes that happened to the state are the size when the number of agents increases or decreases. However, most of the traffic control papers have one goal a global reward that not much affects the training but in my case, I need every agent to get his local reward.
  https://arxiv.org/abs/1908.03761
Answers.UserDisplayName: I_Al-thamary
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-08-29T04:19:13.903Z
Answers.Text: I have another problem where  the random in most cases is larger than the state itself and I have tried it but the results are getting worse. I generate the random depends on the state.
Answers.UserDisplayName: I_Al-thamary
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-08-29T18:18:14.833Z
Answers.Text: In the paper you linked, which method is the one you were trying to implement so far? I definitely see the problem with the approach of 1 "stand-alone" controller per agent when having a variable nr of agents. A more exploratory research direction would be to test how supposedly single-agent distributed RL algorithms (e.g. https://arxiv.org/pdf/1507.04296.pdf) would do when not using different agents in different parallel environments that get trained, but different agents generating data in parallel in the same environment. But I honestly don't have any idea if this would work or not.
Answers.UserDisplayName: Daniel B.
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-08-29T18:29:13.783Z
Answers.Text: I found the paper I mentioned earlier and clarified the answer a bit. I hope this helps.
Answers.UserDisplayName: Daniel B.
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-08-30T06:27:13.560Z
Answers.Text: Thanks again, Padding with zeros improve the results but it is not as expected and I use to genrate the random like this `if n_features > 0:
      statappend = (np.concatenate((np.random.choice(state, size=n_features, p=p), state)))
  else:
      statappend = np.zeros(n_input)
      statappend[:state.shape[0]] = state `
  where the `n_features` is the missing random size.
Answers.UserDisplayName: I_Al-thamary
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-08-30T11:53:45.077Z
Answers.Text: When I said "zero/meaningless random data", I wasn't entirely sure anymore how the authors had described their method in the paper. But now, that I have found the paper, it clearly says they used zero padding. So, I'd recommend sticking to that.  Regarding suboptimal performance, that may have also to do with reasons other than just how you realize group control. For example, maybe a different RL algo would perform better. Or trying different exploration strategies might lead to better results etc. So I am not sure whether subopt. performance is still due to the original issue of this thread.
Answers.UserDisplayName: Daniel B.
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-08-30T12:43:08.903Z
Answers.Text: I'm trying different ways. I already tried both ways and the results are better but it is not as expected because when I make the number of agents constant, it shows a big difference.
Answers.UserDisplayName: I_Al-thamary
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-08-30T19:25:22.727Z
Answers.Text: But actually, I would not be surprised to see results being worse in a condition where the number of agents is variable (given same amount of training). This is simply because it adds to the complexity of the environment each agent has to function in if the number of controlled agents in an environment varies from time to time. It just requires more training to figure out the best policy per number of agents present in the environment if that nr changes (which is all fixed when the nr of agents is fixed). A variable number of agents just 'complexifies' the dynamics of the overall RL system.
Answers.UserDisplayName: Daniel B.
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-08-30T19:51:31.680Z
Answers.Text: You are right and thank you for your answer and comments.
Answers.UserDisplayName: I_Al-thamary
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-09-06T15:08:16.160Z
Answers.Text: Actually, I'm facing a problem with padding with zero when the state is small, and choosing the maximums expected agents changes the results. I already asked the question https://stackoverflow.com/questions/63728800/how-to-deal-with-different-state-space-size-in-reinforcement-learning, I hope that I can found an answer.
Answers.UserDisplayName: I_Al-thamary
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-08-28T15:50:55.973Z
Answers.LastActivityDate: 2020-08-29T18:28:29.923Z
Answers.LastEditDate: 2020-08-29T18:28:29.923Z
Answers.UserDisplayName: Daniel B.
Body: <p>I'm working in <strong>A2C</strong> and I have an environment where there is increasing or decreasing in the number of agents. The action space in the environment will not change but the state will change when new agents join or leave the game.
  I have tried
  <strong>encoder-decoder model with attention</strong> but the problem is that the state and the model will change when the number of agents is changing.
  I also tried <a href="https://github.com/livey/scalable_maddpg/blob/master/env/envs.py" rel="nofollow noreferrer">this way</a> where they use LSTM to get the Q value for the agent but I got this message</p>
  <pre><code>Cannot interpret feed_dict key as Tensor: Tensor Tensor(&quot;state:0&quot;, shape=(137,), dtype=float32) is not an element of this graph.
  </code></pre>
  <p>or error like this because of changing of state size</p>
  <pre><code>ValueError: Cannot feed value of shape (245,) for Tensor 'state:0', which has shape '(161,)'
  </code></pre>
  <p>(1) Are there any reference papers that deal with such a problem?</p>
  <p>(2) What is the best way to deal with the new agents that join or leave the game?</p>
  <p>(3) How to deal with the changing of state space?</p>
  
Tags: <reinforcement-learning><tensorflow><actor-critic-methods><environment><multi-agent-systems>
Title: How to handle a changing in the Reinforcement Learning environment where there is increasing or decreasing in number of agents?
UserDisplayName: I_Al-thamary
ThreadText: # How to handle a changing in the Reinforcement Learning environment where there is increasing or decreasing in number of agents?
  
  (Asked by: username_0 on 2020-08-25)
  
  I'm working in **A2C** and I have an environment where there is increasing or decreasing in the number of agents. The action space in the environment will not change but the state will change when new agents join or leave the game.
  I have tried
  **encoder-decoder model with attention** but the problem is that the state and the model will change when the number of agents is changing.
  I also tried [this way](https://github.com/livey/scalable_maddpg/blob/master/env/envs.py) where they use LSTM to get the Q value for the agent but I got this message
  
  ```
  Cannot interpret feed_dict key as Tensor: Tensor Tensor("state:0", shape=(137,), dtype=float32) is not an element of this graph.
  
  ```
  
  or error like this because of changing of state size
  
  ```
  ValueError: Cannot feed value of shape (245,) for Tensor 'state:0', which has shape '(161,)'
  
  ```
  
  (1) Are there any reference papers that deal with such a problem?
  
  (2) What is the best way to deal with the new agents that join or leave the game?
  
  (3) How to deal with the changing of state space?
  
  **Tags:** reinforcement-learning, tensorflow, actor-critic-methods, environment, multi-agent-systems
  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2020-08-28. Score: 1
  
  I depends on your overall model architecture (and problem specification). As I understand it, you take the observations of all agents together and feed it into one model, a central controller, which then predicts the action per available agent.
  
  I believe that this varying number of applicable observations (depending on the number of currently present agents) is what you mean by *changing state spaces*?
  
  One option that works in these kinds of settings is to use padding. So, a fixed number of individual observations gets predetermined before training, being set to the average number of agents expected to be present in the environment at any timestep.
  
  Then, during training, local, individual observations of all agents (for the current time step) get stacked together, resulting in an overall observation being fed into central controller.
  
  Then, the controller predicts (concurrently/in a single pass) one action (or multiple, depends on what you want it to predict per agent...) per agent.
  
  In the case that the number of present agents currently (=in a given time step) is smaller than the predetermined number of expected agents, then padding is used to fill the empty observation slots by zero-pseudo-observations. The quality of the predicted actions is consequently only assessed for the present agents, neglecting predictions for absent agents, indicated by their empty observation slots.
  If the number of controlled agents happens to exceed the expected maximal number of agents present in the simulation, excess agents are selected at random and randomly controlled for the next time step. Then, this (randomized control of a random selection of some agents) continues until no excess agents are present in the simulation anymore. Alternatively, if available, other reasonable controllers might be used to temporarily steer excess agents.
  
  I have seen researchers doing this in the context of traffic control, where a variable number of cars was controlled by a central controller.
  See [this paper](http://proceedings.mlr.press/v87/vinitsky18a/vinitsky18a.pdf) (Section 3.2) for an example of the aforementioned technique.
  
  Alternatively, there exist approaches where each agent gets a periodically updated local copy of some central controller and then contributes to jointly updating the central controller. [This paper](http://proceedings.mlr.press/v80/liang18b/liang18b.pdf) shortly summarizes a lot of different options and possibilities to choose from in terms of different RL algorithms, where different options might apply depending on the concrete task at hand. Also, the paper discusses different options with respect to algorithms implemented in the python Reinforcement Learning (RL) library RLlib, which might already provide a suitable implementation for a class of algorithm you want to work with.
  
  Unfortunately, a lot of papers don't reveal explicitly whether their (partly) distributed RL training procedures are more suitable, i.e. converge stably to a good solution, for single-agent environments or equally suitable for multi-agent environments as well.
  
  When it comes to introducing new agents to the simulation, this depends entirely on your concrete problem at hand. In the context of vehicle control, you could position a vehicle at the start of the simulated road, steer it (safely) a few time steps straight ahead (to initialize its observation space) and then hand over control to the RL controller. For other types of agents (e.g. soccer playing robots etc.) that might look entirely different. In that case, you might have them sitting or standing next to the field and then have them enter the play ground or so. So, that really depends on your task at hand.
  
  > **Comments:**
  > **username_0**: Thanks for your answer. Actually, every agent has a different state. Also,  feeding the state with random information may affect the learning results. Also, The only changes that happened to the state are the size when the number of agents increases or decreases. However, most of the traffic control papers have one goal a global reward that not much affects the training but in my case, I need every agent to get his local reward. https://arxiv.org/abs/1908.03761  
  > **username_0**: I have another problem where  the random in most cases is larger than the state itself and I have tried it but the results are getting worse. I generate the random depends on the state.  
  > **username_1**: In the paper you linked, which method is the one you were trying to implement so far? I definitely see the problem with the approach of 1 "stand-alone" controller per agent when having a variable nr of agents. A more exploratory research direction would be to test how supposedly single-agent distributed RL algorithms (e.g. https://arxiv.org/pdf/1507.04296.pdf) would do when not using different agents in different parallel environments that get trained, but different agents generating data in parallel in the same environment. But I honestly don't have any idea if this would work or not.  
  > **username_1**: I found the paper I mentioned earlier and clarified the answer a bit. I hope this helps.  
  > **username_0**: Thanks again, Padding with zeros improve the results but it is not as expected and I use to genrate the random like this `if n_features > 0:     statappend = (np.concatenate((np.random.choice(state, size=n_features, p=p), state))) else:     statappend = np.zeros(n_input)     statappend[:state.shape[0]] = state ` where the `n_features` is the missing random size.  
  > **username_1**: When I said "zero/meaningless random data", I wasn't entirely sure anymore how the authors had described their method in the paper. But now, that I have found the paper, it clearly says they used zero padding. So, I'd recommend sticking to that.  Regarding suboptimal performance, that may have also to do with reasons other than just how you realize group control. For example, maybe a different RL algo would perform better. Or trying different exploration strategies might lead to better results etc. So I am not sure whether subopt. performance is still due to the original issue of this thread.  
  > **username_0**: I'm trying different ways. I already tried both ways and the results are better but it is not as expected because when I make the number of agents constant, it shows a big difference.  
  > **username_1**: But actually, I would not be surprised to see results being worse in a condition where the number of agents is variable (given same amount of training). This is simply because it adds to the complexity of the environment each agent has to function in if the number of controlled agents in an environment varies from time to time. It just requires more training to figure out the best policy per number of agents present in the environment if that nr changes (which is all fixed when the nr of agents is fixed). A variable number of agents just 'complexifies' the dynamics of the overall RL system.  
  > **username_0**: You are right and thank you for your answer and comments.  
  > **username_0**: Actually, I'm facing a problem with padding with zero when the state is small, and choosing the maximums expected agents changes the results. I already asked the question https://stackoverflow.com/questions/63728800/how-to-deal-with-different-state-space-size-in-reinforcement-learning, I hope that I can found an answer.  
  
  ---
  
  