Answers.Body: <p>I didn't find a way to improve state design, but I did find a workaround in making my PG network modular.</p>
  <p>I simply separated my PG network into two parts -- one taking in just the flattened grid matrix part from the aforementioned state, and the other taking in just the relative goal locations. Then I concatenated the outputs from the two sub-networks and passed them through a softmax layer to get the final policy.</p>
  <p>If you want more details you can check out my codes <a href="https://github.com/szhaovas/MARLNavGrid" rel="nofollow noreferrer">here</a> (The relevant codes are in <code>MARL_PolicyGradient.py</code>, <code>MARL_env.py</code>, and <code>MARL_networks.py</code>). Good luck!</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-12-27T05:39:40.863Z
Answers.LastActivityDate: 2021-12-27T05:39:40.863Z
Answers.UserDisplayName: definitelynotabot
Body: <p>I'm trying to do multi-agent reinforcement learning on the grid world navigation task where multiple agents try to collectively reach multiple goals while avoiding collisions with stationary obstacles and each other. As a constraint, each agent can only see within a limited range around itself.</p>
  <p>So on a high level, the state of each agent should contain both information to help it avoid collision and information to guide it towards the goals. I'm thinking of implementing the former by including into the agent's state a matrix consisted of the grid cells surrounding the agent, which would show the agent where the obstacles are. However, I'm not sure how to include goal navigation information on top of this matrix. Currently I just flatten the matrix and append all relative goal locations at the end, and use this as the state.</p>
  <p>For example, for a grid world as shown below (0 means empty cell, 1 means agents, 2 means obstacles, and 3 represents goals):</p>
  <pre><code>[[0 0 0 0 0 0 2 2 0 0]
   [0 0 0 0 0 0 0 0 0 0]
   [0 0 2 2 0 0 0 0 0 0]
   [0 3 2 2 0 0 0 0 0 2]
   [0 0 0 0 0 0 0 0 0 2]
   [0 0 0 0 1 0 0 0 0 2]
   [2 0 0 0 0 2 2 0 3 0]
   [2 0 0 0 0 2 2 0 0 0]
   [0 0 0 0 0 0 0 0 0 0]
   [0 0 2 0 0 1 0 0 0 0]]
  </code></pre>
  <p>The agent at row5 col4 sees the following cells that are within distance1 around it:</p>
  <pre><code>[[0. 0. 0.]
   [0. 1. 0.]
   [0. 0. 2.]]
  </code></pre>
  <p>flattened, the matrix becomes:</p>
  <pre><code>[0,0,0,0,1,0,0,0,2]
  </code></pre>
  <p>The location of the goal at row3 col1 relative to the aforementioned agent is (5-3=2, 4-1=3)</p>
  <p>The location of the goal at row6 col8 relative to the aforementioned agent is (5-6=-1, 4-8=-4)</p>
  <p>So after appending the relative locations, the state of the agent becomes:</p>
  <pre><code>[0,0,0,0,1,0,0,0,2,2,3,-1,-4]
  </code></pre>
  <p>(Similar process for the other agent)</p>
  <p>Is this a reasonable way of designing the state? My primary concern is that the flattened grid matrix and the relative goal locations need to be handled quite differently, but it can be hard for RL to figure out the difference.</p>
  <p>Thanks in advance!</p>
  <p>Edit: To validate my concern, I trained an agent using PG REINFORCE algorithm. As I feared, the agent learned to avoid obstacles but otherwise just moved randomly without navigating towards the goals.</p>
  
Tags: <reinforcement-learning><policy-gradients>
Title: How to mix grid matrix and explicit values when designing RL state?
UserDisplayName: definitelynotabot
ThreadText: # How to mix grid matrix and explicit values when designing RL state?
  
  (Asked by: username_0 on 2021-12-24)
  
  I'm trying to do multi-agent reinforcement learning on the grid world navigation task where multiple agents try to collectively reach multiple goals while avoiding collisions with stationary obstacles and each other. As a constraint, each agent can only see within a limited range around itself.
  
  So on a high level, the state of each agent should contain both information to help it avoid collision and information to guide it towards the goals. I'm thinking of implementing the former by including into the agent's state a matrix consisted of the grid cells surrounding the agent, which would show the agent where the obstacles are. However, I'm not sure how to include goal navigation information on top of this matrix. Currently I just flatten the matrix and append all relative goal locations at the end, and use this as the state.
  
  For example, for a grid world as shown below (0 means empty cell, 1 means agents, 2 means obstacles, and 3 represents goals):
  
  ```
  [[0 0 0 0 0 0 2 2 0 0]
   [0 0 0 0 0 0 0 0 0 0]
   [0 0 2 2 0 0 0 0 0 0]
   [0 3 2 2 0 0 0 0 0 2]
   [0 0 0 0 0 0 0 0 0 2]
   [0 0 0 0 1 0 0 0 0 2]
   [2 0 0 0 0 2 2 0 3 0]
   [2 0 0 0 0 2 2 0 0 0]
   [0 0 0 0 0 0 0 0 0 0]
   [0 0 2 0 0 1 0 0 0 0]]
  
  ```
  
  The agent at row5 col4 sees the following cells that are within distance1 around it:
  
  ```
  [[0. 0. 0.]
   [0. 1. 0.]
   [0. 0. 2.]]
  
  ```
  
  flattened, the matrix becomes:
  
  ```
  [0,0,0,0,1,0,0,0,2]
  
  ```
  
  The location of the goal at row3 col1 relative to the aforementioned agent is (5-3=2, 4-1=3)
  
  The location of the goal at row6 col8 relative to the aforementioned agent is (5-6=-1, 4-8=-4)
  
  So after appending the relative locations, the state of the agent becomes:
  
  ```
  [0,0,0,0,1,0,0,0,2,2,3,-1,-4]
  
  ```
  
  (Similar process for the other agent)
  
  Is this a reasonable way of designing the state? My primary concern is that the flattened grid matrix and the relative goal locations need to be handled quite differently, but it can be hard for RL to figure out the difference.
  
  Thanks in advance!
  
  Edit: To validate my concern, I trained an agent using PG REINFORCE algorithm. As I feared, the agent learned to avoid obstacles but otherwise just moved randomly without navigating towards the goals.
  
  **Tags:** reinforcement-learning, policy-gradients
  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_0 on 2021-12-27. Score: 2 | Accepted answer
  
  I didn't find a way to improve state design, but I did find a workaround in making my PG network modular.
  
  I simply separated my PG network into two parts -- one taking in just the flattened grid matrix part from the aforementioned state, and the other taking in just the relative goal locations. Then I concatenated the outputs from the two sub-networks and passed them through a softmax layer to get the final policy.
  
  If you want more details you can check out my codes [here](https://github.com/szhaovas/MARLNavGrid) (The relevant codes are in `MARL_PolicyGradient.py`, `MARL_env.py`, and `MARL_networks.py`). Good luck!
  
  
  ---
  
  