Answers.Body: <p>You seem to use 0.0 for &quot;free&quot;, which is exactly in the middle between the two &quot;danger&quot; values -1.0 and +1.0. This makes it impossible to separate danger vs free with just a linear mapping.</p>
  <p>So every of your inputs requires learning to solve a XOR problem first, it seems. This means multiple layers are a must to make progress. This should be learnable given enough time, but... did you intentionally make the problem so very hard?</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-10-12T17:12:31.243Z
Answers.LastActivityDate: 2023-10-12T17:12:31.243Z
Answers.UserDisplayName: maxy
Answers.Body: <p>When you trained your agents separately they never saw squares with opposing values (1/-1). So agents don't really know what to expect from visiting that square. 
  I'd try adding (1/-1) to the condition on which you base your squares availability. Also try increasing the reward. It's hard to give suggestions without looking at the code.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-07-30T14:27:23.490Z
Answers.Text: I could add a wall of the opposite number that would kill them on single agent, would that adapt them to the other value? even if it's not a snake but a wall?
  I heard it's best to keep the rewards between -1 and 1, what would you suggest?
Answers.UserDisplayName: Milky
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-07-30T13:20:48.837Z
Answers.LastActivityDate: 2019-07-30T13:20:48.837Z
Answers.UserDisplayName: A_P
Body: <p>So I have 2 models trained with the DQN algorithm that I want to train in a multi-agent environment to see how they react with each other. The models were trained in an environment consisting of 0's and 1's (-1's for the other model)where 1 means that square is filled and 0 is empty. It is a map filling environment, where at each step the agent can move up, down, left or right and for each step, it stays alive without turning into itself (1 or -1 for the other) or the boundary of the environment it gets 0.005 rewards and for "dying" it gets -1. You can think of the player as in the game Tron, where it just leaves a trail behind. I stack the last 4 frames on top of each other so it knows which end is the "head". With a single agent, after training, I didn't get an optimal model which uses all the squares but it does manage to fill about 30% of the environment, which I think is the limit for this algorithm (let me know if you have thoughts on this)</p>
  
  <p>Now, I put the two models in one environment where there are two players, one represented with 1's and the other with -1s. As one model is trained with -1's and the other with 1's I thought they could find their own player, however even before training if I just run the models on the environment without any exploration, they seem to affect the actions of each other. One just goes straight and dies and the other just turns once then dies at the wall (whereas in a single-agent environment these 2 models can fill about 30%). And if I do training, they just diverge to this exact behavior from random without seemingly not learning anything. So, I just wanted to ask is there anything wrong about my approach with the representation of the players (1 and -1) because I thought they would just play as they did in the single-agent environment but they don't and I couldn't get them to learn anything </p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2023-10-12T17:05:37.867Z
Comments.Text: What inputs do you use, the whole game area at once or just the local neighbourhood?
Comments.UserDisplayName: maxy
Tags: <dqn><multi-agent-systems><environment>
Title: How to represent players in a multi agent environment so each model can distinguish its own player
UserDisplayName: Milky
ThreadText: # How to represent players in a multi agent environment so each model can distinguish its own player
  
  (Asked by: username_0 on 2019-07-29)
  
  So I have 2 models trained with the DQN algorithm that I want to train in a multi-agent environment to see how they react with each other. The models were trained in an environment consisting of 0's and 1's (-1's for the other model)where 1 means that square is filled and 0 is empty. It is a map filling environment, where at each step the agent can move up, down, left or right and for each step, it stays alive without turning into itself (1 or -1 for the other) or the boundary of the environment it gets 0.005 rewards and for "dying" it gets -1. You can think of the player as in the game Tron, where it just leaves a trail behind. I stack the last 4 frames on top of each other so it knows which end is the "head". With a single agent, after training, I didn't get an optimal model which uses all the squares but it does manage to fill about 30% of the environment, which I think is the limit for this algorithm (let me know if you have thoughts on this)
  
  Now, I put the two models in one environment where there are two players, one represented with 1's and the other with -1s. As one model is trained with -1's and the other with 1's I thought they could find their own player, however even before training if I just run the models on the environment without any exploration, they seem to affect the actions of each other. One just goes straight and dies and the other just turns once then dies at the wall (whereas in a single-agent environment these 2 models can fill about 30%). And if I do training, they just diverge to this exact behavior from random without seemingly not learning anything. So, I just wanted to ask is there anything wrong about my approach with the representation of the players (1 and -1) because I thought they would just play as they did in the single-agent environment but they don't and I couldn't get them to learn anything
  
  **Tags:** dqn, multi-agent-systems, environment
  
  > **Comments:**
  > **username_1**: What inputs do you use, the whole game area at once or just the local neighbourhood?  
  
  ---
  
  ## 2 answers
  
  ### Answer by username_1 on 2023-10-12. Score: 0
  
  You seem to use 0.0 for "free", which is exactly in the middle between the two "danger" values -1.0 and +1.0. This makes it impossible to separate danger vs free with just a linear mapping.
  
  So every of your inputs requires learning to solve a XOR problem first, it seems. This means multiple layers are a must to make progress. This should be learnable given enough time, but... did you intentionally make the problem so very hard?
  
  
  ---
  
  ### Answer by username_2 on 2019-07-30. Score: 0
  
  When you trained your agents separately they never saw squares with opposing values (1/-1). So agents don't really know what to expect from visiting that square.
  I'd try adding (1/-1) to the condition on which you base your squares availability. Also try increasing the reward. It's hard to give suggestions without looking at the code.
  
  > **Comments:**
  > **username_0**: I could add a wall of the opposite number that would kill them on single agent, would that adapt them to the other value? even if it's not a snake but a wall? I heard it's best to keep the rewards between -1 and 1, what would you suggest?  
  
  ---
  
  