Answers.Body: <p>Your environment should handle it, not the agent.</p>
  <p>If I'm playing a game, and my player dies, then if I press buttons, nothing happens... it's not the player supposed to stop playing, it's the game that is supposed to stop you from doing anything</p>
  <p>In other words, the network can still observe whatever is supposed to be seeing, it's the environment and your training algorithm that should take special cautions for such corner case</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-09-27T15:11:16.917Z
Answers.Text: I guess you misunderstand my question. I mean how to handle observation of dead agents. Suppose we have 3 agents a, b, c. Agent a's obs is [a's state, b's state, c's state].  If b is destroyed, how should the a's obs be? By the way, the environment is created by myself, so how should the environment give the agent observation when some other agents die. And this is not a corner case, it is expected to see agents destroyed by others.
Answers.UserDisplayName: zhixin
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-09-27T14:31:32.550Z
Answers.LastActivityDate: 2023-09-27T14:31:32.550Z
Answers.UserDisplayName: Alberto
Answers.Body: <p>A dead agent is a state of that agent, assuming the episode continues.</p>
  <p>Using the existing state or observation vector, and adding special values should work, but you could be making extra work for any value function approximator. The usual way to solve this is to expand the observation vector and add e.g. 1 for alive and 0 for dead, as a new feature.</p>
  <p>If the agent is also removed from any possible interactions, you should also set other variables to some standard fixed values, because there should be only one observation vector matching a dead agent. Exceptions to that rule might might apply for static features describing a agent in general terms, such team affiliation (knowing it's your own or opposing team member that is out of action is important knowledge for predicting outcomes).</p>
  <p>This also has the advantage of supporting extensions to the environment as your agents become more sophisticated. For example, if instead of being removed from the map, visiting a dead agent's location allowed scavenging resources (you may also need more observation features for whether this has been done already).</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-09-28T02:13:02.533Z
Answers.Text: What do you mean by "making extra work for any value function approximator"? Do you think expanding the observation vector is better choice than adding special values?
Answers.UserDisplayName: zhixin
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-09-28T06:45:04.577Z
Answers.Text: @zhixin I mean that the function shape is usually more difficult to learn if it contains a big change at some special values, where otherwise it would have been simpler and smoother. An NN can learn the shape of e.g. a circle easily, but would find a circle with a single spike on one side slightly harder to learn.
Answers.UserDisplayName: Neil Slater
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-09-27T18:33:04.020Z
Answers.LastActivityDate: 2023-09-27T18:42:29.510Z
Answers.LastEditDate: 2023-09-27T18:42:29.510Z
Answers.UserDisplayName: Neil Slater
Body: <p>I try to implement deep reinforcement learning on a defender-vs-attacker problem, where agents can be destroyed by enemies. <em>I am coding both the environment and the RL algorithm</em>. The agent can observe his own state and other's. As we know, the input size of a network should be constant. Here comes the question: <strong>how to handle the observation of a dead agent?</strong> i.e., if agent1 dies, what should be the agent2's observation of agent1?</p>
  <p>My first thought is to set the state to be zero. But I don't think it is a good idea. let's take the state of a agent as: [x, y, theta] and normalize it to [0, 1]. Apparently [0, 0, 0] is also a valid state. If we set the state to be other strange numbers like -1, How can the network realized this is not a normal input of alive agent?</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2023-09-28T01:35:46.450Z
Comments.Text: How to handle the dead agent? Borrow him with honors. Or just keep a track on a separate variable (just guessing)
Comments.UserDisplayName: Ernesto Iglesias
Tags: <deep-rl>
Title: How to handle the dead agent in multi-agent environment?
UserDisplayName: zhixin
ThreadText: # How to handle the dead agent in multi-agent environment?
  
  (Asked by: username_0 on 2023-09-27)
  
  I try to implement deep reinforcement learning on a defender-vs-attacker problem, where agents can be destroyed by enemies. *I am coding both the environment and the RL algorithm*. The agent can observe his own state and other's. As we know, the input size of a network should be constant. Here comes the question: **how to handle the observation of a dead agent?** i.e., if agent1 dies, what should be the agent2's observation of agent1?
  
  My first thought is to set the state to be zero. But I don't think it is a good idea. let's take the state of a agent as: [x, y, theta] and normalize it to [0, 1]. Apparently [0, 0, 0] is also a valid state. If we set the state to be other strange numbers like -1, How can the network realized this is not a normal input of alive agent?
  
  **Tags:** deep-rl
  
  > **Comments:**
  > **username_1**: How to handle the dead agent? Borrow him with honors. Or just keep a track on a separate variable (just guessing)  
  
  ---
  
  ## 2 answers
  
  ### Answer by username_3 on 2023-09-27. Score: 2 | Accepted answer
  
  A dead agent is a state of that agent, assuming the episode continues.
  
  Using the existing state or observation vector, and adding special values should work, but you could be making extra work for any value function approximator. The usual way to solve this is to expand the observation vector and add e.g. 1 for alive and 0 for dead, as a new feature.
  
  If the agent is also removed from any possible interactions, you should also set other variables to some standard fixed values, because there should be only one observation vector matching a dead agent. Exceptions to that rule might might apply for static features describing a agent in general terms, such team affiliation (knowing it's your own or opposing team member that is out of action is important knowledge for predicting outcomes).
  
  This also has the advantage of supporting extensions to the environment as your agents become more sophisticated. For example, if instead of being removed from the map, visiting a dead agent's location allowed scavenging resources (you may also need more observation features for whether this has been done already).
  
  > **Comments:**
  > **username_0**: What do you mean by "making extra work for any value function approximator"? Do you think expanding the observation vector is better choice than adding special values?  
  > **username_3**: @username_0 I mean that the function shape is usually more difficult to learn if it contains a big change at some special values, where otherwise it would have been simpler and smoother. An NN can learn the shape of e.g. a circle easily, but would find a circle with a single spike on one side slightly harder to learn.  
  
  ---
  
  ### Answer by username_2 on 2023-09-27. Score: 0
  
  Your environment should handle it, not the agent.
  
  If I'm playing a game, and my player dies, then if I press buttons, nothing happens... it's not the player supposed to stop playing, it's the game that is supposed to stop you from doing anything
  
  In other words, the network can still observe whatever is supposed to be seeing, it's the environment and your training algorithm that should take special cautions for such corner case
  
  > **Comments:**
  > **username_0**: I guess you misunderstand my question. I mean how to handle observation of dead agents. Suppose we have 3 agents a, b, c. Agent a's obs is [a's state, b's state, c's state].  If b is destroyed, how should the a's obs be? By the way, the environment is created by myself, so how should the environment give the agent observation when some other agents die. And this is not a corner case, it is expected to see agents destroyed by others.  
  
  ---
  
  