Body: <p>I'm modeling a problem using Reinforcement Learning (RL). Formally, I have two agents: one of them is the one that I have to program and model, the other one is unpredictable (random). With unpredictable, I mean that I can't even define a set of possible actions for it.</p>
  <p>Given that, I thought of modeling the scenario as a single-agent scenario in which the environment changes even without the intervention of my agent. Can I do it?</p>
  <p>Reformulating: in RL, should the environment be modified (and then change) only by the agent in a single-agent scenario? Is it possible that even if the agent makes no action the environment changes or should I add some entity (the random agent) to my model?</p>
  <p>I tried to find the answer in the book &quot;Reinforcement Learning: An Introduction&quot;, 2nd edition, by Sutton and Barto, but I found nothing answering the question in the introductory chapters (1 and 3). With &quot;I found nothing&quot;, I mean that the authors neither say that I can do it, nor that I can't.</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2022-03-07T20:07:42.067Z
Comments.Text: For the second agent, have you been set any task such as predicting its expected return? How are time steps arranged - do both agents act at once, or take turns, or is it more complicated? If you cannot define the action set of the second agent, can you at least still measure the changes it makes to the environment during training?
Comments.UserDisplayName: Neil Slater
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2022-03-08T17:34:19.530Z
Comments.Text: @NeilSlater following the order in which you asked:
  - no task set for the second agent
  - random time steps (I could see one action of one agent, the n actions of the other, m of the first and so on)
  - I cannot see all the changes made by the second agent
Comments.UserDisplayName: Francesco
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2022-03-08T18:26:46.443Z
Comments.Text: Thanks for the updaate. When you say "cannot see the changes", does that mean the state includes hidden information (related to what the other agent did) that is important and can affect the success of the first agent? E.g. is it like a poker game where you don't know how many draws the other player has made? Or is everything you need to know for success observable in the state description? E.g. is it like Blackjack where you know your and the dealer's draws, but don't know what any other players have done (so you can generally ignore whatever cards they have drawn)?
Comments.UserDisplayName: Neil Slater
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2022-03-09T09:58:39.903Z
Comments.Text: Thank you for your interest. The model is about a defender of a network and an attacker of the network, so the attacker may be inside the network and it could have modified something, but the defender may not be aware of it. In other words, the other agent (attacker) could have acted or not - if it made an action, I can't be 100% sure to see that it acted and what it did.
Comments.UserDisplayName: Francesco
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2022-03-09T11:28:46.587Z
Comments.Text: So, to clarify, the observer can see a state that might include network stats, machine stats etc, but has no idea whether any of those stats have been influenced by the attacker, who is in essence invisible? This is more like Poker, unfortunately. Is the first agent supposed to actively defend against adversarial attacks (that it cannot see), or just raise the alarm or simply shut services down?
Comments.UserDisplayName: Neil Slater
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2022-03-09T11:35:18.700Z
Comments.Text: The agent is supposed to actively defend. One of the goals of the defender is: given the state (network stats, machine stats, etc), which action should I take? Implicitly, it has to understand if this state is influenced by the attacker. Therefore, some actions of the attackers will be visible and some others not, because some actions are noisier than others.
Comments.UserDisplayName: Francesco
Tags: <reinforcement-learning><models>
Title: Can the environment change even without the intervention of the agent in Reinforcement Learning?
UserDisplayName: Francesco
ThreadText: # Can the environment change even without the intervention of the agent in Reinforcement Learning?
  
  (Asked by: username_0 on 2022-03-07)
  
  I'm modeling a problem using Reinforcement Learning (RL). Formally, I have two agents: one of them is the one that I have to program and model, the other one is unpredictable (random). With unpredictable, I mean that I can't even define a set of possible actions for it.
  
  Given that, I thought of modeling the scenario as a single-agent scenario in which the environment changes even without the intervention of my agent. Can I do it?
  
  Reformulating: in RL, should the environment be modified (and then change) only by the agent in a single-agent scenario? Is it possible that even if the agent makes no action the environment changes or should I add some entity (the random agent) to my model?
  
  I tried to find the answer in the book "Reinforcement Learning: An Introduction", 2nd edition, by Sutton and Barto, but I found nothing answering the question in the introductory chapters (1 and 3). With "I found nothing", I mean that the authors neither say that I can do it, nor that I can't.
  
  **Tags:** reinforcement-learning, models
  
  > **Comments:**
  > **username_1**: For the second agent, have you been set any task such as predicting its expected return? How are time steps arranged - do both agents act at once, or take turns, or is it more complicated? If you cannot define the action set of the second agent, can you at least still measure the changes it makes to the environment during training?  
  > **username_0**: @username_1 following the order in which you asked: - no task set for the second agent - random time steps (I could see one action of one agent, the n actions of the other, m of the first and so on) - I cannot see all the changes made by the second agent  
  > **username_1**: Thanks for the updaate. When you say "cannot see the changes", does that mean the state includes hidden information (related to what the other agent did) that is important and can affect the success of the first agent? E.g. is it like a poker game where you don't know how many draws the other player has made? Or is everything you need to know for success observable in the state description? E.g. is it like Blackjack where you know your and the dealer's draws, but don't know what any other players have done (so you can generally ignore whatever cards they have drawn)?  
  > **username_0**: Thank you for your interest. The model is about a defender of a network and an attacker of the network, so the attacker may be inside the network and it could have modified something, but the defender may not be aware of it. In other words, the other agent (attacker) could have acted or not - if it made an action, I can't be 100% sure to see that it acted and what it did.  
  > **username_1**: So, to clarify, the observer can see a state that might include network stats, machine stats etc, but has no idea whether any of those stats have been influenced by the attacker, who is in essence invisible? This is more like Poker, unfortunately. Is the first agent supposed to actively defend against adversarial attacks (that it cannot see), or just raise the alarm or simply shut services down?  
  > **username_0**: The agent is supposed to actively defend. One of the goals of the defender is: given the state (network stats, machine stats, etc), which action should I take? Implicitly, it has to understand if this state is influenced by the attacker. Therefore, some actions of the attackers will be visible and some others not, because some actions are noisier than others.  
  
  ---
  
  ## 0 answers
  
  