Body: <p>How to handle a problem where the agents' actions are changing the environment? For instance, a trading agent can hypothetically make a huge transaction that is enough to push the stock prices to go up or down.</p>
  <p>Another example could be an electric grid environment. Assume a sufficient number of large consumers utilize an RL agent in their energy management system. If these agents happen to have similar trained policies they might collectively affect stable operation of the electric grid and even lead to a blackout.</p>
  <p>Are there specific mechanisms, methods or algorithms to handle this type environments?</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2024-07-04T08:48:10.037Z
Comments.Text: I'm not sure what you mean, are you not just referring to transitioning to a new state? ideally the state space would encapsulate all possible states of the environment, so there's not really a notion of the environment 'changing' with the agent's actions.
Comments.UserDisplayName: David
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2024-07-04T10:53:20.937Z
Comments.Text: @David I think this is an issue of what is being modelled during training. The two scenarios presented imply that the agents are trained against a "large, robust" environment where their actions don't usually cause the kinds of changes that the OP is concerned about. Modelling such effects in a training environment, or repeating online training enough times to learn from them, would be next-level hard in terms of resources or complexity required.
Comments.UserDisplayName: Neil Slater
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2024-07-04T11:52:00.687Z
Comments.Text: i’d be worried if an RL agent would not change/influence the environment…
Comments.UserDisplayName: Alberto
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2024-07-04T12:16:14.350Z
Comments.Text: @Alberto True, but read the question body. Perhaps you could suggest a title improvement?
Comments.UserDisplayName: Neil Slater
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2024-07-04T12:54:02.173Z
Comments.Text: I've edited the title and tried to capture what I think is the essence of the question in more exacting RL terminology. Feel free to roll back or further adjust my changes.
Comments.UserDisplayName: Neil Slater
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2024-07-08T04:36:12.373Z
Comments.Text: @NeilSlater To my knowledge, non-stationarity is inherent to certain environments, i.e. they change not due to agents' actions but because of their specific inherent dynamics. Here I consider an environment that is stationary by nature, as you mentioned "large, robust". Happy to hear your thoughts on this before I change the title.
Comments.UserDisplayName: Mika
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2024-07-08T06:40:31.200Z
Comments.Text: Non-stationarity means the environment is not identical over time, and fits here - in general it is not necessary to identify the nature/cause of the environment changes in order to apply the label. The problem you have does not have a standard name in RL as far as I know, and could equally be framed as non-stationarity or as not having a complete model for training purposes. Which is more useful way to frame it depends on the solutions available to you
Comments.UserDisplayName: Neil Slater
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2024-07-09T09:29:57.613Z
Comments.Text: @NeilSlater I'm still not entirely sure on OP's problem. Do they mean whereby an action space could consist of taking multiple discrete actions per time step, e.g. like seen in [DecQN](https://openreview.net/forum?id=U5XOGxAgccS&referrer=%5Bthe%20profile%20of%20Tim%20Seyde%5D(%2Fprofile%3Fid%3D~Tim_Seyde1)) ? here the non-stationarity is caused from the fact that each sub-action selection policy (i.e. the policies derived from utility values) changes.
Comments.UserDisplayName: David
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2024-07-09T15:35:22.490Z
Comments.Text: @David - from what I can tell, the issue is that the OP is concerned about a shift in environment impact from agent behaviour between "small" and "large" changes, where large can be driven in at least two ways (a single agent which has gathered lots of resources and can deploy them in a single action, or multiple similar agents). What these changes both have in common is that they are harder to model or train in advance for, and are potentially much rarer to experience. I don't think there is an easy solution here, definitely not a general one that fits all scenarios
Comments.UserDisplayName: Neil Slater
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2024-07-09T15:50:25.313Z
Comments.Text: @David However, I do think that something useful could be written about issues of this kind, approaches and limitations that impact RL.
Comments.UserDisplayName: Neil Slater
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2024-07-20T03:17:13.703Z
Comments.Text: @NeilSlater, what are the methods for handling non-stationary environments?
Comments.UserDisplayName: Mika
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2024-07-20T11:15:02.587Z
Comments.Text: This is too complex to answer in comments. I think one reason you don't have any answer is that a lot depends on details, so it is hard to write a good general answer to your question. It is likely any correct answer will not lead you to a technical fix, but be more of a statement on limitations of ML in the scenarios you present
Comments.UserDisplayName: Neil Slater
Tags: <reinforcement-learning><deep-rl>
Title: Handling large or multiple agent actions that cause non-stationarity
UserDisplayName: Mika
ThreadText: # Handling large or multiple agent actions that cause non-stationarity
  
  (Asked by: username_0 on 2024-07-04)
  
  How to handle a problem where the agents' actions are changing the environment? For instance, a trading agent can hypothetically make a huge transaction that is enough to push the stock prices to go up or down.
  
  Another example could be an electric grid environment. Assume a sufficient number of large consumers utilize an RL agent in their energy management system. If these agents happen to have similar trained policies they might collectively affect stable operation of the electric grid and even lead to a blackout.
  
  Are there specific mechanisms, methods or algorithms to handle this type environments?
  
  **Tags:** reinforcement-learning, deep-rl
  
  > **Comments:**
  > **username_1**: I'm not sure what you mean, are you not just referring to transitioning to a new state? ideally the state space would encapsulate all possible states of the environment, so there's not really a notion of the environment 'changing' with the agent's actions.  
  > **username_2**: @username_1 I think this is an issue of what is being modelled during training. The two scenarios presented imply that the agents are trained against a "large, robust" environment where their actions don't usually cause the kinds of changes that the OP is concerned about. Modelling such effects in a training environment, or repeating online training enough times to learn from them, would be next-level hard in terms of resources or complexity required.  
  > **username_3**: i’d be worried if an RL agent would not change/influence the environment…  
  > **username_2**: @username_3 True, but read the question body. Perhaps you could suggest a title improvement?  
  > **username_2**: I've edited the title and tried to capture what I think is the essence of the question in more exacting RL terminology. Feel free to roll back or further adjust my changes.  
  > **username_0**: @username_2 To my knowledge, non-stationarity is inherent to certain environments, i.e. they change not due to agents' actions but because of their specific inherent dynamics. Here I consider an environment that is stationary by nature, as you mentioned "large, robust". Happy to hear your thoughts on this before I change the title.  
  > **username_2**: Non-stationarity means the environment is not identical over time, and fits here - in general it is not necessary to identify the nature/cause of the environment changes in order to apply the label. The problem you have does not have a standard name in RL as far as I know, and could equally be framed as non-stationarity or as not having a complete model for training purposes. Which is more useful way to frame it depends on the solutions available to you  
  > **username_1**: @username_2 I'm still not entirely sure on OP's problem. Do they mean whereby an action space could consist of taking multiple discrete actions per time step, e.g. like seen in [DecQN](https://openreview.net/forum?id=U5XOGxAgccS&referrer=%5Bthe%20profile%20of%20Tim%20Seyde%5D(%2Fprofile%3Fid%3D~Tim_Seyde1)) ? here the non-stationarity is caused from the fact that each sub-action selection policy (i.e. the policies derived from utility values) changes.  
  > **username_2**: @username_1 - from what I can tell, the issue is that the OP is concerned about a shift in environment impact from agent behaviour between "small" and "large" changes, where large can be driven in at least two ways (a single agent which has gathered lots of resources and can deploy them in a single action, or multiple similar agents). What these changes both have in common is that they are harder to model or train in advance for, and are potentially much rarer to experience. I don't think there is an easy solution here, definitely not a general one that fits all scenarios  
  > **username_2**: @username_1 However, I do think that something useful could be written about issues of this kind, approaches and limitations that impact RL.  
  > **username_0**: @username_2, what are the methods for handling non-stationary environments?  
  > **username_2**: This is too complex to answer in comments. I think one reason you don't have any answer is that a lot depends on details, so it is hard to write a good general answer to your question. It is likely any correct answer will not lead you to a technical fix, but be more of a statement on limitations of ML in the scenarios you present  
  
  ---
  
  ## 0 answers
  
  