Answers.Body: <p>An environment is not a magical concept in Reinforcement Learning (RL).  When the textbooks were written on RL the python &quot;gym&quot; concept certainly did not exist much less during the time of some of the earlier research work.</p>
  <p>An environment is anything which the agent can sense, interact with and receive feedback from.  One could throw an agent into a robotic body an let it learn from the real world in real time.  There's no need even for a simulation.  It's just common because we can run a simulation at faster than real time and/or spread learning across many agents.</p>
  <p>So, for your simulation to be an &quot;environment&quot; it need only be modified to do those three things.  Give the agent an input it can sense, accept and enact the agent's actions and evaluate it's performance in the form of a reward signal.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-03-20T00:44:05.187Z
Answers.Text: I see. I had come to understand it would take an interface that would output a state vector. However, I would like to use a multi-agent RL technique, with varying types of units. One step would be full information, knowing the whole state of the environment to decide initial placements. And the future steps would be partial. It would have interesting things like deploying an energy capture device, with the reward delayed. But from what you say it seems like you would just modify or replace the current algorithms used to generate each units behavior.
Answers.UserDisplayName: michael-c-michael
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-03-20T02:14:13.240Z
Answers.Text: So yes, the environment must present the agent with a state vector.  That is the "sensing" part.  One might have an agent which is the general, pilot and sailor all at once, sensing all of the information simultaneously.  But it is also possible for each to be their own agent, with their own collection of things they can sense.  Which is better is something to test.
Answers.UserDisplayName: foreverska
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-03-20T03:23:55.940Z
Answers.Text: What about a battleship, a plane, and a missile launcher, MARL can handle all of that? They'll all have partial information, communicated to each other in timesteps.
  Then the only question I have is how do I do this. Is there a guide, a paper?
Answers.UserDisplayName: michael-c-michael
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-03-20T12:53:28.850Z
Answers.Text: I don’t have a lot of MARL experience.  I found a couple interesting papers starting here, hope it helps.   https://arxiv.org/abs/1912.03558
Answers.UserDisplayName: foreverska
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-03-19T16:19:50.160Z
Answers.LastActivityDate: 2024-03-19T16:19:50.160Z
Answers.UserDisplayName: foreverska
Body: <p>I am wanting to train an RL multi-agent model to run in a propietary simulation, which is written in C++. Is there a way to change the simulation itself to create an agent, or must I use a reinforcement learning environment to rebuild my simulation, train the model, and then extract that agent to be used in the simulation? My preferred route would be to change the simulation to enable reinforcement learning, but I have not found examples of that on the web or in literature. Is that possible? Or <em>must</em> the route be that we recreate our simulation in an environment?</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2024-03-19T16:57:48.897Z
Comments.Text: Please clarify - are you intending to write your own learning agent, and if so do you intend to do this in C++ (with e.g. TensorFlow C++ API for a neural network) or are you hoping to use something pre-built in Python (e.g. one of OpenAI's baseline implementations)?
Comments.UserDisplayName: Neil Slater
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2024-03-19T17:09:03.270Z
Comments.Text: Also important - can the environment and agent interact over reasonably well-defined time steps/decision points, e.g. when training it is possible to move the simulation forward a known amount and use that to generate next state, reward etc?
Comments.UserDisplayName: Neil Slater
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2024-03-20T00:29:46.800Z
Comments.Text: If I can use python, my main language, that would be fantastic. If there is some benefit to using C++, I would use that. I'm not sure of the difficulty of the task, perhaps I would start easy for prototyping, then go hard (Tensorflow C++ API) after that.
Comments.UserDisplayName: michael-c-michael
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2024-03-20T00:31:13.570Z
Comments.Text: Regarding the second question, I believe this is possible. Also, I might add, I am planning to do multi-agent RL, with agents that can be of different categories (planes, cruise-liners, air-balloons, etc).
Comments.UserDisplayName: michael-c-michael
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2024-03-20T04:51:26.723Z
Comments.Text: According to the question, your environment is written in C++. Does it have an interface that you can script in Python for inputs (agent actions) and outputs (observations of state)? That seems to be what your question is about - how to adapt an RL system around the existing environment. If there is no suitable interface, do you have the C++ source code and able to compile some Python bindings, or only the executable?
Comments.UserDisplayName: Neil Slater
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2024-03-20T13:45:51.650Z
Comments.Text: I do not have such an interface, but I do have the source code. I agree, that is the main question. Tho it seems the other answerer of this  question says you can just modify your simulation to add reinforcement learning. But on the interface question, I have the source code and can modify it on my branch. Not at all clear on how to do that technically, haven't really seen resources online about that, it seems like a niche thing. You search RL and get a million medium posts about cart-pole.
Comments.UserDisplayName: michael-c-michael
Tags: <reinforcement-learning><deep-rl><multi-agent-systems><multi-agent-rl>
Title: Use your own simulation to train a reinforcement learning multi-agent
UserDisplayName: michael-c-michael
ThreadText: # Use your own simulation to train a reinforcement learning multi-agent
  
  (Asked by: username_0 on 2024-03-19)
  
  I am wanting to train an RL multi-agent model to run in a propietary simulation, which is written in C++. Is there a way to change the simulation itself to create an agent, or must I use a reinforcement learning environment to rebuild my simulation, train the model, and then extract that agent to be used in the simulation? My preferred route would be to change the simulation to enable reinforcement learning, but I have not found examples of that on the web or in literature. Is that possible? Or *must* the route be that we recreate our simulation in an environment?
  
  **Tags:** reinforcement-learning, deep-rl, multi-agent-systems, multi-agent-rl
  > **Comments:**
  > **username_1**: Please clarify - are you intending to write your own learning agent, and if so do you intend to do this in C++ (with e.g. TensorFlow C++ API for a neural network) or are you hoping to use something pre-built in Python (e.g. one of OpenAI's baseline implementations)?  
  > **username_1**: Also important - can the environment and agent interact over reasonably well-defined time steps/decision points, e.g. when training it is possible to move the simulation forward a known amount and use that to generate next state, reward etc?  
  > **username_0**: If I can use python, my main language, that would be fantastic. If there is some benefit to using C++, I would use that. I'm not sure of the difficulty of the task, perhaps I would start easy for prototyping, then go hard (Tensorflow C++ API) after that.  
  > **username_0**: Regarding the second question, I believe this is possible. Also, I might add, I am planning to do multi-agent RL, with agents that can be of different categories (planes, cruise-liners, air-balloons, etc).  
  > **username_1**: According to the question, your environment is written in C++. Does it have an interface that you can script in Python for inputs (agent actions) and outputs (observations of state)? That seems to be what your question is about - how to adapt an RL system around the existing environment. If there is no suitable interface, do you have the C++ source code and able to compile some Python bindings, or only the executable?  
  > **username_0**: I do not have such an interface, but I do have the source code. I agree, that is the main question. Tho it seems the other answerer of this  question says you can just modify your simulation to add reinforcement learning. But on the interface question, I have the source code and can modify it on my branch. Not at all clear on how to do that technically, haven't really seen resources online about that, it seems like a niche thing. You search RL and get a million medium posts about cart-pole.  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_2 on 2024-03-19. Score: 0 | Accepted answer
  
  An environment is not a magical concept in Reinforcement Learning (RL). When the textbooks were written on RL the python "gym" concept certainly did not exist much less during the time of some of the earlier research work.
  
  An environment is anything which the agent can sense, interact with and receive feedback from. One could throw an agent into a robotic body an let it learn from the real world in real time. There's no need even for a simulation. It's just common because we can run a simulation at faster than real time and/or spread learning across many agents.
  
  So, for your simulation to be an "environment" it need only be modified to do those three things. Give the agent an input it can sense, accept and enact the agent's actions and evaluate it's performance in the form of a reward signal.
  
  > **Comments:**
  > **username_0**: I see. I had come to understand it would take an interface that would output a state vector. However, I would like to use a multi-agent RL technique, with varying types of units. One step would be full information, knowing the whole state of the environment to decide initial placements. And the future steps would be partial. It would have interesting things like deploying an energy capture device, with the reward delayed. But from what you say it seems like you would just modify or replace the current algorithms used to generate each units behavior.  
  > **username_2**: So yes, the environment must present the agent with a state vector.  That is the "sensing" part.  One might have an agent which is the general, pilot and sailor all at once, sensing all of the information simultaneously.  But it is also possible for each to be their own agent, with their own collection of things they can sense.  Which is better is something to test.  
  > **username_0**: What about a battleship, a plane, and a missile launcher, MARL can handle all of that? They'll all have partial information, communicated to each other in timesteps. Then the only question I have is how do I do this. Is there a guide, a paper?  
  > **username_2**: I don’t have a lot of MARL experience.  I found a couple interesting papers starting here, hope it helps.   https://arxiv.org/abs/1912.03558  
  
  ---
  
  